1. Bibliographic Information

1.1. Title

A-LAMP: Agentic LLM-Based Framework for Automated MDP Modeling and Policy Generation

1.2. Authors

• Hong Je-Gal: Department of AI & Robotics, Sejong University, Seoul.
• Chan-Bin Yi: Department of AI, Sejong University, Seoul.
• Hyun-Suk Lee: Department of AI & Robotics, Sejong University, Seoul.

1.3. Journal/Conference

This paper was published on arXiv as a preprint.

• Publication Date (UTC): 2025-12-12.
• Source Link: https://arxiv.org/abs/2512.11270v1

1.4. Abstract

The paper addresses the challenge of applying Reinforcement Learning (RL) to real-world tasks, which typically requires converting informal natural language descriptions into formal Markov Decision Processes (MDPs) and executable code. The authors introduce A-LAMP (Agentic LLM-based Automated MDP Modeling and Policy generation), a framework that utilizes multiple specialized Large Language Model (LLM) agents to decompose this process into verifiable stages: modeling, coding, and training. Experimental results show that A-LAMP significantly outperforms single state-of-the-art LLMs (like GPT-4o) in generating viable policies across both classic control and custom RL domains. The framework also enables smaller models (e.g., Gemma3-27B) to achieve performance comparable to larger models.

2. Executive Summary

2.1. Background & Motivation

• Core Problem: Reinforcement Learning (RL) is powerful for sequential decision-making, but applying it requires a rigorous formulation of the problem as a Markov Decision Process (MDP). This involves defining states, actions, rewards, and transition dynamics mathematically and then implementing them in code (e.g., Python/Gym environments).
• Key Challenges:
  1. Translation Gap: Real-world tasks are often described in informal natural language or scattered documentation, not mathematical notation.
  2. Complexity: Converting these descriptions into a working RL loop is expertise-intensive, error-prone, and time-consuming.
  3. Fragility: A single misalignment—such as a reward function that doesn't match the goal or a coding syntax error—can cause the entire training process to fail.
  4. Inflexibility: Manual designs are hard to reuse. A slight change in task objectives often requires a complete manual re-engineering of the MDP and code.
• Innovation: The paper proposes automating this end-to-end pipeline using an agentic framework. Instead of asking one LLM to "solve the problem," A-LAMP assigns specific roles (e.g., "Parameter Extractor," "Coder") to different LLM instances, mirroring the workflow of a human expert.

2.2. Main Contributions & Findings

• Contributions:
  1. A-LAMP Framework: A modular, multi-agent system that automates the transition from natural language descriptions to trained RL policies.
  2. Decomposition Strategy: The framework breaks down the complex task into three phases—Abstract Idea, Formulation, and Coding—ensuring semantic alignment and allowing intermediate verification.
  3. Interpretability: Each step produces human-readable outputs (e.g., JSON schemas, mathematical equations), making the "black box" of automation transparent.
• Findings:
  1. Superior Performance: A-LAMP consistently achieves higher success rates in policy generation compared to single-prompt baselines using GPT-4o.
  2. Model Agnosticism: A lightweight version of A-LAMP using the smaller Gemma3-27B model approaches the performance of the much larger GPT-4o, proving the efficacy of the structural decomposition.
  3. Reliability: Analysis reveals that A-LAMP specifically reduces "spurious" successes (where code runs but is semantically wrong) and increases training stability.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a novice needs to grasp the following concepts:

• Reinforcement Learning (RL): A machine learning paradigm where an agent learns to make decisions by interacting with an environment. The agent takes an action, receives a reward, and the environment transitions to a new state. The goal is to maximize the cumulative reward over time.
• Markov Decision Process (MDP): The mathematical framework used to model RL problems. It is defined by the tuple $(S, A, P, R, \gamma)$:
  • $S$: The set of States (what the agent sees).
  • $A$: The set of Actions (what the agent can do).
  • $P$: The Transition Probability (how the state changes based on actions).
  • $R$: The Reward Function (immediate feedback for an action).
  • $\gamma$: The discount factor (importance of future rewards).
• Large Language Models (LLMs) & Agentic Frameworks:
  • LLMs: AI models like GPT-4 trained on vast text data, capable of generating code and reasoning.
  • Agentic Framework: Instead of a simple "question-answer" interaction, LLMs are configured as "agents" with specific roles, tools, and workflows (e.g., one agent writes code, another reviews it).
• Deep Q-Network (DQN): A specific RL algorithm used in this paper as the standard solver. It uses a neural network to estimate the value (Q-value) of taking a specific action in a specific state.

3.2. Previous Works

• General Reasoning: The paper builds on the reasoning capabilities of LLMs demonstrated in works like Sparks of AGI and Generative Agents.
• Agentic Decomposition: It adopts the philosophy of decomposing tasks into sub-agents, similar to:
  • Toolformer and ChatDev: Which show that specialized agents collaborating can solve complex software engineering tasks better than a single model.
  • OptiMUS: A framework for optimization modeling that decomposes workflows into specialized agents.
• RL-Specific LLM Applications:
  • EUREKA (Ma et al., 2023): Focuses specifically on automating reward design using LLMs.
  • Voyager (Wang et al., 2023): Uses LLMs for open-ended exploration and skill acquisition in Minecraft.
  • Codex: Demonstrates the code-generation capability essential for implementing environments.

3.3. Differentiation Analysis

While previous works like EUREKA focus on specific components (like reward shaping) or high-level planning (Voyager), A-LAMP distinguishes itself by automating the entire end-to-end pipeline. It starts from a completely free-form text description and results in a trained policy. It uniquely combines:

1. Mathematical Formalization: Explicitly generating LaTeX equations for objectives and constraints.
2. Full Environment Implementation: Writing the complete Python/Gym code for the environment, not just calling existing APIs.
3. Holistic Alignment: Ensuring the code matches the math, and the math matches the text description.

4. Methodology

4.1. Principles

The core principle of A-LAMP is to mimic the cognitive process of a human expert solving an RL problem. A human does not jump from a paragraph of text directly to writing Python code. Instead, they follow a structured reasoning path:

1. Abstraction: Identify key parameters and goals.

2. Formulation: Translate these into mathematical models (MDPs).

3. Coding: Implement the math in code and debug.

   A-LAMP replaces each of these cognitive steps with a specialized LLM agent.

The following figure (Figure 1 from the original paper) illustrates this comparison between the human manual pipeline and the A-LAMP automated pipeline:

该图像是一个示意图，比较了手动人工专家流程与自动化的 A-LAMP 流程在策略生成过程中的区别。两者均分为三个阶段——抽象思想阶段、公式化阶段和编码阶段，其中人类专家输入领域知识和任务想法，而 A-LAMP 则使用自由形式的自然语言描述任务。图中还展示了 A-LAMP 中的不同智能体，如参数代理、变量代理和建模代理等。

4.2. Core Methodology In-depth (Layer by Layer)

The framework is divided into three sequential phases.

Phase 1: Abstract Idea (Structuring the Text)

The input is a free-form natural language description. This phase breaks the text into structured components.

1. Parameter Agent: Scans the text to identify constant values known ahead of time (e.g., "maximum capacity," "gravity," "number of users").

2. Objective Agent: Extracts the goal of the task (e.g., "maximize throughput," "balance the pole").

3. Variable Agent: Identifies:

   • Decision Variables: What the agent controls (Actions).
   • System Variables: What changes in the environment (States).

4. Constraint Agent: Extracts rules that must not be violated (e.g., "energy cannot be negative").

   Output: A structured JSON object containing these elements.

Phase 2: Formulation (Mathematical MDP Design)

This phase converts the structured ideas into formal mathematics.

1. Modeling Agent:

   • Takes extracted objectives and constraints.
   • Step 1: Formulates the optimization objective $J(\pi)$ mathematically. For example, if the goal is maximizing cumulative reward, it generates: $ \operatorname* { m a x } _ { \pi : S \to A } J ( \pi ) $ where $\pi$ is the policy, $S$ is the state space, $\mathcal { A }$ is the action space, and $J ( \pi )$ is the expected cumulative reward.
   • Step 2: Formulates constraints as algebraic inequalities.

2. State-Action-Reward (SAR) Agent:

   • Explicitly defines the MDP tuple components.
   • State ($s$): Defines the vector structure (e.g., vector of channel gains).
   • Action ($a$): Defines the feasible choices (e.g., select user $i$).
   • Reward ($r$): Defines the scalar feedback signal derived from the objective.

Error Correction (EC) Module: Attached to these agents is a verification loop. The agent assigns itself a "self-confidence score." If the score is low, it re-examines the output or asks for clarification. This helps prevent "hallucinations" in the extraction phase.

The workflow for a specific use case (Wireless Network Scheduling) is shown below (Figure 2 from the original paper), highlighting how agents pass information to form the MDP and eventually the code:

该图像是一个示意图，展示了A-LAMP在无线网络调度问题中的应用。图中包含多个代理，如参数代理、目标代理和变量代理，分别负责处理参数、目标和变量，同时展示了组合环境与策略的结果，包含可执行环境和训练策略。此外，图中还标示了包含公式 R_t = rac{ ext{log}_2(1 + rac{P imes G_{ ext{scheduledUser}, t}}{10^{-10} imes ext{NoiseDensity}})} 的奖励信息。

Phase 3: Coding (Implementation & Training)

This phase bridges the gap between math and executable Python code.

1. Environment Agent:

   • Defines the dynamics. It specifies how the system evolves from time $t$ to $t+1$ (e.g., physics updates, customer demand logic).
   • Specifies termination conditions (e.g., time limit $T$ or failure state).

2. Coding Agent:

   • Takes the definitions from the SAR and Environment agents.
   • Generates a complete Python script.
   • Structure:
     • Implements a custom OpenAI Gym environment class.
     • Implements reset() and step() functions based on the dynamics.
     • Implements the Reward Logic inside step().
     • Sets up a Deep Q-Network (DQN) training loop.

3. Code Executor & Debugging Loop:

   • Executes the generated code.
   • If a runtime error occurs, the feedback (error trace) is sent back to the Coding Agent to regenerate or fix the code.

5. Experimental Setup

5.1. Datasets (Tasks)

The authors evaluated A-LAMP on five distinct tasks defined solely by natural language descriptions.

1. Cart-pole: A classic control task where a pole must be balanced on a moving cart.
2. Mountain-car: A classic control task where an underpowered car must reach a hilltop by building momentum.
   • Note: These two use standard Gym environments, testing A-LAMP's ability to recognize and interface with existing libraries.
3. Wireless: A domain-specific resource allocation task.
   • Scenario: A base station schedules users to transmit data.
   • Physics: Involves Shannon capacity, path loss, and fading.
   • Objective: Maximize sum-rate (throughput).
4. Drone-delivery (Drone-del.): A $50 \times 50$ grid world.
   • Goal: Deliver packages to targets.
   • Constraints: Energy consumption.
5. Inventory-management (Inv.-mgmt.): A supply chain task.
   • Dynamics: Poisson demand distribution.
   • Action: Decide reorder quantities.
   • Goal: Maximize profit (revenue minus holding/ordering costs).

5.2. Evaluation Metrics

The paper uses three hierarchical success rates to evaluate the pipeline:

1. Modeling Success Rate:

   • Conceptual Definition: Measures if the extracted MDP components (State, Action, Reward) are logically consistent with the text description.
   • Calculation: $\frac{\text{Number of trials with logically correct MDPs}}{\text{Total number of trials}}$

2. Coding Success Rate:

   • Conceptual Definition: Measures if the generated Python code is syntactically correct and runs without crashing.
   • Calculation: $\frac{\text{Number of trials with successful code execution}}{\text{Total number of trials}}$

3. Policy Generation Success Rate (Primary Metric):

   • Conceptual Definition: Measures the ultimate goal—does the generated code actually train an agent that solves the task? It requires convergence to a reward-maximizing policy that satisfies the original task objectives.
   • Calculation: $\frac{\text{Number of trials with successful policy convergence}}{\text{Total number of trials}}$

5.3. Baselines

The authors compared A-LAMP against single-model approaches to isolate the benefit of the agentic framework.

1. A-LAMP (GPT-4o): The full framework using GPT-4o.
2. Light A-LAMP (Gemma3-27B): The framework using a smaller, open-weights model.
3. Single GPT-4o: A standard prompting approach where GPT-4o is asked to write the training code directly from the description.
4. Single Gemma3-27B: The same baseline using the smaller model.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results demonstrate that the A-LAMP framework significantly outperforms single-model approaches, particularly in complex, custom domains.

The following are the results from Table 1 of the original paper. The table reports success rates as a triplet: Modeling / Coding / Policy Generation.

Analysis:

1. Policy Generation Dominance: In the custom tasks (Wireless, Drone, Inventory), A-LAMP (0.45, 0.45, 0.30) consistently achieves much higher policy generation rates than single GPT-4o (0.20, 0.10, 0.05).
2. Framework over Model Size: Remarkably, Light A-LAMP (using the smaller Gemma model) often outperforms or matches the single GPT-4o model (e.g., in Drone-delivery, Light A-LAMP gets 0.15 vs Single GPT-4o's 0.10). This proves that the multi-agent structure is a more powerful driver of performance than raw model size alone.
3. Role of Error Correction (EC): The "A-LAMP" column (with EC) shows improvements over "A-LAMP w/o EC," especially in coding and policy generation, validating the utility of the self-reflection loop.

6.2. Failure Distribution Analysis

To understand why A-LAMP is better, the authors analyzed failure cases. They categorized failures into Modeling (M), Coding (C), and Policy Training (P).

The following are the results from Table 2 of the original paper. Note: $P \times$ denotes Policy Generation Failure.

The failure analysis (visualized in Figure 3 of the paper) highlights that A-LAMP drastically reduces "Spurious Coding Success" (cases where code runs but the model is wrong, denoted as $M \times, C \circ$). By ensuring the modeling ($M$) is correct first, A-LAMP ensures that if the code runs, it is likely doing the right thing.

The following figure (Figure 3 from the original paper) visualizes these failure distributions:

该图像是一个柱状图，展示了不同RL任务中A-LAMP与单一GPT-4o的失败分布。左侧(a)的A-LAMP没有EC，右侧(b)展示了单一的GPT-4o。图中列出了任务类型如Cart-pole、Mountain-car等，颜色分别代表不同的计算模式。该图直观比较了两种方法在不同情境下的表现。

6.3. Case Study Validation (Wireless Task)

The authors verified not just that the code runs, but that it learns the correct policy. In the Wireless task, the optimal strategy is a "Greedy Scheduler" (always pick the user with the best channel).

The following figure (Figure 4 from the original paper) compares the training return of the A-LAMP generated agent against the greedy baseline:

该图像是一个图表，展示了A-LAMP生成的DQN在训练和评估阶段的回报。左侧图（a）表示训练进展，回报随着迭代次数逐渐增加；右侧图（b）则对比了DQN与贪婪基线的策略评估，两者回报相近但存在波动。

• Training (a): The return increases and stabilizes, showing successful learning.
• Evaluation (b): The A-LAMP agent (DQN) achieves performance very close to the optimal Greedy Scheduler. This confirms the generated MDP and code accurately reflected the physical constraints and objectives of the task.

7. Conclusion & Reflections

7.1. Conclusion Summary

A-LAMP successfully demonstrates that agentic decomposition is a viable path to automating the complex pipeline of Reinforcement Learning. By breaking down the problem into Abstract Idea, Formulation, and Coding phases, the framework:

1. Aligns unstructured text with formal mathematical definitions.
2. Generates executable, verifiable code.
3. Outperforms single-prompt LLMs significantly, even when using smaller underlying models.

7.2. Limitations & Future Work

• Coding Fragility: The authors note that the coding stage remains the bottleneck. Syntactic errors or library mismatches (e.g., Gym version conflicts) can still cause failures ($C \times$) even if the modeling is perfect ($M \circ$).
• Future Work:
  • Fine-grained Coding Agents: Implementing specialized sub-agents for specific parts of the code (e.g., one agent just for the step function, another for the training loop) to improve robustness.
  • Hyperparameter Tuning: Currently, the framework uses standard defaults (like generic DQN parameters). Integrating an automated tuning agent could improve performance on harder tasks.

7.3. Personal Insights & Critique

• Methodological Transferability: The core strength of this paper is not just for RL, but for any Scientific Machine Learning task. The pattern of "Text $\to$ Math $\to$ Code" using specialized agents is highly transferrable to fields like optimization (Operations Research) or simulation design.
• Verification Gap: While the "Modeling" phase produces LaTeX, there is no automated "Theorem Prover" or symbolic checker mentioned that verifies the LaTeX equations are solvable or consistent. The verification relies on the LLM's self-reflection. Integrating a symbolic math engine (like Wolfram or SymPy) into the verification loop could be a powerful enhancement.
• Dependency on Description Quality: The system assumes the natural language description contains all necessary information. If the prompt is vague (e.g., "optimize the network" without defining variables), the agents might hallucinate parameters. Robustness to ambiguous inputs is a critical area for real-world deployment.