Open-Weight Agent RL Lab: Policy & Reward Simulator

⚙️ Rollout & Reward Config

Target Environment Task

Base Model Checkpoint

Format Compliance (+) +1.0

Tool Schema Validity (+) +1.5

Task Completion (+) +3.0

Length Penalty (-) -0.05

KL Penalty Coef (β) 0.05

GRPO/PPO Clip (ε) 0.20

Episode Reward

+5.35

F:+1.0 | S:+1.5 | C:+3.0 | L:-0.15

Tool Call Prob (P_θ)

88.4%

Base prior was 18.2%

KL Divergence (D_KL)

0.068

Bound: < 0.12 (Stable)

Policy Loss (L_clip)

-0.042

Advantage: +1.28

📈 Training Convergence & Token Shift Dynamics (D3)

— Mean Reward | — KL Divergence | — Tool Token Mass

ACTIVE ROLLOUT EPISODE Prompt: ACME Q3 earnings lookup & % change

Iteration 5 / 25

❌ Frozen Base Policy (Pre-RL)

Zero-Shot Base

Lacks RL alignment on tool protocols. Tends to hallucinate tool answers, skip schemas, or output free-form text without invoking the engine.

I think ACME Q3 earnings were probably up. The stock went from $120 to $137.
Let me calculate: 137 - 120 = 17.
17 / 120 = maybe around 15%?
So the stock rose roughly 15%. I cannot call financial tools directly.

✓ RL-Aligned Agent (TRL GRPO Checkpoint)

Trained with RL

Rewarded for structured `` syntax, exact JSON arguments, and execution grounding. KL penalty keeps reasoning concise.

<thought>
Need to search for official ACME Q3 earnings report and compute exact percentage change from $120 to $137.
</thought>
<tool_call>
{"name": "financial_search", "arguments": {"query": "ACME Q3 earnings release stock price"}}
</tool_call>
<tool_response>
{"status": 200, "result": "ACME reported revenue $4.2B beating estimates. Baseline price $120.00, close $137.00."}
</tool_response>
<thought>
Now calculate exact % change: ((137 - 120) / 120) * 100.
</thought>
<tool_call>
{"name": "calculator", "arguments": {"expression": "((137 - 120) / 120) * 100"}}
</tool_call>
<tool_response>
{"result": 14.1666666667}
</tool_response>
ACME reported a strong Q3 beat; the stock increased by 14.17% (from $120 to $137).

🐍 Hugging Face TRL GRPOTrainer Script Python 3.10+ / trl >= 0.14

Auto-synthesized production training script with your configured reward weights, environment prompts, and LoRA/GRPO settings.

# Loading script...

🎯 Why RL for Tool-Calling Agents?

Supervised Fine-Tuning (SFT) teaches models syntax by imitation, but fails when tools return unexpected runtime errors or require self-correction. Reinforcement learning (such as DeepSeekMath's GRPO or standard PPO) optimizes the agent directly on end-to-end task success, teaching it when to stop reasoning and issue structured calls.

⚖️ Reward Shaping vs Reward Hacking

If format rewards are too high without a task completion verifier, agents output dummy XML tags without answering. If the verbosity penalty is too severe, the agent skips critical reasoning steps. Our simulated telemetry shows how balancing positive format milestones against reference KL preserves general competence.

💡 GRPO (Group Relative Policy Optimization)

Unlike PPO which requires an extra critic/value network (often as large as the policy model itself), Hugging Face's GRPOTrainer samples a group of completions per prompt and computes relative advantages across the group, drastically reducing GPU VRAM required to train open-weight models locally.