The Training Pipeline
The blueprint from the paper's diagram: a query flows through the policy model, its answer is scored on two axes — task success and metacognitive accuracy — and both rewards drive the RL update. Tap each stage to expand it, or press Play.
Reward Simulator
You are the reward function. Pick whether the model's sampled answer is actually correct, then drag its stated confidence. Watch the task reward and the metacognitive (calibration) reward — and see which behavior the RL update reinforces.
Why Calibration Emerges
The metacognitive reward peaks when stated confidence equals actual accuracy. Overclaiming on wrong answers is punished hardest — exactly the hallucination-with-bravado failure mode. The chart shows reward as a function of confidence for correct vs. wrong answers.
The dot tracks your simulator settings above. The honest policy is a truthful map from internal knowledge to expressed confidence — that is faithful uncertainty expression.
Key Takeaways
- Self-judgment is trainable. Adding a metacognitive feedback signal to RL teaches the policy model to assess its own answers, not just produce them.
- Honesty and accuracy are not a tradeoff. Models trained with metacognitive rewards improved on tasks while also expressing uncertainty that matched their real error rates.
- Faithful uncertainty beats confident hallucination. Because overconfident wrong answers earn the worst combined reward, the policy learns to hedge exactly when it should.
- It generalizes. Calibrated confidence learned in training carries to expressing honest "I'm not sure" behavior at inference time.