Teaching an LLM to judge itself — and be honest about it
\nThis paper trains a language model with reinforcement learning using two signals at once: did it get the task right, and did it accurately judge how likely it was to be right. The result is a model that solves more problems and expresses uncertainty you can actually trust.
\nConcept: \"Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs\" — source post by @pybeebee. Diagram below follows the paper's training blueprint.
\n1The training loop, step by step
\nTap any stage to expand it. This mirrors the paper's diagram: a query flows into the policy model, the response earns two rewards, and both push an RL update back into the model.
\n\n\n
\n▾\n\n
\nQuery
A question or task arrives
Input. Training questions range from easy facts to genuinely hard problems. That mix matters: the model must learn to feel the difference between \"I know this\" and \"I am guessing.\"
\n\n
\n▾\n\n
\nPolicy model
The LLM being trained
The learner. The policy model does two things in one response: it answers the query, and it reports a confidence — a metacognitive self-judgment of how likely its own answer is correct.
\n\n
\n▾\n\n
\nAnswer + stated confidence
\"The answer is X — I am about 80% sure\"
The output being graded. Both parts count. A right answer with a fitting confidence is ideal. A wrong answer delivered with high confidence is the worst case — that is exactly the overconfident behavior the training is designed to punish.
\n\n
\n
\n▾\n\n
\nTask reward
Was the answer correct?
Signal one. The classic RL signal: reward for solving the task. On its own, this tends to breed confident-sounding answers whether or not they are right.
\n▾\n
Metacognitive reward
Was the self-judgment accurate?
Signal two — the paper's key