Model Confidence & Error Calibrator Uncertainty Lab

Can models measure confidence to flag their own errors?

Explore how modern LLMs translate internal token logits, self-consistency sample entropy, and post-hoc temperature scaling into calibrated confidence signals to automatically catch hallucinations and incorrect reasoning before execution.

Expected Calibration Error (ECE)
0.042
Gap between predicted confidence & true accuracy
Error Detection AUROC
0.891
Ability to rank incorrect vs correct outputs
Selective Accuracy @ Threshold
94.8%
Accuracy on retained (unflagged) generations
Retained Coverage Rate
81.4%
Fraction of total queries passed to user
Inference & Calibration Controls Live Engine
Logit Temperature (T) 1.00
Underconfident (Cold) Default (1.0) Overconfident (Hot)
Confidence Abstention Cutoff (τ) 0.75
High Coverage (Riskier) Strict Abstention (Safe)
Generated Response Stream — Click any token to inspect top logit alternatives:
High Conf (>85%) Uncertain (50-85%) High Entropy (<50%)
Selected Token Logit Probabilities
Uncertainty Diagnostics
PASSED • 88% CONF
Aggregate uncertainty score remains comfortably above threshold τ=0.75. Response accepted for downstream user display.

Why Measuring Confidence in Language Models Is Difficult

Modern autoregressive models output normalized softmax probabilities over vocabulary tokens, but these do not directly represent true calibration or semantic certainty.

◆ Token Logprobs vs Semantic Entropy

An LLM might use multiple valid synonyms (e.g. "Paris", "The capital of France", "Ville Lumière"), dividing token probability across lexical choices despite having 100% semantic certainty.

H(s) = -∑ p(c) log p(c)

Clustering multiple stochastic generation paths into semantic equivalence classes removes trivial phrasing entropy to reveal true knowledge doubt.

◆ The RLHF Overconfidence Distortion

Reinforcement Learning from Human Feedback (RLHF) and DPO tune models to sound decisive, authoritative, and polite. This often compresses softmax logits near 1.0 even when facts are fabricated.

p_i = exp(z_i / T) / ∑ exp(z_j / T)

Temperature scaling (T > 1.0) and Platt scaling can recalibrate probability scales without changing argmax output order.

◆ Selective Abstention & Guardrails

By establishing a confidence abstention threshold τ, production AI agents can route low-confidence tasks to human escalation, retrieval verification, or external tool execution.

Risk(τ) = E[Loss | Conf(x) ≥ τ]

Optimal thresholding maximizes retained throughput (coverage) while capping tolerable hallucination risk to safety thresholds.