| Outcome Event | True Prob P(x) | Model Prob Q(x) | Optimal Bits -log₂(p) | Model Bits -log₂(q) | Assigned Code P | P Entropy Term | Cross-Entropy Term | KL Penalty |
|---|
Why 8-sided die with 4 eights needs only 2 bits? An 8-sided fair die has 8 equiprobable outcomes requiring log₂(8) = 3 bits each (000 through 111). When 4 sides are labeled '8' and the other 4 are distinct (1, 2, 3, 4), outcome '8' happens 50% of the time (p=0.5), requiring only 1 bit ("0"). The remaining outcomes each happen 12.5% of the time (p=0.125), requiring 3 bits ("100", "101", "110", "111").
The average optimal length is: 0.5 × 1 + 4 × (0.125 × 3) = 2.0 bits.
The Machine Learning Loss Connection: When a classifier model incorrectly assumes a uniform distribution Q across the 5 distinct outcomes (q=0.20), it allocates log₂(5) = 2.3219 bits per outcome. Testing this mismatched model on true events P produces Cross-Entropy H(P, Q) = 2.3219 bits, introducing a relative entropy tax of D_KL = 0.3219 bits (+16.1% bandwidth waste). Minimizing Cross-Entropy during training directly eliminates this coding redundancy.