| Outcome Event | True Prob P(x) | Model Prob Q(x) | Optimal Bits -log₂(p) | Model Bits -log₂(q) | Assigned Code P | P Entropy Term | Cross-Entropy Term | KL Penalty |
|---|
Why 8-sided die with 4 eights needs only 2 bits? An 8-sided fair die has 8 equiprobable outcomes requiring log₂(8) = 3 bits each (000 through 111). When 4 sides are labeled '8' and the other 4 are distinct (1, 2, 3, 4), outcome '8' happens 50% of the time (p=0.5), requiring only 1 bit ("0"). The remaining outcomes each happen 12.5% of the time (p=0.125), requiring 3 bits ("100", "101", "110", "111").
The average optimal length is: 0.5 × 1 + 4 × (0.125 × 3) = 2.0 bits.
The Machine Learning Loss Connection: When a classifier model incorrectly assumes a uniform distribution Q across the 5 distinct outcomes (q=0.20), it allocates log₂(5) = 2.3219 bits per outcome. Testing this mismatched model on true events P produces Cross-Entropy H(P, Q) = 2.3219 bits, introducing a relative entropy tax of D_KL = 0.3219 bits (+16.1% bandwidth waste). Minimizing Cross-Entropy during training directly eliminates this coding redundancy.
The saved entropy source normalizes raw weights into probabilities. Its default source has probabilities one half and four outcomes of one eighth. Entropy sums minus p times log base two of p. The half-probability outcome contributes one half bit; each one-eighth outcome contributes three eighths. Summing gives two bits. Those contributions are the bar lengths here. Entropy is an average information quantity rather than one fixed-length code word for every outcome. Its model Q initially assigns one fifth to every outcome. Every modeled surprise is log base two of five, about two point three two one nine bits. Averaging those surprises under the actual P still gives two point three two one nine, because they are all equal. Cross entropy therefore exceeds source entropy by point three two one nine bits. The source displays that difference as KL divergence and relative overhead about sixteen point one percent. Match the modeled distribution Q to the source P and the two logarithms become equal. Cross entropy returns to two bits and KL becomes zero. The mathematical identity is cross entropy equals source entropy plus KL. The source also constructs prefix code books and can stream sampled symbols. Actual integer codeword lengths must be distinguished from fractional ideal log-loss bits; a simulated code stream is extra evidence, not the definition of entropy.