z

AI Information Bottleneck

Latent Abstraction Sim
Scenario & Controls
Neural Abstraction Flow (Input → Filter → Latent → Reconstruction) Active Flow
IB Diagnostics
Input Shannon Entropy 5.82 bits Raw uncompressed noise content
Latent Info Retention 64.2% Target concept signals preserved
Discarded Noise Ratio 72.8% Redundant & jitter paths pruned
Concept Classification Acc 94.1% Downstream task invariance score
Node & Vector Inspector Hover or Click Node
[System Ready] Hover over network nodes in the flow visualization above to inspect individual feature activation vectors, cosine similarity scores, and local Shannon entropy decay.
Core Principle: Information Discarding

The Essence of AI: Real-world inputs are saturated with high-dimensional noise. Neural networks do not retain raw inputs; they systematically discard unnecessary information (Information Bottleneck Principle: min I(X;Z) s.t. max I(Z;Y)).

By shrinking the central layer (Latent Bottleneck), the network forces high-frequency noise and background clutter to drop out, distilling invariant topological concepts.

Squeeze to see: the information bottleneck in machine learning

A one-minute painted explainer of how this simulator models the information bottleneck, followed by what the research literature actually says.

Read the narration

A neural network sees far more than it needs. The information bottleneck idea says a good model squeezes its input down, keeping only what predicts the answer and throwing the rest away. This simulator pushes a noisy sixty-four pixel digit through four layers: twelve raw inputs, eight filters, a narrow latent core, and four outputs. Two dials do the squeezing. The noise filter threshold, at zero point six five, decides how much jitter gets pruned. The bottleneck, here just four units wide, caps how much can pass at all. At the defaults, the tool keeps about sixty percent of the signal, discards about a third, and classifies at nearly ninety-six percent. Squeeze down to two units and accuracy falls to seventy-eight percent. At one unit it drops to sixty. Too narrow, and the concept itself no longer fits. In real networks, a bottleneck that's too wide can let a model memorize noise, a cost this simple model leaves out. Try the presets and hunt for the sweet spot.

What the model assumes, and what the research says

The principle

Tishby, Pereira and Bialek (1999) framed learning as finding a compressed representation Z of an input X that keeps as much information as possible about a target Y: minimise I(X;Z) while keeping I(Z;Y) high, with a single parameter trading the two off.

The tool's formulas

Retention = 45% × (bottleneck ÷ 16) + 55% × (1 − noise × (1 − threshold)). Accuracy starts at 99%, loses 18 points per unit below a bottleneck of 3, and loses noise × 25 × (1 − threshold). At the defaults (4 units, threshold 0.65, noise 0.35) that gives 59.5% retained, 34.1% discarded and 95.9% accuracy. These are illustrative rules of thumb, not measurements from a trained network.

What's random on the page

The 64-cell input matrix, the 'Input Shannon Entropy' figure and which links are drawn as pruned are regenerated with random numbers on every pass, so they change each time you move a slider. The entropy value is also divided by 10 for display, so treat it as a visual cue rather than a real measurement in bits.

A debated theory of deep learning

Shwartz-Ziv and Tishby (2017) reported that deep networks first fit their training data and then compress their internal representations. Saxe and colleagues (ICLR 2018) found that this compression phase depends on the activation function and is not needed for good generalisation, so the bottleneck view is a useful lens rather than a settled law.

Bottlenecks you can train

The Deep Variational Information Bottleneck (Alemi et al., 2016) turns the idea into a practical training objective, and a narrow latent layer is the core of autoencoders. In real models, too wide a bottleneck lets noise through and too narrow a one loses the signal; this tool only models the second failure.

Try it in the tool

  1. Drag Latent Bottleneck Size from 4 down to 1 and watch Concept Classification Acc fall from 95.9% to 59.9%.
  2. Raise Input Noise Level to 0.80, then push Noise Filter Threshold up to 0.95, and see how much accuracy the filter wins back.
  3. Switch the Domain Preset to Text Sentiment vs Jargon Filter (a 2-unit bottleneck) and compare its accuracy with Customer Behavior Noise Filter (8 units).

Sources and further reading

  1. The information bottleneck method · Tishby, Pereira & Bialek, arXiv (1999/2000)

    The original statement of the information bottleneck principle.

  2. Deep Learning and the Information Bottleneck Principle · Tishby & Zaslavsky, arXiv 2015

    Proposes analysing deep networks layer by layer through the bottleneck lens.

  3. Opening the Black Box of Deep Neural Networks via Information · Shwartz-Ziv & Tishby, arXiv 2017

    The fitting-then-compression account of training.

  4. On the Information Bottleneck Theory of Deep Learning · Saxe et al., ICLR 2018

    Challenges whether compression is universal or needed for generalisation.

  5. Deep Variational Information Bottleneck · Alemi, Fischer, Dillon & Murphy, arXiv 2016

    A trainable variational version of the bottleneck objective.

Numbers in the video come from the simulator's simplified formulas at the Noisy Digit Recognition defaults; real networks do not report retention or accuracy this way.

Super generates helpful tools and automates fact-checking across the internet proactively. If you enjoyed this tool, build your own with Super and share it with a friend.