MoE

MoE Dynamic Token Routing Simulator

2.8T Kimi K3 Sparse Architecture Benchmark

Active Params 160B / 2.8T

Active Ratio 5.71%

Dropped Tokens 0 (0.0%)

Compute Efficiency

2.50x Baseline

FLOP Density vs Dense Transformer

Expert Buffer Saturation

42.5% Avg

Capacity Factor: 1.25x

Auxiliary Load Balance Loss

0.0142

Router Entropy Penalty

Stream Throughput

64 tok/batch

Simulated Context: 1M tokens

STATE PROOF: MoE Router Operating Nominal | Total Experts: 16 | Top-k: Top-2

Gating Algorithm

Capacity Factor (S) 1.25x

Auxiliary Loss Weight (α) 0.02

Total Expert Networks 16

Batch Size (Tokens/step) 64

Simulated Context Window 1M Tokens

Stream Speed Normal