Empirical Model Synthesis

Neutral AI Benchmark Studio

Benchmark scores alone mislead when cost, latency, hallucination drift, and test set contamination are concealed. Run multi-objective sensitivity audits to find the true Pareto frontier.

Objective Weighting

Sum: 100%

Accuracy & Reasoning (MMLU / GSM8k) 35%

Hallucination Suppression (Faithfulness) 25%

Inference Speed & TTFT (Latency) 20%

Cost Efficiency ($/1M Tokens) 20%

Audit Presets

Contamination Penalty High (1.5x) Penalizes models suspected of memorizing public benchmark test sets.

Add Model Candidate

Top Neutral Leader Claude 3.5 Sonnet

Score: 84.6 pts (Pareto Optimal)

Best Cost / Performance DeepSeek-V2.5

$0.28 / M tokens at 79.2 pts

Frontier Trade-off 4 Models

on un-dominated efficiency border

Scatter plots Cost (log-scale) vs Neutral Composite Score.

Ranked Multi-Criteria Assessment

Rank & Model	Raw Accuracy	Hallucination	Latency (TTFT)	Cost / 1M	Neutral Score	Status

Benchmarking 8 models across 4 neutral criteria.

Why AI Benchmarks Are Broken

Public LLM leaderboards frequently incentivize model builders to train directly on test sets (eval contamination) or cherry-pick evaluation prompts with extreme few-shot scaffolding.

MMLU Saturation: High scores on standardized benchmarks no longer guarantee real-world reliability.
Vendor Weighting: Standard leaderboards often omit cost and latency, making $30/M token models seem equivalent to $0.50/M token alternatives.
Hallucination Penalties: A model that answers with 92% confidence but hallucinates 18% of facts is dangerous for regulated workflows.

How NeutralBench Restores Trust

This studio implements multi-objective Pareto optimization and contamination discounts:

Pareto Dominance: Identifies models that cannot be beaten in accuracy without paying more or running slower.
De-Contamination Multiplier: Applies a discount factor based on historical test-leakage flags and synthetic benchmark perplexity shifts.
Normalized Composite Scoring: Converts disparate scales (tokens/sec, USD, error rate) into normalized percentiles.

Frequently Asked Questions

How is the neutral composite score calculated?

Each metric is normalized into a standard [0, 100] percentile score across the active candidate pool. Inverted metrics (cost, latency, hallucination rate) are transformed so that lower is better. User-selected objective weights then compute the weighted harmonic mean, adjusted by the contamination penalty.

Can I upload my own internal benchmark dataset?

Yes. Use the "Add Model Candidate" form to insert internal fine-tunes or custom endpoints. You can export the full synthesized scorecard as a JSON or CSV report for executive and engineering reviews.

What defines a "Pareto Optimal" model?

A model is Pareto optimal if no other active model achieves higher neutral quality at a lower or equal operational cost and latency. These models represent true efficient frontiers rather than marketing hype.

A saved scorecard uses weighted means and a two dimensional frontier

Your browser does not support video.

Read the explanation

The saved Claude profile has hallucination seven point two on fixed five to sixteen bounds. Its inverted score is one hundred minus two point two divided by eleven times one hundred, giving eighty. Latency four hundred twenty milliseconds on fixed one hundred fifty to nine hundred fifty bounds gives sixty six point two five. At three pixels per point hallucination score measures two hundred forty, speed one hundred ninety eight point seven five and gap thirteen point seven five measures forty one point two five. Accuracy is used directly, while cost uses a fixed logarithmic scale. These are transformed saved values, not percentiles estimated from current candidates or fresh measurements of a real model. Balanced weights are thirty five accuracy, twenty five hallucination, twenty speed and twenty cost. The script adds weighted terms and divides by their total: an arithmetic mean, despite the FAQ calling it harmonic. For the saved Claude profile cost score is about fifteen point eight seven, giving raw score about sixty seven point four seven. Contamination risk point two times point one two times factor one point five discounts three point six percent, yielding about sixty five point zero four and displayed sixty five point zero. At four pixels per rounded point raw sixty seven point five measures two hundred seventy, final sixty five two hundred sixty and rounded loss two point five ten. Assigned contamination risk is not measured test leakage. Eight saved profiles become nine when a custom candidate is inserted. The form accepts name, accuracy and cost, but assigns hallucination nine point five, latency four hundred milliseconds and contamination risk point one regardless of the endpoint label. At thirty pixels per count original eight measures two hundred forty, new nine two hundred seventy and added candidate thirty. No endpoint is called. Pareto comparison checks final score and cost only; latency influences the composite but is not an independent dominance dimension. Reset removes custom profiles and restores balanced weights, and native chart tabs and JSON export work locally. The vendor labels, prices and scores are saved illustrative fixtures, not a current purchasing recommendation or independent benchmark.