Evaluate real physical-world risk in frontier multimodal agents. Inspect refusal adherence, hazardous execution attempts, and net completion probability under standardized chemical, thermal, and kinetic threat vectors.
| Model | Attempt % | Success given Attempt | Refusal % | Net Completion % | Wilson 95% CI | Risk Tier |
|---|
Evaluating an AI agent solely on refusal rate ignores physical execution efficacy. Even if an agent refuses 80% of attempts (like Fable 5.1), a 34% completion rate on remaining attempts still yields a 27.2% overall harm completion rate—far above acceptable industrial safety thresholds.
Small trial counts (n < 100) introduce wide binomial variance. This analyzer computes continuous 95% Wilson confidence intervals to ensure safety assurances are not artifacts of statistical noise before robotic or API deployment.
Unlike text-only jailbreaks, physical safety tests require multi-step planning, tool manipulation, and override of physical containment. Tracking kinetic, chemical, and pressure vectors prevents deceptive single-domain compliance.