>_

AI Agent Database Task Eval Matrix

Supabase Evals Benchmark v1.0
Overall Accuracy Rate 83.3% 5 of 6 target SQL assertions passed
RLS Security Pass Rate 66.7% 1 vulnerability detected (Infinite Loop)
Average Execution Latency 1420ms Per agent trace synthesis
Top Scoring Agent Claude Code (3.7) 100% precision on RLS & DDL
📊 Execution Log Matrix (6 Tasks) Showing all agents
Agent Task Name Category Status Latency
🔍 SQL / Security Diff Inspector eval-rls-01
Target Benchmark Tenant-Isolated RLS Policy
Agent Evaluated Claude Code (3.7 Sonnet)
Error Category None (Exact Match)
Generated SQL Output vs Benchmark Assertion
Security & Optimization Analysis

Policy correctly extracts org_id from JWT context and casts to UUID without subquery overhead or recursive loop risks.

Model Benchmark Comparison

RUNNABLE EVALUATION PROOF
Overall Accuracy Rate: 83.3% | RLS Security Pass Rate: 66.7% | Average Latency: 1420ms | Top Agent: Claude Code (3.7 Sonnet) | Total Executed: 6

Database Agent Benchmark Mechanics

How do database eval harnesses test AI coding agents on Row Level Security and vector index generation?

This harness compares AI agent output against reference assertions for PostgreSQL DDL, Row Level Security (RLS) policies, and pgvector HNSW index creation. In database evaluations, syntactically plausible SQL can still cause critical runtime faults—such as querying a protected table within its own policy check, which triggers infinite recursion error 42P17. When agents are filtered or re-evaluated, aggregate accuracy, category pass rates, and latency metrics recompute across the active test logs.

The execution log latencies and model accuracy scores shown in this interface are pre-scripted demonstration fixtures rather than live LLM API calls or isolated PostgreSQL container executions. Live database agent benchmarks require sandboxed test environments running pgTAP or CLI integration suites.

Try a worked example

Uncheck Codex CLI under Agent Models Under Test. The benchmark immediately filters out Codex CLI's logs—including its failing recursive tenant policy—raising the Overall Accuracy Rate from 83.3% to 100.0% and the RLS Security Pass Rate from 50.0% to 100.0% across the 4 remaining assertions.

Row Level Security and Recursive Policy Detection

PostgreSQL Row Level Security evaluates policy expressions as implicit WHERE filters on incoming queries. When an agent crafts a policy that queries the target relation within its own USING condition (such as querying documents while evaluating row permissions for documents), PostgreSQL detects circular rule evaluation and throws error 42P17 (infinite recursion detected in policy for relation). Row Level Security | Supabase Docs

Production architectures break policy recursion by reading JWT context claims like auth.jwt() directly, or by wrapping relation queries into SECURITY DEFINER functions with fixed search paths that bypass secondary policy invocation under controlled privileges. Row Level Security | Supabase Docs

pgvector Indexing and Operator Classes

Evaluating pgvector schema synthesis requires asserting both index algorithm choice (such as HNSW for hierarchical navigable small world graphs) and the matching distance operator class. When creating an HNSW index for cosine distance queries (<=>), the definition must explicitly designate vector_cosine_ops rather than Euclidean (vector_l2_ops) or inner product (vector_ip_ops). pgvector: Open-source vector similarity search for Postgres

Sources and further reading

Database evaluation metrics summarize supplied logs, not agent execution

Read the explanation

The page loads six supplied log rows, five labeled pass and one fail. Overall accuracy is therefore eighty three point three percent. Only two rows are Security and one passes, so the computed security rate is fifty percent, despite an initially declared sixty six point seven metric. Their six assigned latencies sum to eight thousand five hundred twenty milliseconds, giving average fourteen twenty. These are summaries of fixtures, not a fresh benchmark of real coding agents or database security. Filtering to the supplied Open Code row leaves one passing migration record. The overall rate becomes one hundred percent. Because there are no Security rows, the security metric also displays one hundred percent through an explicit empty-subset fallback. That second number is absence of observations, not evidence of perfect security. The top agent tie uses first encountered record order. A leaderboard built from unequal fixed samples cannot establish comparative model capability. Run Evaluation marks selected rows running, then after seven hundred milliseconds restores pass or fail from their existing error category. It neither executes SQL nor calls coding agents. A custom task copies your target SQL into generated SQL for each selected agent, assigns pass and a random latency, then reports success. With three selected agents, one custom task adds three rows and changes six rows to nine. Exporting these logs does not turn them into independent execution receipts or a valid database security audit.

Super generates helpful tools and automates fact-checking across the internet proactively. If you enjoyed this tool, build your own with Super and share it with a friend.