Supabase Releases Evals: An Open Source Benchmark for AI Coding Agents
Supabase has launched Evals, an open-source benchmarking tool running locally via pnpm to score AI systems like Claude Code and Codex on real tasks.
Stock photo for illustration only, not from the actual event
- Supabase Evals is a new open-source benchmarking tool under Apache-2.0.
- Evaluates AI agents across products, topics, and development stages.
- Runs real environments in Docker containers utilizing actual MCP servers and CLI.
- Reveals performance metrics for top-tier models including Opus 5, Kimi K3, and Sonnet 5.
Supabase has officially released Supabase Evals, a public open-source benchmarking framework available under the Apache-2.0 license. Designed to run locally via pnpm, the tool measures the performance and accuracy of AI coding agents such as Claude Code, Codex, and OpenCode using scenarios grounded in actual bug reports, support tickets, and GitHub issues.
To ensure thorough evaluation, Supabase structured the benchmark across three core dimensions: products (encompassing databases, auth, storage, edge functions, and more), topics (such as RLS, security, migrations, and SQL), and stages (ranging from build and deploy to investigate and resolve). The framework then selects a concise scenario set touching each dimension.
Stock photo for illustration only, not from the actual event
The scenarios are split into two distinct suites. Benchmark scenarios test breadth and are publicly published, while regression scenarios target known failure modes, refresh daily, and leave published scores unaffected. Every scenario runs against a live environment where the framework boots a hosted-like stack and a local CLI project inside containers, allowing agents to interact directly with the real MCP server and CLI.
The introduction of Supabase Evals highlights a crucial shift toward domain-specific benchmarking for generative AI in backend engineering. Generic coding benchmarks often fail to capture the intricacies of managing databases, security policies, and cloud infrastructure. By anchoring tests in real-world support tickets and containerized local stacks, Supabase provides an invaluable yardstick for measuring how well AI assistants handle complex developer workflows.
Interestingly, the framework's design—combining deterministic checks with an LLM-as-a-judge approach—sets a new standard for evaluating non-deterministic agent outputs in enterprise software environments.
Test results demonstrated that agents successfully clear most scenarios without any pre-loaded skills. During the build stage, both Opus 5 and Kimi K3 achieved a perfect 100% unaided. Integrating custom skills successfully closed remaining performance gaps for other models, lifting Sonnet 5 from 78% to 100% and GPT-5.6 Sol from 89% to 100%.
Despite strong performances, the evaluation uncovered three notable weaknesses: agents tend to hand-write database migrations rather than using declarative schemas, they verify authentication manually instead of utilizing the @supabase/server package, and documentation consumption varies wildly—with Codex reading around 8 documentation pages per scenario compared to Claude Code, which checks documentation in under 40% of scenarios.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment