Anthropic Adds Plugin Evals to Claude Code
Anthropic updates Claude Code v2.1.269 with plugin evaluations, supporting 6 grader types, baseline comparisons, and CI gates for skills.
Stock photo for illustration only, not from the actual event
- Compatible with Claude Code v2.1.269 and later with billed API model calls
- Evaluation suites reside in evals/ directories inside each plugin folder
- Features 6 grader types combining computed checks and LLM judge models
- Measures true plugin impact by comparing runs with and without the plugin
Anthropic has introduced plugin evaluations to Claude Code starting with version v2.1.269, equipping developers with systematic tools to test and measure the performance of plugins and custom skills. The framework operates on any directory featuring a plugin.json manifest or a designated skills directory. Every evaluation execution constitutes a live model call billed directly to the user's plan or API account.
An evaluation suite is structured within an evals/ directory located inside the plugin folder. Each test case occupies a subdirectory containing a prompt.md file and a graders/ folder. The prompt body is transmitted to Claude exactly as written without expanding @path references, while frontmatter configurations dictate parameters such as max_turns (default 10), timeout_seconds (default 300), model specifications, tags, and allowed tools.
Stock photo for illustration only, not from the actual event
Graders are authored as markdown files utilizing frontmatter to declare their type, optional weight, and optional arm attributes. The framework provides 6 distinct grader types:
- regex: Computes checks from transcripts and disk files at zero cost
- tool_used: Validates tool invocation at zero cost
- tool_order: Checks the execution sequence of tools at zero cost
- file_exists: Verifies target file presence at zero cost
- llm: Invokes a judge model to score replies against prose criteria
- baseline: Compares outputs against a reference answer
Integrating plugin evaluations directly into Claude Code addresses a crucial visibility gap in AI agent development. Historically, developers often struggled when models failed to invoke custom skills naturally despite correct syntax schemas. By establishing a delta-based comparison between active and inactive plugin states, this feature bridges the divide between controlled laboratory tests and unpredictable production behavior.
By default, each test case executes twice: once with the plugin loaded and once without it. The resulting difference, denoted as delta, isolates the actual contribution provided by the plugin. If a case scores 1.0 in both scenarios, the plugin was not the driving factor behind the success. Additionally, the initialization command claude plugin eval init inspects plugins, proposes valid test cases, and generates necessary files, while the --bare flag accommodates headless CI pipelines.
"Anthropic calls out the most common first finding: a Δ near zero with the tool_used: Skill grader failing, which means Claude is not choosing the skill on natural phrasing."
MarkTechPost
Execution reports are generated under evals/results/ containing granular verdicts and judge votes. Supported accounts will automatically publish these reports to claude.ai unless suppressed via the --no-publish flag. Documented CI runner invocations utilize cost control flags such as --max-cost-usd 20, requiring a functional Claude Code installation alongside active credentials like ANTHROPIC_API_KEY to execute successfully.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment