Microsoft Open Sources Code-Testing Agent That Hits 92.1% Task Completion
Microsoft releases a polyglot unit-test agent definition that runs locally, outperforming stock GitHub Copilot significantly in internal benchmarks.
Stock photo for illustration only, not from the actual event
- Microsoft open-sourced a code-testing agent definition complete with skills.
- Achieved 92.1% task completion on internal benchmarks versus 78.9% for stock Copilot.
- Executes work through a Research-Plan-Implement (RPI) pipeline with a 5-check validation.
- Excels significantly on vague prompts and precise diff-targeted tasks.
Microsoft has released and open-sourced a new code-testing generation tool structured as an agent definition along with specific skills. Because it operates as a local definition rather than a hosted service, it runs directly inside an existing coding agent on the developer's machine, ensuring all source code remains strictly local.
The workflow is coordinated via a Research-Plan-Implement (RPI) pipeline. It begins by scanning the repository for code needing test coverage, detecting programming languages and testing frameworks, reading existing test conventions, and identifying actual build and test execution commands. This specifically addresses a common breakdown where test projects build locally but fail to run in CI pipelines because they were never registered.
Stock photo for illustration only, not from the actual event
The agent then selects one of three distinct strategies: direct writes with immediate validation, a single pass execution, or an iterative loop for larger scopes and coverage targets. Notably, it strictly avoids modifying production code and steers clear of tests that trigger external URLs, bind ports, or rely on timing dependencies.
By releasing this tool as an open-source agent definition rather than a proprietary SaaS product, Microsoft empowers developers to maintain strict data privacy while integrating advanced autonomous testing directly into their local environments. This architecture highlights a shift toward modular, agentic workflows embedded within existing developer toolchains.
Before reporting task completion, the agent enforces five rigorous checks. It reasons about minor code changes that should trigger test failures as a lightweight mutation testing approach, scans for weak or missing assertions, maps every requested scenario to a test case, builds the entire workspace, and verifies that the repository's native test command successfully discovers the newly generated tests.
On Microsoft's internal benchmark consisting of 152 tasks derived from real repositories, the agent successfully completed 140 tasks (92.1%), outperforming the stock GitHub Copilot which scored 120 tasks (78.9%) using the exact same models and prompts, representing a 63% reduction in failures. Performance gains were especially concentrated in vague prompts and targeted diff scenarios.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment