Google Research Opens Sources RRSI for AI Agent Harnesses

Google Research has officially unveiled RRSI, a new open-source research framework released under the Apache 2.0 license designed to let AI agents improve their own evaluation harnesses while avoiding severe overfitting. The codebase requires Python 3.10 or higher and accepts any LiteLLM model string, with defaults pointing directly to Claude Opus 4.8 hosted on Vertex AI.

Harness evolution loops operate by proposing specific code modifications, scoring them against a fixed evaluation set, and retaining the winning version. Because identical tasks are reused across rounds, the loop risks memorizing them. The RRSI research highlights three distinct failure modes: benchmark-specific fitting, noise chasing, and complexity accumulation. Each of these factors widens the performance gap between evolve-set scores and genuine out-of-distribution transfer.

Instead of locking down components, RRSI keeps every harness element fully editable while regularizing how the search space transitions. The researchers map these mechanics directly to classic regularizers: edit budgets correspond to L0, code pruning maps to Lasso (L1), and cost rules align with Ridge (L2).

Stock photo for illustration only, not from the actual event

Performance metrics across all six held-out splits showed notable improvements. Utilizing Gemini 3.5 Flash as the underlying policy, Terminal-Bench 2.1 scores rose from 64.6 to 78.7, while SWE-bench Verified advanced from 76.8 to 79.0. Furthermore, the overall harness footprint became significantly lighter. Running on agentic workspace instances, RRSI consumed 2.42 million policy tokens per trial compared to 3.80 million used by unregularized evolution loops—representing a 30 percent reduction in the abstract and up to 36 percent on the project page.