ByteDance Seed's HarnessDev: LLMs Engineer Agent Harness
New research by ByteDance Seed tests six LLMs in engineering their own agent harnesses, finding that only 34 of 64 changes generalize.
Stock photo for illustration only, not from the actual event
- Tested six creator LLMs in building their own agent harnesses
- Found only 34 out of 64 modifications successfully generalized
- Significant portions of generated code and features remained dead code
ByteDance Seed has released findings on HarnessDev, evaluating whether large language models can engineer their own agent harnesses. Starting from a basic, weak seed with no loops, planners, or verifiers, creators built full harnesses evaluated across multiple benchmark tasks.
The evaluation tested six prominent creator LLMs—Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro—operating inside Claude Code 2.1.177. The tests spanned four domains and five benchmarks totaling 2,207 instances, including SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp.
Results highlighted that much of the generated machinery was inert. Out of 108 code component instances, several state and memory elements never fired, and over a hundred writing features turned out to be dead code. Feedback and held-out scores moved in the same direction only 53.1% of the time, revealing generalization hurdles.
Allowing language models to self-engineer their execution harnesses represents a bold step toward autonomous AI systems. However, these findings indicate that models frequently overfit to specific feedback loops, producing redundant structures or hard-coded limits that fail when executors change.
Under self-evaluation, Opus 4.8 achieved the highest average score of 67.8 against a human reference of 86.2. Swapping the execution engine to Gemini significantly shifted rankings and performance metrics, demonstrating how tightly coupled generated code can be to its original environment.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment