Verdict: An Evidence-First Agent Harness for Bug Fixes
Discover Verdict, an open-source framework from the WeMakeDevs x TrueFoundry Hackathon that turns bug reproduction into verifiable evidence.
Stock photo for illustration only, not from the actual event
- Verdict turns GitHub issues into bounded, verifiable investigations
- It is an evidence-producing agent harness, not an autonomous patch generator
- Features structured components like Hunter, Surgeon, and Insurance
- Built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon
Most flaky bug reports typically conclude in one of two frustrating outcomes: a status of "cannot reproduce" or a patch that nobody can prove actually resolved the issue. The Verdict system was engineered to address this exact dilemma by transforming a GitHub issue into a strictly bounded investigation workflow. It executes approved commands repeatedly under approved conditions, logs every single observation, and refuses to claim a reproduction unless the gathered evidence crosses a strict deterministic threshold.
This framework is purposefully not an autonomous patch generator. Instead, it operates as an evidence-producing agent harness designed for the difficult diagnostic phase that must precede any code patching. While a Large Language Model (LLM) can read a stack trace and quickly propose a plausible explanation, a plausible theory is fundamentally different from a verified reproduction.
Stock photo for illustration only, not from the actual event
Verdict's architecture relies on distinct operational components that work together to maintain transparency and strict traceability throughout the investigation:
- Hunter: Searches exclusively within the condition matrix and command budget approved by the maintainer, keeping all successful, failed, and partial runs in an immutable ledger.
- Surgeon: Narrows the reproduced conditions down to the tightest suspect range supported by records, without authoring any patches.
- Insurance: Converts the reproduction into a regression plan encompassing test names, fixtures, failing assertions, and publication manifests.
The underlying philosophy of Verdict highlights a critical challenge in applying AI to software engineering. True certainty in code maintenance cannot be established merely through conversational confidence or model predictions; it requires a transparent, reproducible ledger of evidence. This methodology significantly mitigates the risk of AI hallucination, ensuring that proposed fixes are grounded strictly in verifiable execution boundaries.
In practical testing, Verdict successfully reproduced TrueForge issue #417, where snapshot registration stalled indefinitely due to an unresolved upstream request. Operating on a pinned runtime at @truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider, it evaluated comparative conditions that provided substantially stronger evidence than unstructured failures.
The verification output yields a deterministic SHA-256 canonical hash, permanently binding the observations to the TrueForge session, Hunter thread, repository commit, and shared source blobs. The AI model gathers candidate observations, but it is strictly restricted from deciding what those observations prove, maintaining a firm boundary between agent exploration and systematic claims.
Developed for the WeMakeDevs x TrueFoundry Agent Harness Hackathon and released under the MIT license, the repository currently maintains a suite of 220 tests spanning agent, protocol, and web packages. Ultimately, Verdict's core mission is not to make an AI agent sound certain, but to make certainty entirely inspectable.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment