I Seeded Bugs Into My Own PR to Test the AI Reviewer
A student developer tests a free AI code reviewer by intentionally seeding a sign-flip logic bug into a discount calculation function.
Stock photo for illustration only, not from the actual event
- A student developer tested an AI reviewer by intentionally planting a logic bug in their code.
- Built a Python-based regression harness utilizing MonkeyCode's free services.
- Discovered that AI model behavior and accuracy can drift over time.
- Emphasizes that AI reviewers are software and require proper regression testing.
A developer was just one merge away from shipping their code when the AI reviewer declared that no issues were found and everything looked good. That moment of approval sparked instant doubt about what a positive review actually meant when the reviewer was a model they had never personally tested.
To find out, they stopped reviewing code and started reviewing the reviewer. They planted a known bug into a function and challenged the free model to catch it, questioning whether a free AI reviewer could spot an error they intentionally placed there.
The function used for the test is a compact Python snippet designed to calculate prices with discounts:
def price_with_discount ( price , discount_pct ): if discount_pct < 0 or discount_pct > 100 : raise ValueError ( " discount must be between 0 and 100 " ) factor = 1 + discount_pct / 100 # seeded bug: should be minus return round ( price * factor , 2 )Spotting the flipped sign means passing the initial test, but the real challenge is whether a model reviewer flags the exact same issue when the function is buried deep inside a much larger file.
Treating an AI reviewer like software that requires regression testing introduces a crucial engineering mindset. Because foundation models undergo silent updates and changes over time, a reviewer that performs well one month might fail the next. Establishing automated harnesses shifts the evaluation from hopeful guesswork to measurable reliability.
The prerequisites required to run this evaluation setup include:
- Python 3.10 or higher.
- An API endpoint compatible with the OpenAI chat-completions format along with an API key.
- The author utilized free model access and server hosting provided by the open-source project MonkeyCode as part of product outreach.
The expected output shape when executing python review_harness.py price.py looks like this:
{ "caught" : true , "response" : "Line 4: `factor = 1 + discount_pct / 100` — the sign is flipped. A 20% discount increases the price by 20% instead of reducing it." }The evaluation harness checks for specific symptom phrases rather than strict exact matches, introducing a deliberate tradeoff that avoids missing correct logic due to minor wording variations.
While a one-off run answers an immediate question, scheduling automated runs via cron jobs answers whether the reviewer remains reliable over time. The turning point was when the model confidently missed a planted sign flip in an unchanged file simply because the underlying model had drifted.
Three major lessons emerged from the experiment: un-testable reviewers are just opinions with good grammar, seeding bugs is a calibration tool rather than sabotage, and one-off tests quickly grow stale.
This approach isn't suitable for codebases you cannot modify, and free tiers come with strict rate limits and latency constraints. However, building a regression harness ensures developers understand what their AI tools are actually capable of catching.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment