I Built an Agent That Asks Your App What Happened to It on SigNoz
Discover Why Did It Break, an investigation agent built on SigNoz that answers natural language app troubleshooting queries with direct trace evidence.
Stock photo for illustration only, not from the actual event
- Built for the Agents of SigNoz hackathon in July 2026.
- Answers production troubleshooting questions using natural language with confidence scores.
- Links every claim directly to exact traces in SigNoz for complete transparency.
- Solves agent runaway loops through strict request budgeting and preflight health checks.
When production breaks at 2 AM, the evidence is already there. Every request handled, database call, timeout, and retry sits recorded inside your observability stack. Reading through it, however, requires query builders, a mental map of the architecture, and about forty minutes that an on-call engineer simply does not have.
That inspired the creation of Why Did It Break, an investigation agent operating on top of SigNoz. Users can ask questions just like talking to a colleague—such as "why is checkout slow?"—and the agent executes live queries, follows leads like a real engineer, and returns the root cause, evidence, suggested fix, and its confidence level. Every single claim links straight to the exact proving trace in SigNoz.
Stock photo for illustration only, not from the actual event
Before writing code, the creator set a strict constraint: nothing gets faked. No seeded demo data or canned answers allowed. If the agent cannot support a claim using live telemetry data, it must admit it does not know and lower its confidence score. This single rule forced every feature to survive actual contact with real data instead of scripted happy paths.
Constructing AI agents for observability represents a pivotal shift from passive dashboard monitoring to active conversational troubleshooting. The primary engineering hurdle remains grounding the AI's outputs to prevent hallucinations. Requiring strict programmatic links to underlying trace spans ensures that operators can verify every automated diagnostic claim inside real production environments.
The demo system runs HotROD, a multi-service ride-dispatch application operating under continuous real HTTP load. Its performance bottlenecks and failures are entirely genuine, featuring MySQL SELECT queries taking over a second under contention and intermittent Redis timeouts during driver lookups.
Beyond reactive queries, the system handles proactive trace-based alerting. When live traffic pushes error rates over thresholds, SigNoz fires a webhook notification. The agent investigates independently, measuring an 18.5% error rate against the 10% alert threshold, checking the alert's claims directly against raw spans before anyone even opens a dashboard.
On deadline day, paid API credits ran out entirely. The developer filtered the OpenRouter catalog for free models supporting genuine tool-calling and switched to a free 550B model. Because the model cited trace IDs in prose rather than structured arrays, a custom backfill was added to pull mentioned trace IDs into clickable evidence cards.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment