Structured Output From LLMs: A Retry-Repair Loop
Learn how to build a robust 3-layer architecture for reliable LLM structured outputs, combining provider constraints, strict validation, and repair loops.
Stock photo for illustration only, not from the actual event
- Relying solely on prompt instructions for LLM JSON output fails under real traffic.
- A stable architecture requires three layers: constraints, validation, and repair.
- Automated repair loops feed exact errors back to the model for quick fixes.
The first time I wired an LLM into a real product feature, I did the naive thing by prompting the model to return JSON and parsing the response immediately. It worked wonderfully in the demo environment. However, once real traffic hit the service, FormatException errors started popping up at 2 AM because the model wrapped its JSON in code fences, added conversational filler, or left trailing commas. A model boasting 97% accuracy still fails thousands of times a day, proving that reliable structured output requires a dedicated pipeline rather than simple prompting tricks.
This is the exact pattern I rely on whenever an LLM needs to return a typed object instead of plain prose, such as contact extraction, invoice parsing, or database row conversion. Internalizing this pattern puts an end to endless firefighting over malformed JSON responses in production.
Stock photo for illustration only, not from the actual event
LLM failure modes when dealing with raw JSON are predictable and repetitive. Treating them as isolated bugs to squash one by one is a mistake; instead, they should be absorbed architecturally as a class of failures.
From a software engineering perspective, relying entirely on an LLM to consistently generate valid JSON in production introduces significant vulnerability. Implementing validation and repair mechanisms acts as defensive programming, ensuring external systems remain resilient even when the AI hallucinates or formats data incorrectly.
You have three primary tools to enforce structure, listed in order of descending strength:
- Provider-level response constraints and schema enforcement.
- Strict validation against your own defined schema source of truth.
- Automated repair loops that feed validation errors back to the model.
A common misconception is that strict schema modes completely eliminate the need for validation. While syntax validation ensures the string can be decoded, it does not guarantee that required fields are filled correctly or that business logic rules are met. Therefore, generating within tight constraints, validating locally, and repairing iteratively remain essential steps.
"Reliable structured output isn't a prompting trick — it's a small pipeline, and the last stage is a repair loop your parser never sees through."
Shakib
Ultimately, achieving reliable structured output is a layered contract rather than a single switch. By constraining the model, validating against your own schemas, and wrapping requests in a targeted repair loop, your parser will never have to deal with malformed JSON again.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment