Structured Output Reliability: JSON Mode vs. Constrained Decoding
A pipeline calls an LLM and parses the response as JSON. The pipeline runs 100,000 times a day. Roughly 8,000 of those responses fail validation — wrong type on a field, missing a required key, an enum value the model invented, a number where a string was expected. Each failure becomes a retry, an alert, or a silently-dropped record. At enough scale, “the model usually returns valid JSON” stops being a feature and starts being a bug.
This is the structured-output reliability problem, and the model on its own will not solve it for you. There are three real strategies that close the gap, in increasing order of cost and effectiveness: provider JSON mode (what you got for free), validate-and-retry (the Pydantic + Instructor pattern), and constrained decoding (Outlines, xgrammar, llguidance, SGLang and vLLM’s grammar backends, llama.cpp’s GBNF support). The choice between them is not a religious question — it is a budget decision driven by the reliability your downstream system actually needs.
This article walks through all three, including a working compiler from Pydantic schemas to GBNF grammars, and ends with a calibrated simulation showing what each strategy actually delivers at scale.
Disclaimer: The opinions expressed in this article are my own and do not represent the views of…
