I Checked 18 Claims in My Spec: 5 Were False
A developer built a tool to evaluate 18 specification claims, uncovering 5 false statements, 4 unknowns, and the risks of vague technical terminology.
Stock photo for illustration only, not from the actual event
- A developer built an automated tool to evaluate 18 formal specification claims.
- Results showed 5 false statements, 9 valid ones, and 4 unknown cases.
- Most errors stemmed from using rigorous mathematical terms with loose meanings.
The specification for this project was deliberately written using precise language, employing terms like this map is a homomorphism or these operations form a lens rather than converting X to Y, because exact phrasing carries strict obligations while vague descriptions do not.
Previously, nobody had ever verified whether those formal obligations were actually met. The author built a tool that reads each claim, resolves terms against actual definitions, and returns valid, invalid, or unknown. Out of 18 specification rows, 9 were valid, 5 were invalid, and 4 remained unknown, with every invalid claim sharing the same core mistake: using a precise word at the strength of its vague synonym.
Stock photo for illustration only, not from the actual event
The clearest case involved a spec stating that the encoder is a homomorphism into byte strings under concatenation, which carries a strict obligation: enc(a · b) = enc(a) ++ enc(b). In reality, the encoder processes pairs as enc(pair(x, y)) = tag ++ enc(x) ++ enc(y), prepending a single byte to identify the subsequent data type.
That single byte enables unambiguous decoding, meaning it performs essential work and cannot be removed. However, it also renders the equation false for every single input, as no pair exists where both sides are mathematically equal. The encoder is actually an injective and prefix-free map, which are properties the system relies on, even though it is not a homomorphism.
The reliance on advanced technical terminology in documentation often creates a false sense of security. Without automated validation, human readers naturally interpret prose through the lens of what they expect to see, allowing hidden contradictions separated by dozens of lines to evade detection for years.
Another issue arose where DAG, strict order, covering relation, and transitive closure were treated as if they named the same concept. While related, they are not interchangeable; a DAG is not transitive, and taking the transitive closure of a covering relation simply restores the original order. Yet the specification contained a sentence written as though a single rule applied to all four terms.
Furthermore, one row asserted round-trip laws requiring get(put(v, s)) = v and put(get(s), s) = s, while a separate section 130 lines away plainly stated that the two maps do not compose to identity. Such contradictions remain completely invisible during manual reading because humans never hold both distant parts of a document in their minds simultaneously.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment