I Audited 2,864 CI Checks and Found Silent Failures

A simple one-line question on Hacker News about security checks run on every pull request prompted a developer to count every single CI check they had ever executed. The total reached 2,864 checks distributed across nine repositories. While the direct answer to the question turned out to be straightforward, what was uncovered along the way was far more surprising.

Failure rates were not spread evenly throughout the timeline. Instead, they appeared in tight three- or four-day clusters separated by months of complete silence. For instance, in late April when these checks were integrated into a shared reusable workflow, eighteen cost-gate failures occurred over four days while sorting out credential and fetch-depth bugs, followed by ten weeks of nothing.

Repositories with the highest failure rates in the entire organization were those stuck permanently in the setup phase without any steady state for baseline comparison. The worst repository sat at a 57% failure rate, serving as the template cloned by others. None of this context existed within the raw numbers alone; it lived entirely in the developer's memory.

Stock photo for illustration only, not from the actual event

The deeper investigation kicked off when a Vale rule named EmDashes in the content pipeline failed to catch fifteen em dashes present in a newly shipped post, reporting zero errors instead. The root cause lay in the regular expression wrapping word boundaries around punctuation marks that lacked adjacent word characters to anchor against.

"Here is why. Vale's existence check compiles a tokens list into a regex with word boundaries wrapped around it."

The Developer

Compounding the issue, the execution command utilized --no-exit to force a zero exit code regardless of findings, paired with a grep command searching for a severity field that did not exist in the output format. Consequently, the check had passed every single build it ever ran without doing any actual gating. Further inspection revealed missing .mdx file mapping causing false alerts in code blocks, an unlinked Semgrep security tool, and an unprovisioned GitHub App token causing persistent cost-gate failures.

Source: Dev.to