Four Ways a Batch Runner Fails: Dev.to Insights
An in-depth look at content pipeline batch runner development on 16 March 2026, exploring four silent bugs and architectural lessons.
Stock photo for illustration only, not from the actual event
- The batch runner for our content pipeline landed on 16 March 2026 with resusability built-in.
- Entities reside in one of five states, with failures categorized into six distinct types.
- An audit uncovered four independent bugs that caused the batch runner to falsely report completion.
On 16 March 2026, the batch runner for our content pipeline landed in a single commit. That commit already contained a resume flag, a state log with per-item status, error classification, a quota pause, and a lock file. Resumability was not bolted on after it hurt. It was there on day one, and it still did not work for another three months.
The order in which we built it is not flattering, so I will put it first. Eleven days before the runner existed, we had a circuit breaker stopping after three consecutive quota failures and a shutdown path killing child processes. Four days after the runner, we shipped a retry continuing a half-written file instead of regenerating it, confusing retry patterns with actual resume capabilities.
Stock photo for illustration only, not from the actual event
An entity is in exactly one of five states, and a failure additionally carries one of six categories: timeout, quota, validation, phase_error, planning_error, and unknown. Keeping the category off the state axis is what lets paused and failed be different verbs at all. Quota exhaustion is not a defect; it tells the system to come back later, separating pausing from failing.
Separating error categories from core entity states is a vital architectural pattern in modern batch processing systems. By distinguishing between temporary metered API limits and deterministic validation errors, systems prevent wasteful retries on unfixable inputs, thereby optimizing overall computational resources.
An audit had turned up four independent ways the resumable runner could be silently wrong, and all four fixes landed in one day. First, a quota error came back as an empty result instead of raising an exception. The parallel aggregator treated it as a valid data value, leading the batch to report success and record missing content as produced.
Second, three separate stack locations recognized rate-limit errors using their own string matching rules, leading to inconsistencies. Third, a corrupted state file was read cheerfully as a fresh start with zero progress, causing us to pay for everything again. Fourth, the lock mechanism suffered from a check-then-write race condition wide enough for two concurrent runners to bypass.
"File-based locking on one machine is not distributed coordination, and I am not going to pretend otherwise."
Jula Markova
The real fix was abandoning the authority of the state file entirely. Instead of querying an untracked local progress file to check completion, status is now derived directly from signals committed to the repository. The state file was demoted to a cache, rebuilt from derivation, and ignored whenever discrepancies arise.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment