Why I stopped guessing at Spark and dbt config values

Spending more than a decade building data pipelines teaches you that the hardest part is rarely the pipeline logic itself. It is the tuning. Executor memory, shuffle partitions, cluster size, and thread counts require you to pick numbers, ship them, and wait for something to break weeks later in a way that clearly points to tuning issues without telling you what to change.

The pattern repeats often enough that experienced engineers recognize it instantly before opening logs. A job runs slowly due to thousands of tiny shuffle tasks from an overestimated partition count. A job dies from an Out-Of-Memory error because memory values were set for last quarter's data volume and forgotten. Cloud bills spike because a cluster was sized for peak loads and sits idle most of the day. This tribal knowledge usually lives in the heads of one or two senior engineers, failing to scale and disappearing entirely when someone leaves.

To fix this, a small tool was built to make that pattern-matching explicit rather than tribal. It reads pipeline configurations alongside actual run metrics to point out likely problems with clear reasoning attached, rather than throwing out arbitrary suggested numbers.

The obvious move in 2026 would be to adopt a machine learning model. However, for this specific problem, a handful of threshold rules already provides most of the value while remaining fully auditable. If a rule triggers, you can point to the exact condition and numbers, such as an average heap usage of 28% and a peak of 47% across five runs with zero OOM errors, proving that memory is over-provisioned and should be cut by roughly twenty percent.

Stock photo for illustration only, not from the actual event

There are currently seven active rules covering memory allocation, shuffle partition sizing, cluster instance counts, and two specific dbt parameters including thread counts and tasks per thread. Each rule operates as a pure function taking current configs and recent metrics as input, returning a recommendation or nothing, completely free of side effects and fully unit tested.