Dev.to developer shares duplicate content check flaws

Managing a network of free calculator tools often comes with the challenge of multiple pages competing for the same search queries. To combat this, a site developer created a custom research script designed to compare target keywords against every existing page before building a new one. If an overlap is detected, the script fails the build, forcing the creator to either expand the existing page or clarify the different search intents, noting that Google Search Console previously revealed 63 queries over 28 days where two separate URLs from the same site appeared.

The initial version of the gate was straightforward: lowercase the text, strip stop words and the word calculator, then compare the remaining word sets. However, on its very first day of operation, the script failed in both directions. The first test query, "quebec paycheck calculator," was blocked due

Stock photo for illustration only, not from the actual event

to a strong word overlap with existing generic payroll pages. Because the existing pages were broad, their word sets were subsets of the keyword, causing the script to flag a match even though the addition of a specific location fundamentally changed the search intent.

The subsequent fix required limiting the subset check to a single direction, meaning an existing page only covers a keyword if it already contains every single word of that keyword. Despite this adjustment, a later plan to build a "bc income tax calculator" with 6,600 monthly searches slipped past the gate because the tool only inspected titles and URL slugs, completely missing the fact that a newly published BC paycheck calculator already contained the relevant phrases multiple times within its body text and meta descriptions.