Four False Positives in One Evening: Lessons from Testing Web Apps
A developer shares insights from testing 27 web applications, detailing four instances where automated tooling falsely flagged healthy apps as broken.
Stock photo for illustration only, not from the actual event
- Automated checks flagged 4 out of 27 web apps as broken when they were fully functional.
- DOM and network checks alone cannot distinguish between real failures and tool limitations.
- Taking a screenshot first before any measurement is the highest-yield step in the process.
- Most false positives stem from aggressive filtering, WAF blocks, or misconfigured test environments.
The investigation began by spending an evening opening 27 product configurators, which included 3D and parametric tools on manufacturer websites, to uncover genuine defects. While the findings were real, the most valuable takeaway was that on four separate occasions during that single evening, automated tooling reported that an application was broken when it was completely fine. Every single one of those four apps had successfully passed automated checks designed to look rigorous, and the only thing that caught the false alarms was a screenshot.
For anyone writing scripts to evaluate external web pages—such as uptime checkers, competitor teardowns, scraping health monitors, or embedded widget QA—these pitfalls are virtually guaranteed. The common symptom across all four false alarms was the absence of a <canvas> element and an almost empty innerText, which appears alarming when a page is explicitly titled Configurator. However, this exact pattern matches three completely healthy scenarios.
Stock photo for illustration only, not from the actual event
This experience established a strict new operational rule before taking any measurements: take the screenshot first and look at the image. Anything invisible in the picture should not be measured. This single step takes only one second and yields the highest return throughout the entire debugging workflow. The key realization is that automation can answer whether a site is reachable, but never whether it actually functions correctly. When designing yet another automated pass, what is really needed is a human opening the page and clicking the obvious element.
"No canvas" and "empty innerText". The two least reliable signals I use. Demoted permanently: they are now a reason to look at a screenshot, never a finding.
Other misleading signals included reading pixels off a third-party WebGL canvas, which yields zero because WebGL clears drawing buffers after compositing unless preserveDrawingBuffer is explicitly enabled. Another misleading case was an ERR_ABORTED error caused by cutting off page loads prematurely before the network settled. Similarly, third-party widgets like chat tools, analytics, and captchas often trigger net::ERR_FAILED and CORS errors simply because the tester's own environment or ad blocker is filtering the requests.
Context and Analysis: False positives in automated web testing frequently occur when scripts evaluate pages outside their direct control. Security mechanisms such as WebFirewalls (WAF), datacenter IP filters, or requests lacking a legitimate User-Agent frequently receive 403 or 503 status codes even when the target website is operating normally. Recognizing these environmental constraints prevents developers from wasting time investigating healthy infrastructure.
Further filtering issues include headless requests lacking a User-Agent, which often return a 403 status and a tiny body while standard browsers receive a 200 OK response. A robust replacement for unreliable external proxies is to fetch a neighbouring object from the same network and request context, since no geo-filter or WAF can selectively 404 a single file while serving its exact neighbour. Additionally, a practical operational tip learned the hard way is to give every tested site its own dedicated browser tab to prevent a single hanging navigation from interrupting subsequent checks.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment