Receiving Webhooks Without Getting Burned: A Production Guide
An in-depth operational guide to safely receiving webhooks, handling retries, race conditions, and out-of-order events.
Stock photo for illustration only, not from the actual event
- Always verify signatures using timingSafeEqual on the raw body
- Check timestamps to prevent replay attacks
- Design your handler to be idempotent to handle retries safely
- Use precedence rules instead of timestamps for out-of-order events
Sending a webhook is trivial. You just POST some JSON to a URL and move on. Receiving one, however, is where the hidden pitfalls lie. The endpoint is public, meaning anyone can call it. It gets retried, so it will run twice. It arrives out-of-order, meaning a 'delivered' status can land before 'sent'. And it sits directly on the critical path of someone else's system, so if you are slow, they time out and retry, making you even slower.
None of this is difficult once you know it, but all of it remains invisible until you hit production. This article covers the complete set of challenges a webhook receiver must handle and the specific failures each one prevents. We use email delivery webhooks (bounces and complaints) as our running example because they pack every awkward property into one: they are security-sensitive, they retry, they arrive out of order, and processing them twice corrupts actual state. These principles apply equally to Stripe, GitHub, Shopify, or any other webhook provider.
Your endpoint is a public URL, which means without verification, anyone who discovers it can post a fake 'hard bounced' event to suppress a customer, or a fake 'payment succeeded' event to grant a free subscription. Providers sign each request with a shared secret, and your job is to recompute the signature and compare it.
Stock photo for illustration only, not from the actual event
You must use timingSafeEqual rather than a standard === string comparison. A normal string comparison returns as soon as it finds a mismatching byte. An attacker who measures that timing can recover a valid signature byte by byte. It is a real, exploitable attack, but easily avoided with a single function call. Furthermore, always sign the raw body rather than the parsed object—this is the single most common webhook bug and notoriously frustrating to debug.
Timing attacks exploit the microscopic differences in execution time when a computer compares strings character by character. By using a constant-time comparison function like timingSafeEqual, you neutralize this vector entirely, ensuring that verification takes the exact same amount of time regardless of how many bytes match.
A valid signature only proves the payload came from your provider; it does not prove it is happening right now. An attacker who captures a legitimate request can replay it a month later with the signature fully intact. This is why providers include a timestamp in the signed payload, which your handler must actively inspect and validate.
Providers also enforce strict timeouts, frequently ranging between 5 and 30 seconds. Exceeding this window causes them to record a failure and trigger a retry. If your handler is sluggish because it dispatches emails, updates three database tables, and calls a downstream API, you will get retried while the first attempt is still executing, instantly doubling your workload.
Returning the correct HTTP status code is vital. A 200 response means 'I have it, stop retrying.' A 4xx response means 'this request is broken, do not bother retrying.' A 5xx response means 'try me again later.' Returning a 200 status on an error you could have recovered from will permanently discard the event. Retries are standard operations caused by timeouts, deployments, or network blips, so your handler must be built to process the same event twice without duplicating work.
Avoid solving this with a simple application-level 'have I seen this?' check. Two concurrent retries will both read 'no' before either can write. Instead, let the database enforce uniqueness. If the provider does not supply a stable event ID, synthesize one from fields identifying the unique occurrence—such as message ID combined with event type and timestamp. Never hash the entire payload, as providers add fields over time, which would invalidate your fingerprint for identical events.
Ordering guarantees do not exist. A retry sent three minutes ago can easily arrive after the delivery event that followed it. Handle this with state precedence rather than raw timestamps, ensuring terminal states outrank late arrivals. For events you must reject, store them safely before the provider gives up on retries and data vanishes into the void.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment