Scaling System Connectivity with Event Hub Partitions, ACA, and Keda

This article represents part three of the User Connectivity Architecture series, revisiting a system pattern consisting of a timer-driven heartbeat, an Event Hub, a worker writing sessions into Redis, and Redis key expiration managing online and offline facility statuses. Normally, the API instructs the client regarding call frequency, defaulting to every 30 seconds. After running smoothly in production for two years, January 2026 brought a month-long operational crisis when heartbeat traffic escalated dramatically.

The anomaly manifested around 7:00 AM PST on January 5, 2026, when traffic patterns shifted from flat to overwhelming. Event volumes multiplied eightfold within a single hour and continued climbing upward. SignalR connections mirrored this surge, holding roughly eleven times the normal volume of active browser sessions that required tracking, maintenance, and status reporting, prompting an intensive multi-week investigation into the underlying triggers.

Upon uncovering the root cause, investigators identified 507 lingering zombie sessions running months-old cached client code alongside a single user account responsible for 33% of all token API traffic. However, the core architectural limitation lay deeper within the system design. In Azure Event Hubs, partition counts dictate parallel processing capability, where each consumer group assigns a single partition to one processor at a time, creating a single-lane bottleneck regardless of added instances or CPU upgrades.

Stock photo for illustration only, not from the actual event

While incoming events poured in at over a hundred per second, the HeartbeatMonitor processed them sequentially at a fraction of that speed, causing facility connectivity metrics relied upon by hospitals and emergency services to lag behind reality. Furthermore, the Consumption plan introduced SNAT port exhaustion as storm traffic depleted outbound ports faster than they could be released, creating a compounding feedback loop of failed calls and client retries.

Before executing a complete rewrite, the team stabilized the environment by modifying the API-driven heartbeat interval from 30 seconds to 90 seconds. This adjustment instantly cut incoming events by approximately two-thirds without requiring client deployments, maintaining operational stability while structural fixes were developed. Connectivity status resolution became coarser, but it kept critical services online.

To permanently resolve the bottleneck, a new Event Hub named heartbeat-p8 was provisioned with 8 partitions and a dedicated heartbeat-monitor consumer group, enabling eight parallel processing streams. KEDA was integrated to monitor unprocessed backlog in the Event Hub and dynamically scale worker replicas accordingly, with maxReplicas strictly matched to the partition count to prevent unnecessary rebalancing churn.

Source: Dev.to