rekuiper: Rust stream engine for edge memory limit

Most stream-processing benchmarks typically optimize for a single metric: peak throughput on a massive server. However, that figure is virtually useless where these engines actually run—such as industrial gateways, ESPHome hubs, vehicle head-units, or electric vehicle chargers. In those environments, you are allocated one or two CPU cores with only a few hundred megabytes of free RAM, while input arrives via MQTT and experiences severe burst traffic as vehicle fleets reconnect simultaneously following an outage.

In such scenarios, two critical questions dictate whether your data pipeline survives, neither of which is peak throughput. The development team built a stream engine called rekuiper to answer both affirmatively and established a rigorous benchmark to verify their success. This post focuses on the benchmark just as much as the engine itself, as it uncovered runtime correctness bugs that throughput-only tests would have rewarded as fast.

Across five MQTT workloads modeled after real-world deployments, rekuiper delivered complete and correct output at 100,000 messages per second on a single CPU core across every workload without hitting a ceiling. More importantly, on windowed workloads, rekuiper's memory remained between 5 and 10 megabytes while Go-based engines climbed to half a gigabyte to a full gigabyte or failed entirely. That performance gap stems from architectural design rather than the programming language itself.

Stock photo for illustration only, not from the actual event

rekuiper is a stream-processing engine written in Rust that reimplements the surface interface of LF Edge eKuiper, including its REST API, SQL dialect, stream definitions, and the kuiper command-line utility. The primary goal was seamless drop-in compatibility so existing eKuiper rules and deployment tools continue working without forcing a complete rewrite. The compatibility scope covers 98 REST paths and 140 operations validated against eKuiper's OpenAPI specs.

The core differences lie underneath, anchored by the principle that memory must remain bounded under heavy load. Three specific design choices enable this performance:

• An in-process stream bus featuring bounded per-subscriber queues of 4,096 records each
• An MQTT source utilizing rumqttc to batch incoming publishes up to 1,024 records to reduce scheduling overhead
• An incremental evaluator computing window aggregates based on device count instead of buffering raw rows

"rekuiper's memory stayed between 5 and 10 MB while the Go-based engines climbed to half a gigabyte to a full gigabyte, or failed."

rekuiper Development Team

For intermittent connectivity in remote or mobile deployments, sinks can leverage eKuiper options to enable caching. Recoverable send failures route records into FIFO queues—residing in memory up to a threshold before spilling to disk pages to prevent silent data loss during prolonged outages.

Source: Dev.to