Ollama keep_alive: Model Reloaded 214 Times a Day
Discover how an Ollama local model reloaded 214 times out of 1,180 requests daily due to keep_alive settings and how to fix it.
Stock photo for illustration only, not from the actual event
- Ollama's default keep_alive is 5 minutes, which is too short for bursty workloads.
- Passing keep_alive via the OpenAI SDK compatibility layer can fail silently on certain versions.
- Setting keep_alive to -1 can spill layers to the CPU, dropping generation speed dramatically.
- Configuring OLLAMA_KEEP_ALIVE=24h at the server environment level permanently resolves reload issues.
A software developer noticed their local chat application felt fast during testing but frustratingly slow during actual daily use. After ignoring the issue for weeks, they decided to instrument every request over a 24-hour period. Out of 1,180 total requests, the model was evicted and reloaded from disk an astonishing 214 times, representing a reload event for every 5.5 requests.
The root cause lies in Ollama's default keep_alive setting of 5 minutes. After five minutes of inactivity, the runner exits, and model weights are flushed from VRAM. Consequently, the next request forces the system to re-read gigabytes from disk and re-allocate VRAM before token generation can even begin—a terrible default for side projects with sporadic traffic patterns.
Stock photo for illustration only, not from the actual event
Verifying active status is straightforward using the ollama ps command. If the output remains empty while you believe a model is running, it actually resides on disk. Additionally, server logs can be queried on Linux using journalctl -u ollama --since "24 hours ago" | grep -ci "llama runner started" to track exact daily load frequencies.
Understanding Ollama's memory management mechanisms is crucial for developers running large language models (LLMs) on constrained hardware. Constantly swapping models between disk and VRAM not only spikes latency but also increases storage wear from reading massive weight files repeatedly. Setting proper environment variables ensures stable performance without requiring hardware upgrades.
Another pitfall encountered was passing the keep_alive parameter through the OpenAI SDK pointing to the /v1 endpoint, where the compatibility layer stripped it without throwing errors. Conversely, setting keep_alive to -1 to keep models permanently loaded introduced a different bottleneck: running both an 8B chat model and a 14B coder model on a 16GB GPU caused the 14B model to spill layers onto the CPU, dropping token generation speed from 42 tokens per second down to just 6.
The effective mitigation steps, ordered by impact, include:
- Setting the server environment variable OLLAMA_KEEP_ALIVE=24h via systemd or launchctl.
- Restricting the GPU to a single resident model to eliminate resource thrashing.
- Offloading embedding tasks to a separate CPU-only instance to protect GPU VRAM.
- Removing custom warmup cron jobs since the native keep_alive parameter handles persistence cleanly.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment