lmcache

What it is and what it does

LMCache decouples KV cache management from inference engines, storing transformer caches persistently across GPU, CPU, and remote storage tiers. It enables reuse of cached key-value blocks across requests and sessions, reducing redundant prefill computation for multi-turn conversations, agentic workloads, and retrieval-augmented generation (RAG). The layer runs as a standalone daemon, survives engine restarts, and provides observability through Kubernetes metrics, cache hit rates, and token-level diagnostics.

The package integrates with mainstream open-source serving engines and supports pluggable storage backends (CPU RAM, local SSD, Redis, S3-compatible systems, and others) and transport layers (NVLink, RDMA, TCP). It includes non-prefix KV reuse via CacheBlend for selective recomputation, prefill-decode disaggregation, and a flexible serialization interface for compression and token dropping. Vendor neutrality means you can switch storage or serving engines without losing cached KV data.

Use it for:

• Multi-turn conversational AI: cache KV blocks from earlier turns to skip prefill recomputation on follow-up messages.
• RAG and knowledge-augmented generation: reuse cached context embeddings across queries with the same knowledge base.
• Agentic workloads: share KV caches across multiple reasoning steps and tool calls within a single session.
• Long-context inference: offload KV caches to CPU or remote storage when GPU memory is exhausted, enabling longer prompts.
• Batch serving with cache sharing: reduce total prefill cost across requests that share common prompt prefixes.
• Cross-engine failover: persist KV caches so inference can resume on a different serving engine without recomputation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

LMCache is a KV cache management layer that stores, reuses, and monitors transformer key-value caches across LLM inference requests, reducing time-to-first-token and improving throughput for long-context workloads.

Yes, with conditions. Install if you run LLM inference at scale with multi-turn, RAG, or long-context workloads and want to reduce latency and cost. Active maintenance, permissive license, and strong community adoption are positive signals. Medium install friction (37 dependencies, Linux-only wheels) and alpha status mean you should test integration with your serving engine and storage backend before production deployment. No known vulnerabilities.

Install

lmcache on PyPI

pip install lmcache

uv add lmcache

poetry add lmcache

Installing lmcache

Before you install

Medium install friction: 37 runtime dependencies including torch, numpy, fastapi, redis, and observability libraries (opentelemetry, prometheus). Wheels available for Python 3.10–3.13 on Linux x86_64. Active maintenance with recent release (9 days old) and strong community signal.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions; you must retain license and copyright notices in distributions.

Quickstart

pip install lmcache

from lmcache import LMCache

# Initialize cache manager
cache = LMCache()

Requires Linux x86_64, Python 3.10–3.13, and torch; GPU recommended for typical LLM inference workloads.

Verify before relying

• Specific integration steps with mainstream serving engines beyond the high-level architecture.
• Performance gains (TTFT reduction, throughput improvement) quantified for representative workloads.
• Compatibility and tested versions for each supported storage backend (Redis, S3, local disk, etc.).
• Overhead of the daemon process and observability stack on latency-sensitive inference.

Package facts

Evidence: lmcache-0.5.3-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; lmcache-0.5.3-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; lmcache-0.5.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; lmcache-0.5.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Tags

More Artificial Intelligence packages

Further reading