---
id: llmlingua
version: "0.2.2"
license: MIT License
license_treatment: permissive
maintenance: active
---

# llmlingua — To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

License: permissive · Maintenance: active · Downloads: 96.0K/mo

## What it is and what it does

LLMLingua is a prompt compression toolkit that uses a small, efficient language model to identify and remove non-essential tokens from prompts before sending them to larger LLMs. It implements three variants—LLMLingua, LongLLMLingua (for long-context scenarios), and LLMLingua-2 (faster, distilled from GPT-4)—each designed to reduce token consumption while preserving task performance. The core use case is lowering costs and latency when working with token-metered LLM APIs or handling long contexts that would otherwise exceed model limits.

The package integrates with popular RAG frameworks like LangChain and LlamaIndex, making it straightforward to drop into existing pipelines. It depends on transformers, torch, accelerate, tiktoken, nltk, and numpy—standard ML infrastructure. The project is actively maintained by Microsoft researchers, with recent releases and integration into production systems.

Use it for:

- Reduce API costs when calling GPT-4 or GPT-3.5 by compressing prompts before submission.
- Handle long documents in RAG systems by compressing retrieved context to fit within token budgets.
- Mitigate the 'lost in the middle' problem in long-context LLM tasks by prioritizing key information.
- Accelerate inference latency by reducing KV-cache size and prompt processing overhead.
- Preserve in-context learning and chain-of-thought reasoning while shrinking prompt size.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

LLMLingua compresses prompts by identifying and removing non-essential tokens, reducing input length to LLMs by up to 20x while preserving task performance and lowering API costs.

Yes. The package solves a real problem—token costs and context limits—with active maintenance, low install friction, and permissive licensing. It integrates cleanly into existing LLM workflows and has been adopted by major frameworks. Install if you work with LLM APIs or long-context tasks and want to reduce costs or fit within token constraints.

## Install

pip install llmlingua
uv add llmlingua
poetry add llmlingua

## Installing llmlingua

Before you install: Low friction install with a pure Python wheel. The package depends on transformers, torch, accelerate, and other standard ML libraries—all widely maintained. Maintenance is active with recent commits and a healthy star count.

License in practice: MIT License permits unrestricted use, modification, and distribution in both open-source and commercial projects with minimal obligations.

Quickstart:

pip install llmlingua

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor()
compressed = llm_lingua.compress_prompt(
    prompt="your prompt here",
    instruction="",
    question="",
    target_token=200
)

Requires Python >=3.8.0 and a compatible version of torch and transformers; model weights are downloaded on first use.

Verify before relying:

- Whether the package works offline after initial model download or requires network access for inference.
- Memory footprint and GPU requirements for different compression methods (LLMLingua vs. LongLLMLingua vs. LLMLingua-2).
- Compatibility with specific LLM APIs (OpenAI, Anthropic, etc.) beyond the examples shown.

## Package facts

- License: MIT License (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 96.0K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

prompt compression for llms, reduce token usage, llm inference acceleration, prompt optimization, token pruning, cost reduction api calls, context compression, rag efficiency, prompt-engineering, cost-optimization, rag

[View on SkillFed](https://skillfed.io/packages/llmlingua) · [View on PyPI](https://pypi.org/project/llmlingua/)