llmlingua

What it is and what it does

LLMLingua is a prompt compression toolkit that uses a small, efficient language model to identify and remove non-essential tokens from prompts before sending them to larger LLMs. It implements three variants—LLMLingua, LongLLMLingua (for long-context scenarios), and LLMLingua-2 (faster, distilled from GPT-4)—each designed to reduce token consumption while preserving task performance. The core use case is lowering costs and latency when working with token-metered LLM APIs or handling long contexts that would otherwise exceed model limits.

The package integrates with popular RAG frameworks like LangChain and LlamaIndex, making it straightforward to drop into existing pipelines. It depends on transformers, torch, accelerate, tiktoken, nltk, and numpy—standard ML infrastructure. The project is actively maintained by Microsoft researchers, with recent releases and integration into production systems.

Use it for:

• Reduce API costs when calling GPT-4 or GPT-3.5 by compressing prompts before submission.
• Handle long documents in RAG systems by compressing retrieved context to fit within token budgets.
• Mitigate the 'lost in the middle' problem in long-context LLM tasks by prioritizing key information.
• Accelerate inference latency by reducing KV-cache size and prompt processing overhead.
• Preserve in-context learning and chain-of-thought reasoning while shrinking prompt size.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

LLMLingua compresses prompts by identifying and removing non-essential tokens, reducing input length to LLMs by up to 20x while preserving task performance and lowering API costs.

Yes. The package solves a real problem—token costs and context limits—with active maintenance, low install friction, and permissive licensing. It integrates cleanly into existing LLM workflows and has been adopted by major frameworks. Install if you work with LLM APIs or long-context tasks and want to reduce costs or fit within token constraints.

Install

llmlingua on PyPI

pip install llmlingua

uv add llmlingua

poetry add llmlingua

Installing llmlingua

Before you install

Low friction install with a pure Python wheel. The package depends on transformers, torch, accelerate, and other standard ML libraries—all widely maintained. Maintenance is active with recent commits and a healthy star count.

License in practice

MIT License permits unrestricted use, modification, and distribution in both open-source and commercial projects with minimal obligations.

Quickstart

pip install llmlingua

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor()
compressed = llm_lingua.compress_prompt(
    prompt="your prompt here",
    instruction="",
    question="",
    target_token=200
)

Requires Python >=3.8.0 and a compatible version of torch and transformers; model weights are downloaded on first use.

Verify before relying

• Whether the package works offline after initial model download or requires network access for inference.
• Memory footprint and GPU requirements for different compression methods (LLMLingua vs. LongLLMLingua vs. LLMLingua-2).
• Compatibility with specific LLM APIs (OpenAI, Anthropic, etc.) beyond the examples shown.

Package facts

Evidence: llmlingua-0.2.2-py3-none-any.whl

Keywords: Prompt Compression, LLMs, Inference Acceleration, Black-box LLMs, Efficient LLMs

Tags

More Artificial Intelligence packages

Further reading