{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/10"}],"enrichment":{"capability":"LLMLingua compresses prompts by identifying and removing non-essential tokens, reducing input length to LLMs by up to 20x while preserving task performance and lowering API costs.","skillfed_tags":["prompt-engineering","cost-optimization","rag"],"use_cases":["Reduce API costs when calling GPT-4 or GPT-3.5 by compressing prompts before submission.","Handle long documents in RAG systems by compressing retrieved context to fit within token budgets.","Mitigate the 'lost in the middle' problem in long-context LLM tasks by prioritizing key information.","Accelerate inference latency by reducing KV-cache size and prompt processing overhead.","Preserve in-context learning and chain-of-thought reasoning while shrinking prompt size."],"what_it_does":"LLMLingua is a prompt compression toolkit that uses a small, efficient language model to identify and remove non-essential tokens from prompts before sending them to larger LLMs. It implements three variants\u2014LLMLingua, LongLLMLingua (for long-context scenarios), and LLMLingua-2 (faster, distilled from GPT-4)\u2014each designed to reduce token consumption while preserving task performance. The core use case is lowering costs and latency when working with token-metered LLM APIs or handling long contexts that would otherwise exceed model limits.\n\nThe package integrates with popular RAG frameworks like LangChain and LlamaIndex, making it straightforward to drop into existing pipelines. It depends on transformers, torch, accelerate, tiktoken, nltk, and numpy\u2014standard ML infrastructure. The project is actively maintained by Microsoft researchers, with recent releases and integration into production systems.","worth_installing":"Yes. The package solves a real problem\u2014token costs and context limits\u2014with active maintenance, low install friction, and permissive licensing. It integrates cleanly into existing LLM workflows and has been adopted by major frameworks. Install if you work with LLM APIs or long-context tasks and want to reduce costs or fit within token constraints."},"id":"llmlingua","links":{"html":"https://skillfed.io/packages/llmlingua","md":"https://skillfed.io/packages/llmlingua.md","pypi":"https://pypi.org/project/llmlingua/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2024-04-09","license_spdx":null,"license_treatment":"permissive","name":"llmlingua","python_support":"supports_current","summary":"To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss."},"popularity":{"monthly_downloads":95983,"position":13238,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.2.2"}