litserve

What it is and what it does

LitServe is a Python framework for building custom inference servers without rigid abstractions or MLOps boilerplate. You define a LitAPI subclass with setup() and predict() methods, then wrap it in a LitServer to handle concurrency, batching, streaming, and deployment. It's built on FastAPI but optimized for AI workloads—you control how models are loaded, how requests are batched, and how outputs are routed, while LitServe manages performance, scaling, and infrastructure concerns.

The framework supports agents, RAG systems, multi-model pipelines, and any PyTorch-based inference workload. You can run it locally, self-host anywhere, or deploy to Lightning Cloud with one command. It includes features like automatic GPU scaling, request batching, streaming responses, and device management, making it suitable for teams that need flexibility beyond single-model serving tools.

Use it for

• Build a multi-model inference API combining text, vision, and audio models in one server with custom routing logic.
• Deploy a RAG system or agent that orchestrates LLM calls, database lookups, and tool use without separate microservices.
• Run a batched inference pipeline for classical ML models (XGBoost, random forests) with custom preprocessing and caching.
• Host a chatbot or LLM proxy that handles streaming responses and concurrent requests with GPU autoscaling.
• Create a media conversion API combining multiple models (e.g., Whisper for audio, Stable Diffusion for images) in one endpoint.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

litserve on PyPI

Before you install

Installation is straightforward with low friction—a pure Python wheel with three runtime dependencies (fastapi, pyzmq, uvicorn). The project is actively maintained with recent commits and steady development since its February 2024 release.

Requires Python 3.10 or later; GPU acceleration requires appropriate drivers and PyTorch installation.

License in practice

Licensed under Apache-2.0 (permissive), which allows commercial use, modification, and distribution with minimal restrictions—suitable for both open-source and proprietary projects.

Quickstart

pip install litserve

import litserve as ls

class InferenceEngine(ls.LitAPI):
    def setup(self, device):
        self.model = lambda x: x**2
    def predict(self, request):
        return {"output": self.model(request["input"])}

server = ls.LitServer(InferenceEngine(), accelerator="auto")
server.run(port=8000)

Package facts

Evidence: litserve-0.2.17-py3-none-any.whl

Tags

More Artificial Intelligence packages

See also litdata · lightning · vllm · mlserver · sagemaker-serve · vllm-tpu · tensorflow-serving-api · tpu-inference · bentoml · smg-grpc-servicer

Further reading