--- id: vllm-cpu version: "0.27.1" license: Apache-2.0 license_treatment: permissive maintenance: active --- # vllm-cpu — A high-throughput and memory-efficient inference and serving engine for LLMs License: permissive · Maintenance: active · Downloads: 255.5K/mo ## What it is and what it does vllm-cpu is a community-maintained CPU inference engine for large language models that unifies multiple CPU instruction set architectures (AVX2, AVX-512, AMX, NEON, BF16, DOTPROD) into a single wheel. It automatically detects and uses the best available instruction set at runtime, eliminating the need for manual ISA-specific builds. The package ships with fallback implementations so the same wheel works across different x86_64 and aarch64 platforms. It enables LLM inference on servers, laptops, and edge devices without requiring a GPU, making it suitable for development, testing, and moderate-scale deployments. The package includes an OpenAI-compatible API server via FastAPI and supports batch processing through the vLLM Python API. It depends on transformers, tokenizers, safetensors for model loading, and specialized libraries like lm-format-enforcer and xgrammar for output constraints. Use it for: - Run inference on development laptops or CI/CD systems without GPU hardware. - Deploy LLM services on ARM-based cloud instances (AWS Graviton, Ampere Altra) with automatic NEON/BF16 detection. - Batch process text through language models on multi-socket CPU servers using NUMA optimization. - Serve an OpenAI-compatible API endpoint on edge devices or on-premises infrastructure. - Prototype and test LLM applications before scaling to GPU clusters. ## Worth the install? AI-flagged interpretation of the facts above — verify before relying. vllm-cpu provides CPU-optimized inference for large language models with automatic detection of available CPU instruction sets (AVX2, AVX-512, AMX on x86; NEON, BF16, DOTPROD on ARM), packaged as a single unified wheel. Yes, with conditions. Install if you need CPU-only LLM inference and have a compatible processor (x86_64 with AVX2 or aarch64 with NEON). The package is actively maintained, has no known vulnerabilities, and uses a permissive license. However, be aware it is community-maintained (not official vLLM), has 63 runtime dependencies creating medium install friction, and performance will be significantly slower than GPU inference—suitable for development, testing, and edge deployment rather than high-throughput production serving. ## Install pip install vllm-cpu uv add vllm-cpu poetry add vllm-cpu ## Installing vllm-cpu Before you install: Medium install friction due to 63 runtime dependencies including transformers, fastapi, and specialized libraries like lm-format-enforcer and xgrammar. Package is actively maintained with recent releases (3 days old) and no archived status, though it is community-maintained rather than part of the official vLLM project. License in practice: Apache-2.0 permissive license allows commercial and private use with minimal restrictions, making it suitable for production deployments. Quickstart: pip3 install vllm-cpu from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen3-0.6B", dtype="bfloat16") outputs = llm.generate(["Hello, my name is"], SamplingParams(max_tokens=50)) print(outputs[0].outputs[0].text) Requires Python 3.10+, Linux with glibc 2.28+ (Debian 10+, Ubuntu 18.04+, RHEL 8+), and x86_64 CPU with AVX2 minimum or aarch64 with NEON. Verify before relying: - Actual performance gains relative to GPU inference or other CPU inference engines on representative models. - Stability and production-readiness claims for the community-maintained fork versus official vLLM. - Memory overhead and latency characteristics for different model sizes on various CPU architectures. ## Package facts - License: Apache-2.0 (permissive) - Python support: supports_current - Install friction: medium - Maintenance: active - Downloads: 255.5K/month (top 15,000 on PyPI) - Known vulnerabilities: none known ## Tags cpu llm inference, large language model without gpu, cpu-optimized transformer serving, avx2 avx512 llm acceleration, arm64 llm inference, edge device language model, cpu inference engine, cpu-inference, llm-serving, arm64-support [View on SkillFed](https://skillfed.io/packages/vllm-cpu) · [View on PyPI](https://pypi.org/project/vllm-cpu/)