llama-cpp
llama-cpp is a lightweight C/C++ inference engine for running large language models on CPUs and non-NVIDIA hardware, including Apple Silicon and AMD/Intel GPUs. It supports GGUF quantization formats (1.5-8 bit) to reduce memory footprint and accelerate performance, making it ideal for edge deployment and environments without CUDA support.
llama-cpp runs LLM inference efficiently on CPUs, Apple Silicon, and non-NVIDIA GPUs without CUDA.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-06-16
llama-cpp runs LLM inference efficiently on CPUs, Apple Silicon, and non-NVIDIA GPUs without CUDA. llama-cpp is a lightweight C/C++ inference engine for running large language models on CPUs and non-NVIDIA hardware, including Apple Silicon and AMD/Intel GPUs. It supports GGUF quantization formats (1.5-8 bit) to reduce memory footprint and accelerate performance, making it ideal for edge deployment and environments without CUDA support.
Use it when
- llama-cpp supports GGUF quantization formats ranging from 1.5-bit to 8-bit precision.
- llama-cpp provides conversion tools to transform Hugging Face models into GGUF format.
Verify before relying
Read SKILL.md below before installing (4 files). Open directory: indexed for reading, not audited.
Install
Orchestra-Research/AI-Research-SKILLs/llama-cpp · repository language: TeX
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
Can llama-cpp run llama on Mac without GPU?
llama-cpp is a lightweight C/C++ inference engine designed to run large language models on CPUs, including Apple Silicon (M1/M2). It natively supports Metal acceleration for Apple devices, enabling efficient inference without requiring a dedicated GPU. The engine handles quantized models in GGUF format, which dramatically reduce memory requirements and improve speed on CPU-only systems.
What quantization formats does llama-cpp support?
llama-cpp supports GGUF quantization formats ranging from 1.5-bit to 8-bit precision. These quantization options allow you to reduce model size and accelerate inference while maintaining reasonable quality. The 4-bit quantization is particularly popular for balancing performance and accuracy on resource-constrained devices like edge hardware and Raspberry Pi systems.
How do I convert Hugging Face models to GGUF for llama-cpp?
llama-cpp provides conversion tools to transform Hugging Face models into GGUF format. The conversion process involves downloading the original model, running the conversion script, and optionally quantizing the result. Once converted to GGUF, models are optimized for llama-cpp's inference engine and can run efficiently on CPUs, Apple Silicon, AMD GPUs, and other non-NVIDIA hardware.
Does llama-cpp support AMD GPU and ROCm acceleration?
llama-cpp supports AMD GPU inference through ROCm acceleration, extending beyond CPU-only and Apple Silicon deployments. This enables efficient inference on AMD hardware alongside Intel and other processors. Combined with GGUF quantization, llama-cpp provides a versatile solution for running language models across diverse hardware ecosystems without NVIDIA CUDA dependency.
Can llama-cpp run as an OpenAI-compatible local server?
llama-cpp can be configured as a local OpenAI-compatible inference server, allowing you to run language models with an API interface matching OpenAI's specifications. This setup is ideal for edge deployment and environments without cloud connectivity. The server supports batch processing, constrained JSON generation, and extended context windows (up to 32k tokens), making it suitable for diverse production use cases.
What is llama-cpp's performance compared to vLLM and TensorRT-LLM?
llama-cpp excels in CPU and non-NVIDIA hardware scenarios where vLLM and TensorRT-LLM are less optimized. While those frameworks prioritize NVIDIA GPU performance, llama-cpp's lightweight C/C++ design delivers strong tokens-per-second benchmarks on CPUs, Apple Silicon, and AMD GPUs. llama-cpp is the preferred choice for edge deployment and resource-constrained environments where CUDA isn't available.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
llama.cpp
Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.
When to use llama.cpp
Use llama.cpp when: - Running on CPU-only machines - Deploying on Apple Silicon (M1/M2/M3/M4) - Using AMD or Intel GPUs (no CUDA) - Edge deployment (Raspberry Pi, embedded systems) - Need simple deployment without Docker/Python
Use TensorRT-LLM instead when: - Have NVIDIA GPUs (A100/H100) - Need maximum throughput (100K+ tok/s) - Running in datacenter with CUDA
Use vLLM instead when: - Have NVIDIA GPUs - Need Python-first API - Want PagedAttention
Quick start
Installation
```bash
(truncated - see the full file via the links below)
File tree — 4 files
12-inference-serving/llama-cpp/SKILL.md
12-inference-serving/llama-cpp/references/optimization.md
12-inference-serving/llama-cpp/references/quantization.md
12-inference-serving/llama-cpp/references/server.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Run LLM inference on CPU or non-NVIDIA hardware”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
Llama Cpp enables efficient LLM inference on resource-constrained hardware—CPUs, Apple Silicon, AMD and Intel GPUs—without requiring NVIDIA or CUDA. It uses GGUF quantization (1.5–8 bit) to dramatically reduce memory footprint and accelerate inference across edge devices, Macs, and embedded systems.
Unsloth accelerates LLM fine-tuning on consumer and datacenter GPUs through optimized LoRA and QLoRA training, cutting both speed and memory requirements dramatically. It handles supervised fine-tuning, reinforcement learning with GRPO, vision model adaptation, and TTS training across 300+ model architectures, with direct export to GGUF for deployment on Ollama and llama.cpp.
This skill guides you through deploying language models with production-grade inference engines tailored to your hardware and use case. Choose between vLLM for maximum throughput on GPUs, llama.cpp for CPU and edge devices, or Ollama for quick local experimentation. Learn quantization strategies, memory optimization, and platform-specific tuning to balance speed and resource constraints.
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
Discover GGUF-compatible models on Hugging Face Hub and run them locally using llama.cpp across CPU, Mac Metal, CUDA, and ROCm hardware. The skill guides you through searching the Hub, selecting the right quantization, and launching models with llama-cli or llama-server, with fallback paths for custom file naming and conversion from Transformers weights when needed.