llama-cpp
llama-cpp is a lightweight C/C++ inference engine for running large language models on CPUs and non-NVIDIA hardware, including Apple Silicon and AMD/Intel GPUs. It supports GGUF quantization formats (1.5-8 bit) to reduce memory footprint and accelerate performance, making it ideal for edge deployment and environments without CUDA support.
llama-cpp runs LLM inference efficiently on CPUs, Apple Silicon, and non-NVIDIA GPUs without CUDA.
AI-generated summary based on this skill's SKILL.md
Install
Orchestra-Research/AI-Research-SKILLs/llama-cpp · repository language: TeX
git clone https://github.com/Orchestra-Research/AI-Research-SKILLs
cp -r AI-Research-SKILLs/12-inference-serving/llama-cpp ~/.claude/skills/llama-cppnpx skillfed install Orchestra-Research/AI-Research-SKILLs/llama-cppFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
Can llama-cpp run llama on Mac without GPU?
llama-cpp is a lightweight C/C++ inference engine designed to run large language models on CPUs, including Apple Silicon (M1/M2). It natively supports Metal acceleration for Apple devices, enabling efficient inference without requiring a dedicated GPU. The engine handles quantized models in GGUF format, which dramatically reduce memory requirements and improve speed on CPU-only systems.
What quantization formats does llama-cpp support?
llama-cpp supports GGUF quantization formats ranging from 1.5-bit to 8-bit precision. These quantization options allow you to reduce model size and accelerate inference while maintaining reasonable quality. The 4-bit quantization is particularly popular for balancing performance and accuracy on resource-constrained devices like edge hardware and Raspberry Pi systems.
How do I convert Hugging Face models to GGUF for llama-cpp?
llama-cpp provides conversion tools to transform Hugging Face models into GGUF format. The conversion process involves downloading the original model, running the conversion script, and optionally quantizing the result. Once converted to GGUF, models are optimized for llama-cpp's inference engine and can run efficiently on CPUs, Apple Silicon, AMD GPUs, and other non-NVIDIA hardware.
Does llama-cpp support AMD GPU and ROCm acceleration?
llama-cpp supports AMD GPU inference through ROCm acceleration, extending beyond CPU-only and Apple Silicon deployments. This enables efficient inference on AMD hardware alongside Intel and other processors. Combined with GGUF quantization, llama-cpp provides a versatile solution for running language models across diverse hardware ecosystems without NVIDIA CUDA dependency.
Can llama-cpp run as an OpenAI-compatible local server?
llama-cpp can be configured as a local OpenAI-compatible inference server, allowing you to run language models with an API interface matching OpenAI's specifications. This setup is ideal for edge deployment and environments without cloud connectivity. The server supports batch processing, constrained JSON generation, and extended context windows (up to 32k tokens), making it suitable for diverse production use cases.
What is llama-cpp's performance compared to vLLM and TensorRT-LLM?
llama-cpp excels in CPU and non-NVIDIA hardware scenarios where vLLM and TensorRT-LLM are less optimized. While those frameworks prioritize NVIDIA GPU performance, llama-cpp's lightweight C/C++ design delivers strong tokens-per-second benchmarks on CPUs, Apple Silicon, and AMD GPUs. llama-cpp is the preferred choice for edge deployment and resource-constrained environments where CUDA isn't available.
SKILL.md
rendered from the published skill — quoted content, verbatim
llama.cpp
Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.
When to use llama.cpp
Use llama.cpp when: - Running on CPU-only machines - Deploying on Apple Silicon (M1/M2/M3/M4) - Using AMD or Intel GPUs (no CUDA) - Edge deployment (Raspberry Pi, embedded systems) - Need simple deployment without Docker/Python
Use TensorRT-LLM instead when: - Have NVIDIA GPUs (A100/H100) - Need maximum throughput (100K+ tok/s) - Running in datacenter with CUDA
Use vLLM instead when: - Have NVIDIA GPUs - Need Python-first API - Want PagedAttention
Quick start
Installation
```bash
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 4 files
12-inference-serving/llama-cpp/SKILL.md
12-inference-serving/llama-cpp/references/optimization.md
12-inference-serving/llama-cpp/references/quantization.md
12-inference-serving/llama-cpp/references/server.md