{"enrichment":{"faq":[{"a":"llama-cpp is a lightweight C/C++ inference engine designed to run large language models on CPUs, including Apple Silicon (M1/M2). It natively supports Metal acceleration for Apple devices, enabling efficient inference without requiring a dedicated GPU. The engine handles quantized models in GGUF format, which dramatically reduce memory requirements and improve speed on CPU-only systems.","q":"Can llama-cpp run llama on Mac without GPU?"},{"a":"llama-cpp supports GGUF quantization formats ranging from 1.5-bit to 8-bit precision. These quantization options allow you to reduce model size and accelerate inference while maintaining reasonable quality. The 4-bit quantization is particularly popular for balancing performance and accuracy on resource-constrained devices like edge hardware and Raspberry Pi systems.","q":"What quantization formats does llama-cpp support?"},{"a":"llama-cpp provides conversion tools to transform Hugging Face models into GGUF format. The conversion process involves downloading the original model, running the conversion script, and optionally quantizing the result. Once converted to GGUF, models are optimized for llama-cpp's inference engine and can run efficiently on CPUs, Apple Silicon, AMD GPUs, and other non-NVIDIA hardware.","q":"How do I convert Hugging Face models to GGUF for llama-cpp?"},{"a":"llama-cpp supports AMD GPU inference through ROCm acceleration, extending beyond CPU-only and Apple Silicon deployments. This enables efficient inference on AMD hardware alongside Intel and other processors. Combined with GGUF quantization, llama-cpp provides a versatile solution for running language models across diverse hardware ecosystems without NVIDIA CUDA dependency.","q":"Does llama-cpp support AMD GPU and ROCm acceleration?"},{"a":"llama-cpp can be configured as a local OpenAI-compatible inference server, allowing you to run language models with an API interface matching OpenAI's specifications. This setup is ideal for edge deployment and environments without cloud connectivity. The server supports batch processing, constrained JSON generation, and extended context windows (up to 32k tokens), making it suitable for diverse production use cases.","q":"Can llama-cpp run as an OpenAI-compatible local server?"},{"a":"llama-cpp excels in CPU and non-NVIDIA hardware scenarios where vLLM and TensorRT-LLM are less optimized. While those frameworks prioritize NVIDIA GPU performance, llama-cpp's lightweight C/C++ design delivers strong tokens-per-second benchmarks on CPUs, Apple Silicon, and AMD GPUs. llama-cpp is the preferred choice for edge deployment and resource-constrained environments where CUDA isn't available.","q":"What is llama-cpp's performance compared to vLLM and TensorRT-LLM?"}],"shadow_tags":["cpu-first-inference","quantized-models","edge-computing","cross-platform-gpu","local-deployment","memory-efficient","openai-compatible-api","hardware-agnostic"],"summary_rewrite":"llama-cpp is a lightweight C/C++ inference engine for running large language models on CPUs and non-NVIDIA hardware, including Apple Silicon and AMD/Intel GPUs. It supports GGUF quantization formats (1.5-8 bit) to reduce memory footprint and accelerate performance, making it ideal for edge deployment and environments without CUDA support."},"files":[{"bytes":5912,"path":"12-inference-serving/llama-cpp/SKILL.md","sha256":"d74caa45483f5b3c6bd6267e39b444627989ff90fdb84b6c2e7c3da4ddf143ef","url":"https://skillfed.io/files/Orchestra-Research/AI-Research-SKILLs/llama-cpp/ff039c22/SKILL.md"}],"id":"Orchestra-Research/AI-Research-SKILLs/llama-cpp","links":{"html":"https://skillfed.io/Orchestra-Research/AI-Research-SKILLs/llama-cpp","md":"https://skillfed.io/Orchestra-Research/AI-Research-SKILLs/llama-cpp.md","repo":"https://github.com/Orchestra-Research/AI-Research-SKILLs"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":818,"language":"TeX","last_updated":"2026-06-16","license":"MIT","name":"llama-cpp","publisher":"Orchestra-Research","stars":11165},"relations":{"similar":[{"id":"synthetic-sciences/openscience/llama-cpp"},{"id":"OpenLAIR/dr-claw/llama-cpp"},{"id":"graniet/kheish/llama-cpp"},{"id":"moltis-org/moltis/llama-cpp"},{"id":"synthetic-sciences/openscience/gguf"},{"id":"OpenLAIR/dr-claw/gguf"},{"id":"Orchestra-Research/AI-Research-SKILLs/gguf"},{"id":"graniet/kheish/gguf"},{"id":"NousResearch/hermes-agent/llama-cpp"},{"id":"synthetic-sciences/openscience/unsloth"}]},"slug":{"owner":"Orchestra-Research","repo":"AI-Research-SKILLs","skill":"llama-cpp"},"version":"ff039c22"}