Llama Cpp
Llama Cpp enables efficient LLM inference on resource-constrained hardware—CPUs, Apple Silicon, AMD and Intel GPUs—without requiring NVIDIA or CUDA. It uses GGUF quantization (1.5–8 bit) to dramatically reduce memory footprint and accelerate inference across edge devices, Macs, and embedded systems.
Llama Cpp runs LLM inference efficiently on CPUs, Apple Silicon, and non-NVIDIA GPUs using quantized models.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-07-26
Llama Cpp runs LLM inference efficiently on CPUs, Apple Silicon, and non-NVIDIA GPUs using quantized models. Llama Cpp enables efficient LLM inference on resource-constrained hardware—CPUs, Apple Silicon, AMD and Intel GPUs—without requiring NVIDIA or CUDA. It uses GGUF quantization (1.5–8 bit) to dramatically reduce memory footprint and accelerate inference across edge devices, Macs, and embedded systems.
Use it when
- Llama Cpp uses GGUF (GPT-Generated Unified Format) quantization to compress models to 1.5–8 bits.
- Llama Cpp provides conversion scripts to transform HuggingFace models into GGUF format.
Verify before relying
Read SKILL.md below before installing (4 files). Open directory: indexed for reading, not audited.
Install
graniet/kheish/llama-cpp · repository language: Rust
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
Can Llama Cpp run LLM on Mac without NVIDIA?
Yes. Llama Cpp is designed to run large language models efficiently on Apple Silicon (M1, M2, M3) and Intel Macs without any NVIDIA hardware or CUDA dependency. It leverages Metal acceleration on Apple Silicon and CPU-optimized inference to deliver fast, local LLM execution on macOS.
What is GGUF quantization and how does it speed up inference?
Llama Cpp uses GGUF (GPT-Generated Unified Format) quantization to compress models to 1.5–8 bits, dramatically reducing memory footprint and accelerating inference. This quantization format is optimized for CPU and edge hardware, enabling models to run on resource-constrained devices while maintaining reasonable accuracy.
How do I convert HuggingFace models to GGUF format?
Llama Cpp provides conversion scripts to transform HuggingFace models into GGUF format. After conversion, you can apply quantization levels (4-bit, 5-bit, etc.) to further optimize the model for your target hardware—CPU, Apple Silicon, or edge devices.
Does Llama Cpp support AMD or Intel GPU inference?
Llama Cpp supports inference on AMD GPUs via ROCm and Intel GPUs. While its primary strength is CPU and Apple Silicon acceleration, these GPU backends enable non-NVIDIA users to leverage hardware acceleration for faster local LLM deployment.
Can Llama Cpp set up a local LLM server with OpenAI-compatible API?
Yes. Llama Cpp includes a built-in server mode that exposes an OpenAI-compatible REST API, allowing you to run a local LLM inference server. This enables seamless integration with existing tools and applications designed for OpenAI endpoints.
Is Llama Cpp suitable for edge deployment on Raspberry Pi?
Llama Cpp is well-suited for embedded and edge devices like Raspberry Pi. Its CPU-first architecture and aggressive quantization (down to 4-bit) allow smaller models to run efficiently on low-power hardware with minimal memory, making it ideal for constrained environments.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
Kheish Compatibility
This skill is repo-local and stays inactive until explicitly activated.
When the original instructions refer to legacy tool names, use these Kheish mappings:
terminal=>bashweb_extract=>web_fetch, plusweb_searchwhen discovery is neededsearch_files=>grep_searchandglob_searchbrowser_*tools require a browser-capable surfaced tool or MCP; if none is available, use the closest available surface and say so explicitly
When the instructions mention local helper files, resolve them from ${KHEISH_SKILL_DIR}.
llama.cpp
Pure
(truncated - see the full file via the links below)
File tree — 4 files
skills/mlops/inference/llama-cpp/SKILL.md
skills/mlops/inference/llama-cpp/references/optimization.md
skills/mlops/inference/llama-cpp/references/quantization.md
skills/mlops/inference/llama-cpp/references/server.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Run LLM inference on CPU and non-NVIDIA hardware”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
llama-cpp is a lightweight C/C++ inference engine for running large language models on CPUs and non-NVIDIA hardware, including Apple Silicon and AMD/Intel GPUs. It supports GGUF quantization formats (1.5-8 bit) to reduce memory footprint and accelerate performance, making it ideal for edge deployment and environments without CUDA support.
Unsloth accelerates LLM fine-tuning on consumer and datacenter GPUs through optimized LoRA and QLoRA training, cutting both speed and memory requirements dramatically. It handles supervised fine-tuning, reinforcement learning with GRPO, vision model adaptation, and TTS training across 300+ model architectures, with direct export to GGUF for deployment on Ollama and llama.cpp.
This skill guides you through deploying language models with production-grade inference engines tailored to your hardware and use case. Choose between vLLM for maximum throughput on GPUs, llama.cpp for CPU and edge devices, or Ollama for quick local experimentation. Learn quantization strategies, memory optimization, and platform-specific tuning to balance speed and resource constraints.
Discover GGUF-compatible models on Hugging Face Hub and run them locally using llama.cpp across CPU, Mac Metal, CUDA, and ROCm hardware. The skill guides you through searching the Hub, selecting the right quantization, and launching models with llama-cli or llama-server, with fallback paths for custom file naming and conversion from Transformers weights when needed.
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
This skill enables dynamic scaling of LLM inference workloads across Kubernetes clusters using KEDA and Prometheus metrics tied to GPU utilization and request queues. It covers vLLM deployment, queue-based job scaling with Redis, spot instance strategies, and cluster autoscaler configuration to handle traffic spikes while optimizing costs.