Llama Cpp

Llama Cpp enables efficient LLM inference on resource-constrained hardware—CPUs, Apple Silicon, AMD and Intel GPUs—without requiring NVIDIA or CUDA. It uses GGUF quantization (1.5–8 bit) to dramatically reduce memory footprint and accelerate inference across edge devices, Macs, and embedded systems.

Llama Cpp runs LLM inference efficiently on CPUs, Apple Silicon, and non-NVIDIA GPUs using quantized models.

AI-generated summary based on this skill's SKILL.md

★ 264 ⁮ 22 Apache-2.0 updated 2026-07-26 by graniet

Install

graniet/kheish/llama-cpp · repository language: Rust

Manual (git clone) CLI (skillfed)

git clone https://github.com/graniet/kheish
cp -r kheish/skills/mlops/inference/llama-cpp ~/.claude/skills/llama-cpp

npx skillfed install graniet/kheish/llama-cpp

Frequently asked questions

AI-generated answers based on this skill's SKILL.md and metadata

Can Llama Cpp run LLM on Mac without NVIDIA?

Yes. Llama Cpp is designed to run large language models efficiently on Apple Silicon (M1, M2, M3) and Intel Macs without any NVIDIA hardware or CUDA dependency. It leverages Metal acceleration on Apple Silicon and CPU-optimized inference to deliver fast, local LLM execution on macOS.

What is GGUF quantization and how does it speed up inference?

Llama Cpp uses GGUF (GPT-Generated Unified Format) quantization to compress models to 1.5–8 bits, dramatically reducing memory footprint and accelerating inference. This quantization format is optimized for CPU and edge hardware, enabling models to run on resource-constrained devices while maintaining reasonable accuracy.

How do I convert HuggingFace models to GGUF format?

Llama Cpp provides conversion scripts to transform HuggingFace models into GGUF format. After conversion, you can apply quantization levels (4-bit, 5-bit, etc.) to further optimize the model for your target hardware—CPU, Apple Silicon, or edge devices.

Does Llama Cpp support AMD or Intel GPU inference?

Llama Cpp supports inference on AMD GPUs via ROCm and Intel GPUs. While its primary strength is CPU and Apple Silicon acceleration, these GPU backends enable non-NVIDIA users to leverage hardware acceleration for faster local LLM deployment.

Can Llama Cpp set up a local LLM server with OpenAI-compatible API?

Yes. Llama Cpp includes a built-in server mode that exposes an OpenAI-compatible REST API, allowing you to run a local LLM inference server. This enables seamless integration with existing tools and applications designed for OpenAI endpoints.

Is Llama Cpp suitable for edge deployment on Raspberry Pi?

Llama Cpp is well-suited for embedded and edge devices like Raspberry Pi. Its CPU-first architecture and aggressive quantization (down to 4-bit) allow smaller models to run efficiently on low-power hardware with minimal memory, making it ideal for constrained environments.

SKILL.md

rendered from the published skill — quoted content, verbatim

Kheish Compatibility

This skill is repo-local and stays inactive until explicitly activated.

When the original instructions refer to legacy tool names, use these Kheish mappings:

terminal => bash
web_extract => web_fetch, plus web_search when discovery is needed
search_files => grep_search and glob_search
browser_* tools require a browser-capable surfaced tool or MCP; if none is available, use the closest available surface and say so explicitly

When the instructions mention local helper files, resolve them from ${KHEISH_SKILL_DIR}.

llama.cpp

Pure

(truncated - see the full file via the links below)

Read as markdown · JSON record · Browse the source repository

File tree — 4 files

skills/mlops/inference/llama-cpp/SKILL.md
skills/mlops/inference/llama-cpp/references/optimization.md
skills/mlops/inference/llama-cpp/references/quantization.md
skills/mlops/inference/llama-cpp/references/server.md

All skills by graniet →