huggingface-local-models
Discover GGUF-compatible models on Hugging Face Hub and run them locally using llama.cpp across CPU, Mac Metal, CUDA, and ROCm hardware. The skill guides you through searching the Hub, selecting the right quantization, and launching models with llama-cli or llama-server, with fallback paths for custom file naming and conversion from Transformers weights when needed.
Hugging Face Local Models helps you find and run GGUF models locally with llama.cpp on CPU, Mac, CUDA, or ROCm.
AI-generated summary based on this skill's SKILL.md
Install
huggingface/skills/huggingface-local-models · repository language: Python
git clone https://github.com/huggingface/skills
cp -r skills/skills/huggingface-local-models ~/.claude/skills/huggingface-local-modelsnpx skillfed install huggingface/skills/huggingface-local-modelsFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I find GGUF models on Hugging Face?
huggingface-local-models helps you search the Hugging Face Hub for GGUF-format models ready for local inference. Use the Hub's search filters to look for "GGUF" in model names or descriptions, or browse quantized model repositories. Many popular models like Llama, Mistral, and Qwen have GGUF variants maintained by community quantizers. Filter by model size and quantization level (Q4, Q5, Q8) to match your hardware constraints.
What quantization level should I choose for local model inference?
huggingface-local-models guides you through quantization selection based on your hardware and speed-vs-quality tradeoff. Q4_K_M offers good balance for most CPUs and GPUs with moderate VRAM. Q5_K_M provides higher quality at the cost of more memory. Q8 is near-original quality but requires substantial resources. Start with Q4_K_M on constrained hardware; use Q5_K_M or higher if you have 8GB+ VRAM and prioritize accuracy.
How do I run llama.cpp models locally on my Mac with Metal?
huggingface-local-models supports Metal acceleration on macOS. Download a GGUF model from Hugging Face, then run it with llama-cli or llama-server compiled with Metal support. Use the `-ngl` flag to offload layers to the GPU. Metal automatically accelerates inference on Apple Silicon and Intel Macs. The skill handles model discovery and setup guidance; refer to llama.cpp's Metal documentation for compiler flags if building from source.
How do I convert a Hugging Face model to GGUF format?
huggingface-local-models provides conversion guidance for transforming Hugging Face Transformers weights into GGUF. Use llama.cpp's `convert.py` script: download the model, run the converter pointing to the model directory, and specify your target quantization. The skill documents the step-by-step process and common pitfalls like handling custom architectures. Pre-quantized GGUF versions exist for most popular models, so conversion is typically a fallback for newer or specialized models.
What's the difference between llama-cli and llama-server for local inference?
huggingface-local-models explains both tools: llama-cli is a command-line interface for one-off inference runs, ideal for scripts and testing. llama-server launches an OpenAI-compatible HTTP API, letting you run a persistent local inference endpoint for applications. Choose llama-cli for simple batch processing; use llama-server when you need a reusable service or want to swap models without restarting.
Can I run GGUF models locally on CPU, CUDA, and ROCm hardware?
huggingface-local-models supports inference across CPU, CUDA (NVIDIA), ROCm (AMD), and Metal (Apple). llama.cpp auto-detects your hardware; compile or download pre-built binaries with the appropriate backend. CPU inference works everywhere but is slower; GPU acceleration is recommended for real-time use. The skill guides hardware-specific setup and performance tuning for each platform.
SKILL.md
rendered from the published skill — quoted content, verbatim
Hugging Face Local Models
Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.
Default Workflow
- Search the Hub with
apps=llama.cpp. - Open
https://huggingface.co/<repo>?local-app=llama.cpp. - Prefer the exact HF local-app snippet and quant recommendation when it is visible.
- Confirm exact
.gguffilenames withhttps://huggingface.co/api/models/<repo>/tree/main?recursive=true. - Launch with
llama-cli -hf <repo>:<QUANT>orllama-server -hf <repo>:<QUANT>. - Fall back to
--hf-repoplus--hf-filewhen the repo uses custom file naming. - Convert from Transformers weights only if the repo does not already expose GGUF files.
Quick Start
Install llama.cpp
brew install llama.cpp
winget install llama.cpp
```bash git clone https://github.com/ggml-org/llama.cpp cd
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 4 files
skills/huggingface-local-models/SKILL.md
skills/huggingface-local-models/references/hardware.md
skills/huggingface-local-models/references/hub-discovery.md
skills/huggingface-local-models/references/quantization.md