huggingface-local-models
Discover GGUF-compatible models on Hugging Face Hub and run them locally using llama.cpp across CPU, Mac Metal, CUDA, and ROCm hardware. The skill guides you through searching the Hub, selecting the right quantization, and launching models with llama-cli or llama-server, with fallback paths for custom file naming and conversion from Transformers weights when needed.
Hugging Face Local Models helps you find and run GGUF models locally with llama.cpp on CPU, Mac, CUDA, or ROCm.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-07-27
Hugging Face Local Models helps you find and run GGUF models locally with llama.cpp on CPU, Mac, CUDA, or ROCm. Discover GGUF-compatible models on Hugging Face Hub and run them locally using llama.cpp across CPU, Mac Metal, CUDA, and ROCm hardware. The skill guides you through searching the Hub, selecting the right quantization, and launching models with llama-cli or llama-server, with fallback paths for custom file naming and conversion from Transformers weights when needed.
Use it when
- huggingface-local-models guides you through quantization selection based on your hardware and speed-vs-quality tradeoff.
- huggingface-local-models supports Metal acceleration on macOS.
Verify before relying
Read SKILL.md below before installing (4 files). Open directory: indexed for reading, not audited.
Install
huggingface/skills/huggingface-local-models · repository language: Python
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I find GGUF models on Hugging Face?
huggingface-local-models helps you search the Hugging Face Hub for GGUF-format models ready for local inference. Use the Hub's search filters to look for "GGUF" in model names or descriptions, or browse quantized model repositories. Many popular models like Llama, Mistral, and Qwen have GGUF variants maintained by community quantizers. Filter by model size and quantization level (Q4, Q5, Q8) to match your hardware constraints.
What quantization level should I choose for local model inference?
huggingface-local-models guides you through quantization selection based on your hardware and speed-vs-quality tradeoff. Q4_K_M offers good balance for most CPUs and GPUs with moderate VRAM. Q5_K_M provides higher quality at the cost of more memory. Q8 is near-original quality but requires substantial resources. Start with Q4_K_M on constrained hardware; use Q5_K_M or higher if you have 8GB+ VRAM and prioritize accuracy.
How do I run llama.cpp models locally on my Mac with Metal?
huggingface-local-models supports Metal acceleration on macOS. Download a GGUF model from Hugging Face, then run it with llama-cli or llama-server compiled with Metal support. Use the `-ngl` flag to offload layers to the GPU. Metal automatically accelerates inference on Apple Silicon and Intel Macs. The skill handles model discovery and setup guidance; refer to llama.cpp's Metal documentation for compiler flags if building from source.
How do I convert a Hugging Face model to GGUF format?
huggingface-local-models provides conversion guidance for transforming Hugging Face Transformers weights into GGUF. Use llama.cpp's `convert.py` script: download the model, run the converter pointing to the model directory, and specify your target quantization. The skill documents the step-by-step process and common pitfalls like handling custom architectures. Pre-quantized GGUF versions exist for most popular models, so conversion is typically a fallback for newer or specialized models.
What's the difference between llama-cli and llama-server for local inference?
huggingface-local-models explains both tools: llama-cli is a command-line interface for one-off inference runs, ideal for scripts and testing. llama-server launches an OpenAI-compatible HTTP API, letting you run a persistent local inference endpoint for applications. Choose llama-cli for simple batch processing; use llama-server when you need a reusable service or want to swap models without restarting.
Can I run GGUF models locally on CPU, CUDA, and ROCm hardware?
huggingface-local-models supports inference across CPU, CUDA (NVIDIA), ROCm (AMD), and Metal (Apple). llama.cpp auto-detects your hardware; compile or download pre-built binaries with the appropriate backend. CPU inference works everywhere but is slower; GPU acceleration is recommended for real-time use. The skill guides hardware-specific setup and performance tuning for each platform.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
Hugging Face Local Models
Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.
Default Workflow
- Search the Hub with
apps=llama.cpp. - Open
https://huggingface.co/<repo>?local-app=llama.cpp. - Prefer the exact HF local-app snippet and quant recommendation when it is visible.
- Confirm exact
.gguffilenames withhttps://huggingface.co/api/models/<repo>/tree/main?recursive=true. - Launch with
llama-cli -hf <repo>:<QUANT>orllama-server -hf <repo>:<QUANT>. - Fall back to
--hf-repoplus--hf-filewhen the repo uses custom file naming. - Convert from Transformers weights only if the repo does not already expose GGUF files.
Quick Start
Install llama.cpp
brew install llama.cpp
winget install llama.cpp
```bash git clone https://github.com/ggml-org/llama.cpp cd
(truncated - see the full file via the links below)
File tree — 4 files
skills/huggingface-local-models/SKILL.md
skills/huggingface-local-models/references/hardware.md
skills/huggingface-local-models/references/hub-discovery.md
skills/huggingface-local-models/references/quantization.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Find and run GGUF models locally using llama.cpp on various hardware”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
Train language models on managed Hugging Face infrastructure using TRL's supervised fine-tuning, preference optimization, and reinforcement learning methods. The skill handles cloud GPU job submission, dataset validation, hardware selection, and automatic model persistence to the Hub, with optional GGUF conversion for local deployment via Ollama or llama.cpp.
Unsloth accelerates LLM fine-tuning on consumer and datacenter GPUs through optimized LoRA and QLoRA training, cutting both speed and memory requirements dramatically. It handles supervised fine-tuning, reinforcement learning with GRPO, vision model adaptation, and TTS training across 300+ model architectures, with direct export to GGUF for deployment on Ollama and llama.cpp.
Llama Cpp enables efficient LLM inference on resource-constrained hardware—CPUs, Apple Silicon, AMD and Intel GPUs—without requiring NVIDIA or CUDA. It uses GGUF quantization (1.5–8 bit) to dramatically reduce memory footprint and accelerate inference across edge devices, Macs, and embedded systems.
llama-cpp is a lightweight C/C++ inference engine for running large language models on CPUs and non-NVIDIA hardware, including Apple Silicon and AMD/Intel GPUs. It supports GGUF quantization formats (1.5-8 bit) to reduce memory footprint and accelerate performance, making it ideal for edge deployment and environments without CUDA support.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
GPU Server Management guides you through provisioning and configuring NVIDIA GPU infrastructure for LLM inference and model training. It covers driver and CUDA toolkit installation, Docker GPU integration, multi-GPU topology setup, and production monitoring with DCGM and Prometheus metrics.