huggingface-local-models
Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.
Best for local LLM inferenceWorks with GitHubLow risk
⌘source
- author
@huggingface- language
- Python
★10kstars
642forks
Apr 22, 2026
✦overview.md
Key Features
- ·Search Hugging Face Hub for llama.cpp GGUF repos
- ·Select right quantization (Q4KM, Q5KM, etc.)
- ·Launch models with llama-cli or llama-server
- ·Exact GGUF file lookup via API
- ·Convert Transformers weights if needed
- ·OpenAI-compatible local serving
Use Cases
- →Run a Qwen2.5 model locally on Mac Metal for testing
- →Serve a GGUF model as an OpenAI-compatible endpoint in CI
- →Find the smallest quantized model that fits RAM constraints
- →Convert a new Transformers model to GGUF for offline use
Best for
- ✓local LLM inference
- ✓GPU-accelerated environments
- ✓CI/CD pipelines needing local models
Not ideal for
- !production cloud serving
- !large-scale distributed inference
FAQs
name
huggingface-local-models
description
Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.
Hugging Face Local Models
Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.
Default Workflow
- Search the Hub with
apps=llama.cpp. - Open
https://huggingface.co/<repo>?local-app=llama.cpp. - Prefer the exact HF local-app snippet and quant recommendation when it is visible.
- Confirm exact
.gguffilenames withhttps://huggingface.co/api/models/<repo>/tree/main?recursive=true. - Launch with
llama-cli -hf <repo>:<QUANT>orllama-server -hf <repo>:<QUANT>. - Fall back to
--hf-repoplus--hf-filewhen the repo uses custom file naming. - Convert from Transformers weights only if the repo does not already expose GGUF files.
Quick Start
Install llama.cpp
brew install llama.cpp
winget install llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make
Authenticate for gated repos
hf auth login
Search the Hub
https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
$install
1-click copynpx skills add huggingface/skills --skill huggingface-local-modelsSafety assessment
★
Clarity score
How clear and easy to understand the SKILL.md instructions are, rated from 1 to 5.
4/ 5
very goodClear and well structured, with only minor parts that might need a second read.
◎
Actionability score
How directly an agent can act on the SKILL.md instructions, rated from 1 to 5.
4/ 5
highMostly actionable with clear steps; only a few small gaps remain.