mirror of
https://github.com/waybarrios/opencode-power-pack.git
synced 2026-11-18 18:03:43 +00:00
Add 25 Hugging Face skills (Apache-2.0)
Ports huggingface/skills: hf CLI, cloud/SageMaker, training (TRL, vision, sentence-transformers), Spaces/Gradio, papers, and Transformers.js. Condensed 4 oversized skills, full UPSTREAMS.json provenance, skill count 20 -> 45.
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "opencode-power-pack",
|
||||
"version": "0.3.0",
|
||||
"description": "Twenty Claude Code workflows packaged for Codex, OpenCode, and Pi.",
|
||||
"description": "Forty-five Claude Code workflows packaged for Codex, OpenCode, and Pi.",
|
||||
"author": {
|
||||
"name": "Wayner Barrios",
|
||||
"url": "https://github.com/waybarrios"
|
||||
@@ -24,7 +24,7 @@
|
||||
"skills": "./skills/",
|
||||
"interface": {
|
||||
"displayName": "OpenCode + Codex Power Pack",
|
||||
"shortDescription": "Twenty workflows for Codex, OpenCode, and Pi",
|
||||
"shortDescription": "Forty-five workflows for Codex, OpenCode, and Pi",
|
||||
"longDescription": "Review code, audit security, design and implement features, build MCP servers, and maintain project guidance with reusable workflows for Codex, OpenCode, and Pi.",
|
||||
"developerName": "Wayner Barrios",
|
||||
"category": "Productivity",
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
<h1 align="center">OpenCode Power Pack for Codex, OpenCode + Pi</h1>
|
||||
|
||||
<p align="center">
|
||||
<i>Twenty Claude Code workflows, adapted for Codex, OpenCode, and Pi.<br/>
|
||||
<i>Forty-five Claude Code workflows, adapted for Codex, OpenCode, and Pi.<br/>
|
||||
Code review, security audit, feature development, frontend design, project memory, and authoring tools.</i>
|
||||
</p>
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
<a href="https://github.com/waybarrios/opencode-power-pack/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/waybarrios/opencode-power-pack?style=flat-square&color=FFD60A"></a>
|
||||
<a href="https://github.com/waybarrios/opencode-power-pack/commits/main"><img alt="Last commit" src="https://img.shields.io/github/last-commit/waybarrios/opencode-power-pack?style=flat-square"></a>
|
||||
<a href="https://github.com/waybarrios/opencode-power-pack/issues"><img alt="GitHub issues" src="https://img.shields.io/github/issues/waybarrios/opencode-power-pack?style=flat-square"></a>
|
||||
<img alt="Skills: 20" src="https://img.shields.io/badge/skills-20-FFD60A?style=flat-square&labelColor=0B0F14">
|
||||
<img alt="Skills: 45" src="https://img.shields.io/badge/skills-45-FFD60A?style=flat-square&labelColor=0B0F14">
|
||||
<img alt="OpenCode 1.18.7+" src="https://img.shields.io/badge/opencode-1.18.7%2B-0B0F14?style=flat-square&labelColor=FFD60A">
|
||||
<img alt="Codex plugin" src="https://img.shields.io/badge/Codex-plugin-0B0F14?style=flat-square&labelColor=FFD60A">
|
||||
<img alt="Pi package" src="https://img.shields.io/badge/Pi-package-0B0F14?style=flat-square&labelColor=FFD60A">
|
||||
@@ -45,7 +45,7 @@ Start a new Codex session, open `/plugins` to confirm the installation, or invok
|
||||
pi install git:github.com/waybarrios/opencode-power-pack
|
||||
```
|
||||
|
||||
Pi discovers the twenty skills declared by the package. Use `pi list` to verify the installation. For a project-local installation recorded in `.pi/settings.json`, add `-l`:
|
||||
Pi discovers the forty-five skills declared by the package. Use `pi list` to verify the installation. For a project-local installation recorded in `.pi/settings.json`, add `-l`:
|
||||
|
||||
```bash
|
||||
pi install git:github.com/waybarrios/opencode-power-pack -l
|
||||
@@ -55,7 +55,7 @@ pi install git:github.com/waybarrios/opencode-power-pack -l
|
||||
|
||||
Codex, OpenCode, and Pi read `SKILL.md` workflows, but many valuable Claude Code workflows originated as Claude-specific commands and agents. Copying those artifacts directly does not preserve their orchestration, permissions, or subagent behavior.
|
||||
|
||||
This package adapts the portable methodology into shared skills, registers feature-development specialist roles as read-only OpenCode subagents, lets Codex execute the same phase assignments with its native subagent workflow, and exposes all twenty skills as a Pi package. It ships immutable provenance for every upstream work.
|
||||
This package adapts the portable methodology into shared skills, registers feature-development specialist roles as read-only OpenCode subagents, lets Codex execute the same phase assignments with its native subagent workflow, and exposes all forty-five skills as a Pi package. It ships immutable provenance for every upstream work.
|
||||
|
||||
It complements [obra/superpowers](https://github.com/obra/superpowers), which provides process skills such as brainstorming, TDD, debugging, and plan execution.
|
||||
|
||||
@@ -81,6 +81,31 @@ It complements [obra/superpowers](https://github.com/obra/superpowers), which pr
|
||||
| Research | `paper-summarizer` | Extract actionable findings and a claim-evidence map from academic/technical papers |
|
||||
| Authoring | `mcp-builder` | Design and build MCP servers in TypeScript or Python |
|
||||
| Authoring | `skill-creator` | Create, test, and improve reusable `SKILL.md` workflows |
|
||||
| Hugging Face | `hf-cli` | Core `hf` CLI usage — auth, cache, repos, jobs, papers, Spaces, and more |
|
||||
| Hugging Face | `hf-mem` | Estimate memory needed to load Safetensors/GGUF weights for inference |
|
||||
| Hugging Face | `huggingface-best` | Find and compare the best model for a task via official benchmark leaderboards |
|
||||
| Hugging Face | `huggingface-datasets` | Dataset Viewer API workflows — metadata, pagination, search, filters |
|
||||
| Hugging Face | `huggingface-papers` | Look up and analyze Hugging Face / arXiv paper pages via the papers API |
|
||||
| Hugging Face | `huggingface-paper-publisher` | Publish and manage research papers on the Hub, including research-article templates |
|
||||
| Hugging Face | `huggingface-tool-builder` | Build reusable scripts/tools around the Hugging Face API |
|
||||
| Hugging Face | `huggingface-local-models` | Select and run GGUF models locally with llama.cpp (CPU/Metal/CUDA/ROCm) |
|
||||
| Hugging Face — Training | `huggingface-llm-trainer` | Fine-tune LLMs (TRL: SFT/DPO/GRPO, or Unsloth) on Hugging Face Jobs |
|
||||
| Hugging Face — Training | `huggingface-vision-trainer` | Fine-tune object detection, image classification, and SAM/SAM2 segmentation models |
|
||||
| Hugging Face — Training | `train-sentence-transformers` | Train bi-encoder/cross-encoder/sparse embedding models |
|
||||
| Hugging Face — Training | `trl-training` | TRL CLI training — SFT, DPO, GRPO, KTO, RLOO, reward modeling |
|
||||
| Hugging Face — Training | `huggingface-community-evals` | Run local evaluations with inspect-ai / lighteval |
|
||||
| Hugging Face — Training | `huggingface-trackio` | Track and visualize ML training experiments with Trackio |
|
||||
| Hugging Face — Spaces | `huggingface-spaces` | Build, deploy, and debug Gradio/Docker/static Spaces |
|
||||
| Hugging Face — Spaces | `huggingface-gradio` | Build Gradio web UIs and demos in Python |
|
||||
| Hugging Face — Spaces | `huggingface-lora-space-builder` | Build and publish a Gradio Space demo for a user-provided LoRA |
|
||||
| Hugging Face — Spaces | `huggingface-zerogpu` | Write/debug ZeroGPU-constrained Gradio Space code |
|
||||
| Hugging Face — Cloud | `hf-cloud-aws-context-discovery` | Discover the user's local AWS context before any AWS work |
|
||||
| Hugging Face — Cloud | `hf-cloud-python-env-setup` | Set up an isolated Python env for SageMaker/AWS work |
|
||||
| Hugging Face — Cloud | `hf-cloud-sagemaker-deployment-planner` | Entry point for planning a SageMaker model deployment |
|
||||
| Hugging Face — Cloud | `hf-cloud-sagemaker-iam-preflight` | Ensure a usable SageMaker execution role exists before deploying |
|
||||
| Hugging Face — Cloud | `hf-cloud-serving-image-selection` | Pick the right SageMaker serving container image |
|
||||
| Hugging Face — Cloud | `hf-cloud-sagemaker-production-defaults` | Create a SageMaker endpoint with autoscaling, alarms, and tagging by default |
|
||||
| JavaScript | `transformers-js` | Run Hugging Face models directly in JS/TS (browser or Node/Bun/Deno) |
|
||||
| Project memory | `agents-md-improver` | Audit project rules and propose targeted improvements |
|
||||
| Project memory | `agents-md-revise` | Capture durable session learnings in project rules |
|
||||
|
||||
@@ -106,7 +131,7 @@ codex plugin add opencode-power-pack@opencode-power-pack
|
||||
codex plugin list --marketplace opencode-power-pack
|
||||
```
|
||||
|
||||
Start a new Codex session after installation so the twenty bundled skills are loaded. Use `/plugins` to inspect the installed plugin or `$` to select one of its skills explicitly. Codex plugin packaging follows the [official plugin structure](https://developers.openai.com/plugins/build/plugins).
|
||||
Start a new Codex session after installation so the forty-five bundled skills are loaded. Use `/plugins` to inspect the installed plugin or `$` to select one of its skills explicitly. Codex plugin packaging follows the [official plugin structure](https://developers.openai.com/plugins/build/plugins).
|
||||
|
||||
### OpenCode From GitHub
|
||||
|
||||
@@ -146,7 +171,7 @@ opencode debug skill
|
||||
opencode debug agent code-explorer
|
||||
```
|
||||
|
||||
The first command should include all twenty unprefixed skill names. The second should report a `subagent` with editing denied. In the TUI, `ctrl+p` should list `/code-review`, `/feature-dev`, `/frontend-design`, and the other skill-derived commands.
|
||||
The first command should include all forty-five unprefixed skill names. The second should report a `subagent` with editing denied. In the TUI, `ctrl+p` should list `/code-review`, `/feature-dev`, `/frontend-design`, and the other skill-derived commands.
|
||||
|
||||
### Verify Codex
|
||||
|
||||
|
||||
@@ -39,6 +39,29 @@ orchestration were removed or generalized to host-agnostic phrasing.
|
||||
Copyright 2026 Vigolium. The upstream notice is available at
|
||||
`LICENSES/Piolium-MIT.txt`.
|
||||
|
||||
## Hugging Face Skills
|
||||
|
||||
The modified `hf-cli`, `hf-cloud-aws-context-discovery`, `hf-cloud-python-env-setup`,
|
||||
`hf-cloud-sagemaker-deployment-planner`, `hf-cloud-sagemaker-iam-preflight`,
|
||||
`hf-cloud-sagemaker-production-defaults`, `hf-cloud-serving-image-selection`,
|
||||
`hf-mem`, `huggingface-best`, `huggingface-community-evals`,
|
||||
`huggingface-datasets`, `huggingface-gradio`, `huggingface-llm-trainer`,
|
||||
`huggingface-local-models`, `huggingface-lora-space-builder`,
|
||||
`huggingface-paper-publisher`, `huggingface-papers`, `huggingface-spaces`,
|
||||
`huggingface-tool-builder`, `huggingface-trackio`, `huggingface-vision-trainer`,
|
||||
`huggingface-zerogpu`, `train-sentence-transformers`, `transformers-js`, and
|
||||
`trl-training` skills derive from the Apache-2.0-licensed `huggingface/skills`
|
||||
repository, including their `scripts/`, `references/`, `templates/`, and
|
||||
`examples/` support files. Four skills (`huggingface-llm-trainer`,
|
||||
`huggingface-paper-publisher`, `huggingface-vision-trainer`, `transformers-js`)
|
||||
were condensed to fit this pack's skill-body length limit. Dangling
|
||||
cross-references to a `hugging-face-jobs` skill not present in the upstream
|
||||
snapshot were generalized to describe the underlying `hf jobs` CLI/API
|
||||
directly.
|
||||
|
||||
Copyright 2026 Hugging Face. Licensed under Apache-2.0. The full license is
|
||||
available at `LICENSES/Apache-2.0.txt`.
|
||||
|
||||
## Anthropic Official Plugins
|
||||
|
||||
The following modified skills derive from Apache-2.0 material in
|
||||
|
||||
+338
-13
@@ -81,19 +81,6 @@
|
||||
"blob": "e0f667ef65f7204399110214defd46ce062a1a51"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "code-review",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-07-27",
|
||||
"source": {
|
||||
"repository": "https://github.com/anthropics/claude-plugins-official",
|
||||
"commit": "bdca23e8e46f8832d0030c05804ae207786ae37f",
|
||||
"committedAt": "2026-04-24T19:52:02Z",
|
||||
"path": "plugins/code-review/commands/code-review.md",
|
||||
"blob": "c46e327f19b6bb59dbf9060e8c9961e701908350"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "code-quality",
|
||||
"license": "MIT",
|
||||
@@ -107,6 +94,19 @@
|
||||
"blob": "a65c247e1a2e452f0ef3a2b9bd63aa7d9ca6c3e3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "code-review",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-07-27",
|
||||
"source": {
|
||||
"repository": "https://github.com/anthropics/claude-plugins-official",
|
||||
"commit": "bdca23e8e46f8832d0030c05804ae207786ae37f",
|
||||
"committedAt": "2026-04-24T19:52:02Z",
|
||||
"path": "plugins/code-review/commands/code-review.md",
|
||||
"blob": "c46e327f19b6bb59dbf9060e8c9961e701908350"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "code-reviewer",
|
||||
"license": "Apache-2.0",
|
||||
@@ -172,6 +172,292 @@
|
||||
"blob": "600b6db41fac7e2081c7528ec6982960892c819d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-cli",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "7039bdcf4510c30ec932637e8b2c1646aee7f185",
|
||||
"committedAt": "2026-07-09T17:06:41+02:00",
|
||||
"path": "skills/hf-cli/SKILL.md",
|
||||
"blob": "4fd111f9df28b11d71c6a4f012a2b898f91c195a"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-cloud-aws-context-discovery",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "89d91fd4d0e6d4330471ba35f1c388489e94b496",
|
||||
"committedAt": "2026-07-07T16:32:21+02:00",
|
||||
"path": "skills/hf-cloud-aws-context-discovery/SKILL.md",
|
||||
"blob": "fbf62203595b187ea833904f67ca0ae2fdaf39a6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-cloud-python-env-setup",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "89d91fd4d0e6d4330471ba35f1c388489e94b496",
|
||||
"committedAt": "2026-07-07T16:32:21+02:00",
|
||||
"path": "skills/hf-cloud-python-env-setup/SKILL.md",
|
||||
"blob": "462a6b99e7767576e1ba6c39c1456873d020de92"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-cloud-sagemaker-deployment-planner",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "89d91fd4d0e6d4330471ba35f1c388489e94b496",
|
||||
"committedAt": "2026-07-07T16:32:21+02:00",
|
||||
"path": "skills/hf-cloud-sagemaker-deployment-planner/SKILL.md",
|
||||
"blob": "567bcd7be7529892217d7e9b07518b61ec78df65"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-cloud-sagemaker-iam-preflight",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "89d91fd4d0e6d4330471ba35f1c388489e94b496",
|
||||
"committedAt": "2026-07-07T16:32:21+02:00",
|
||||
"path": "skills/hf-cloud-sagemaker-iam-preflight/SKILL.md",
|
||||
"blob": "8f15164327ff28e8086ae275a54b068ff32025e8"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-cloud-sagemaker-production-defaults",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "89d91fd4d0e6d4330471ba35f1c388489e94b496",
|
||||
"committedAt": "2026-07-07T16:32:21+02:00",
|
||||
"path": "skills/hf-cloud-sagemaker-production-defaults/SKILL.md",
|
||||
"blob": "8e4f7f30f816f40b3eb364485de0b3c886f108f6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-cloud-serving-image-selection",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "89d91fd4d0e6d4330471ba35f1c388489e94b496",
|
||||
"committedAt": "2026-07-07T16:32:21+02:00",
|
||||
"path": "skills/hf-cloud-serving-image-selection/SKILL.md",
|
||||
"blob": "05f70d82c00da9160a086f477f31ff77e9bddea5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "hf-mem",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "c68f1b08d9eb3af22cdc1d3fb60e9cdb78522556",
|
||||
"committedAt": "2026-06-12T11:25:39+02:00",
|
||||
"path": "skills/hf-mem/SKILL.md",
|
||||
"blob": "fadc48b0e2a510e21345e3860c69ac799255d9c0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-best",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "be02c15f4b81c901b6d71a8896e56f93837d8050",
|
||||
"committedAt": "2026-04-22T15:19:36+02:00",
|
||||
"path": "skills/huggingface-best/SKILL.md",
|
||||
"blob": "6073590ebf8b136bb405a567fed9d47840164713"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-community-evals",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "386571e3e65ef0c129af2747ff36d1fa6fc5b8a4",
|
||||
"committedAt": "2026-03-23T15:24:34+00:00",
|
||||
"path": "skills/huggingface-community-evals/SKILL.md",
|
||||
"blob": "a68b2e79958930b3913cb8e7667e07000031e40b"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-datasets",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "b3df145a3a5a3e64ac075d781c625896d9d2bfcd",
|
||||
"committedAt": "2026-04-30T09:17:59-04:00",
|
||||
"path": "skills/huggingface-datasets/SKILL.md",
|
||||
"blob": "abafddcbe62bd6e549c095ee9ce23b921081fe22"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-gradio",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "ff28908172004ae4def67d7fe8b31b77482aa594",
|
||||
"committedAt": "2026-03-24T19:17:07+00:00",
|
||||
"path": "skills/huggingface-gradio/SKILL.md",
|
||||
"blob": "81dfb05834ab34a13cb0171a2fb03e4557b1e148"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-llm-trainer",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "d0d3f4301358fc1351d408a58596d5644b5a8114",
|
||||
"committedAt": "2026-04-09T11:58:58+01:00",
|
||||
"path": "skills/huggingface-llm-trainer/SKILL.md",
|
||||
"blob": "74ad29af05eee633fb4c04376118f751c24cdd5e"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-local-models",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "f4ddab46b9847c7005d7979fdfec36da1aec4e26",
|
||||
"committedAt": "2026-04-22T08:42:11+02:00",
|
||||
"path": "skills/huggingface-local-models/SKILL.md",
|
||||
"blob": "ece849559de75eaa8e9709a51e0e957e6ae630d3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-lora-space-builder",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "86cdeee824b73e504198b6005bb113552cdfa7ba",
|
||||
"committedAt": "2026-07-16T09:07:15+01:00",
|
||||
"path": "skills/huggingface-lora-space-builder/SKILL.md",
|
||||
"blob": "c1d0cbbfa60999ced1f97e136af5010ab86c814c"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-paper-publisher",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "386571e3e65ef0c129af2747ff36d1fa6fc5b8a4",
|
||||
"committedAt": "2026-03-23T15:24:34+00:00",
|
||||
"path": "skills/huggingface-paper-publisher/SKILL.md",
|
||||
"blob": "01b7db0cb9e30912eda8aa2b73a744c17ab1e176"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-papers",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "386571e3e65ef0c129af2747ff36d1fa6fc5b8a4",
|
||||
"committedAt": "2026-03-23T15:24:34+00:00",
|
||||
"path": "skills/huggingface-papers/SKILL.md",
|
||||
"blob": "e65fc90acabc30208883238ec89a27c4cdc1b57e"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-spaces",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "87f9ee5b670a630b022809a8742606c6da21812c",
|
||||
"committedAt": "2026-07-25T06:31:31+02:00",
|
||||
"path": "skills/huggingface-spaces/SKILL.md",
|
||||
"blob": "9c054c071bc4d2bc2ba1d67b96b0d93825a559ce"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-tool-builder",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "95640aa22f447f49ff1321dfa221873644415171",
|
||||
"committedAt": "2026-04-01T15:07:10-04:00",
|
||||
"path": "skills/huggingface-tool-builder/SKILL.md",
|
||||
"blob": "0634af706451240f82faf0871a15b0faac5f39a2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-trackio",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "88e864e27586c1b43eb43610fd3d14252522400f",
|
||||
"committedAt": "2026-07-06T19:39:01+02:00",
|
||||
"path": "skills/huggingface-trackio/SKILL.md",
|
||||
"blob": "df231aa06d661bf031aaad00dfa286ca83df4940"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-vision-trainer",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "386571e3e65ef0c129af2747ff36d1fa6fc5b8a4",
|
||||
"committedAt": "2026-03-23T15:24:34+00:00",
|
||||
"path": "skills/huggingface-vision-trainer/SKILL.md",
|
||||
"blob": "5a2c554b04218a9c411c184fd749ffb6eb0daabe"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "huggingface-zerogpu",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "5e3b4d2226d1beaa6a8a4df3739b6f68bd36521b",
|
||||
"committedAt": "2026-05-20T16:18:47+01:00",
|
||||
"path": "skills/huggingface-zerogpu/SKILL.md",
|
||||
"blob": "7171d2b71c94eab597cbd8a4b38ba5ca489276af"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "insecure-defaults",
|
||||
"license": "MIT",
|
||||
@@ -263,6 +549,45 @@
|
||||
"blob": "1389de7b1c43aa6a9273478ea37dfc585e5ea6ca"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "train-sentence-transformers",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "3ac5910959e2b9fae3aea94badcc3b280781c9c2",
|
||||
"committedAt": "2026-05-07T14:50:17+02:00",
|
||||
"path": "skills/train-sentence-transformers/SKILL.md",
|
||||
"blob": "6af33bdea3fd59e540cc852b65d515f282cd4a21"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "transformers-js",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "4db6015affd7779ca959b7c7c53cdcd3c8f4737b",
|
||||
"committedAt": "2026-04-10T17:57:42+02:00",
|
||||
"path": "skills/transformers-js/SKILL.md",
|
||||
"blob": "bc62aa1a63bc31ea9aea9456fed65b334e5010bd"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "trl-training",
|
||||
"license": "Apache-2.0",
|
||||
"adaptation": "adapted",
|
||||
"reviewedAt": "2026-08-02",
|
||||
"source": {
|
||||
"repository": "https://github.com/huggingface/skills",
|
||||
"commit": "df04e76a36a84c20fb08267b0378465d64835f82",
|
||||
"committedAt": "2026-06-08T08:34:04+02:00",
|
||||
"path": "skills/trl-training/SKILL.md",
|
||||
"blob": "1941385812d82abbbe1f466aa1204e0f8d775ea9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "vuln-report",
|
||||
"license": "MIT",
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "opencode-power-pack",
|
||||
"version": "0.3.0",
|
||||
"description": "Twenty Claude Code workflows packaged for Codex, OpenCode, and Pi: code review, feature development, security review, frontend design, authoring, and project memory.",
|
||||
"description": "Forty-five Claude Code workflows packaged for Codex, OpenCode, and Pi: code review, feature development, security review, frontend design, authoring, and project memory.",
|
||||
"type": "module",
|
||||
"main": ".opencode/plugins/opencode-power-pack.js",
|
||||
"engines": {
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
name: hf-cli
|
||||
description: "Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`."
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
Install: `curl -LsSf https://hf.co/cli/install.sh | bash -s`.
|
||||
|
||||
The Hugging Face Hub CLI tool `hf` is available. IMPORTANT: The `hf` command replaces the deprecated `huggingface-cli` command.
|
||||
|
||||
Use `hf --help` to view available functions. Note that auth commands are now all under `hf auth` e.g. `hf auth whoami`.
|
||||
|
||||
Generated with `huggingface_hub v1.23.0`. Run `hf skills add --force` to regenerate.
|
||||
|
||||
## Commands
|
||||
|
||||
- `hf cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf download REPO_ID` — Download files from the Hub. `[--type [model|dataset|space] --revision TEXT --include TEXT --exclude TEXT --cache-dir TEXT --local-dir TEXT --force-download --dry-run --max-workers INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
- `hf env` — Print information about the environment. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --format [auto|human|agent|json|quiet]]`
|
||||
- `hf update` — Update the `hf` CLI to the latest version. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf upload REPO_ID` — Upload a file or a folder to the Hub. Recommended for single-commit uploads. `[--type [model|dataset|space] --revision TEXT --private --include TEXT --exclude TEXT --delete TEXT --commit-message TEXT --commit-description TEXT --create-pr --every FLOAT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf upload-large-folder REPO_ID LOCAL_PATH` — [Deprecated] Upload a large folder to the Hub. Use `hf upload` instead. `[--type [model|dataset|space] --revision TEXT --private --include TEXT --exclude TEXT --num-workers INTEGER --no-report --no-bars --format [auto|human|agent|json|quiet]]`
|
||||
- `hf version` — Print information about the hf version. `[--format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf auth` — Manage authentication (login, logout, etc.).
|
||||
|
||||
- `hf auth list` — List all stored access tokens. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf auth login` — Login from your browser, or using a token from huggingface.co/settings/tokens. `[--add-to-git-credential --force --format [auto|human|agent|json|quiet]]`
|
||||
- `hf auth logout` — Logout from a specific token. `[--token-name TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf auth switch` — Switch between access tokens. `[--token-name TEXT --add-to-git-credential --format [auto|human|agent|json|quiet]]`
|
||||
- `hf auth token` — Print the current access token to stdout. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf auth whoami` — Find out which huggingface.co account you are logged in as. `[--format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf buckets` — Commands to interact with buckets.
|
||||
|
||||
- `hf buckets cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf buckets create BUCKET_ID` — Create a new bucket. `[--private --region [us|eu] --exist-ok --format [auto|human|agent|json|quiet]]`
|
||||
- `hf buckets delete BUCKET_ID` — Delete a bucket. `[--yes --missing-ok --format [auto|human|agent|json|quiet]]`
|
||||
- `hf buckets info BUCKET_ID` — Get info about a bucket. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf buckets list` — List buckets or files in a bucket. `[--human-readable --tree --recursive --search TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf buckets move FROM_ID TO_ID` — Move (rename) a bucket to a new name or namespace. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf buckets remove ARGUMENT` — Remove files from a bucket. `[--recursive --yes --dry-run --include TEXT --exclude TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf buckets sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf cache` — Manage local cache directory.
|
||||
|
||||
- `hf cache list` — List cached repositories or revisions. `[--cache-dir TEXT --revisions --filter TEXT --sort [accessed|accessed:asc|accessed:desc|modified|modified:asc|modified:desc|name|name:asc|name:desc|size|size:asc|size:desc] --limit INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
- `hf cache prune` — Remove detached revisions and incomplete downloads from the cache. `[--cache-dir TEXT --yes --dry-run --format [auto|human|agent|json|quiet]]`
|
||||
- `hf cache rm TARGETS` — Remove cached repositories or revisions. `[--cache-dir TEXT --yes --dry-run --format [auto|human|agent|json|quiet]]`
|
||||
- `hf cache verify REPO_ID` — Verify checksums for a single repo revision from cache or a local directory. `[--type [model|dataset|space] --revision TEXT --cache-dir TEXT --local-dir TEXT --fail-on-missing-files --fail-on-extra-files --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf collections` — Interact with collections on the Hub.
|
||||
|
||||
- `hf collections add-item COLLECTION_SLUG ITEM_ID ITEM_TYPE` — Add an item to a collection. `[--note TEXT --exists-ok --format [auto|human|agent|json|quiet]]`
|
||||
- `hf collections create TITLE` — Create a new collection on the Hub. `[--namespace TEXT --description TEXT --private --exists-ok --format [auto|human|agent|json|quiet]]`
|
||||
- `hf collections delete COLLECTION_SLUG` — Delete a collection from the Hub. `[--missing-ok --format [auto|human|agent|json|quiet]]`
|
||||
- `hf collections delete-item COLLECTION_SLUG ITEM_OBJECT_ID` — Delete an item from a collection. `[--missing-ok --format [auto|human|agent|json|quiet]]`
|
||||
- `hf collections info COLLECTION_SLUG` — Get info about a collection on the Hub. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf collections list` — List collections on the Hub. `[--owner TEXT --item TEXT --sort [lastModified|trending|upvotes] --limit INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
- `hf collections update COLLECTION_SLUG` — Update a collection's metadata on the Hub. `[--title TEXT --description TEXT --position INTEGER --private --theme TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf collections update-item COLLECTION_SLUG ITEM_OBJECT_ID` — Update an item in a collection. `[--note TEXT --position INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf datasets` — Interact with datasets on the Hub.
|
||||
|
||||
- `hf datasets card DATASET_ID` — Get the dataset card (README) for a dataset on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]`
|
||||
- `hf datasets info DATASET_ID` — Get info about a dataset on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf datasets leaderboard DATASET_ID` — List model scores from a dataset leaderboard. This command helps find the best models for a task or compare models by benchmark scores. Use 'hf datasets ls --filter benchmark:official' to list available leaderboards. `[--limit INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
- `hf datasets list` — List datasets on the Hub, or files in a dataset repo. `[--search TEXT --author TEXT --filter TEXT --sort [created_at|downloads|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf datasets parquet DATASET_ID` — List parquet file URLs available for a dataset. `[--subset TEXT --split TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf datasets sql SQL` — Execute a raw SQL query with DuckDB against dataset parquet URLs. `[--format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf discussions` — Manage discussions and pull requests on the Hub.
|
||||
|
||||
- `hf discussions close REPO_ID NUM` — Close a discussion or pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions comment REPO_ID NUM` — Comment on a discussion or pull request. `[--body TEXT --body-file PATH --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions create REPO_ID --title TEXT` — Create a new discussion or pull request on a repo. `[--body TEXT --body-file PATH --pull-request --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions diff REPO_ID NUM` — Show the diff of a pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions edit REPO_ID NUM COMMENT_ID` — Edit an existing comment on a discussion or pull request. `[--body TEXT --body-file PATH --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions info REPO_ID NUM` — Get info about a discussion or pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions list REPO_ID` — List discussions and pull requests on a repo. `[--status [open|closed|merged|draft|all] --kind [all|discussion|pull_request] --author TEXT --limit INTEGER --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions merge REPO_ID NUM` — Merge a pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions rename REPO_ID NUM NEW_TITLE` — Rename a discussion or pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf discussions reopen REPO_ID NUM` — Reopen a closed discussion or pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf endpoints` — Manage Hugging Face Inference Endpoints.
|
||||
|
||||
- `hf endpoints catalog deploy --repo TEXT` — Deploy an Inference Endpoint from the Model Catalog. `[--name TEXT --accelerator TEXT --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints catalog list` — List available Catalog models. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints delete NAME` — Delete an Inference Endpoint permanently. `[--namespace TEXT --yes --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints deploy NAME --repo TEXT --framework TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --region TEXT --vendor TEXT` — Deploy an Inference Endpoint from a Hub repository. `[--namespace TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric [pendingRequests|hardwareUsage] --scaling-threshold FLOAT --revision TEXT --custom-image TEXT --health-route TEXT --port INTEGER --container-command TEXT --container-args TEXT --env TEXT --env-file TEXT --secrets TEXT --secrets-file TEXT --type [public|protected|authenticated|private] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints describe NAME` — Get information about an existing endpoint. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints list` — Lists all Inference Endpoints for the given namespace. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints pause NAME` — Pause an Inference Endpoint. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints resume NAME` — Resume an Inference Endpoint. `[--namespace TEXT --fail-if-already-running --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints scale-to-zero NAME` — Scale an Inference Endpoint to zero. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf endpoints update NAME` — Update an existing endpoint. `[--namespace TEXT --repo TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --framework TEXT --revision TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric [pendingRequests|hardwareUsage] --scaling-threshold FLOAT --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf extensions` — Manage hf CLI extensions.
|
||||
|
||||
- `hf extensions exec NAME` — Execute an installed extension.
|
||||
- `hf extensions install REPO_ID` — Install an extension from a public GitHub repository. `[--force --format [auto|human|agent|json|quiet]]`
|
||||
- `hf extensions list` — List installed extension commands. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf extensions remove NAME` — Remove an installed extension. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf extensions search` — Search extensions available on GitHub (tagged with 'hf-extension' topic). `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf extensions update` — Update installed extension(s) to their latest version. `[--format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf jobs` — Run and manage Jobs on the Hub.
|
||||
|
||||
- `hf jobs cancel JOB_ID` — Cancel a Job `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs hardware` — List available hardware options for Jobs `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs inspect JOB_IDS` — Display detailed information on one or more Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs labels JOB_ID` — Update labels on a Job. Replaces all existing labels. `[--label TEXT --clear --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs list` — List Jobs. `[--all --status [COMPLETED|CANCELED|ERROR|DELETED|SCHEDULING|RUNNING] --label TEXT --limit INTEGER --namespace TEXT --filter TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs logs JOB_ID` — Fetch the logs of a Job. `[--follow --tail INTEGER --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs run IMAGE COMMAND` — Run a Job. `[--env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --timeout TEXT --detach --expose INTEGER --ssh --namespace TEXT]`
|
||||
- `hf jobs scheduled delete SCHEDULED_JOB_ID` — Delete a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs scheduled inspect SCHEDULED_JOB_IDS` — Display detailed information on one or more scheduled Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs scheduled labels SCHEDULED_JOB_ID` — Update labels on a scheduled Job. Replaces all existing labels. `[--label TEXT --clear --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs scheduled list` — List scheduled Jobs `[--all --namespace TEXT --filter TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs scheduled resume SCHEDULED_JOB_ID` — Resume (unpause) a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs scheduled run SCHEDULE IMAGE COMMAND` — Schedule a Job. `[--suspend --concurrency --env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --timeout TEXT --expose INTEGER --namespace TEXT]`
|
||||
- `hf jobs scheduled suspend SCHEDULED_JOB_ID` — Suspend (pause) a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs scheduled trigger SCHEDULED_JOB_ID` — Trigger a scheduled Job to run immediately (does not change the schedule). `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs scheduled uv run SCHEDULE SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--suspend --concurrency --image TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --expose INTEGER --namespace TEXT --with TEXT --python TEXT]`
|
||||
- `hf jobs ssh JOB_ID` — SSH into a running Job. `[--identity-file PATH --dry-run --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs stats` — Fetch the resource usage statistics and metrics of Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf jobs uv run SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--image TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --detach --expose INTEGER --ssh --namespace TEXT --with TEXT --python TEXT]`
|
||||
- `hf jobs wait JOB_IDS` — Wait for one or more Jobs to reach a terminal state. `[--timeout TEXT --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf models` — Interact with models on the Hub.
|
||||
|
||||
- `hf models card MODEL_ID` — Get the model card (README) for a model on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]`
|
||||
- `hf models info MODEL_ID` — Get info about a model on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf models list` — List models on the Hub, or files in a model repo. `[--search TEXT --author TEXT --filter TEXT --pipeline-tag TEXT --gated --apps TEXT --num-parameters TEXT --inference-provider [cerebras|cohere|deepinfra|fal-ai|featherless-ai|fireworks-ai|groq|hf-inference|novita|nscale|openai|ovhcloud|publicai|replicate|scaleway|together|wavespeed|zai-org] --warm --sort [created_at|downloads|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf papers` — Interact with papers on the Hub.
|
||||
|
||||
- `hf papers info PAPER_ID` — Get info about a paper on the Hub. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf papers list` — List daily papers on the Hub. `[--date TEXT --week TEXT --month TEXT --submitter TEXT --sort [publishedAt|trending] --limit INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
- `hf papers read PAPER_ID` — Read a paper as markdown. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf papers search QUERY` — Search papers on the Hub. `[--limit INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf repos` — Manage repos on the Hub.
|
||||
|
||||
- `hf repos branch create REPO_ID BRANCH` — Create a new branch for a repo on the Hub. `[--revision TEXT --type [model|dataset|space] --exist-ok --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos branch delete REPO_ID BRANCH` — Delete a branch from a repo on the Hub. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos create REPO_ID` — Create a new repo on the Hub. `[--type [model|dataset|space] --sdk TEXT --template TEXT --private --public --protected --exist-ok --resource-group-id TEXT --region [us|eu] --flavor [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --storage [small|medium|large] --sleep-time INTEGER --secrets TEXT --secrets-file TEXT --env TEXT --env-file TEXT --volume TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos delete REPO_ID` — Delete a repo from the Hub. This is an irreversible operation. `[--type [model|dataset|space] --missing-ok --yes --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos delete-files REPO_ID PATTERNS` — Delete files from a repo on the Hub. `[--type [model|dataset|space] --revision TEXT --commit-message TEXT --commit-description TEXT --create-pr --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos duplicate FROM_ID` — Duplicate a repo on the Hub (model, dataset, or Space). `[--type [model|dataset|space] --private --public --protected --exist-ok --flavor [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --storage [small|medium|large] --sleep-time INTEGER --secrets TEXT --secrets-file TEXT --env TEXT --env-file TEXT --volume TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos list` — List all repos (models, datasets, spaces, buckets) with storage info. `[--namespace TEXT --type [model|dataset|space|bucket] --search TEXT --limit INTEGER --explore --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos move FROM_ID TO_ID` — Move a repository from a namespace to another namespace. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos settings REPO_ID` — Update the settings of a repository. `[--gated [auto|manual|false] --private --public --protected --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos tag create REPO_ID TAG` — Create a tag for a repo. `[--message TEXT --revision TEXT --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos tag delete REPO_ID TAG` — Delete a tag for a repo. `[--yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf repos tag list REPO_ID` — List tags for a repo. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf sandbox` — Run and manage sandboxes on Hugging Face Jobs.
|
||||
|
||||
- `hf sandbox cp SRC DST` — Copy a file between the local machine and a sandbox (docker-style). `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox create` — Create a sandbox: a dedicated VM by default, or a cheap shared one with `--pool`. `[--pool TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --idle-timeout TEXT --env TEXT --secrets TEXT --env-file TEXT --secrets-file TEXT --volume TEXT --namespace TEXT --forward-hf-token --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox exec SANDBOX_ID COMMAND` — Run a command in a sandbox, streaming output. Exits with the command's exit code. `[--workdir TEXT --env TEXT --env-file TEXT --timeout FLOAT --namespace TEXT]`
|
||||
- `hf sandbox kill` — Terminate a sandbox, a whole shared host, or everything (--all). `[--all --yes --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox pool create` — Warm a pool: boot one host VM now, tagged so it can be found later by its pool id. `[--flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --per-host INTEGER RANGE --max-hosts INTEGER RANGE --idle-timeout TEXT --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox pool delete POOL_ID` — Terminate every host VM of a pool (and therefore all its sandboxes). `[--yes --namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox pool ls` — List running sandbox pools (grouped from their host VMs). `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox process kill SANDBOX_ID PID` — Stop a background process running in a sandbox. `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox process ls SANDBOX_ID` — List the background processes running in a sandbox (started with `hf sandbox spawn`). `[--namespace TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf sandbox spawn SANDBOX_ID COMMAND` — Start a long-running command in the background and return its pid (don't wait). `[--workdir TEXT --env TEXT --env-file TEXT --namespace TEXT]`
|
||||
|
||||
### `hf skills` — Manage skills for AI assistants.
|
||||
|
||||
- `hf skills add` — Install a Hugging Face skill for an AI assistant. `[--claude --global --dest PATH --force --format [auto|human|agent|json|quiet]]`
|
||||
- `hf skills list` — List available skills from the Hugging Face marketplace. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf skills preview` — Print the generated `hf-cli` SKILL.md to stdout. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf skills update` — Update installed Hugging Face marketplace skills. `[--claude --global --dest PATH --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf spaces` — Interact with spaces on the Hub.
|
||||
|
||||
- `hf spaces card SPACE_ID` — Get the Space card (README) for a Space on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces dev-mode SPACE_ID` — Enable or disable dev mode on a Space. `[--stop --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces hardware` — List available hardware options for Spaces. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces hot-reload SPACE_ID` — Hot-reload any Python file of a Space without a full rebuild + restart. `[--local-file PATH --skip-checks --skip-summary --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces info SPACE_ID` — Get info about a space on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces list` — List spaces on the Hub, or files in a space repo. `[--search TEXT --author TEXT --filter TEXT --sort [created_at|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces logs SPACE_ID` — Fetch the run or build logs of a Space. `[--build --follow --tail INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces pause SPACE_ID` — Pause a Space. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces restart SPACE_ID` — Restart a Space. `[--factory-reboot --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces search QUERY` — Search spaces on the Hub using semantic search. `[--filter TEXT --sdk TEXT --include-non-running --description --limit INTEGER --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces secrets add SPACE_ID` — Add or update secrets for a Space. `[--secrets TEXT --secrets-file TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces secrets delete SPACE_ID KEY` — Remove a secret from a Space. `[--yes --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces secrets list SPACE_ID` — List secrets for a Space. Secret values are write-only and not returned. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces settings SPACE_ID` — Update the settings of a Space. `[--sleep-time INTEGER --hardware [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces ssh SPACE_ID` — SSH into a Space's Dev Mode container. `[--identity-file PATH --dry-run --auto --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces templates` — List the available Space templates. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces variables add SPACE_ID` — Add or update environment variables for a Space. `[--env TEXT --env-file TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces variables delete SPACE_ID KEY` — Remove an environment variable from a Space. `[--yes --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces variables list SPACE_ID` — List environment variables for a Space. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces volumes delete SPACE_ID` — Remove all volumes from a Space. `[--yes --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces volumes list SPACE_ID` — List volumes mounted in a Space. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces volumes set SPACE_ID` — Set (replace) volumes for a Space. `[--volume TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf spaces wait SPACE_ID` — Wait for a Space to finish building/starting. `[--timeout TEXT --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
### `hf webhooks` — Manage webhooks on the Hub.
|
||||
|
||||
- `hf webhooks create --watch TEXT` — Create a new webhook. `[--url TEXT --job-id TEXT --domain [repo|discussions] --secret TEXT --format [auto|human|agent|json|quiet]]`
|
||||
- `hf webhooks delete WEBHOOK_ID` — Delete a webhook permanently. `[--yes --format [auto|human|agent|json|quiet]]`
|
||||
- `hf webhooks disable WEBHOOK_ID` — Disable an active webhook. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf webhooks enable WEBHOOK_ID` — Enable a disabled webhook. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf webhooks info WEBHOOK_ID` — Show full details for a single webhook. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf webhooks list` — List all webhooks for the current user. `[--format [auto|human|agent|json|quiet]]`
|
||||
- `hf webhooks update WEBHOOK_ID` — Update an existing webhook. Only provided options are changed. `[--url TEXT --watch TEXT --domain [repo|discussions] --secret TEXT --format [auto|human|agent|json|quiet]]`
|
||||
|
||||
## Common options
|
||||
|
||||
- `--format` — Output format: `--format json` (or `--json`) or `--format table` (default).
|
||||
- `-q / --quiet` — Quiet output (one ID per line).
|
||||
- `--revision` — Git revision id which can be a branch name, a tag, or a commit hash.
|
||||
- `--token` — Use a User Access Token. Prefer setting `HF_TOKEN` env var instead of passing `--token`.
|
||||
- `--type` — The type of repository (model, dataset, or space).
|
||||
|
||||
## Mounting repos as local filesystems
|
||||
|
||||
To mount Hub repositories or buckets as local filesystems — no download, no copy, no waiting — use `hf-mount`. Files are fetched on demand. GitHub: https://github.com/huggingface/hf-mount
|
||||
|
||||
Install: `curl -fsSL https://raw.githubusercontent.com/huggingface/hf-mount/main/install.sh | sh`
|
||||
|
||||
Some command examples:
|
||||
- `hf-mount start repo openai-community/gpt2 /tmp/gpt2` — mount a repo (read-only)
|
||||
- `hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket /tmp/data` — mount a bucket (read-write)
|
||||
- `hf-mount status` / `hf-mount stop /tmp/data` — list or unmount
|
||||
|
||||
## Tips
|
||||
|
||||
- Use `hf <command> --help` for full options, descriptions, usage, and real-world examples
|
||||
- Authenticate with `HF_TOKEN` env var (recommended) or with `--token`
|
||||
- Update the CLI with `hf update` (uses the correct command for the detected install method)
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
name: hf-cloud-aws-context-discovery
|
||||
description: Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like "use my AWS account", "deploy to AWS", "use my profile", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# AWS Context Discovery
|
||||
|
||||
Before doing any AWS work, read the user's local AWS config. Don't guess the region, and don't ask the user for things their config already answers.
|
||||
|
||||
## What to discover
|
||||
|
||||
Run these at the start of the AWS work and remember the results for the rest of the session.
|
||||
|
||||
### 1. Active profile
|
||||
|
||||
`AWS_PROFILE` env var, else `default`. If the user mentioned a profile in their prompt, that overrides. If the named profile doesn't exist in `~/.aws/config`, surface that clearly.
|
||||
|
||||
### 2. Region
|
||||
|
||||
Resolution order — stop at the first one that produces a value:
|
||||
1. Region the user explicitly named in this conversation
|
||||
2. `AWS_REGION` env var
|
||||
3. `AWS_DEFAULT_REGION` env var
|
||||
4. `region` field on the active profile in `~/.aws/config`
|
||||
5. Ask the user — but only after the first four have failed
|
||||
|
||||
Do not fall back to `us-east-1` or any other hardcoded default.
|
||||
|
||||
### 3. Credentials, account ID, caller ARN
|
||||
|
||||
```bash
|
||||
aws sts get-caller-identity --profile <profile> --region <region>
|
||||
```
|
||||
|
||||
Three purposes in one call: confirms credentials are valid (stop if not), returns the `Account` ID (needed for ARN construction), returns the `Arn` of the caller.
|
||||
|
||||
### 4. Identify SSO / assumed-role principals
|
||||
|
||||
The `Arn` field tells you what kind of principal this is. The pattern matters because it determines what IAM operations the caller can do.
|
||||
|
||||
| ARN pattern | Type | IAM write capability |
|
||||
|---|---|---|
|
||||
| `arn:aws:iam::<acct>:user/<name>` | IAM user | Depends on attached policies |
|
||||
| `arn:aws:sts::<acct>:assumed-role/AWSReservedSSO_<...>/<email>` | **SSO assumed-role** | Typically **none** — can't create/modify IAM roles |
|
||||
| `arn:aws:sts::<acct>:assumed-role/<role>/<session>` | Regular assumed-role | Depends on the role |
|
||||
|
||||
**If the caller is SSO**, surface this immediately before later skills hit `iam:CreateRole` and fail:
|
||||
|
||||
> Heads up: you're authenticated via SSO (`AWSReservedSSO_<PermissionSet>_...`). SSO principals usually can't create IAM roles directly. If we need a SageMaker execution role, I'll look for an existing one first — if none exists, you'll need to ask whoever manages your AWS access to create one.
|
||||
|
||||
This is the highest-leverage thing this skill does. Surfacing it now turns a confusing mid-deployment error into a five-second conversation.
|
||||
|
||||
## Commands to run
|
||||
|
||||
```bash
|
||||
# Effective profile and region (faster than parsing config files)
|
||||
aws configure list
|
||||
|
||||
# Validate credentials and get identity
|
||||
aws sts get-caller-identity
|
||||
aws sts get-caller-identity --profile <profile-name> # if a profile was named
|
||||
```
|
||||
|
||||
`aws configure list` handles env-var overrides and shows the resolved effective values. Prefer it over parsing `~/.aws/config` yourself. If you need to read raw config (e.g. to list profiles), `~/.aws/config` and `~/.aws/credentials` are plain INI files — read-only.
|
||||
|
||||
## What to report back
|
||||
|
||||
One or two lines, not a wall of text:
|
||||
|
||||
> Working with profile `my-profile` in `eu-west-1`, account `123456789012`. You're authenticated via SSO, so we'll need to use an existing IAM role rather than create one.
|
||||
|
||||
Don't ask the user to confirm the region you just read from their config — they configured it; that is the confirmation.
|
||||
|
||||
If something is wrong (credentials expired, profile doesn't exist, no region anywhere), stop and surface the specific error before continuing.
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: hf-cloud-python-env-setup
|
||||
description: 'Set up an isolated Python environment for SageMaker / AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to "set up the environment". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.'
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Python Environment Setup for SageMaker
|
||||
|
||||
Most SageMaker deployment failures that look like AWS problems are actually Python environment problems: wrong Python version, broken dependency resolution, stale SDK that doesn't know about a current API. This skill makes env setup boring and correct.
|
||||
|
||||
## Core rules
|
||||
|
||||
1. **Never use the system Python.** Always work inside an isolated environment.
|
||||
2. **Pin the Python version, not the package versions.** Use 3.10, 3.11, or 3.12. Avoid 3.13+ — ML libraries lag on wheel availability and dependency resolution breaks in confusing ways.
|
||||
3. **Install the latest of each package.** Don't defensively pin `boto3` or `awscli`. Newer ones have current API surfaces and security fixes. Only pin if the user explicitly requires a specific version.
|
||||
4. **Check installed versions correctly.** Use `importlib.metadata.version("package-name")`, never `module.__version__`. The latter is inconsistent across packages.
|
||||
5. **The bundled scripts use `boto3` directly.** The SageMaker Python SDK is a valid alternative — see "boto3 vs the SageMaker SDK" below.
|
||||
|
||||
## boto3 vs the SageMaker SDK
|
||||
|
||||
The bundled deploy scripts (`deploy.py`, `deploy_async.py`, `teardown.py`) use `boto3` directly and read image URIs from [AWS's published Deep Learning Containers catalog](https://aws.github.io/deep-learning-containers/reference/available_images/). That fits this workflow's explicit-stages design — each skill produces a concrete value (region, role ARN, image URI) that the next one consumes — and `boto3` is the stable underlying API client.
|
||||
|
||||
The SageMaker Python SDK (v3) is fine to use when the user prefers it or their project already does. Since [PR #5960](https://github.com/aws/sagemaker-python-sdk/pull/5960) (June 2026), `ModelBuilder` auto-routes HuggingFace models to the current containers (text-generation → HuggingFace vLLM, multimodal → vLLM-Omni, embeddings → TEI). Don't avoid the SDK over stale-image or wrong-container concerns — that routing is fixed.
|
||||
|
||||
Two specific SDK cases that still need care:
|
||||
|
||||
- **Generative rerankers**: the SDK routes the `text-ranking` task to TEI unconditionally, which is wrong for causal-LM rerankers like Qwen3-Reranker — those need vLLM (see `hf-cloud-serving-image-selection`). Pass the container explicitly for these models.
|
||||
- **SSO assumed-role credentials**: v3 has had credential-resolution regressions in `ModelTrainer` / `FrameworkProcessor` under SSO profiles. If SDK calls fail with credential errors while `aws sts get-caller-identity` succeeds in the same shell, suspect this rather than your AWS config.
|
||||
|
||||
If you use the SDK, install it into the isolated env like everything else (`.venv/bin/python -m pip install sagemaker`). The bundled scripts don't require it.
|
||||
|
||||
## How to set up
|
||||
|
||||
The fastest path is the bundled script — it's Python, so it runs the same on Windows, macOS, and Linux:
|
||||
|
||||
```bash
|
||||
python3 scripts/setup_env.py # macOS / Linux
|
||||
python scripts/setup_env.py # Windows (PowerShell / cmd)
|
||||
```
|
||||
|
||||
This script detects `uv` and uses it if available (faster), falls back to the stdlib `venv` module, creates `.venv/` with Python 3.12 (override: `python3 setup_env.py .venv 3.11`), refuses unsupported Python versions, installs from the bundled `requirements.txt`, and is idempotent. It also prints the correct interpreter path for the host OS (see below).
|
||||
|
||||
Manual equivalent:
|
||||
|
||||
```bash
|
||||
# Preferred: uv
|
||||
uv venv --python 3.12 .venv
|
||||
uv pip install --python .venv/bin/python --upgrade boto3 awscli # Windows: .venv\Scripts\python.exe
|
||||
|
||||
# Fallback: stdlib venv
|
||||
python3.12 -m venv .venv
|
||||
.venv/bin/python -m pip install --upgrade pip boto3 awscli
|
||||
```
|
||||
|
||||
After setup, **invoke the env's Python explicitly** rather than activating the venv. The interpreter path differs by platform:
|
||||
|
||||
```bash
|
||||
.venv/bin/python deploy.py # macOS / Linux
|
||||
.venv\Scripts\python.exe deploy.py # Windows
|
||||
```
|
||||
|
||||
This works the same in scripts, interactive shells, and agent tool calls. The rest of this skill writes `.venv/bin/python` for brevity — on Windows substitute `.venv\Scripts\python.exe`.
|
||||
|
||||
## Verifying
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/check_versions.py
|
||||
```
|
||||
|
||||
Prints versions of `boto3`, `botocore`, `awscli`. Uses `importlib.metadata.version()` so it works on every package, including ones without `__version__`. Pass arbitrary names: `... check_versions.py transformers huggingface_hub`.
|
||||
|
||||
## Deployment-specific extras
|
||||
|
||||
Default `requirements.txt` covers SageMaker orchestration. Some deployments need extras (`huggingface_hub` for model inspection, `transformers` for tokenizer validation). Add these to a deployment-specific requirements file in the project, install with the env's Python, don't pin unless there's a reason.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
**Mysterious `pip install` resolution errors**
|
||||
Almost always Python 3.13+ trying to install packages without wheels yet, or installing into a polluted system Python. Recreate at 3.12: delete `.venv` and re-run `python3 setup_env.py .venv 3.12` (the script recreates the env when the version doesn't match, so you can also just re-run it).
|
||||
|
||||
**`pip install` succeeded but the script says "module not found"**
|
||||
You installed into a different interpreter than the one running the script. Always invoke Python explicitly: `.venv/bin/python -m pip install ...` and `.venv/bin/python deploy.py`.
|
||||
|
||||
**Inline `python -c "..."` one-liners fail in PowerShell**
|
||||
PowerShell's quoting rules mangle nested/escaped quotes in inline Python. Don't debug the quoting — write the snippet to a small `.py` file and run that. (All bundled helpers are files for exactly this reason.)
|
||||
|
||||
**boto3 call fails with "unknown parameter"**
|
||||
Your boto3 is older than the API surface. Upgrade with `.venv/bin/python -m pip install --upgrade boto3`. Don't downgrade the script to match an old version.
|
||||
|
||||
**`sagemaker` (the SDK) installed but the bundled scripts fail**
|
||||
The bundled scripts don't use the SDK — they only need `boto3`/`awscli` from `requirements.txt`. Installing `sagemaker` alongside is harmless, but it doesn't replace the requirements install.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Core AWS dependencies for SageMaker deployment.
|
||||
#
|
||||
# No version pins. We want the latest of each — if scripts break against the
|
||||
# latest, fix the script, not the version. Newer boto3/awscli have current
|
||||
# API surfaces and security fixes.
|
||||
#
|
||||
# We do NOT depend on the SageMaker Python SDK (sagemaker / sagemaker-core /
|
||||
# sagemaker-serve). The deployment scripts call boto3 directly, and image URIs
|
||||
# are read from AWS's published catalog page rather than resolved via the SDK.
|
||||
# See hf-cloud-python-env-setup SKILL.md ("Why no SageMaker SDK").
|
||||
#
|
||||
# Add per-deployment extras (huggingface_hub for model inspection, transformers
|
||||
# for tokenizer validation) to a deployment-specific requirements file alongside
|
||||
# this one. Do not pin versions here unless you have a documented reason.
|
||||
|
||||
boto3
|
||||
awscli
|
||||
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env python
|
||||
"""Report installed versions of key dependencies.
|
||||
|
||||
Uses importlib.metadata.version() (works for every installed package, unlike
|
||||
the inconsistently-defined `module.__version__`).
|
||||
|
||||
Usage:
|
||||
python check_versions.py
|
||||
python check_versions.py boto3 botocore transformers
|
||||
"""
|
||||
|
||||
import sys
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
|
||||
DEFAULT_PACKAGES = ["boto3", "botocore", "awscli"]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
packages = sys.argv[1:] if len(sys.argv) > 1 else DEFAULT_PACKAGES
|
||||
for pkg in packages:
|
||||
try:
|
||||
print(f"{pkg}=={version(pkg)}")
|
||||
except PackageNotFoundError:
|
||||
print(f"{pkg}: NOT INSTALLED")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Create an isolated Python environment for SageMaker work. Cross-platform.
|
||||
|
||||
Idempotent. Prefers uv, falls back to the stdlib venv module.
|
||||
|
||||
Handles the platform difference in venv layout: the interpreter lives at
|
||||
`<venv>/bin/python` on macOS/Linux and `<venv>\\Scripts\\python.exe` on Windows.
|
||||
The path it prints is the correct one for the host OS.
|
||||
|
||||
Usage:
|
||||
python setup_env.py [VENV_DIR=.venv] [PYTHON_VERSION=3.12]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 3.10-3.12 is the safe zone for modern boto3 + awscli. 3.13+ may work but ML
|
||||
# libs lag on wheel availability.
|
||||
SUPPORTED_MIN = (3, 10)
|
||||
SUPPORTED_MAX = (3, 12)
|
||||
|
||||
IS_WINDOWS = os.name == "nt"
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(f"[setup_env] {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def parse_version(v: str) -> tuple[int, int]:
|
||||
try:
|
||||
major, minor = (int(p) for p in v.split(".")[:2])
|
||||
return (major, minor)
|
||||
except ValueError:
|
||||
log(f"ERROR: invalid Python version '{v}' (expected e.g. 3.12)")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def venv_python(venv_dir: Path) -> Path:
|
||||
"""Path to the interpreter inside a venv, per platform."""
|
||||
if IS_WINDOWS:
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def interpreter_version(python_path: Path) -> str | None:
|
||||
if not python_path.exists():
|
||||
return None
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c",
|
||||
"import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
return proc.stdout.strip() if proc.returncode == 0 else None
|
||||
|
||||
|
||||
def find_base_python(version: str) -> str | None:
|
||||
"""Locate a base interpreter for the requested version for `python -m venv`."""
|
||||
# Try `pythonX.Y` (Unix) / `pythonX.Y.exe`, then the Windows `py` launcher.
|
||||
candidate = shutil.which(f"python{version}")
|
||||
if candidate:
|
||||
return candidate
|
||||
if IS_WINDOWS and shutil.which("py"):
|
||||
probe = subprocess.run(["py", f"-{version}", "--version"], capture_output=True, text=True)
|
||||
if probe.returncode == 0:
|
||||
return f"py -{version}" # sentinel; expanded by caller
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
venv_dir = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(".venv")
|
||||
python_version = sys.argv[2] if len(sys.argv) > 2 else "3.12"
|
||||
|
||||
ver = parse_version(python_version)
|
||||
if not (SUPPORTED_MIN <= ver <= SUPPORTED_MAX):
|
||||
lo = ".".join(map(str, SUPPORTED_MIN))
|
||||
hi = ".".join(map(str, SUPPORTED_MAX))
|
||||
log(f"ERROR: Python {python_version} outside supported range {lo}-{hi}")
|
||||
log("Use 3.10, 3.11, or 3.12.")
|
||||
return 1
|
||||
|
||||
installer = "uv" if shutil.which("uv") else "venv"
|
||||
if installer == "venv":
|
||||
log("uv not found — falling back to python + venv. Consider: https://docs.astral.sh/uv/")
|
||||
|
||||
py = venv_python(venv_dir)
|
||||
|
||||
# Reuse existing env only if Python version matches.
|
||||
current = interpreter_version(py)
|
||||
if current == python_version:
|
||||
log(f"Env exists at {venv_dir} with Python {current} — reusing")
|
||||
else:
|
||||
if current is not None:
|
||||
log(f"Env at {venv_dir} uses Python {current} (wanted {python_version}) — recreating")
|
||||
shutil.rmtree(venv_dir)
|
||||
|
||||
if installer == "uv":
|
||||
uv = shutil.which("uv")
|
||||
if subprocess.run([uv, "venv", "--python", python_version, str(venv_dir)]).returncode != 0:
|
||||
log("ERROR: `uv venv` failed.")
|
||||
return 1
|
||||
else:
|
||||
base = find_base_python(python_version)
|
||||
if not base:
|
||||
log(f"ERROR: python{python_version} not found on PATH.")
|
||||
log("Install via pyenv/asdf/brew/system package manager, the Windows installer, or install uv.")
|
||||
return 1
|
||||
base_cmd = base.split() if base.startswith("py ") else [base]
|
||||
if subprocess.run([*base_cmd, "-m", "venv", str(venv_dir)]).returncode != 0:
|
||||
log("ERROR: `python -m venv` failed.")
|
||||
return 1
|
||||
log(f"Created env at {venv_dir}")
|
||||
|
||||
requirements = Path(__file__).resolve().parent.parent / "requirements.txt"
|
||||
if not requirements.is_file():
|
||||
log(f"ERROR: requirements.txt not found at {requirements}")
|
||||
return 1
|
||||
|
||||
log(f"Installing from {requirements}")
|
||||
if installer == "uv":
|
||||
uv = shutil.which("uv")
|
||||
rc = subprocess.run(
|
||||
[uv, "pip", "install", "--python", str(py), "--upgrade", "-r", str(requirements)]
|
||||
).returncode
|
||||
else:
|
||||
subprocess.run([str(py), "-m", "pip", "install", "--upgrade", "pip"])
|
||||
rc = subprocess.run(
|
||||
[str(py), "-m", "pip", "install", "--upgrade", "-r", str(requirements)]
|
||||
).returncode
|
||||
if rc != 0:
|
||||
log("ERROR: dependency install failed.")
|
||||
return 1
|
||||
|
||||
log(f"Done. Invoke directly: {py} <script>")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
name: hf-cloud-sagemaker-deployment-planner
|
||||
description: 'Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like "deploy a model", "host this LLM on AWS", "serve this embedding model", "deploy a reranker", "deploy a text-to-image / diffusion model", "host this for async inference", "create an endpoint", "serve my fine-tuned model", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. "I just want to get this running on AWS, you figure it out"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image / diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.'
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# SageMaker Deployment Planner
|
||||
|
||||
You are helping a user deploy a model to Amazon SageMaker. Most users invoking this skill want the model deployed with reasonable defaults, in as few questions as possible. Ask only what you need, recommend a pathway honestly, and hand off to the specialized skills.
|
||||
|
||||
## Workflow phases
|
||||
|
||||
1. **Discovery** — what is being deployed and what are the constraints (this skill)
|
||||
2. **Pathway selection** — real-time / serverless / async / batch / Bedrock CMI (this skill)
|
||||
3. **Context preflight** — `hf-cloud-aws-context-discovery`, then `hf-cloud-python-env-setup`
|
||||
4. **IAM preflight** — `hf-cloud-sagemaker-iam-preflight`
|
||||
5. **Image selection** — `hf-cloud-serving-image-selection`
|
||||
6. **Deployment** — `hf-cloud-sagemaker-production-defaults`
|
||||
|
||||
Phases 1–2 are this skill's job. The others activate when their patterns match.
|
||||
|
||||
## Discovery: ask only what you need
|
||||
|
||||
You will eventually need to know:
|
||||
|
||||
- **What model**: HuggingFace ID, S3 path to artifacts, or model name. If the user is vague ("the model I fine-tuned"), ask for the artifact location.
|
||||
- **Model type**: text-generation LLM, embedding/reranker, or other (classifier, NER, etc.). This determines the serving stack — usually inferable from the model name (anything ending in `-embed-*`, starting with `BAAI/bge-`, `sentence-transformers/*` etc. is embeddings; chat/instruct models are LLMs). Only ask if it's genuinely ambiguous.
|
||||
- **Traffic shape**: roughly how often will this be called?
|
||||
- **Latency tolerance**: interactive, near-real-time, or async?
|
||||
- **Cost sensitivity**: ask only if the user signals it or the traffic pattern is ambiguous.
|
||||
|
||||
Region comes from `hf-cloud-aws-context-discovery` — don't ask unless the user volunteers it.
|
||||
|
||||
Do **not** front-load all of these. A common minimal set is just: *what model, and roughly how often will it be called?* The model name usually settles the model-type question. That alone is often enough to narrow the pathway to two candidates. If the user already told you something, don't ask again.
|
||||
|
||||
## Pathway selection
|
||||
|
||||
| Pathway | When it fits | When it does not |
|
||||
|---|---|---|
|
||||
| **Real-time endpoint** | Steady traffic, sub-second to few-second latency, always-on | Very spiky or very sparse traffic (wastes money on idle) |
|
||||
| **Serverless inference** | Spiky/intermittent, tolerates cold starts (~10s+), simpler models | LLMs above a few B params (memory/cold-start limits), strict SLAs |
|
||||
| **Async inference** | Long inference (>60s), large payloads, queue-friendly | Interactive synchronous calls |
|
||||
| **Batch transform** | Offline scoring over a dataset | Anything online or interactive |
|
||||
| **Bedrock Custom Model Import** | Wants Bedrock-compatible API, supported base family, weights only | Custom inference logic, unsupported architectures |
|
||||
|
||||
For LLMs, **real-time endpoints are the default** unless traffic is explicitly spiky/sparse or inference is long-running. Serverless looks attractive for "low traffic" cases but most LLMs exceed its memory limits.
|
||||
|
||||
For **embeddings**, real-time is again the default — but CPU instances are usually the right choice (much cheaper, fast enough for most embedding workloads). Don't reflexively recommend GPU instances for embedding models; ask `hf-cloud-serving-image-selection` to consider CPU variants if the model is small (<1B params) and traffic is moderate.
|
||||
|
||||
For **text-to-image, video generation, or other long-inference workloads** (>30s per request) where traffic is also bursty: async inference is the right answer. It supports genuine scale-to-zero between batches and queues requests via S3, so you don't pay for idle GPU. `hf-cloud-sagemaker-production-defaults` has a dedicated `deploy_async.py` for this.
|
||||
|
||||
Real-time and async are the two scripted pathways. Serverless, batch transform, and Bedrock Custom Model Import are not currently scripted — for those, hand the user off with a brief explanation rather than trying to deploy them through this workflow.
|
||||
|
||||
If two pathways are both reasonable, say so in one sentence each and pick one. Don't bury the recommendation in options.
|
||||
|
||||
## Instance selection: check quota before recommending
|
||||
|
||||
Endpoint quotas are per instance type, per region, and default to **0** for GPU types in many accounts. Recommending an instance the account can't launch wastes a full deploy cycle on `ResourceLimitExceeded`. Check first:
|
||||
|
||||
```bash
|
||||
aws service-quotas list-service-quotas --service-code sagemaker --region <region> \
|
||||
--query "Quotas[?contains(QuotaName, 'for endpoint usage') && Value > \`0\`].[QuotaName, Value]" \
|
||||
--output table
|
||||
```
|
||||
|
||||
If the type you want isn't in the result, recommend one that is — or tell the user to request an increase (hours to days) *before* creating anything.
|
||||
|
||||
GPU family notes for the common 24 GB tier:
|
||||
|
||||
- `ml.g5.*` (A10G) and `ml.g6.*` (L4) both work with current vLLM images when the gpu-3-1 AMI is set (see `hf-cloud-serving-image-selection`). g6 is the newer generation and slightly cheaper per hour; g5 has roughly double the memory bandwidth, which usually means better LLM token throughput. Pick whichever has quota; when both do, either is defensible — g5 for throughput, g6 for cost.
|
||||
- `ml.g6e.*` (L40S, 48 GB) when the model doesn't fit in 24 GB.
|
||||
|
||||
Once you have enough to recommend, state it plainly:
|
||||
|
||||
> Based on what you've told me, I'd recommend a real-time endpoint on `ml.g5.xlarge`. The model is small enough that this is cost-effective, and your traffic pattern is steady enough that you won't be paying for idle. Alternative: serverless would be cheaper if traffic dries up for hours at a time, but Qwen3-0.6B is at the edge of serverless memory limits and cold starts would be 15–30s. Want me to proceed with the real-time endpoint?
|
||||
|
||||
Then wait for confirmation. The user should know what they're about to spend money on before you create anything.
|
||||
|
||||
The plan lives in the conversation — don't generate `plan.yaml` or similar artifacts unless explicitly asked.
|
||||
|
||||
## Style
|
||||
|
||||
- Users invoking this skill are deferring to the agent because they don't want to do AWS plumbing. Match that energy: efficient, not exhaustive.
|
||||
- One round of clarifying questions is usually enough. Three rounds is interrogation.
|
||||
- When you don't know something specific (current image URI, SDK API surface, quotas), check it rather than guess. Other skills handle the "how to check" details.
|
||||
- If the user pushes back on a recommendation, accept it. They know their constraints better than you do.
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
name: hf-cloud-sagemaker-iam-preflight
|
||||
description: 'Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.'
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# SageMaker IAM Preflight
|
||||
|
||||
Every SageMaker resource needs an **execution role** — the IAM role SageMaker assumes to read model artifacts from S3, pull serving containers from ECR, and write logs. Most deployments fail here because the script tried to create a new role without checking if a usable one already existed, then blew up because the caller is an SSO principal.
|
||||
|
||||
This skill encodes the right order: discover, validate, only create if necessary.
|
||||
|
||||
## Running the helpers (cross-platform)
|
||||
|
||||
The helpers are Python so they run identically on Windows, macOS, and Linux:
|
||||
|
||||
```bash
|
||||
python3 scripts/check_role.py # macOS / Linux
|
||||
python scripts/check_role.py # Windows (PowerShell / cmd)
|
||||
```
|
||||
|
||||
**Run them from the shell where the AWS CLI already works** — i.e. wherever `aws sts get-caller-identity` succeeds. The script shells out to that same `aws` binary and inherits the shell's profile, region, SSO session, proxy, and credential chain.
|
||||
|
||||
> **Windows / WSL / Git Bash caveat.** Do **not** invoke these through a Bash shim (WSL, Git Bash, MSYS) on Windows. Those Bash environments frequently do **not** share the Windows AWS config, credentials, SSO sessions, environment variables, or proxy settings — so `aws sts get-caller-identity` fails inside Bash even when it works natively in PowerShell. (This is exactly why the old `.sh` helpers failed on Windows and were replaced with Python.) If you're in PowerShell, run `python ...\check_role.py` directly in PowerShell. If the helper still can't see your identity, run the same discovery natively (see "Native AWS CLI equivalent" below) in the shell where `aws sts get-caller-identity` returns your ARN.
|
||||
|
||||
## Order of operations
|
||||
|
||||
### Step 1 — Did the user provide a role?
|
||||
|
||||
Validate that one specifically:
|
||||
|
||||
```bash
|
||||
python3 scripts/check_role.py "<role-name-or-arn>"
|
||||
```
|
||||
|
||||
On success it prints the ARN to stdout (exit 0). On failure it logs why on stderr. Don't try to silently fix a broken role — surface the problem.
|
||||
|
||||
### Step 2 — Discover existing roles
|
||||
|
||||
```bash
|
||||
python3 scripts/check_role.py
|
||||
```
|
||||
|
||||
Lists roles matching common SageMaker patterns (`AmazonSageMaker-ExecutionRole-*`, `SageMakerExecutionRole*`, etc.), **ranks by last-used date** (most recent first), validates trust policy in that order, returns the first usable ARN. Most accounts that have used SageMaker before already have one.
|
||||
|
||||
Why rank by last-used: in accounts with multiple roles (auto-generated 2021 role + manual project role + etc.), the alphabetically-first one is rarely the actively-maintained one. The most-recently-used role is more likely to have current policies — including cross-account ECR pull. The script prints the ranking so you can see which got picked.
|
||||
|
||||
IAM frequently reports **no** `RoleLastUsed` at all (tracking only covers recent activity). When every candidate ties at "never used", the script falls back to **newest creation date** — a newer role is more likely to have current policies than a 2021 leftover.
|
||||
|
||||
### Step 3 — Create, only if discovery found nothing
|
||||
|
||||
**If the user can create** (has IAM permissions):
|
||||
|
||||
```bash
|
||||
python3 scripts/create_role.py "<role-name>" "<model-bucket>"
|
||||
```
|
||||
|
||||
Second arg scopes S3 access to a specific bucket. Omit if unknown; script warns and the user can update the policy later.
|
||||
|
||||
**If the user cannot create** (SSO principal — `hf-cloud-aws-context-discovery` will have flagged this):
|
||||
|
||||
Stop and surface this clearly. Don't retry alternative IAM operations hoping one works:
|
||||
|
||||
> I can't find an existing SageMaker execution role, and you're authenticated via SSO so you can't create one directly. Please either:
|
||||
> - Ask your AWS admin for a SageMaker execution role ARN, or
|
||||
> - Have them grant your SSO permission set `iam:CreateRole`, `iam:AttachRolePolicy`, `iam:PutRolePolicy`
|
||||
|
||||
Specific instructions get unblocked fast; vague "permission denied" messages don't.
|
||||
|
||||
## What "validated" means
|
||||
|
||||
A role is usable when (1) it exists, (2) its trust policy allows `sagemaker.amazonaws.com` to `sts:AssumeRole` — see `references/trust-policy.json` for the canonical form.
|
||||
|
||||
`check_role.py` verifies these two. It does **not** deep-check permissions because comprehensive analysis is expensive (`iam:SimulatePrincipalPolicy` per action) and most existing SageMaker roles are over-permissioned via `AmazonSageMakerFullAccess`. If you suspect a permissions issue at deploy time, the deployment error will tell you which action was denied — fix it then, not preemptively.
|
||||
|
||||
## Minimum permissions
|
||||
|
||||
`references/minimum-permissions.json` covers what SageMaker actually needs:
|
||||
- `s3:GetObject` + `s3:ListBucket` on the model artifact bucket
|
||||
- ECR pull permissions
|
||||
- CloudWatch logs and metrics
|
||||
|
||||
Layered on top of `AmazonSageMakerFullAccess` (attached by `create_role.py`). Replace `REPLACE_WITH_MODEL_BUCKET` in the template with the actual bucket name — `create_role.py` does this automatically when given a bucket as its second argument.
|
||||
|
||||
## Native AWS CLI equivalent (fallback)
|
||||
|
||||
If the Python helper can't run or can't see your identity (rare — usually a broken PATH or running under a Bash shim that lacks AWS context), do the same preflight by hand in the shell where `aws sts get-caller-identity` works. The logic is just AWS CLI calls; the helper exists only to bundle and rank them.
|
||||
|
||||
PowerShell:
|
||||
|
||||
```powershell
|
||||
# 1. List candidate SageMaker roles
|
||||
aws iam list-roles --query "Roles[?contains(RoleName,'SageMaker') || contains(RoleName,'sagemaker')]" --output json
|
||||
|
||||
# 2. For each candidate, confirm the trust policy allows sagemaker.amazonaws.com
|
||||
aws iam get-role --role-name <role-name> --query "Role.AssumeRolePolicyDocument" --output json
|
||||
|
||||
# 3. Prefer the most-recently-used role with SageMaker-execution naming
|
||||
# (LastUsedDate is often None for every role — then prefer newest CreateDate)
|
||||
aws iam get-role --role-name <role-name> --query "Role.[RoleLastUsed.LastUsedDate, CreateDate]" --output text
|
||||
```
|
||||
|
||||
Pick the most-recently-used role whose trust policy contains `sagemaker.amazonaws.com`. Use the resulting ARN exactly as if `check_role.py` had returned it. Bash/macOS/Linux use the same commands.
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "S3ModelArtifactAccess",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::REPLACE_WITH_MODEL_BUCKET",
|
||||
"arn:aws:s3:::REPLACE_WITH_MODEL_BUCKET/*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Sid": "ECRPullServingImages",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ecr:GetAuthorizationToken",
|
||||
"ecr:BatchCheckLayerAvailability",
|
||||
"ecr:GetDownloadUrlForLayer",
|
||||
"ecr:BatchGetImage"
|
||||
],
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Sid": "CloudWatchLogs",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"logs:CreateLogStream",
|
||||
"logs:CreateLogGroup",
|
||||
"logs:PutLogEvents",
|
||||
"logs:DescribeLogStreams"
|
||||
],
|
||||
"Resource": "arn:aws:logs:*:*:log-group:/aws/sagemaker/*"
|
||||
},
|
||||
{
|
||||
"Sid": "CloudWatchMetrics",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"cloudwatch:PutMetricData"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": {
|
||||
"Service": "sagemaker.amazonaws.com"
|
||||
},
|
||||
"Action": "sts:AssumeRole"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Discover and validate a SageMaker execution role. Cross-platform (Windows/macOS/Linux).
|
||||
|
||||
Without args: searches the account for usable roles, ranks by last-used date
|
||||
(newest creation date when last-used is unrecorded), returns the first one
|
||||
with a valid trust policy.
|
||||
With arg: validates the named role.
|
||||
|
||||
Does NOT create roles — see create_role.py.
|
||||
|
||||
Why Python (not a shell script): this runs `aws` from the *same* shell the user
|
||||
launched it in, inheriting their AWS profile, region, SSO session, proxy, and
|
||||
credential chain. A bundled Bash helper on Windows (Git Bash/WSL/MSYS) often
|
||||
does NOT share that context, so `aws sts get-caller-identity` fails there even
|
||||
when the native CLI works fine. Run this from the shell where `aws sts
|
||||
get-caller-identity` already succeeds.
|
||||
|
||||
Usage:
|
||||
python check_role.py # discover
|
||||
python check_role.py <role-name-or-arn> # validate a specific role
|
||||
|
||||
(On Windows the launcher is usually `python`; on macOS/Linux it's `python3`.)
|
||||
|
||||
Exit: 0 = usable role found (ARN printed to stdout)
|
||||
1 = no usable role
|
||||
2 = AWS CLI / credentials error
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROLE_NAME_PATTERNS = [
|
||||
"AmazonSageMaker-ExecutionRole-*",
|
||||
"SageMakerExecutionRole*",
|
||||
"*SageMaker*Execution*",
|
||||
"*sagemaker*execution*",
|
||||
]
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(f"[check_role] {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def aws_bin() -> str:
|
||||
"""Resolve the `aws` executable, honoring PATHEXT (finds aws.exe/aws.cmd on Windows)."""
|
||||
exe = shutil.which("aws")
|
||||
if not exe:
|
||||
log("ERROR: the 'aws' CLI was not found on PATH.")
|
||||
log("Install AWS CLI v2 and confirm `aws sts get-caller-identity` works in this shell.")
|
||||
sys.exit(2)
|
||||
return exe
|
||||
|
||||
|
||||
def run_aws(args: list[str]) -> subprocess.CompletedProcess:
|
||||
"""Run an aws CLI command, capturing output. Never raises on nonzero exit."""
|
||||
return subprocess.run(
|
||||
[aws_bin(), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
|
||||
def get_role(role_name: str) -> dict | None:
|
||||
"""Return the Role dict from `aws iam get-role`, or None if it can't be read."""
|
||||
proc = run_aws(["iam", "get-role", "--role-name", role_name, "--output", "json"])
|
||||
if proc.returncode != 0:
|
||||
return None
|
||||
try:
|
||||
return json.loads(proc.stdout)["Role"]
|
||||
except (json.JSONDecodeError, KeyError):
|
||||
return None
|
||||
|
||||
|
||||
def trust_allows_sagemaker(role: dict) -> bool:
|
||||
"""True if the trust policy lets sagemaker.amazonaws.com assume the role."""
|
||||
trust = role.get("AssumeRolePolicyDocument", {})
|
||||
# Substring check over the serialized doc — matches the original shell helper
|
||||
# and is robust to single/list Principal.Service shapes.
|
||||
return "sagemaker.amazonaws.com" in json.dumps(trust)
|
||||
|
||||
|
||||
def validate_role(role_name: str, quiet: bool = False) -> str | None:
|
||||
"""Return the role ARN if it exists and trusts SageMaker, else None."""
|
||||
role = get_role(role_name)
|
||||
if role is None:
|
||||
if not quiet:
|
||||
log(f"Role '{role_name}' does not exist or you cannot describe it.")
|
||||
return None
|
||||
if not trust_allows_sagemaker(role):
|
||||
if not quiet:
|
||||
log(f"Role '{role_name}': trust policy does not allow sagemaker.amazonaws.com")
|
||||
return None
|
||||
return role.get("Arn")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if run_aws(["sts", "get-caller-identity"]).returncode != 0:
|
||||
log("ERROR: 'aws sts get-caller-identity' failed. Run hf-cloud-aws-context-discovery first.")
|
||||
log("Run this helper from the shell where the AWS CLI is configured (e.g. PowerShell on Windows).")
|
||||
return 2
|
||||
|
||||
# Path 1: validate a user-supplied role
|
||||
if len(sys.argv) >= 2:
|
||||
supplied = sys.argv[1]
|
||||
role_name = supplied.rsplit("/", 1)[-1] if supplied.startswith("arn:aws:iam::") else supplied
|
||||
log(f"Validating: {role_name}")
|
||||
arn = validate_role(role_name)
|
||||
if arn:
|
||||
log(f"OK: {arn}")
|
||||
print(arn)
|
||||
return 0
|
||||
return 1
|
||||
|
||||
# Path 2: discover candidates
|
||||
log("Searching for SageMaker execution roles in the account...")
|
||||
proc = run_aws(["iam", "list-roles", "--query", "Roles[*].RoleName", "--output", "json"])
|
||||
if proc.returncode != 0:
|
||||
log("Could not list roles (caller likely lacks iam:ListRoles).")
|
||||
log("Ask the user for an existing role ARN, or have someone with IAM access run this.")
|
||||
return 1
|
||||
try:
|
||||
all_roles = json.loads(proc.stdout) or []
|
||||
except json.JSONDecodeError:
|
||||
all_roles = []
|
||||
|
||||
candidates = [
|
||||
name
|
||||
for name in all_roles
|
||||
if any(fnmatch.fnmatchcase(name, pat) for pat in ROLE_NAME_PATTERNS)
|
||||
]
|
||||
|
||||
if not candidates:
|
||||
log("No matching roles. Options:")
|
||||
log(" 1. Ask the user for an ARN (role might have an unusual name)")
|
||||
log(" 2. Create one (see create_role.py, requires iam:CreateRole)")
|
||||
return 1
|
||||
|
||||
log(f"Found {len(candidates)} candidate(s): {' '.join(candidates)}")
|
||||
|
||||
# Rank by RoleLastUsed (most recent first) — alphabetical order rarely picks
|
||||
# the actively-maintained role in accounts with multiple SageMaker roles.
|
||||
# Fetch each role once and reuse the result for both ranking and validation.
|
||||
log("Ranking by last-used date (fallback: creation date)...")
|
||||
roles_by_name: dict[str, dict] = {}
|
||||
ranked: list[tuple[str, str, str]] = [] # (last_used, create_date, role_name)
|
||||
for name in candidates:
|
||||
role = get_role(name)
|
||||
if role is None:
|
||||
continue
|
||||
roles_by_name[name] = role
|
||||
last_used = (role.get("RoleLastUsed") or {}).get("LastUsedDate") or ""
|
||||
# IAM often reports no RoleLastUsed at all (tracking only covers recent
|
||||
# activity); when every candidate ties at "", newest CreateDate wins.
|
||||
create_date = str(role.get("CreateDate") or "")
|
||||
# ISO-8601 timestamps sort chronologically as strings; "" (never used)
|
||||
# sorts before any timestamp, so descending order puts it last.
|
||||
ranked.append((last_used, create_date, name))
|
||||
|
||||
ranked.sort(reverse=True)
|
||||
|
||||
log("Ranking (most recent first):")
|
||||
for last_used, create_date, name in ranked:
|
||||
when = f"last used {last_used}" if last_used else f"never used, created {create_date or '?'}"
|
||||
log(f" - {name} ({when})")
|
||||
|
||||
for _, _, name in ranked:
|
||||
if trust_allows_sagemaker(roles_by_name[name]):
|
||||
arn = roles_by_name[name].get("Arn")
|
||||
log(f"Using: {arn}")
|
||||
print(arn)
|
||||
return 0
|
||||
|
||||
log("No candidate passed validation (they exist but lack correct trust policy).")
|
||||
log("Fix the trust policy or create a new role (see create_role.py).")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Create a SageMaker execution role. Cross-platform (Windows/macOS/Linux).
|
||||
|
||||
Run only after check_role.py confirms no usable role exists.
|
||||
Requires iam:CreateRole, iam:AttachRolePolicy, iam:PutRolePolicy.
|
||||
|
||||
SSO principals typically lack these — this script will fail with AccessDenied
|
||||
in that case, and the right answer is to ask an AWS admin for a role.
|
||||
|
||||
Like check_role.py, this calls `aws` from the current shell so it inherits the
|
||||
same AWS context (profile, region, SSO session, proxy). Policy documents are
|
||||
passed inline rather than via `file://` paths, which sidesteps Windows path
|
||||
translation entirely.
|
||||
|
||||
Usage:
|
||||
python create_role.py <role-name> [<model-s3-bucket>]
|
||||
|
||||
Without bucket, the inline policy keeps a placeholder for later editing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REFERENCES = Path(__file__).resolve().parent.parent / "references"
|
||||
TRUST_POLICY = REFERENCES / "trust-policy.json"
|
||||
PERMISSIONS_TEMPLATE = REFERENCES / "minimum-permissions.json"
|
||||
|
||||
FULL_ACCESS_ARN = "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(f"[create_role] {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def aws_bin() -> str:
|
||||
exe = shutil.which("aws")
|
||||
if not exe:
|
||||
log("ERROR: the 'aws' CLI was not found on PATH. Install AWS CLI v2.")
|
||||
sys.exit(2)
|
||||
return exe
|
||||
|
||||
|
||||
def run_aws(args: list[str], check: bool = False) -> subprocess.CompletedProcess:
|
||||
proc = subprocess.run([aws_bin(), *args], capture_output=True, text=True)
|
||||
if check and proc.returncode != 0:
|
||||
log(f"AWS command failed: aws {' '.join(args)}")
|
||||
if proc.stderr.strip():
|
||||
log(proc.stderr.strip())
|
||||
sys.exit(1)
|
||||
return proc
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) < 2:
|
||||
log(f"Usage: {Path(sys.argv[0]).name} <role-name> [<model-s3-bucket>]")
|
||||
return 64
|
||||
|
||||
role_name = sys.argv[1]
|
||||
model_bucket = sys.argv[2] if len(sys.argv) > 2 else ""
|
||||
|
||||
if run_aws(["iam", "get-role", "--role-name", role_name]).returncode == 0:
|
||||
log(f"Role '{role_name}' already exists. Use check_role.py to validate it.")
|
||||
return 1
|
||||
|
||||
ident = run_aws(["sts", "get-caller-identity", "--query", "Arn", "--output", "text"], check=True)
|
||||
caller_arn = ident.stdout.strip()
|
||||
if ":assumed-role/AWSReservedSSO_" in caller_arn:
|
||||
log(f"WARNING: SSO caller ({caller_arn}) — IAM creation likely to fail with AccessDenied.")
|
||||
log("If it does, ask an AWS admin to create the role.")
|
||||
|
||||
trust_doc = TRUST_POLICY.read_text(encoding="utf-8")
|
||||
|
||||
log(f"Creating role: {role_name}")
|
||||
run_aws(
|
||||
[
|
||||
"iam", "create-role",
|
||||
"--role-name", role_name,
|
||||
"--assume-role-policy-document", trust_doc,
|
||||
"--description", "SageMaker execution role (hf-cloud-sagemaker-iam-preflight skill)",
|
||||
],
|
||||
check=True,
|
||||
)
|
||||
|
||||
log("Attaching AmazonSageMakerFullAccess")
|
||||
run_aws(
|
||||
["iam", "attach-role-policy", "--role-name", role_name, "--policy-arn", FULL_ACCESS_ARN],
|
||||
check=True,
|
||||
)
|
||||
|
||||
inline_policy = PERMISSIONS_TEMPLATE.read_text(encoding="utf-8")
|
||||
if model_bucket:
|
||||
inline_policy = inline_policy.replace("REPLACE_WITH_MODEL_BUCKET", model_bucket)
|
||||
log(f"Inline policy will grant S3 access to: {model_bucket}")
|
||||
else:
|
||||
log("WARNING: No model bucket specified — policy contains placeholder.")
|
||||
log("Update before deployment, or pass the bucket name as the 2nd argument.")
|
||||
|
||||
log("Attaching inline policy: SageMakerDeploymentMinimum")
|
||||
run_aws(
|
||||
[
|
||||
"iam", "put-role-policy",
|
||||
"--role-name", role_name,
|
||||
"--policy-name", "SageMakerDeploymentMinimum",
|
||||
"--policy-document", inline_policy,
|
||||
],
|
||||
check=True,
|
||||
)
|
||||
|
||||
arn = run_aws(
|
||||
["iam", "get-role", "--role-name", role_name, "--query", "Role.Arn", "--output", "text"],
|
||||
check=True,
|
||||
).stdout.strip()
|
||||
log(f"Created: {arn}")
|
||||
print(arn)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,304 @@
|
||||
---
|
||||
name: hf-cloud-sagemaker-production-defaults
|
||||
description: 'Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.'
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# SageMaker Production Defaults
|
||||
|
||||
The difference between a demo endpoint and one you can leave running is: it scales with traffic, it tells you when it breaks, and you can debug it later. This skill makes those three the default rather than optional extras.
|
||||
|
||||
By the time this skill runs, the planner has chosen a real-time endpoint, IAM has a usable role, and image-selection has resolved a container URI + AMI version. This skill turns those into an actual deployment.
|
||||
|
||||
## What gets created
|
||||
|
||||
For every endpoint, the skill creates these as a unit:
|
||||
|
||||
1. **SageMaker Model** — image + env vars + execution role + S3 artifacts
|
||||
2. **Endpoint config** — instance type, initial count, optional data capture
|
||||
3. **Endpoint** — the real-time endpoint serving inference
|
||||
4. **Autoscaling target + policy** — target tracking on invocations per instance
|
||||
5. **CloudWatch alarms** — latency, errors, platform overhead
|
||||
|
||||
Data capture (logging requests/responses to S3) is **off by default** — useful for debugging but creates ongoing S3 costs the user didn't necessarily ask for. Enable with `--enable-data-capture`.
|
||||
|
||||
All resources get a consistent tag set including `CreatedBy=agentic-deploy-skills` for later cleanup.
|
||||
|
||||
Defaults and reasoning in `references/deployment-template.md`.
|
||||
|
||||
## Running the deployment
|
||||
|
||||
For a text-generation LLM (vLLM):
|
||||
|
||||
```bash
|
||||
python scripts/deploy.py \
|
||||
--model-name qwen3-medical \
|
||||
--image-uri "$IMAGE_URI" \
|
||||
--inference-ami-version "$AMI" \
|
||||
--role-arn "$ROLE_ARN" \
|
||||
--instance-type ml.g5.xlarge \
|
||||
--region "$REGION" \
|
||||
--env SM_VLLM_MODEL=Qwen/Qwen3-0.6B \
|
||||
--env SM_VLLM_HOST=0.0.0.0 \
|
||||
--env SM_VLLM_TRUST_REMOTE_CODE=true \
|
||||
--env SM_VLLM_MAX_MODEL_LEN=4096
|
||||
```
|
||||
|
||||
For an embedding model (TEI, often on CPU):
|
||||
|
||||
```bash
|
||||
python scripts/deploy.py \
|
||||
--model-name bge-large-embeddings \
|
||||
--image-uri "$IMAGE_URI" \
|
||||
--role-arn "$ROLE_ARN" \
|
||||
--instance-type ml.c6i.2xlarge \
|
||||
--region "$REGION" \
|
||||
--env HF_MODEL_ID=BAAI/bge-large-en-v1.5
|
||||
```
|
||||
|
||||
Note: TEI deployments **do not** need `--inference-ami-version`. That flag is vLLM-specific. TEI env vars are also simpler (`HF_MODEL_ID` instead of `SM_VLLM_*`, no host or trust-remote-code to configure).
|
||||
|
||||
Where each value comes from:
|
||||
|
||||
| Parameter | Source |
|
||||
|---|---|
|
||||
| `--image-uri` | `hf-cloud-serving-image-selection` — agent reads from the AWS DLC catalog page |
|
||||
| `--inference-ami-version` | `hf-cloud-serving-image-selection` — required for vLLM tags containing cu130+ |
|
||||
| `--role-arn` | `hf-cloud-sagemaker-iam-preflight` (`check_role.py`) |
|
||||
| `--region` | `hf-cloud-aws-context-discovery` |
|
||||
| `--instance-type` | User input or planner recommendation |
|
||||
| `--env` | Model-specific; see `hf-cloud-serving-image-selection` for required `SM_VLLM_*` vars |
|
||||
| `--model-s3-uri` | Optional — S3 path to model artifacts; omit if loading from HF Hub |
|
||||
|
||||
The script creates resources in order with error handling, waits for `InService` (up to 30 min), surfaces failure reasons, registers autoscaling and alarms, and prints a summary including the teardown command. Outputs a JSON blob on stdout with endpoint/config/model names for downstream scripting.
|
||||
|
||||
The scripts ship with this skill. If the installed copy is missing the `scripts/` directory (some harnesses copy only SKILL.md on install), fetch them from the source repo rather than re-implementing them from this description.
|
||||
|
||||
**Cold-start expectation**: when the model loads from HF Hub, the download happens inside the container after the endpoint starts — 5–15+ minutes to InService is normal, not a failure. `deploy.py` waits 30 minutes; if you write custom wait code, don't time out at 15. Pre-staging weights in S3 (`--model-s3-uri`) cuts this and removes the Hub dependency.
|
||||
|
||||
## InService is not success — smoke-test before declaring victory
|
||||
|
||||
`InService` only means the container answered `/ping`. In MMS-based containers (HF Inference Toolkit) the Java front-end answers pings even while the Python worker crash-loops — an endpoint can be InService and serve nothing. Two checks, always:
|
||||
|
||||
1. **One real invocation.**
|
||||
- Real-time: `invoke_endpoint.py` (below) with a minimal payload; require an HTTP 200 with a sane body.
|
||||
- Async: upload one input to S3, call `invoke-endpoint-async`, poll the output URI for a few minutes (see "Invoking async endpoints"). A result object = success; an object at the failure URI, or nothing appearing, = broken.
|
||||
2. **Scan the endpoint logs for worker-crash markers** — catches the crash-loop case even when the smoke request merely times out:
|
||||
|
||||
```bash
|
||||
aws logs filter-log-events \
|
||||
--log-group-name /aws/sagemaker/Endpoints/<endpoint-name> \
|
||||
--filter-pattern '?"Worker died" ?"Load model failed" ?"ImportError"' \
|
||||
--region <region> --max-items 5
|
||||
```
|
||||
|
||||
Only report the deployment complete after both pass. If the log scan hits, surface the actual traceback from CloudWatch — not the InService status.
|
||||
|
||||
## Testing a real-time endpoint
|
||||
|
||||
Once the endpoint is `InService`, test it with the bundled helper. It is cross-platform and **BOM-safe** — use it instead of hand-writing a payload file and calling `invoke-endpoint` directly:
|
||||
|
||||
```bash
|
||||
# macOS / Linux
|
||||
python3 scripts/invoke_endpoint.py \
|
||||
--endpoint-name <endpoint-name> \
|
||||
--payload '{"inputs": "Hello"}' \
|
||||
--region "$REGION"
|
||||
```
|
||||
|
||||
```powershell
|
||||
# Windows (PowerShell)
|
||||
python scripts\invoke_endpoint.py `
|
||||
--endpoint-name <endpoint-name> `
|
||||
--payload-file payload.json `
|
||||
--region $REGION
|
||||
```
|
||||
|
||||
It accepts either `--payload '<json>'` (inline) or `--payload-file <path>`, validates JSON, writes the request body as plain UTF-8, invokes the endpoint, and prints the response body to stdout.
|
||||
|
||||
### The UTF-8 BOM gotcha (Windows)
|
||||
|
||||
If you write the request payload yourself on Windows, **do not** use `Set-Content -Encoding UTF8` — depending on the PowerShell version it prepends a UTF-8 byte-order mark (BOM). SageMaker's JSON parser rejects a BOM with a 400 `ModelError`:
|
||||
|
||||
```
|
||||
Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
|
||||
```
|
||||
|
||||
This is **not** a model, endpoint-health, or image problem — only the file encoding of the request body. `invoke_endpoint.py` avoids it entirely (it even strips a BOM from a `--payload-file` that already has one). If you must call the CLI directly, write the body as BOM-free UTF-8:
|
||||
|
||||
```powershell
|
||||
# BOM-free UTF-8 — use this
|
||||
[System.IO.File]::WriteAllText((Resolve-Path "payload.json"), $json, [System.Text.UTF8Encoding]::new($false))
|
||||
|
||||
aws sagemaker-runtime invoke-endpoint `
|
||||
--endpoint-name <endpoint-name> `
|
||||
--content-type application/json `
|
||||
--body fileb://payload.json `
|
||||
--region $REGION `
|
||||
response.json
|
||||
```
|
||||
|
||||
**Fallback:** if any invocation fails with `Unexpected UTF-8 BOM`, rewrite the payload as BOM-free UTF-8 (or re-run via `invoke_endpoint.py`) and retry once before treating the endpoint or model as broken.
|
||||
|
||||
### Invoking a generative reranker (vLLM)
|
||||
|
||||
Generative rerankers (Qwen3-Reranker etc. — routed to the HuggingFace vLLM DLC by `hf-cloud-serving-image-selection`) are causal LMs scored by their first generated token, not chat models. Use the **completions API with a raw `prompt`**, not the messages/chat API: chat templating does not reliably honor `chat_template_kwargs` such as `{"enable_thinking": false}`, and a wrong template silently returns near-identical scores for every query–document pair instead of erroring.
|
||||
|
||||
Payload shape (Qwen3-Reranker's expected format — substitute `{query}` / `{document}`):
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: {query}\n<Document>: {document}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
|
||||
"max_tokens": 1,
|
||||
"temperature": 0,
|
||||
"logprobs": 20
|
||||
}
|
||||
```
|
||||
|
||||
The trailing `<|im_start|>assistant\n<think>\n\n</think>\n\n` suffix is load-bearing: it pre-fills an empty thinking block so the first generated token is the yes/no judgment. Score from the returned logprobs: `P("yes") / (P("yes") + P("no"))`. Sanity check the endpoint with one relevant pair (expect >0.9) and one irrelevant pair (expect <0.05) — near-identical scores across pairs mean the prompt template is wrong, not that the model is broken.
|
||||
|
||||
The same rule generalizes: for any thinking-mode model where the prompt must be byte-exact, prefer the raw completions API over chat.
|
||||
|
||||
### Picking the image URI
|
||||
|
||||
The agent reads the image URI from AWS's [Deep Learning Containers catalog](https://aws.github.io/deep-learning-containers/reference/available_images/) — pick the row that matches the model family (HuggingFace vLLM for LLMs, TEI for embeddings, etc.), substitute `<region>` with the deployment region, and pass to `deploy.py --image-uri`.
|
||||
|
||||
For vLLM images specifically (both `huggingface-vllm` and the AWS `vllm` fallback), also check the tag's CUDA version:
|
||||
|
||||
```bash
|
||||
# Example: HuggingFace vLLM 0.21.0 from the catalog
|
||||
IMAGE_URI="763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04"
|
||||
|
||||
# cu130 tag → must pass --inference-ami-version
|
||||
python deploy.py --image-uri "$IMAGE_URI" \
|
||||
--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1 \
|
||||
...
|
||||
```
|
||||
|
||||
For tags with `cu129` or lower, omit `--inference-ami-version`. See `hf-cloud-serving-image-selection` for the full vLLM AMI lookup table and the env-var requirements for each image family.
|
||||
|
||||
## Async inference deployments
|
||||
|
||||
For long-running inferences (>60s), large payloads, or workloads that are bursty/sparse enough to benefit from scale-to-zero, use `deploy_async.py` instead of `deploy.py`. Async genuinely supports `MinCapacity=0` — real-time autoscaling can't.
|
||||
|
||||
```bash
|
||||
python scripts/deploy_async.py \
|
||||
--model-name flux-text-to-image \
|
||||
--image-uri "$IMAGE_URI" \
|
||||
--role-arn "$ROLE_ARN" \
|
||||
--instance-type ml.g5.2xlarge \
|
||||
--region "$REGION" \
|
||||
--output-s3-uri s3://my-bucket/async-output/ \
|
||||
--env HF_MODEL_ID=black-forest-labs/FLUX.1-dev
|
||||
```
|
||||
|
||||
Required extras over `deploy.py`:
|
||||
- `--output-s3-uri` — where async results land (results are not returned synchronously)
|
||||
|
||||
Optional async-specific flags:
|
||||
- `--failure-s3-uri` — separate path for failed invocations
|
||||
- `--success-sns-topic`, `--error-sns-topic` — get notified when async results are ready or fail
|
||||
- `--min-capacity 0` (the default) — scale to zero between batches
|
||||
- `--backlog-per-instance-target N` — target queue depth per instance (default 5)
|
||||
- `--max-concurrent-invocations-per-instance N` — default 4
|
||||
|
||||
### How scale-to-zero works
|
||||
|
||||
The async script registers **two** autoscaling policies on the variant:
|
||||
|
||||
1. **Target-tracking** on `ApproximateBacklogSizePerInstance` — handles ongoing scaling between min and max
|
||||
2. **Step-scaling** triggered by a `HasBacklogWithoutCapacity` CloudWatch alarm — handles `0→1` wake-from-zero
|
||||
|
||||
Both are needed. Target-tracking alone cannot transition from zero (it can't divide by zero instances), so without the step policy the endpoint comes up, scales to zero after the first batch, and never wakes again. The script wires this up automatically.
|
||||
|
||||
### Async alarms
|
||||
|
||||
The script creates three CloudWatch alarms:
|
||||
- `ApproximateBacklogSize > 50` — queue is building faster than capacity can drain it
|
||||
- `InvocationsFailed > 5` — repeated processing failures
|
||||
- `HasBacklogWithoutCapacity` — drives the wake-from-zero policy (not a notification alarm; its action is the step-scaling policy, not the SNS topic)
|
||||
|
||||
If you pass `--sns-alarm-topic <arn>`, the first two notify on that topic. The wake alarm always points at the step policy.
|
||||
|
||||
### Invoking async endpoints
|
||||
|
||||
Async endpoints aren't called synchronously. You upload the input to S3, call `invoke-endpoint-async` with the S3 input location, and SageMaker writes the result to your `--output-s3-uri` when done:
|
||||
|
||||
```bash
|
||||
# Upload your input first
|
||||
aws s3 cp input.json s3://my-input-bucket/job1/input.json
|
||||
|
||||
# Invoke
|
||||
aws sagemaker-runtime invoke-endpoint-async \
|
||||
--endpoint-name <endpoint-name> \
|
||||
--input-location s3://my-input-bucket/job1/input.json \
|
||||
--content-type application/json \
|
||||
--region <region>
|
||||
|
||||
# Poll for the result at your output URI
|
||||
aws s3 cp s3://my-bucket/async-output/<inference-id>.out result.json
|
||||
```
|
||||
|
||||
The same UTF-8 BOM caveat applies to the `input.json` you upload (see "The UTF-8 BOM gotcha" above) — if you build it on Windows, write it as BOM-free UTF-8 or the container's JSON parser will reject it.
|
||||
|
||||
Teardown works the same as real-time: `python3 scripts/teardown.py <endpoint-name>` (the teardown script discovers policies and alarms by name prefix, so it handles both deployment modes).
|
||||
|
||||
## Defaults at a glance
|
||||
|
||||
| Setting | Default | Override |
|
||||
|---|---|---|
|
||||
| Initial instance count | 1 | `--initial-instance-count` |
|
||||
| Autoscaling min / max | 1 / 4 | `--min-capacity`, `--max-capacity` |
|
||||
| Autoscaling target | 20 invocations/min/instance | `--target-invocations-per-instance` |
|
||||
| Data capture | disabled (opt-in) | `--enable-data-capture` |
|
||||
| CloudWatch alarms | 3 alarms | `--no-alarms` |
|
||||
| SNS notification | none (alarms created but won't notify) | `--sns-alarm-topic <arn>` |
|
||||
| Environment tag | `dev` | `--environment` |
|
||||
| InferenceAmiVersion | none (SageMaker default) | `--inference-ami-version` (REQUIRED for vLLM CUDA 13+) |
|
||||
|
||||
Not defaulted (user-specific input needed): VPC config, KMS key, multi-variant, async inference.
|
||||
|
||||
### Autoscaling target — tune by model type
|
||||
|
||||
The default `--target-invocations-per-instance 20` is conservative and tuned for LLM workloads where each request takes 1–5 seconds. For embedding deployments (TEI), each request is much faster (typically <100ms on CPU, <20ms on GPU), so a single instance can handle far more throughput. **For embedding deployments, raise the target to 100–500** depending on instance and model size. The default of 20 will trigger autoscaling far too aggressively for embeddings and waste money.
|
||||
|
||||
A rule of thumb: target value ≈ 60 / (typical request latency in seconds). LLM at 3s latency → target 20. Embedding at 100ms → target 600. Generative rerankers sit in between — they generate a single token per request, so ~40–100 is a reasonable target.
|
||||
|
||||
## Data capture + IAM gotcha
|
||||
|
||||
If the user enables data capture, the execution role needs S3 write access to the capture prefix. The default URI (`s3://sagemaker-<region>-<account>/<endpoint>/data-capture/`) is typically a different bucket than the model artifact bucket. If `hf-cloud-sagemaker-iam-preflight` scoped the inline policy narrowly to just the model bucket, capture writes fail silently — endpoint keeps serving but no data appears.
|
||||
|
||||
If the user reports "data capture isn't showing up", check the role's S3 access. Either widen the inline policy or pass `--data-capture-s3-uri` pointing to a bucket the role can write.
|
||||
|
||||
## Teardown
|
||||
|
||||
```bash
|
||||
python3 scripts/teardown.py <endpoint-name> <region> # macOS / Linux
|
||||
python scripts\teardown.py <endpoint-name> <region> # Windows
|
||||
```
|
||||
|
||||
Deletes in safe order: alarms → autoscaling → endpoint (stops billing) → endpoint config → model. Idempotent.
|
||||
|
||||
Does **not** delete: the IAM execution role (might be shared), data capture S3 objects (user might want to keep), SNS topic, original model artifacts.
|
||||
|
||||
Always tell the user about the teardown command after the deployment summary. Users forget; endpoints accrue cost.
|
||||
|
||||
## When the deployment fails
|
||||
|
||||
**`CannotStartContainerError` + no CloudWatch logs ever created** — the InferenceAmiVersion problem. If the image tag contains `cu130` or later and you didn't pass `--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1`, this is the cause. See `hf-cloud-serving-image-selection`. Do NOT chase images, IAM roles, env vars, or instance types — the failure signature is identical for many other things but the cause here is the AMI.
|
||||
|
||||
**"Failed to pass ping health check"** — the container *did* start and produced logs, but `/ping` isn't responding. Check CloudWatch at `/aws/sagemaker/Endpoints/<endpoint-name>`. Usually: wrong image for model architecture, missing HF token, or OOM.
|
||||
|
||||
**"Container failed to start" (with logs present)** — entrypoint ran, then exited. Check CloudWatch. Common: missing required env vars (`SM_VLLM_MODEL`, `SM_VLLM_HOST`, `SM_VLLM_TRUST_REMOTE_CODE`), wrong `ModelDataUrl` format, unreadable model artifacts.
|
||||
|
||||
**`ResourceLimitExceeded`** — no quota for the instance type in this region. Request increase or pick a different type (the planner should have checked quotas up front — see `hf-cloud-sagemaker-deployment-planner`).
|
||||
|
||||
**`ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` in CloudWatch logs** — known packaging defect in `huggingface-pytorch-inference` GPU images (see "Known-broken images" in `hf-cloud-serving-image-selection`). Inside the container, so no env var, AMI, instance type, or sibling tag fixes it. Switch to DJL Inference.
|
||||
|
||||
**InService, but invocations time out / async outputs never appear** — dead Python worker behind a live MMS front-end. Run the log scan from "InService is not success" above; the traceback in CloudWatch is the real error.
|
||||
|
||||
**`403 Forbidden` downloading weights from HF Hub during startup** — the container's bundled `huggingface_hub` predates HF's XET CDN auth. Add `--env HF_HUB_ENABLE_HF_TRANSFER=0`, or pre-stage the weights in S3. Note: this can *mask* a deeper failure (the worker may still crash after the download succeeds) — re-check logs after fixing it.
|
||||
|
||||
**Diagnostic rule**: when failures look identical across multiple configurations (different images, roles, instance types) and **no logs are ever produced**, the cause is almost always below the container — host AMI, networking, account-level — not the deployment config. Stop iterating on config; check the AMI version and account state.
|
||||
|
||||
Don't retry blindly. The script prints the specific `FailureReason` from `describe-endpoint` — fix the root cause before retrying.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Production Defaults — What and Why
|
||||
|
||||
These defaults turn a working demo into something you can leave running. They favor "you'll find out about problems" over "absolute minimum cost". Tune any of them away if needed — the deployment script should be the starting point, not bare `create_endpoint`.
|
||||
|
||||
## Autoscaling — enabled, target tracking on invocations
|
||||
|
||||
| Setting | Default | Why |
|
||||
|---|---|---|
|
||||
| `MinCapacity` | 1 | Keep one warm. Min=0 enables scale-to-zero but adds cold-start latency on every gap. |
|
||||
| `MaxCapacity` | 4 | Cap on scale-out. Raise for known peaks. |
|
||||
| Target metric | `SageMakerVariantInvocationsPerInstance` | Scales on actual request load, not noisy GPU/CPU. |
|
||||
| Target value | 20 invocations/min/instance | Conservative. Tune to model latency: faster model = higher target. |
|
||||
| `ScaleInCooldown` | 300s | Long enough to avoid flapping. |
|
||||
| `ScaleOutCooldown` | 60s | Scale out faster than in — under spikes, you want capacity now. |
|
||||
|
||||
For LLMs, the right target depends heavily on latency. A 100ms model handles 600/min easily; a 5s model maxes at ~12/min. 20 is a safe default for typical LLM latencies.
|
||||
|
||||
For embeddings (TEI), each request is much faster — typically <100ms on CPU and <20ms on GPU. The default of 20 will autoscale far too aggressively and waste money. Use `--target-invocations-per-instance 100–500` for embedding deployments. Rule of thumb: target ≈ 60 / (typical request latency in seconds).
|
||||
|
||||
## CloudWatch alarms — 3 by default
|
||||
|
||||
- **`ModelLatency` p99 > 30s for 5min** — slow inference, runaway requests, model loading issues.
|
||||
- **`Invocation5XXErrors` sum > 5 in 5min** — container crashes, OOM, ping failures.
|
||||
- **`OverheadLatency` p99 > 2s for 5min** — SageMaker platform issues, separate from model.
|
||||
|
||||
Without `--sns-alarm-topic`, alarms exist but don't notify. The deploy script warns when this happens.
|
||||
|
||||
## Data capture — disabled (opt-in via `--enable-data-capture`)
|
||||
|
||||
Useful for debugging "why did the model say that?" weeks later, building eval datasets, and audit trails for sensitive domains. But it creates ongoing S3 costs the user didn't necessarily ask for and stores data they may not realize is being kept — opt-in is the right default.
|
||||
|
||||
When enabled: 100% sampling, both input and output captured. URI defaults to `s3://sagemaker-<region>-<account>/<endpoint>/data-capture/`. Execution role needs write access to this prefix — if IAM was scoped narrowly to the model bucket, capture fails silently.
|
||||
|
||||
## Resource tagging
|
||||
|
||||
| Tag | Value |
|
||||
|---|---|
|
||||
| `Project` | User-supplied or model name |
|
||||
| `Owner` | Caller ARN's user/email portion |
|
||||
| `Environment` | `dev` (default) |
|
||||
| `CreatedBy` | `agentic-deploy-skills` |
|
||||
| `ModelArtifact` | S3 URI of the model |
|
||||
|
||||
`CreatedBy` matters most — `aws resourcegroupstaggingapi get-resources --tag-filters Key=CreatedBy,Values=agentic-deploy-skills` finds every resource these skills created.
|
||||
|
||||
## Endpoint naming
|
||||
|
||||
`<model-name>-<YYYYMMDD-HHMM>` by default. Timestamped names enable blue-green-ish iteration (bring up new, validate, delete old). Override with `--endpoint-name` if you have naming conventions.
|
||||
|
||||
## Not defaulted (need user input)
|
||||
|
||||
- **VPC config** — needs VPC ID + subnets
|
||||
- **KMS encryption** — needs a specific key
|
||||
- **Multi-variant endpoints** — uncommon enough that defaulting adds complexity
|
||||
- **Async inference config** — different deployment pathway entirely
|
||||
|
||||
## Teardown
|
||||
|
||||
`python3 scripts/teardown.py <endpoint-name>` deletes endpoint → endpoint config → model in safe order. Autoscaling targets and alarms cleaned up separately. Data capture S3 objects are NOT deleted — that's the user's call.
|
||||
@@ -0,0 +1,148 @@
|
||||
"""Shared helpers for the deployment scripts (deploy.py, deploy_async.py).
|
||||
|
||||
Anything that should look identical between real-time and async deployments
|
||||
lives here — tag schema, name format, env-var parsing, model creation,
|
||||
endpoint waiting. Mode-specific logic (endpoint config shape, autoscaling
|
||||
policies, alarm metrics) stays in the calling script.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
|
||||
# Shared tagging convention. CreatedBy lets us find every resource these
|
||||
# scripts have ever created via:
|
||||
# aws resourcegroupstaggingapi get-resources \
|
||||
# --tag-filters Key=CreatedBy,Values=agentic-deploy-skills
|
||||
CREATED_BY_TAG_VALUE = "agentic-deploy-skills"
|
||||
|
||||
|
||||
def log(prefix: str, msg: str) -> None:
|
||||
"""Stream a log line to stderr with a per-script prefix."""
|
||||
print(f"[{prefix}] {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def parse_env(env_args: list[str]) -> dict[str, str]:
|
||||
"""Parse --env KEY=VALUE flags from argparse into a dict."""
|
||||
env: dict[str, str] = {}
|
||||
for item in env_args:
|
||||
if "=" not in item:
|
||||
raise SystemExit(f"--env must be KEY=VALUE, got: {item}")
|
||||
k, v = item.split("=", 1)
|
||||
env[k] = v
|
||||
return env
|
||||
|
||||
|
||||
def make_endpoint_name(model_name: str, override: str | None) -> str:
|
||||
"""Generate a SageMaker-legal endpoint name.
|
||||
|
||||
Default format: <model-name>-<YYYYMMDD-HHMM>. SageMaker names are limited
|
||||
to 63 chars and must be DNS-friendly (lowercase, hyphens, no underscores).
|
||||
"""
|
||||
if override:
|
||||
return override
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M")
|
||||
base = model_name.replace("_", "-").lower()
|
||||
return f"{base}-{stamp}"[:63]
|
||||
|
||||
|
||||
def build_tags(
|
||||
*,
|
||||
project: str,
|
||||
caller_arn: str,
|
||||
environment: str,
|
||||
model_s3_uri: str | None = None,
|
||||
extra: dict[str, str] | None = None,
|
||||
) -> list[dict[str, str]]:
|
||||
"""Build the shared tag set applied to every resource we create.
|
||||
|
||||
`extra` lets the caller add deployment-mode-specific tags (e.g.
|
||||
`{"InferenceMode": "async"}`) without forking the schema.
|
||||
"""
|
||||
owner = caller_arn.split("/")[-1] if "/" in caller_arn else caller_arn
|
||||
tags = [
|
||||
{"Key": "Project", "Value": project},
|
||||
{"Key": "Owner", "Value": owner},
|
||||
{"Key": "Environment", "Value": environment},
|
||||
{"Key": "CreatedBy", "Value": CREATED_BY_TAG_VALUE},
|
||||
]
|
||||
if model_s3_uri:
|
||||
tags.append({"Key": "ModelArtifact", "Value": model_s3_uri})
|
||||
if extra:
|
||||
for k, v in extra.items():
|
||||
tags.append({"Key": k, "Value": v})
|
||||
return tags
|
||||
|
||||
|
||||
def create_model(
|
||||
sm: Any,
|
||||
*,
|
||||
model_name: str,
|
||||
image_uri: str,
|
||||
role_arn: str,
|
||||
model_s3_uri: str | None,
|
||||
env: dict[str, str],
|
||||
tags: list[dict[str, str]],
|
||||
log_prefix: str = "deploy",
|
||||
) -> str:
|
||||
"""Create a SageMaker Model. Idempotent — reuses on AlreadyExists.
|
||||
|
||||
Returns the model name on success. The model definition itself is mode-
|
||||
independent: same call, same parameters, whether the eventual endpoint
|
||||
is real-time or async.
|
||||
"""
|
||||
log(log_prefix, f"Creating model: {model_name}")
|
||||
primary_container: dict[str, Any] = {"Image": image_uri}
|
||||
if env:
|
||||
primary_container["Environment"] = env
|
||||
if model_s3_uri:
|
||||
primary_container["ModelDataUrl"] = model_s3_uri
|
||||
|
||||
try:
|
||||
sm.create_model(
|
||||
ModelName=model_name,
|
||||
PrimaryContainer=primary_container,
|
||||
ExecutionRoleArn=role_arn,
|
||||
Tags=tags,
|
||||
)
|
||||
except ClientError as e:
|
||||
if "Cannot create already existing model" in str(e):
|
||||
log(log_prefix, f"Model {model_name} already exists — reusing")
|
||||
else:
|
||||
raise
|
||||
return model_name
|
||||
|
||||
|
||||
def wait_for_endpoint(
|
||||
sm: Any,
|
||||
endpoint_name: str,
|
||||
timeout_minutes: int = 30,
|
||||
log_prefix: str = "deploy",
|
||||
) -> None:
|
||||
"""Poll DescribeEndpoint until InService, or raise on Failed/timeout."""
|
||||
log(log_prefix, f"Waiting for {endpoint_name} to reach InService (up to {timeout_minutes} min)...")
|
||||
start = time.time()
|
||||
deadline = start + (timeout_minutes * 60)
|
||||
|
||||
while time.time() < deadline:
|
||||
resp = sm.describe_endpoint(EndpointName=endpoint_name)
|
||||
status = resp["EndpointStatus"]
|
||||
elapsed = int(time.time() - start)
|
||||
|
||||
if status == "InService":
|
||||
log(log_prefix, f"InService after {elapsed}s")
|
||||
return
|
||||
if status == "Failed":
|
||||
reason = resp.get("FailureReason", "(no reason given)")
|
||||
raise RuntimeError(f"Endpoint creation failed after {elapsed}s: {reason}")
|
||||
|
||||
log(log_prefix, f" status={status} elapsed={elapsed}s")
|
||||
time.sleep(30)
|
||||
|
||||
raise TimeoutError(f"Endpoint did not reach InService within {timeout_minutes} minutes")
|
||||
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env python
|
||||
"""Create a SageMaker real-time endpoint with production defaults.
|
||||
|
||||
Minimal usage:
|
||||
python deploy.py --model-name <name> --image-uri <uri> \
|
||||
--inference-ami-version <ami> --role-arn <arn> \
|
||||
--instance-type ml.g5.xlarge --region <region>
|
||||
|
||||
See the SKILL.md for full flags and the recommended chained-with-resolver pattern.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import boto3
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
from _common import (
|
||||
build_tags,
|
||||
create_model,
|
||||
log as _log,
|
||||
make_endpoint_name,
|
||||
parse_env,
|
||||
wait_for_endpoint,
|
||||
)
|
||||
|
||||
|
||||
# Defaults — change here, not at call sites.
|
||||
# Reasoning lives in references/deployment-template.md.
|
||||
DEFAULTS = {
|
||||
"initial_instance_count": 1,
|
||||
"min_capacity": 1,
|
||||
"max_capacity": 4,
|
||||
"target_invocations_per_instance": 20,
|
||||
"scale_in_cooldown_seconds": 300,
|
||||
"scale_out_cooldown_seconds": 60,
|
||||
"data_capture_sampling_percent": 100,
|
||||
"alarm_latency_threshold_ms": 30_000,
|
||||
"alarm_5xx_threshold_count": 5,
|
||||
"alarm_overhead_threshold_ms": 2_000,
|
||||
"alarm_evaluation_periods": 1,
|
||||
"alarm_period_seconds": 300,
|
||||
"environment_tag": "dev",
|
||||
}
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
_log("deploy", msg)
|
||||
|
||||
|
||||
def create_endpoint_config(
|
||||
sm: Any, *, config_name: str, model_name: str, instance_type: str,
|
||||
initial_instance_count: int, inference_ami_version: str | None,
|
||||
data_capture_enabled: bool, data_capture_s3_uri: str | None,
|
||||
tags: list[dict],
|
||||
) -> str:
|
||||
log(f"Creating endpoint config: {config_name}")
|
||||
|
||||
production_variant: dict[str, Any] = {
|
||||
"VariantName": "AllTraffic",
|
||||
"ModelName": model_name,
|
||||
"InstanceType": instance_type,
|
||||
"InitialInstanceCount": initial_instance_count,
|
||||
"InitialVariantWeight": 1.0,
|
||||
}
|
||||
|
||||
# InferenceAmiVersion required for vLLM DLC with CUDA 13+. Without it the
|
||||
# container dies on startup with no logs. See hf-cloud-serving-image-selection skill.
|
||||
if inference_ami_version:
|
||||
production_variant["InferenceAmiVersion"] = inference_ami_version
|
||||
log(f" InferenceAmiVersion set to: {inference_ami_version}")
|
||||
|
||||
kwargs: dict[str, Any] = {
|
||||
"EndpointConfigName": config_name,
|
||||
"ProductionVariants": [production_variant],
|
||||
"Tags": tags,
|
||||
}
|
||||
|
||||
if data_capture_enabled:
|
||||
if not data_capture_s3_uri:
|
||||
raise ValueError("data_capture_enabled but no data_capture_s3_uri provided")
|
||||
kwargs["DataCaptureConfig"] = {
|
||||
"EnableCapture": True,
|
||||
"InitialSamplingPercentage": DEFAULTS["data_capture_sampling_percent"],
|
||||
"DestinationS3Uri": data_capture_s3_uri,
|
||||
"CaptureOptions": [{"CaptureMode": "Input"}, {"CaptureMode": "Output"}],
|
||||
"CaptureContentTypeHeader": {"JsonContentTypes": ["application/json"]},
|
||||
}
|
||||
|
||||
try:
|
||||
sm.create_endpoint_config(**kwargs)
|
||||
except ClientError as e:
|
||||
if "Cannot create already existing endpoint configuration" in str(e):
|
||||
log(f"Endpoint config {config_name} already exists — reusing")
|
||||
else:
|
||||
raise
|
||||
return config_name
|
||||
|
||||
|
||||
def create_endpoint(sm: Any, *, endpoint_name: str, config_name: str, tags: list[dict]) -> None:
|
||||
log(f"Creating endpoint: {endpoint_name}")
|
||||
sm.create_endpoint(
|
||||
EndpointName=endpoint_name,
|
||||
EndpointConfigName=config_name,
|
||||
Tags=tags,
|
||||
)
|
||||
|
||||
|
||||
def register_autoscaling(
|
||||
*, endpoint_name: str, variant_name: str, min_capacity: int, max_capacity: int,
|
||||
target_invocations: int, scale_in_cooldown: int, scale_out_cooldown: int, region: str,
|
||||
) -> None:
|
||||
log(f"Registering autoscaling: min={min_capacity} max={max_capacity} target={target_invocations}/min")
|
||||
appscaling = boto3.client("application-autoscaling", region_name=region)
|
||||
resource_id = f"endpoint/{endpoint_name}/variant/{variant_name}"
|
||||
|
||||
appscaling.register_scalable_target(
|
||||
ServiceNamespace="sagemaker",
|
||||
ResourceId=resource_id,
|
||||
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
|
||||
MinCapacity=min_capacity,
|
||||
MaxCapacity=max_capacity,
|
||||
)
|
||||
appscaling.put_scaling_policy(
|
||||
PolicyName=f"{endpoint_name}-target-tracking",
|
||||
ServiceNamespace="sagemaker",
|
||||
ResourceId=resource_id,
|
||||
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
|
||||
PolicyType="TargetTrackingScaling",
|
||||
TargetTrackingScalingPolicyConfiguration={
|
||||
"TargetValue": float(target_invocations),
|
||||
"PredefinedMetricSpecification": {
|
||||
"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance",
|
||||
},
|
||||
"ScaleInCooldown": scale_in_cooldown,
|
||||
"ScaleOutCooldown": scale_out_cooldown,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def create_alarms(*, endpoint_name: str, variant_name: str, sns_topic_arn: str | None, region: str) -> None:
|
||||
log(f"Creating CloudWatch alarms for {endpoint_name}")
|
||||
cw = boto3.client("cloudwatch", region_name=region)
|
||||
actions = [sns_topic_arn] if sns_topic_arn else []
|
||||
common_dims = [
|
||||
{"Name": "EndpointName", "Value": endpoint_name},
|
||||
{"Name": "VariantName", "Value": variant_name},
|
||||
]
|
||||
|
||||
alarms = [
|
||||
{
|
||||
"AlarmName": f"{endpoint_name}-ModelLatencyP99",
|
||||
"MetricName": "ModelLatency",
|
||||
"ExtendedStatistic": "p99",
|
||||
"Threshold": DEFAULTS["alarm_latency_threshold_ms"] * 1000, # microseconds
|
||||
"ComparisonOperator": "GreaterThanThreshold",
|
||||
"AlarmDescription": "Model inference latency p99 > 30s",
|
||||
},
|
||||
{
|
||||
"AlarmName": f"{endpoint_name}-Invocation5XXErrors",
|
||||
"MetricName": "Invocation5XXErrors",
|
||||
"Statistic": "Sum",
|
||||
"Threshold": DEFAULTS["alarm_5xx_threshold_count"],
|
||||
"ComparisonOperator": "GreaterThanThreshold",
|
||||
"AlarmDescription": "5XX errors > 5 in 5min",
|
||||
},
|
||||
{
|
||||
"AlarmName": f"{endpoint_name}-OverheadLatencyP99",
|
||||
"MetricName": "OverheadLatency",
|
||||
"ExtendedStatistic": "p99",
|
||||
"Threshold": DEFAULTS["alarm_overhead_threshold_ms"] * 1000,
|
||||
"ComparisonOperator": "GreaterThanThreshold",
|
||||
"AlarmDescription": "Platform overhead latency p99 > 2s",
|
||||
},
|
||||
]
|
||||
|
||||
for spec in alarms:
|
||||
params = {
|
||||
"AlarmName": spec["AlarmName"],
|
||||
"AlarmDescription": spec["AlarmDescription"],
|
||||
"MetricName": spec["MetricName"],
|
||||
"Namespace": "AWS/SageMaker",
|
||||
"Dimensions": common_dims,
|
||||
"Period": DEFAULTS["alarm_period_seconds"],
|
||||
"EvaluationPeriods": DEFAULTS["alarm_evaluation_periods"],
|
||||
"Threshold": spec["Threshold"],
|
||||
"ComparisonOperator": spec["ComparisonOperator"],
|
||||
"TreatMissingData": "notBreaching",
|
||||
"AlarmActions": actions,
|
||||
}
|
||||
if "Statistic" in spec:
|
||||
params["Statistic"] = spec["Statistic"]
|
||||
if "ExtendedStatistic" in spec:
|
||||
params["ExtendedStatistic"] = spec["ExtendedStatistic"]
|
||||
cw.put_metric_alarm(**params)
|
||||
|
||||
if not sns_topic_arn:
|
||||
log("WARNING: no --sns-alarm-topic — alarms exist but won't notify anyone.")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
|
||||
# Required
|
||||
p.add_argument("--model-name", required=True)
|
||||
p.add_argument("--image-uri", required=True, help="From hf-cloud-serving-image-selection")
|
||||
p.add_argument("--role-arn", required=True, help="From hf-cloud-sagemaker-iam-preflight")
|
||||
p.add_argument("--instance-type", required=True, help="e.g. ml.g5.xlarge")
|
||||
p.add_argument("--region", required=True, help="From hf-cloud-aws-context-discovery")
|
||||
|
||||
# Conditional
|
||||
p.add_argument("--model-s3-uri", default=None, help="Omit when loading from HF Hub")
|
||||
p.add_argument("--env", action="append", default=[], help="KEY=VALUE; repeatable")
|
||||
p.add_argument(
|
||||
"--inference-ami-version", default=None,
|
||||
help=(
|
||||
"REQUIRED for vLLM DLC with CUDA 13+ (e.g. al2-ami-sagemaker-inference-gpu-3-1). "
|
||||
"Without this, container dies on startup with no logs. "
|
||||
"See hf-cloud-serving-image-selection's 'vLLM AMI requirement' table to map a tag to the AMI version."
|
||||
),
|
||||
)
|
||||
|
||||
# Naming
|
||||
p.add_argument("--endpoint-name", default=None, help="Default: <model-name>-<timestamp>")
|
||||
p.add_argument("--project", default=None, help="Tag value (default: model name)")
|
||||
p.add_argument("--environment", default=DEFAULTS["environment_tag"])
|
||||
|
||||
# Capacity / scaling
|
||||
p.add_argument("--initial-instance-count", type=int, default=DEFAULTS["initial_instance_count"])
|
||||
p.add_argument("--min-capacity", type=int, default=DEFAULTS["min_capacity"])
|
||||
p.add_argument("--max-capacity", type=int, default=DEFAULTS["max_capacity"])
|
||||
p.add_argument("--target-invocations-per-instance", type=int, default=DEFAULTS["target_invocations_per_instance"])
|
||||
p.add_argument("--no-autoscaling", action="store_true", help="NOT RECOMMENDED")
|
||||
|
||||
# Data capture (off by default)
|
||||
p.add_argument("--enable-data-capture", action="store_true", help="Log requests/responses to S3")
|
||||
p.add_argument("--data-capture-s3-uri", default=None)
|
||||
|
||||
# Alarms
|
||||
p.add_argument("--sns-alarm-topic", default=None, help="SNS topic ARN for alarm notifications")
|
||||
p.add_argument("--no-alarms", action="store_true")
|
||||
|
||||
args = p.parse_args()
|
||||
|
||||
env_dict = parse_env(args.env)
|
||||
endpoint_name = make_endpoint_name(args.model_name, args.endpoint_name)
|
||||
config_name = f"{endpoint_name}-config"
|
||||
|
||||
sts = boto3.client("sts", region_name=args.region)
|
||||
sm = boto3.client("sagemaker", region_name=args.region)
|
||||
caller_arn = sts.get_caller_identity()["Arn"]
|
||||
account_id = sts.get_caller_identity()["Account"]
|
||||
|
||||
if args.enable_data_capture and not args.data_capture_s3_uri:
|
||||
args.data_capture_s3_uri = f"s3://sagemaker-{args.region}-{account_id}/{endpoint_name}/data-capture/"
|
||||
log(f"Data capture URI defaulted to: {args.data_capture_s3_uri}")
|
||||
|
||||
tags = build_tags(
|
||||
project=args.project or args.model_name,
|
||||
caller_arn=caller_arn,
|
||||
environment=args.environment,
|
||||
model_s3_uri=args.model_s3_uri,
|
||||
)
|
||||
|
||||
create_model(
|
||||
sm, model_name=args.model_name, image_uri=args.image_uri,
|
||||
role_arn=args.role_arn, model_s3_uri=args.model_s3_uri,
|
||||
env=env_dict, tags=tags, log_prefix="deploy",
|
||||
)
|
||||
create_endpoint_config(
|
||||
sm, config_name=config_name, model_name=args.model_name,
|
||||
instance_type=args.instance_type, initial_instance_count=args.initial_instance_count,
|
||||
inference_ami_version=args.inference_ami_version,
|
||||
data_capture_enabled=args.enable_data_capture,
|
||||
data_capture_s3_uri=args.data_capture_s3_uri, tags=tags,
|
||||
)
|
||||
create_endpoint(sm, endpoint_name=endpoint_name, config_name=config_name, tags=tags)
|
||||
wait_for_endpoint(sm, endpoint_name, log_prefix="deploy")
|
||||
|
||||
if not args.no_autoscaling:
|
||||
register_autoscaling(
|
||||
endpoint_name=endpoint_name, variant_name="AllTraffic",
|
||||
min_capacity=args.min_capacity, max_capacity=args.max_capacity,
|
||||
target_invocations=args.target_invocations_per_instance,
|
||||
scale_in_cooldown=DEFAULTS["scale_in_cooldown_seconds"],
|
||||
scale_out_cooldown=DEFAULTS["scale_out_cooldown_seconds"],
|
||||
region=args.region,
|
||||
)
|
||||
else:
|
||||
log("WARNING: autoscaling skipped. Endpoint won't scale with traffic.")
|
||||
|
||||
if not args.no_alarms:
|
||||
create_alarms(
|
||||
endpoint_name=endpoint_name, variant_name="AllTraffic",
|
||||
sns_topic_arn=args.sns_alarm_topic, region=args.region,
|
||||
)
|
||||
|
||||
# Summary
|
||||
log("")
|
||||
log(f"Deployment complete: {endpoint_name}")
|
||||
log(f" Instance: {args.instance_type}")
|
||||
log(f" Autoscaling: {'OFF' if args.no_autoscaling else f'{args.min_capacity}-{args.max_capacity} instances'}")
|
||||
log(f" Data capture: {args.data_capture_s3_uri if args.enable_data_capture else 'OFF (pass --enable-data-capture)'}")
|
||||
log("")
|
||||
log(f"Test: python3 invoke_endpoint.py --endpoint-name {endpoint_name} \\")
|
||||
log(f" --payload '{{\"prompt\": \"hello\"}}' --region {args.region}")
|
||||
log(" (BOM-safe + cross-platform; use 'python' on Windows)")
|
||||
log(f"Teardown: python3 teardown.py {endpoint_name} {args.region}")
|
||||
|
||||
# Machine-readable summary for downstream scripting
|
||||
print(json.dumps({
|
||||
"endpoint_name": endpoint_name,
|
||||
"endpoint_config_name": config_name,
|
||||
"model_name": args.model_name,
|
||||
"region": args.region,
|
||||
"instance_type": args.instance_type,
|
||||
}))
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,433 @@
|
||||
#!/usr/bin/env python
|
||||
"""Create a SageMaker async-inference endpoint with production defaults.
|
||||
|
||||
Minimal usage:
|
||||
python deploy_async.py --model-name <name> --image-uri <uri> \
|
||||
--role-arn <arn> --instance-type ml.g5.xlarge --region <region> \
|
||||
--output-s3-uri s3://bucket/path/
|
||||
|
||||
Async differs from real-time in three structural ways:
|
||||
1. AsyncInferenceConfig on the endpoint config (S3 output path, optional
|
||||
SNS notification topics, optional failure path)
|
||||
2. Autoscaling MinCapacity=0 is supported — endpoints can genuinely scale
|
||||
to zero between batches. Target-tracking on ApproximateBacklogSizePerInstance.
|
||||
3. Scale-from-zero (0→1) requires a SEPARATE step-scaling policy bound to
|
||||
a HasBacklogWithoutCapacity alarm. Target-tracking alone can't transition
|
||||
from zero, so without the step policy the endpoint won't wake up.
|
||||
|
||||
See SKILL.md for the recommended chained-with-resolver pattern.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
import boto3
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
from _common import (
|
||||
build_tags,
|
||||
create_model,
|
||||
log as _log,
|
||||
make_endpoint_name,
|
||||
parse_env,
|
||||
wait_for_endpoint,
|
||||
)
|
||||
|
||||
|
||||
# Defaults — change here, not at call sites.
|
||||
DEFAULTS = {
|
||||
"initial_instance_count": 1,
|
||||
"min_capacity": 0, # async genuinely supports scale-to-zero
|
||||
"max_capacity": 4,
|
||||
"backlog_per_instance_target": 5, # target queue depth per instance
|
||||
"scale_in_cooldown_seconds": 600, # slower scale-in for async — batches are bursty
|
||||
"scale_out_cooldown_seconds": 60,
|
||||
"wake_from_zero_step_size": 1, # 0→1 instance when backlog appears
|
||||
"max_concurrent_invocations_per_instance": 4,
|
||||
# Alarms
|
||||
"alarm_backlog_size_threshold": 50, # queue too deep
|
||||
"alarm_failed_invocations_threshold": 5, # repeated failures
|
||||
"alarm_evaluation_periods": 1,
|
||||
"alarm_period_seconds": 60, # async needs faster eval for wake-from-zero
|
||||
"environment_tag": "dev",
|
||||
}
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
_log("deploy_async", msg)
|
||||
|
||||
|
||||
def create_async_endpoint_config(
|
||||
sm: Any, *, config_name: str, model_name: str, instance_type: str,
|
||||
initial_instance_count: int, inference_ami_version: str | None,
|
||||
output_s3_uri: str, failure_s3_uri: str | None,
|
||||
success_topic_arn: str | None, error_topic_arn: str | None,
|
||||
max_concurrent_invocations: int,
|
||||
tags: list[dict],
|
||||
) -> str:
|
||||
log(f"Creating async endpoint config: {config_name}")
|
||||
|
||||
production_variant: dict[str, Any] = {
|
||||
"VariantName": "AllTraffic",
|
||||
"ModelName": model_name,
|
||||
"InstanceType": instance_type,
|
||||
"InitialInstanceCount": initial_instance_count,
|
||||
"InitialVariantWeight": 1.0,
|
||||
}
|
||||
if inference_ami_version:
|
||||
production_variant["InferenceAmiVersion"] = inference_ami_version
|
||||
log(f" InferenceAmiVersion set to: {inference_ami_version}")
|
||||
|
||||
output_config: dict[str, Any] = {"S3OutputPath": output_s3_uri}
|
||||
if failure_s3_uri:
|
||||
output_config["S3FailurePath"] = failure_s3_uri
|
||||
if success_topic_arn or error_topic_arn:
|
||||
notification: dict[str, str] = {}
|
||||
if success_topic_arn:
|
||||
notification["SuccessTopic"] = success_topic_arn
|
||||
if error_topic_arn:
|
||||
notification["ErrorTopic"] = error_topic_arn
|
||||
output_config["NotificationConfig"] = notification
|
||||
|
||||
async_config: dict[str, Any] = {
|
||||
"OutputConfig": output_config,
|
||||
"ClientConfig": {
|
||||
"MaxConcurrentInvocationsPerInstance": max_concurrent_invocations,
|
||||
},
|
||||
}
|
||||
|
||||
kwargs: dict[str, Any] = {
|
||||
"EndpointConfigName": config_name,
|
||||
"ProductionVariants": [production_variant],
|
||||
"AsyncInferenceConfig": async_config,
|
||||
"Tags": tags,
|
||||
}
|
||||
|
||||
try:
|
||||
sm.create_endpoint_config(**kwargs)
|
||||
except ClientError as e:
|
||||
if "Cannot create already existing endpoint configuration" in str(e):
|
||||
log(f"Endpoint config {config_name} already exists — reusing")
|
||||
else:
|
||||
raise
|
||||
return config_name
|
||||
|
||||
|
||||
def create_endpoint(sm: Any, *, endpoint_name: str, config_name: str, tags: list[dict]) -> None:
|
||||
log(f"Creating endpoint: {endpoint_name}")
|
||||
sm.create_endpoint(
|
||||
EndpointName=endpoint_name,
|
||||
EndpointConfigName=config_name,
|
||||
Tags=tags,
|
||||
)
|
||||
|
||||
|
||||
def register_async_autoscaling(
|
||||
*, endpoint_name: str, variant_name: str, min_capacity: int, max_capacity: int,
|
||||
backlog_per_instance_target: int, scale_in_cooldown: int, scale_out_cooldown: int,
|
||||
wake_step_size: int, region: str,
|
||||
) -> None:
|
||||
"""Register two autoscaling policies on the variant:
|
||||
|
||||
1. Target-tracking on ApproximateBacklogSizePerInstance — handles
|
||||
ongoing scaling between MinCapacity and MaxCapacity.
|
||||
2. Step-scaling that increments capacity by `wake_step_size` when
|
||||
triggered — needed for 0→1 wake-from-zero, because target-tracking
|
||||
can't transition from zero capacity.
|
||||
|
||||
The step-scaling policy is invoked by a HasBacklogWithoutCapacity alarm
|
||||
(created separately in create_async_alarms).
|
||||
"""
|
||||
log(f"Registering async autoscaling: min={min_capacity} max={max_capacity} backlog-target={backlog_per_instance_target}")
|
||||
appscaling = boto3.client("application-autoscaling", region_name=region)
|
||||
resource_id = f"endpoint/{endpoint_name}/variant/{variant_name}"
|
||||
|
||||
appscaling.register_scalable_target(
|
||||
ServiceNamespace="sagemaker",
|
||||
ResourceId=resource_id,
|
||||
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
|
||||
MinCapacity=min_capacity,
|
||||
MaxCapacity=max_capacity,
|
||||
)
|
||||
|
||||
# (1) Target-tracking for ongoing scaling between min and max
|
||||
appscaling.put_scaling_policy(
|
||||
PolicyName=f"{endpoint_name}-backlog-target-tracking",
|
||||
ServiceNamespace="sagemaker",
|
||||
ResourceId=resource_id,
|
||||
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
|
||||
PolicyType="TargetTrackingScaling",
|
||||
TargetTrackingScalingPolicyConfiguration={
|
||||
"TargetValue": float(backlog_per_instance_target),
|
||||
"CustomizedMetricSpecification": {
|
||||
"MetricName": "ApproximateBacklogSizePerInstance",
|
||||
"Namespace": "AWS/SageMaker",
|
||||
"Dimensions": [{"Name": "EndpointName", "Value": endpoint_name}],
|
||||
"Statistic": "Average",
|
||||
},
|
||||
"ScaleInCooldown": scale_in_cooldown,
|
||||
"ScaleOutCooldown": scale_out_cooldown,
|
||||
},
|
||||
)
|
||||
|
||||
# (2) Step-scaling for wake-from-zero. The CloudWatch alarm bound to
|
||||
# this policy is created in create_async_alarms.
|
||||
appscaling.put_scaling_policy(
|
||||
PolicyName=f"{endpoint_name}-step-wake-from-zero",
|
||||
ServiceNamespace="sagemaker",
|
||||
ResourceId=resource_id,
|
||||
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
|
||||
PolicyType="StepScaling",
|
||||
StepScalingPolicyConfiguration={
|
||||
"AdjustmentType": "ChangeInCapacity",
|
||||
"MetricAggregationType": "Maximum",
|
||||
"Cooldown": scale_out_cooldown,
|
||||
"StepAdjustments": [
|
||||
{
|
||||
"MetricIntervalLowerBound": 0,
|
||||
"ScalingAdjustment": wake_step_size,
|
||||
},
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def create_async_alarms(
|
||||
*, endpoint_name: str, variant_name: str, sns_topic_arn: str | None,
|
||||
region: str, wake_alarm_arns_for_step_policy: list[str],
|
||||
) -> None:
|
||||
"""Create CloudWatch alarms for the async endpoint, including the
|
||||
HasBacklogWithoutCapacity alarm that drives the wake-from-zero policy.
|
||||
"""
|
||||
log(f"Creating CloudWatch alarms for {endpoint_name}")
|
||||
cw = boto3.client("cloudwatch", region_name=region)
|
||||
actions = [sns_topic_arn] if sns_topic_arn else []
|
||||
endpoint_dim = [{"Name": "EndpointName", "Value": endpoint_name}]
|
||||
|
||||
# 1. Backlog too deep — capacity is keeping up poorly with incoming requests
|
||||
cw.put_metric_alarm(
|
||||
AlarmName=f"{endpoint_name}-ApproximateBacklogSize",
|
||||
AlarmDescription="Async queue backlog too deep",
|
||||
MetricName="ApproximateBacklogSize",
|
||||
Namespace="AWS/SageMaker",
|
||||
Dimensions=endpoint_dim,
|
||||
Statistic="Average",
|
||||
Period=DEFAULTS["alarm_period_seconds"],
|
||||
EvaluationPeriods=DEFAULTS["alarm_evaluation_periods"],
|
||||
Threshold=DEFAULTS["alarm_backlog_size_threshold"],
|
||||
ComparisonOperator="GreaterThanThreshold",
|
||||
TreatMissingData="notBreaching",
|
||||
AlarmActions=actions,
|
||||
)
|
||||
|
||||
# 2. InvocationsFailed — repeated client/server errors in async processing
|
||||
cw.put_metric_alarm(
|
||||
AlarmName=f"{endpoint_name}-InvocationsFailed",
|
||||
AlarmDescription="Async invocations failing repeatedly",
|
||||
MetricName="InvocationsFailed",
|
||||
Namespace="AWS/SageMaker",
|
||||
Dimensions=endpoint_dim,
|
||||
Statistic="Sum",
|
||||
Period=DEFAULTS["alarm_period_seconds"],
|
||||
EvaluationPeriods=DEFAULTS["alarm_evaluation_periods"],
|
||||
Threshold=DEFAULTS["alarm_failed_invocations_threshold"],
|
||||
ComparisonOperator="GreaterThanThreshold",
|
||||
TreatMissingData="notBreaching",
|
||||
AlarmActions=actions,
|
||||
)
|
||||
|
||||
# 3. HasBacklogWithoutCapacity — drives the step-scaling wake-from-zero
|
||||
# policy. The Alarm action is the step policy ARN, not the SNS topic.
|
||||
cw.put_metric_alarm(
|
||||
AlarmName=f"{endpoint_name}-HasBacklogWithoutCapacity",
|
||||
AlarmDescription="Async backlog exists but no instances running — triggers wake-from-zero",
|
||||
MetricName="HasBacklogWithoutCapacity",
|
||||
Namespace="AWS/SageMaker",
|
||||
Dimensions=endpoint_dim,
|
||||
Statistic="Average",
|
||||
Period=60, # tight evaluation: we want fast wake-up
|
||||
EvaluationPeriods=1,
|
||||
Threshold=1,
|
||||
ComparisonOperator="GreaterThanOrEqualToThreshold",
|
||||
TreatMissingData="notBreaching",
|
||||
AlarmActions=wake_alarm_arns_for_step_policy, # step policy ARNs
|
||||
)
|
||||
|
||||
if not sns_topic_arn:
|
||||
log("WARNING: no --sns-alarm-topic — backlog/failure alarms exist but won't notify anyone.")
|
||||
|
||||
|
||||
def get_step_policy_arn(*, endpoint_name: str, variant_name: str, region: str) -> str:
|
||||
"""Look up the ARN of the step-scaling policy we just created.
|
||||
|
||||
`put_scaling_policy` returns the ARN on creation but we don't capture it
|
||||
above to keep the function signature small. Re-fetch via describe.
|
||||
"""
|
||||
appscaling = boto3.client("application-autoscaling", region_name=region)
|
||||
resp = appscaling.describe_scaling_policies(
|
||||
ServiceNamespace="sagemaker",
|
||||
ResourceId=f"endpoint/{endpoint_name}/variant/{variant_name}",
|
||||
PolicyNames=[f"{endpoint_name}-step-wake-from-zero"],
|
||||
)
|
||||
policies = resp.get("ScalingPolicies", [])
|
||||
if not policies:
|
||||
raise RuntimeError(f"Step-scaling policy not found for {endpoint_name} — was register_async_autoscaling called?")
|
||||
return policies[0]["PolicyARN"]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
|
||||
# Required
|
||||
p.add_argument("--model-name", required=True)
|
||||
p.add_argument("--image-uri", required=True, help="From hf-cloud-serving-image-selection")
|
||||
p.add_argument("--role-arn", required=True, help="From hf-cloud-sagemaker-iam-preflight")
|
||||
p.add_argument("--instance-type", required=True, help="e.g. ml.g5.xlarge")
|
||||
p.add_argument("--region", required=True, help="From hf-cloud-aws-context-discovery")
|
||||
p.add_argument("--output-s3-uri", required=True,
|
||||
help="S3 path where async results are written. e.g. s3://my-bucket/async-output/")
|
||||
|
||||
# Conditional
|
||||
p.add_argument("--model-s3-uri", default=None, help="Omit when loading from HF Hub")
|
||||
p.add_argument("--env", action="append", default=[], help="KEY=VALUE; repeatable")
|
||||
p.add_argument("--inference-ami-version", default=None,
|
||||
help="REQUIRED for vLLM DLC with CUDA 13+ (e.g. al2-ami-sagemaker-inference-gpu-3-1)")
|
||||
p.add_argument("--failure-s3-uri", default=None,
|
||||
help="S3 path for failed async invocations (optional; default: errors written to OutputConfig.S3OutputPath)")
|
||||
p.add_argument("--success-sns-topic", default=None,
|
||||
help="SNS topic ARN notified when async invocation succeeds")
|
||||
p.add_argument("--error-sns-topic", default=None,
|
||||
help="SNS topic ARN notified when async invocation fails")
|
||||
|
||||
# Naming
|
||||
p.add_argument("--endpoint-name", default=None, help="Default: <model-name>-<timestamp>")
|
||||
p.add_argument("--project", default=None, help="Tag value (default: model name)")
|
||||
p.add_argument("--environment", default=DEFAULTS["environment_tag"])
|
||||
|
||||
# Capacity / scaling
|
||||
p.add_argument("--initial-instance-count", type=int, default=DEFAULTS["initial_instance_count"])
|
||||
p.add_argument("--min-capacity", type=int, default=DEFAULTS["min_capacity"],
|
||||
help="Async supports 0 (scale-to-zero between batches). Default 0.")
|
||||
p.add_argument("--max-capacity", type=int, default=DEFAULTS["max_capacity"])
|
||||
p.add_argument("--backlog-per-instance-target", type=int,
|
||||
default=DEFAULTS["backlog_per_instance_target"],
|
||||
help="Target queue depth per instance for autoscaling")
|
||||
p.add_argument("--max-concurrent-invocations-per-instance", type=int,
|
||||
default=DEFAULTS["max_concurrent_invocations_per_instance"])
|
||||
p.add_argument("--no-autoscaling", action="store_true",
|
||||
help="NOT RECOMMENDED for async — endpoint won't scale to zero")
|
||||
|
||||
# Alarms
|
||||
p.add_argument("--sns-alarm-topic", default=None, help="SNS topic ARN for backlog/failure alarms")
|
||||
p.add_argument("--no-alarms", action="store_true")
|
||||
|
||||
args = p.parse_args()
|
||||
|
||||
env_dict = parse_env(args.env)
|
||||
endpoint_name = make_endpoint_name(args.model_name, args.endpoint_name)
|
||||
config_name = f"{endpoint_name}-config"
|
||||
|
||||
sts = boto3.client("sts", region_name=args.region)
|
||||
sm = boto3.client("sagemaker", region_name=args.region)
|
||||
caller_arn = sts.get_caller_identity()["Arn"]
|
||||
|
||||
if args.min_capacity == 0 and args.no_autoscaling:
|
||||
raise SystemExit(
|
||||
"--min-capacity 0 requires autoscaling to be enabled (the wake-from-zero "
|
||||
"policy is what brings instances up). Either remove --no-autoscaling or "
|
||||
"set --min-capacity 1."
|
||||
)
|
||||
|
||||
tags = build_tags(
|
||||
project=args.project or args.model_name,
|
||||
caller_arn=caller_arn,
|
||||
environment=args.environment,
|
||||
model_s3_uri=args.model_s3_uri,
|
||||
extra={"InferenceMode": "async"},
|
||||
)
|
||||
|
||||
create_model(
|
||||
sm, model_name=args.model_name, image_uri=args.image_uri,
|
||||
role_arn=args.role_arn, model_s3_uri=args.model_s3_uri,
|
||||
env=env_dict, tags=tags, log_prefix="deploy_async",
|
||||
)
|
||||
create_async_endpoint_config(
|
||||
sm, config_name=config_name, model_name=args.model_name,
|
||||
instance_type=args.instance_type,
|
||||
initial_instance_count=args.initial_instance_count,
|
||||
inference_ami_version=args.inference_ami_version,
|
||||
output_s3_uri=args.output_s3_uri,
|
||||
failure_s3_uri=args.failure_s3_uri,
|
||||
success_topic_arn=args.success_sns_topic,
|
||||
error_topic_arn=args.error_sns_topic,
|
||||
max_concurrent_invocations=args.max_concurrent_invocations_per_instance,
|
||||
tags=tags,
|
||||
)
|
||||
create_endpoint(sm, endpoint_name=endpoint_name, config_name=config_name, tags=tags)
|
||||
wait_for_endpoint(sm, endpoint_name, log_prefix="deploy_async")
|
||||
|
||||
if not args.no_autoscaling:
|
||||
register_async_autoscaling(
|
||||
endpoint_name=endpoint_name, variant_name="AllTraffic",
|
||||
min_capacity=args.min_capacity, max_capacity=args.max_capacity,
|
||||
backlog_per_instance_target=args.backlog_per_instance_target,
|
||||
scale_in_cooldown=DEFAULTS["scale_in_cooldown_seconds"],
|
||||
scale_out_cooldown=DEFAULTS["scale_out_cooldown_seconds"],
|
||||
wake_step_size=DEFAULTS["wake_from_zero_step_size"],
|
||||
region=args.region,
|
||||
)
|
||||
# The wake-from-zero alarm needs the step policy's ARN as its action.
|
||||
step_policy_arn = get_step_policy_arn(
|
||||
endpoint_name=endpoint_name, variant_name="AllTraffic", region=args.region,
|
||||
)
|
||||
else:
|
||||
log("WARNING: autoscaling skipped. Endpoint will NOT scale (in either direction).")
|
||||
step_policy_arn = None
|
||||
|
||||
if not args.no_alarms:
|
||||
create_async_alarms(
|
||||
endpoint_name=endpoint_name, variant_name="AllTraffic",
|
||||
sns_topic_arn=args.sns_alarm_topic, region=args.region,
|
||||
wake_alarm_arns_for_step_policy=[step_policy_arn] if step_policy_arn else [],
|
||||
)
|
||||
|
||||
# Summary
|
||||
log("")
|
||||
log(f"Async deployment complete: {endpoint_name}")
|
||||
log(f" Instance: {args.instance_type}")
|
||||
if args.no_autoscaling:
|
||||
autoscaling_summary = "OFF"
|
||||
else:
|
||||
zero_label = "enabled" if args.min_capacity == 0 else "disabled"
|
||||
autoscaling_summary = f"{args.min_capacity}-{args.max_capacity} instances (scale-to-zero {zero_label})"
|
||||
log(f" Autoscaling: {autoscaling_summary}")
|
||||
log(f" Output S3 path: {args.output_s3_uri}")
|
||||
log(f" Notifications: success={args.success_sns_topic or 'none'} error={args.error_sns_topic or 'none'}")
|
||||
log("")
|
||||
log("Invoke (async, via S3 input location):")
|
||||
log(f" aws sagemaker-runtime invoke-endpoint-async \\")
|
||||
log(f" --endpoint-name {endpoint_name} \\")
|
||||
log(f" --input-location s3://YOUR-INPUT-BUCKET/path/to/input.json \\")
|
||||
log(f" --content-type application/json --region {args.region}")
|
||||
log(f" # Result will land at: {args.output_s3_uri}")
|
||||
log(" # Write input.json as BOM-free UTF-8 (on Windows, NOT 'Set-Content -Encoding UTF8')")
|
||||
log("")
|
||||
log(f"Teardown: python3 teardown.py {endpoint_name} {args.region}")
|
||||
|
||||
print(json.dumps({
|
||||
"endpoint_name": endpoint_name,
|
||||
"endpoint_config_name": config_name,
|
||||
"model_name": args.model_name,
|
||||
"region": args.region,
|
||||
"instance_type": args.instance_type,
|
||||
"inference_mode": "async",
|
||||
"output_s3_uri": args.output_s3_uri,
|
||||
}))
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,134 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Invoke a real-time SageMaker endpoint with a JSON payload. Cross-platform.
|
||||
|
||||
This is the BOM-safe way to test an endpoint. On Windows, writing a payload with
|
||||
PowerShell's `Set-Content -Encoding UTF8` can prepend a UTF-8 byte-order mark
|
||||
(BOM). SageMaker's JSON parser rejects it with:
|
||||
|
||||
Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
|
||||
|
||||
This helper never produces that error: it reads the payload with `utf-8-sig`
|
||||
(stripping any BOM the source file may already have) and writes the request body
|
||||
as plain BOM-free UTF-8 before calling `aws sagemaker-runtime invoke-endpoint`.
|
||||
|
||||
Usage:
|
||||
python invoke_endpoint.py --endpoint-name NAME --payload '{"inputs": "hi"}'
|
||||
python invoke_endpoint.py --endpoint-name NAME --payload-file body.json
|
||||
python invoke_endpoint.py --endpoint-name NAME --payload-file img.json \\
|
||||
--content-type application/json --region eu-west-1
|
||||
|
||||
The endpoint response body is printed to stdout.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(f"[invoke] {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def aws_bin() -> str:
|
||||
exe = shutil.which("aws")
|
||||
if not exe:
|
||||
log("ERROR: the 'aws' CLI was not found on PATH. Install AWS CLI v2.")
|
||||
sys.exit(2)
|
||||
return exe
|
||||
|
||||
|
||||
def resolve_region(arg_region: str | None) -> str:
|
||||
if arg_region:
|
||||
return arg_region
|
||||
for var in ("AWS_REGION", "AWS_DEFAULT_REGION"):
|
||||
if os.environ.get(var):
|
||||
return os.environ[var]
|
||||
proc = subprocess.run(
|
||||
[aws_bin(), "configure", "get", "region"], capture_output=True, text=True
|
||||
)
|
||||
return proc.stdout.strip() if proc.returncode == 0 else ""
|
||||
|
||||
|
||||
def load_payload(args: argparse.Namespace) -> str:
|
||||
"""Return the raw payload text, BOM stripped. Validates JSON when applicable."""
|
||||
if args.payload_file:
|
||||
# utf-8-sig decodes and discards a leading BOM if the file has one.
|
||||
raw = open(args.payload_file, "r", encoding="utf-8-sig").read()
|
||||
else:
|
||||
raw = args.payload
|
||||
|
||||
# If it's meant to be JSON, validate and re-serialize so the body is clean.
|
||||
if "json" in args.content_type:
|
||||
try:
|
||||
return json.dumps(json.loads(raw))
|
||||
except json.JSONDecodeError as e:
|
||||
log(f"ERROR: payload is not valid JSON: {e}")
|
||||
sys.exit(1)
|
||||
return raw
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--endpoint-name", required=True)
|
||||
parser.add_argument("--region")
|
||||
parser.add_argument("--content-type", default="application/json")
|
||||
src = parser.add_mutually_exclusive_group(required=True)
|
||||
src.add_argument("--payload", help="Inline request body (e.g. a JSON string)")
|
||||
src.add_argument("--payload-file", help="Path to a file containing the request body")
|
||||
args = parser.parse_args()
|
||||
|
||||
region = resolve_region(args.region)
|
||||
if not region:
|
||||
log("ERROR: no AWS region. Pass --region or set AWS_REGION.")
|
||||
return 1
|
||||
|
||||
body = load_payload(args)
|
||||
|
||||
# Write the request body as BOM-free UTF-8. NamedTemporaryFile + explicit
|
||||
# utf-8 encoding guarantees no BOM regardless of platform.
|
||||
body_path = None
|
||||
out_path = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
"w", suffix=".json", encoding="utf-8", delete=False
|
||||
) as f:
|
||||
f.write(body)
|
||||
body_path = f.name
|
||||
out_fd, out_path = tempfile.mkstemp(suffix=".out")
|
||||
os.close(out_fd)
|
||||
|
||||
log(f"Invoking {args.endpoint_name} in {region}...")
|
||||
proc = subprocess.run(
|
||||
[
|
||||
aws_bin(), "sagemaker-runtime", "invoke-endpoint",
|
||||
"--endpoint-name", args.endpoint_name,
|
||||
"--content-type", args.content_type,
|
||||
"--body", f"fileb://{body_path}",
|
||||
"--region", region,
|
||||
out_path,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
log("Invocation failed:")
|
||||
log(proc.stderr.strip())
|
||||
return 1
|
||||
|
||||
# The response body landed in out_path; print it to stdout.
|
||||
print(open(out_path, "r", encoding="utf-8-sig").read())
|
||||
return 0
|
||||
finally:
|
||||
for p in (body_path, out_path):
|
||||
if p and os.path.exists(p):
|
||||
os.remove(p)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,174 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tear down a SageMaker endpoint and its associated resources. Cross-platform.
|
||||
|
||||
Deletes in safe order: alarms -> autoscaling -> endpoint (stops billing)
|
||||
-> endpoint config -> model.
|
||||
|
||||
Does NOT delete: IAM role, data capture S3 objects, SNS topic, model artifacts.
|
||||
Idempotent — missing resources are skipped, not errors.
|
||||
|
||||
Calls the `aws` CLI from the current shell, so it inherits the same AWS context
|
||||
(profile, region, SSO session) — no Bash/WSL context-sharing problem on Windows.
|
||||
|
||||
Usage:
|
||||
python teardown.py <endpoint-name> [<region>]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
RESOURCE_DIM = "sagemaker:variant:DesiredInstanceCount"
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(f"[teardown] {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def aws_bin() -> str:
|
||||
exe = shutil.which("aws")
|
||||
if not exe:
|
||||
log("ERROR: the 'aws' CLI was not found on PATH. Install AWS CLI v2.")
|
||||
sys.exit(2)
|
||||
return exe
|
||||
|
||||
|
||||
def run_aws(args: list[str]) -> subprocess.CompletedProcess:
|
||||
return subprocess.run([aws_bin(), *args], capture_output=True, text=True)
|
||||
|
||||
|
||||
def resolve_region(arg_region: str | None) -> str:
|
||||
"""Region from arg, then env, then the active profile's config."""
|
||||
if arg_region:
|
||||
return arg_region
|
||||
for var in ("AWS_REGION", "AWS_DEFAULT_REGION"):
|
||||
if os.environ.get(var):
|
||||
return os.environ[var]
|
||||
proc = run_aws(["configure", "get", "region"])
|
||||
return proc.stdout.strip() if proc.returncode == 0 else ""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) < 2:
|
||||
log(f"Usage: {os.path.basename(sys.argv[0])} <endpoint-name> [<region>]")
|
||||
return 64
|
||||
|
||||
endpoint_name = sys.argv[1]
|
||||
region = resolve_region(sys.argv[2] if len(sys.argv) > 2 else None)
|
||||
if not region:
|
||||
log("ERROR: no AWS region. Pass region as 2nd arg or set AWS_REGION.")
|
||||
return 1
|
||||
|
||||
reg = ["--region", region]
|
||||
log(f"Tearing down endpoint: {endpoint_name} in {region}")
|
||||
|
||||
# Discover what's attached to this endpoint.
|
||||
config_name = ""
|
||||
model_name = ""
|
||||
desc = run_aws(["sagemaker", "describe-endpoint", "--endpoint-name", endpoint_name, *reg])
|
||||
endpoint_exists = desc.returncode == 0
|
||||
if endpoint_exists:
|
||||
try:
|
||||
config_name = json.loads(desc.stdout).get("EndpointConfigName", "")
|
||||
except json.JSONDecodeError:
|
||||
config_name = ""
|
||||
else:
|
||||
log("Endpoint not found — checking for orphan resources anyway")
|
||||
|
||||
if config_name:
|
||||
cfg = run_aws(
|
||||
["sagemaker", "describe-endpoint-config", "--endpoint-config-name", config_name, *reg]
|
||||
)
|
||||
if cfg.returncode == 0:
|
||||
try:
|
||||
variants = json.loads(cfg.stdout).get("ProductionVariants", [])
|
||||
model_name = variants[0]["ModelName"] if variants else ""
|
||||
except (json.JSONDecodeError, KeyError, IndexError):
|
||||
model_name = ""
|
||||
|
||||
# Alarms — discover by name prefix. Both real-time and async deploys create
|
||||
# alarms named "<endpoint-name>-<something>", so this handles either mode.
|
||||
alarms_proc = run_aws(
|
||||
["cloudwatch", "describe-alarms", "--alarm-name-prefix", f"{endpoint_name}-",
|
||||
"--query", "MetricAlarms[*].AlarmName", "--output", "json", *reg]
|
||||
)
|
||||
alarms: list[str] = []
|
||||
if alarms_proc.returncode == 0:
|
||||
try:
|
||||
alarms = json.loads(alarms_proc.stdout) or []
|
||||
except json.JSONDecodeError:
|
||||
alarms = []
|
||||
if alarms:
|
||||
run_aws(["cloudwatch", "delete-alarms", "--alarm-names", *alarms, *reg])
|
||||
log(f"Deleted alarms: {' '.join(alarms)}")
|
||||
|
||||
# Autoscaling policies — discover all on this variant. Real-time has 1 policy;
|
||||
# async has 2 (target-tracking + step-scaling for wake-from-zero).
|
||||
resource_id = f"endpoint/{endpoint_name}/variant/AllTraffic"
|
||||
policies_proc = run_aws(
|
||||
["application-autoscaling", "describe-scaling-policies",
|
||||
"--service-namespace", "sagemaker", "--resource-id", resource_id,
|
||||
"--query", "ScalingPolicies[*].PolicyName", "--output", "json", *reg]
|
||||
)
|
||||
policies: list[str] = []
|
||||
if policies_proc.returncode == 0:
|
||||
try:
|
||||
policies = json.loads(policies_proc.stdout) or []
|
||||
except json.JSONDecodeError:
|
||||
policies = []
|
||||
for policy in policies:
|
||||
run_aws(
|
||||
["application-autoscaling", "delete-scaling-policy",
|
||||
"--service-namespace", "sagemaker", "--resource-id", resource_id,
|
||||
"--scalable-dimension", RESOURCE_DIM, "--policy-name", policy, *reg]
|
||||
)
|
||||
log(f"Deleted autoscaling policy: {policy}")
|
||||
|
||||
targets = run_aws(
|
||||
["application-autoscaling", "describe-scalable-targets",
|
||||
"--service-namespace", "sagemaker", "--resource-ids", resource_id,
|
||||
"--query", "ScalableTargets[*].ResourceId", "--output", "json", *reg]
|
||||
)
|
||||
has_target = False
|
||||
if targets.returncode == 0:
|
||||
try:
|
||||
has_target = resource_id in (json.loads(targets.stdout) or [])
|
||||
except json.JSONDecodeError:
|
||||
has_target = False
|
||||
if has_target:
|
||||
run_aws(
|
||||
["application-autoscaling", "deregister-scalable-target",
|
||||
"--service-namespace", "sagemaker", "--resource-id", resource_id,
|
||||
"--scalable-dimension", RESOURCE_DIM, *reg]
|
||||
)
|
||||
log("Deregistered scalable target")
|
||||
|
||||
# Endpoint (stops billing)
|
||||
if endpoint_exists:
|
||||
run_aws(["sagemaker", "delete-endpoint", "--endpoint-name", endpoint_name, *reg])
|
||||
log(f"Deleted endpoint: {endpoint_name} (billing stopped)")
|
||||
|
||||
# Endpoint config
|
||||
if config_name and run_aws(
|
||||
["sagemaker", "describe-endpoint-config", "--endpoint-config-name", config_name, *reg]
|
||||
).returncode == 0:
|
||||
run_aws(["sagemaker", "delete-endpoint-config", "--endpoint-config-name", config_name, *reg])
|
||||
log(f"Deleted endpoint config: {config_name}")
|
||||
|
||||
# Model
|
||||
if model_name and run_aws(
|
||||
["sagemaker", "describe-model", "--model-name", model_name, *reg]
|
||||
).returncode == 0:
|
||||
run_aws(["sagemaker", "delete-model", "--model-name", model_name, *reg])
|
||||
log(f"Deleted model: {model_name}")
|
||||
|
||||
log("Teardown complete. Data capture S3 objects (if any) NOT deleted — manage separately.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,219 @@
|
||||
---
|
||||
name: hf-cloud-serving-image-selection
|
||||
description: 'Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says "deploy this LLM", "host this HuggingFace model", "serve this fine-tuned model", "deploy this embedding model", "host a reranker", "serve a sentence-transformers model", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings/cross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.'
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Serving Image Selection
|
||||
|
||||
The serving container is the single thing most likely to break a SageMaker deployment that "looked correct on paper". Wrong container, stale tag, or the wrong AMI — all produce the same opaque `Failed to pass health check` error.
|
||||
|
||||
## Rule zero: HuggingFace images always win
|
||||
|
||||
When both a HuggingFace-curated family (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-sglang`, `tei`, `huggingface-pytorch-inference`) and a generic family (`vllm`, `vllm-omni`, `sglang`, `djl-inference`) can serve the model, **the HuggingFace one is mandatory, not preferred**. The only valid reasons to use a generic image:
|
||||
|
||||
1. **Verified incompatibility** — the model needs an architecture/modality/feature no available HuggingFace tag supports, confirmed against the catalog (not assumed).
|
||||
2. **No HuggingFace tag exists in the target region** and mirroring is not an option.
|
||||
3. **The HuggingFace image is in "Known-broken images"** below.
|
||||
|
||||
A **newer version number on the generic repo is not a reason**. The AWS `vllm` repo often publishes a higher vLLM version than `huggingface-vllm`; an older-but-compatible `huggingface-vllm` tag still wins. "Latest vLLM" is not a requirement anyone stated — compatibility with the model is. If you fall back, record in the deployment log which of the three reasons applied.
|
||||
|
||||
## Where image URIs come from
|
||||
|
||||
**Primary source: AWS's official Deep Learning Containers catalog.**
|
||||
|
||||
URL: https://aws.github.io/deep-learning-containers/reference/available_images/
|
||||
|
||||
This page is AWS-maintained and lists every image family with example URIs, tags, CUDA versions, Python versions, and platform (SageMaker vs EC2/ECS/EKS). When picking a URI for a deployment, **read it from this page directly** — copy the example URL, substitute `<region>` with the user's region, and pass it to `deploy.py --image-uri`.
|
||||
|
||||
The example URLs use `763104351884` as the account ID for most regions. A few regions use different accounts (e.g. `eu-south-1` uses `692866216735`). Check the [Region Availability page](https://aws.github.io/deep-learning-containers/reference/region_availability/) when in doubt.
|
||||
|
||||
**Exception: none currently.** Every image family used by this workflow is now on the AWS catalog page (TEI was added in late 2026). If you encounter a new family that isn't there, mirror it via `mirror_image.py` and pass the resulting URI directly.
|
||||
|
||||
## Quick decision
|
||||
|
||||
| Model | Container family | How to get the URI |
|
||||
|---|---|---|
|
||||
| HuggingFace text-generation LLM (Llama, Qwen, Mistral, etc.) | **HuggingFace vLLM** | AWS catalog → "HuggingFace vLLM Inference" (ECR repo `huggingface-vllm`) |
|
||||
| Same as above, multimodal | **HuggingFace vLLM-Omni** | AWS catalog → "HuggingFace vLLM-Omni Inference" (ECR repo `huggingface-vllm-omni`) |
|
||||
| HuggingFace embeddings | TEI | AWS catalog → "HuggingFace Text Embeddings Inference" |
|
||||
| Encoder / cross-encoder rerankers (BERT-family `*ForSequenceClassification`) | TEI | Same as embeddings |
|
||||
| **Generative rerankers** (causal-LM, e.g. Qwen3-Reranker) | HuggingFace vLLM | Same as text-generation LLMs — **not TEI**, see "Rerankers: TEI or vLLM?" |
|
||||
| Text-to-image / diffusion (Stable Diffusion, FLUX) | DJL Inference | AWS catalog → "DJL Inference" — **not** HF Inference Toolkit, see "Known-broken images" |
|
||||
| HuggingFace classifiers, NER, QA, summarization | HF Inference Toolkit (CPU) | AWS catalog → "HuggingFace PyTorch Inference"; GPU tags currently broken — see "Known-broken images" |
|
||||
| User specifically wants SGLang | HuggingFace SGLang | AWS catalog → "HuggingFace SGLang Inference" |
|
||||
| No compatible `huggingface-vllm` tag (verified incompatibility or region gap — see "Rule zero") | vLLM (AWS) | AWS catalog → "vLLM" section — fallback only, never for version freshness |
|
||||
| User specifically wants DJL-LMI | DJL Inference | AWS catalog → "DJL Inference" |
|
||||
| Amazon Nova | SageMaker JumpStart | Use JumpStart, not raw endpoint creation |
|
||||
| Custom inference code | BYOC | User provides URI |
|
||||
|
||||
**HuggingFace-curated DLCs are mandatory when one is compatible (see "Rule zero").** `huggingface-vllm` is layered directly on the AWS vLLM DLC — **identical `SM_VLLM_*` env contract and the same cu130 AMI rule** — and adds current `transformers`, current `huggingface_hub` + `hf_xet` (avoids the XET-CDN 403 download failures older images hit), and HF performance defaults. It is also what SageMaker SDK v3 auto-routes to. The AWS `vllm` image is a compatibility escape hatch only; it usually shows a higher vLLM version than `huggingface-vllm`, and that is not a reason to pick it.
|
||||
|
||||
**Do not use TGI.** Text Generation Inference is archived. Models released after the archive (Qwen3 most famously) fail ping health checks on TGI. Use vLLM instead. (The SageMaker SDK v3 agrees: since [PR #5960](https://github.com/aws/sagemaker-python-sdk/pull/5960), June 2026, its `ModelBuilder` auto-routes `text-generation` to the HuggingFace vLLM DLC and multimodal tasks to HuggingFace vLLM-Omni.)
|
||||
|
||||
Full reasoning for each family in `references/model-to-image.md`.
|
||||
|
||||
## Rerankers: TEI or vLLM?
|
||||
|
||||
"Reranker" covers two very different architectures, and picking wrong wastes a full endpoint-creation cycle (~20 min) before TEI rejects the model:
|
||||
|
||||
- **Encoder cross-encoders** (BAAI/bge-reranker-*, mixedbread, most `sentence-transformers` rerankers) — BERT-family models with a classification head. `config.json` has `architectures: [..ForSequenceClassification]` on a TEI-supported encoder type. → **TEI**.
|
||||
- **Generative rerankers** (Qwen/Qwen3-Reranker-*, and similar causal-LM judges) — decoder LLMs that score relevance via the logprob of a yes/no token. `config.json` has `architectures: [..ForCausalLM]`. → **HuggingFace vLLM**, deployed exactly like a text-generation LLM. TEI will load the architecture then reject the `classifier` model type (Qwen3 support in TEI is *embeddings-only*). Invocation pattern (raw completions API, `max_tokens=1`, logprobs scoring) is in `hf-cloud-sagemaker-production-defaults`.
|
||||
|
||||
**Preflight before creating any resources** — one HTTP GET settles it:
|
||||
|
||||
```bash
|
||||
curl -s https://huggingface.co/<model-id>/raw/main/config.json
|
||||
# "architectures": ["Qwen3ForCausalLM"] → vLLM
|
||||
# "architectures": ["XLMRobertaForSequenceClassification"] → TEI
|
||||
```
|
||||
|
||||
For TEI also confirm the *(architecture, task)* pair: an architecture appearing in TEI's supported list means embeddings support, not necessarily classification/reranking support.
|
||||
|
||||
Heads-up: SageMaker SDK v3 (PR #5960) routes the `text-ranking` task to TEI **unconditionally** — correct for cross-encoders, wrong for generative rerankers. Don't treat the SDK's routing as evidence that TEI can serve a given reranker.
|
||||
|
||||
## Workflow
|
||||
|
||||
For every family: **read the URI from the AWS catalog page**.
|
||||
|
||||
1. Open https://aws.github.io/deep-learning-containers/reference/available_images/
|
||||
2. Find the section for the right family (e.g. "HuggingFace vLLM Inference" for HuggingFace LLMs, "HuggingFace Text Embeddings Inference" for embeddings)
|
||||
3. Pick the newest row marked `SageMaker` for the platform column — newest **within that family**. Do not switch to another family's section because it lists a higher engine version (see "Rule zero")
|
||||
4. Substitute `<region>` with the user's region (from `hf-cloud-aws-context-discovery`)
|
||||
5. For vLLM: also check the AMI requirement (see "vLLM AMI requirement" below)
|
||||
6. Pass the URI to `deploy.py --image-uri` (real-time) or `deploy_async.py --image-uri` (async)
|
||||
|
||||
### TEI: pick the right variant
|
||||
|
||||
The TEI catalog row lists two URIs — GPU (`tei` repo) and CPU (`tei-cpu` repo). Pick based on the instance type:
|
||||
|
||||
- `ml.g*`, `ml.p*`, `ml.inf*` → GPU variant
|
||||
- `ml.c*`, `ml.m*`, `ml.t*` → CPU variant
|
||||
|
||||
Mixing them fails: CPU image on a GPU instance wastes hardware, GPU image on a CPU instance fails to start.
|
||||
|
||||
**Note on the TEI account ID**: the catalog page shows `683313688378` as the example account, but TEI is published from a different account namespace than the main AWS DLCs and the per-region account IDs vary. If `683313688378.dkr.ecr.<region>.amazonaws.com/tei:...` returns an ECR pull error for a region other than us-east-1, check the [Region Availability page](https://aws.github.io/deep-learning-containers/reference/region_availability/) for the correct account ID for that region.
|
||||
|
||||
## vLLM AMI requirement
|
||||
|
||||
vLLM DLC images with **CUDA 13 or higher** (current default: `cu130`) require setting `InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1` on the ProductionVariant. This applies equally to `huggingface-vllm` and `huggingface-vllm-omni` (layered on the same cu130 base) and to the AWS `vllm` repo. Without it the container dies on startup with no CloudWatch logs ever created. The failure looks identical to many other things (account-level issues, quota, networking) and routinely sends people down wrong diagnostic paths.
|
||||
|
||||
Lookup table:
|
||||
|
||||
| Tag contains | InferenceAmiVersion to pass |
|
||||
|---|---|
|
||||
| `cu130` (or higher) | `al2-ami-sagemaker-inference-gpu-3-1` |
|
||||
| `cu129` or lower | (omit the flag; default AMI works) |
|
||||
|
||||
Rule of thumb: if the vLLM tag you picked contains `cu130` or later, pass `--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1` to `deploy.py`. If a future CUDA version (cu140+) needs a different AMI, add a row to the table when AWS publishes the new image.
|
||||
|
||||
This is a vLLM-specific concern. TEI and HF Inference Toolkit images don't need an AMI override.
|
||||
|
||||
## Configuring the vLLM DLCs (HuggingFace vLLM and AWS vLLM)
|
||||
|
||||
Both images share the same contract: configuration as environment variables on the SageMaker model definition, `SM_VLLM_*` mapped to vLLM CLI flags. The `huggingface-vllm` entrypoint additionally auto-detects the model when `SM_VLLM_MODEL` is unset — from `/opt/ml/model` if artifacts are mounted, else from `HF_MODEL_ID` — but setting `SM_VLLM_MODEL` explicitly works on both and is what our examples use.
|
||||
|
||||
### Required for every HuggingFace LLM deployment
|
||||
|
||||
| Env var | Purpose | Notes |
|
||||
|---|---|---|
|
||||
| `SM_VLLM_MODEL` | HF model ID (e.g. `Qwen/Qwen3-0.6B`) or `/opt/ml/model` if loading from S3 | — |
|
||||
| `SM_VLLM_HOST` | **Must be `0.0.0.0`** | Otherwise vLLM binds localhost only, ping fails, container dies before logs. Top cause of mystery failures with this image. |
|
||||
| `SM_VLLM_TRUST_REMOTE_CODE` | `true` for Qwen and several recent architectures | Set unconditionally — downside negligible, upside is the model loads. |
|
||||
| `HUGGING_FACE_HUB_TOKEN` | HF token | Required for gated models. |
|
||||
|
||||
### Tuning (optional)
|
||||
|
||||
| Env var | Purpose |
|
||||
|---|---|
|
||||
| `SM_VLLM_MAX_MODEL_LEN` | Max sequence length — set this; defaults can be wrong for fine-tunes |
|
||||
| `SM_VLLM_GPU_MEMORY_UTILIZATION` | Float 0.0–1.0, ~0.9 reasonable |
|
||||
| `SM_VLLM_TENSOR_PARALLEL_SIZE` | GPU count for multi-GPU instances |
|
||||
| `SM_VLLM_DTYPE` | `auto`, `bfloat16`, `float16` |
|
||||
|
||||
Any vLLM CLI flag works — uppercase, replace dashes with underscores, prepend `SM_VLLM_`.
|
||||
|
||||
## Configuring TEI
|
||||
|
||||
Simpler env contract than vLLM:
|
||||
|
||||
| Env var | Purpose | Required |
|
||||
|---|---|---|
|
||||
| `HF_MODEL_ID` | HF model ID (e.g. `BAAI/bge-large-en-v1.5`) or `/opt/ml/model` | Yes |
|
||||
| `HF_TOKEN` | HF auth token | Only for gated models |
|
||||
| `MAX_BATCH_TOKENS` | Max tokens per batch (default 16384) | No |
|
||||
| `MAX_CLIENT_BATCH_SIZE` | Max requests per client batch (default 32) | No |
|
||||
|
||||
No host-binding to configure, no trust-remote-code flag. The architectures TEI supports (BERT, CamemBERT, RoBERTa, XLM-RoBERTa, NomicBert, JinaBert, JinaCodeBert, Mistral, Qwen2/3, Gemma2/3, ModernBert) are baked into the image.
|
||||
|
||||
## CUDA / instance compatibility
|
||||
|
||||
Critical and easy to get wrong:
|
||||
|
||||
| CUDA in image tag | Default AMI | With `al2-ami-sagemaker-inference-gpu-3-1` |
|
||||
|---|---|---|
|
||||
| cu124 / cu128 | g5, g6, p5 all work | (not needed) |
|
||||
| cu129 | g6, p5; g5 fails (driver mismatch → CannotStartContainerError) | expected to fix g5 (unverified) |
|
||||
| cu130+ | fails everywhere — AMI flag is mandatory | g5, g6, p5 all work (cu130-on-g5 verified June 2026) |
|
||||
|
||||
The driver comes from the host AMI, not the instance family — so passing the gpu-3-1 AMI (which vLLM cu130 images require anyway) also makes `ml.g5.*` viable for cu129+ images.
|
||||
|
||||
## VPC / NAT gateway problem
|
||||
|
||||
SageMaker endpoints inside a VPC **without** a NAT gateway can't pull from `public.ecr.aws`. The deployment fails with an image-pull error that doesn't mention "VPC" or "egress".
|
||||
|
||||
For images on AWS's regional ECR (everything in the catalog): SageMaker reaches them through built-in routing, no NAT needed. Use the regional URI pattern (`<account>.dkr.ecr.<region>.amazonaws.com/...`), not the `public.ecr.aws/...` pattern.
|
||||
|
||||
For images requiring `public.ecr.aws` access (less common): mirror to a private ECR repo in your account with `scripts/mirror_image.py` (cross-platform; needs Docker + the `aws` CLI). Run it from the shell where the AWS CLI works.
|
||||
|
||||
```bash
|
||||
# macOS / Linux
|
||||
PRIVATE_URI=$(python3 scripts/mirror_image.py \
|
||||
public.ecr.aws/deep-learning-containers/vllm:<tag> \
|
||||
vllm-mirror)
|
||||
```
|
||||
|
||||
```powershell
|
||||
# Windows (PowerShell) — capture stdout into a variable
|
||||
$PRIVATE_URI = python scripts\mirror_image.py `
|
||||
public.ecr.aws/deep-learning-containers/vllm:<tag> vllm-mirror
|
||||
```
|
||||
|
||||
## When the catalog page won't render, is stale, or is wrong
|
||||
|
||||
**The page won't render / fetch returns junk**: the catalog page is JavaScript-heavy and some fetch tools get an empty shell. Fallbacks, in order:
|
||||
|
||||
1. **The catalog's source data on GitHub** — the page is generated from one YAML file per version, listing exact tags, CUDA, and Python versions. List a family's files, then fetch the newest:
|
||||
```bash
|
||||
curl -s https://api.github.com/repos/aws/deep-learning-containers/contents/docs/src/data/huggingface-vllm
|
||||
curl -s https://raw.githubusercontent.com/aws/deep-learning-containers/main/docs/src/data/huggingface-vllm/0.21.0-gpu-sagemaker.yml
|
||||
```
|
||||
Directory names match ECR repos (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-tei`, `vllm`, `djl-inference`, ...).
|
||||
2. **Query ECR directly** for current tags in the target region (works with credentials that can read the DLC registry; if it returns AccessDenied, use the YAML files):
|
||||
```bash
|
||||
aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \
|
||||
--region <region> --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output json
|
||||
```
|
||||
3. [Release notes on the DLC GitHub repo](https://github.com/aws/deep-learning-containers/releases).
|
||||
|
||||
**A tag was just released and isn't on the page yet**: rare; AWS updates the page on each release. Check the release notes above.
|
||||
|
||||
**An architecture you need isn't supported by the listed image yet**: for TEI specifically, you can mirror the upstream image from GHCR (`ghcr.io/huggingface/text-embeddings-inference:<version>`) into private ECR and pass the resulting URI directly to `deploy.py --image-uri`. Same `mirror_image.py` script.
|
||||
|
||||
## Known-broken images (last checked July 2026)
|
||||
|
||||
| Image | Defect | Use instead |
|
||||
|---|---|---|
|
||||
| `huggingface-pytorch-inference` **GPU** tags — all recent ones tested (PT 2.3–2.6, cu121/cu124, transformers 4.48–5.5.3) | `ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` at `import torch`. The NCCL bundled in the image is older than what torch links against — a packaging defect *inside the container*, on g5 **and** g6, regardless of AMI, model, or inference code. The MMS Java front-end keeps answering `/ping`, so the endpoint can reach InService while the Python worker crash-loops and serves nothing. | DJL Inference (bundles its own complete CUDA/NCCL stack) or BYOC. CPU tags are unaffected. |
|
||||
|
||||
Re-check when AWS publishes new `huggingface-pytorch-inference` GPU tags — remove the row once a fixed image is confirmed.
|
||||
|
||||
**General fallback rule**: when an HF DLC fails with CUDA/NCCL linker errors, switch to DJL Inference rather than iterating over sibling tags — the defect class is per-repo, not per-tag (three different tags were tried for the case above; all broken).
|
||||
|
||||
**Related HF Hub gotcha**: older DLCs can fail model download with `403 Forbidden` from HF's XET CDN (their bundled `huggingface_hub` predates XET auth). Set `HF_HUB_ENABLE_HF_TRANSFER=0` to force the standard download path, or pre-stage weights in S3.
|
||||
|
||||
## Hub download time at first boot
|
||||
|
||||
Loading the model from HF Hub happens *inside the container after the endpoint starts* — expect **5–15+ minutes** before InService even for small models, longer for multi-GB ones. A slow first boot is not a failure; don't tear down or re-diagnose before the deploy script's 30-minute wait expires.
|
||||
|
||||
For production or repeated deployments, pre-stage the weights in S3 and pass `--model-s3-uri` to `deploy.py` (the model then loads from `/opt/ml/model`) — faster, immune to Hub rate limits/outages, and no `HUGGING_FACE_HUB_TOKEN` needed at runtime.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Model Family → Serving Container Decision Table
|
||||
|
||||
Consult this **before** writing deployment code that hardcodes an image URI.
|
||||
|
||||
The canonical source for AWS Deep Learning Container image URIs is:
|
||||
**https://aws.github.io/deep-learning-containers/reference/available_images/**
|
||||
|
||||
This page is AWS-maintained and lists every published image family with example URIs, tags, CUDA versions, and platform (SageMaker vs EC2/ECS/EKS). Read URIs from there directly — substitute `<region>` with the user's region and pass to `deploy.py --image-uri`. The doc below explains *which* family to pick for which use case; the URI itself comes from the AWS page.
|
||||
|
||||
## Decision summary
|
||||
|
||||
| Model family | Container | Source |
|
||||
|---|---|---|
|
||||
| HuggingFace text-generation LLMs (Llama, Qwen, Mistral, Mixtral, DeepSeek, Phi, Gemma, GPT-OSS, etc.) | **HuggingFace vLLM DLC** | AWS catalog → "HuggingFace vLLM Inference" |
|
||||
| Same as above, multimodal (vision-language, audio, any-to-any) | **HuggingFace vLLM-Omni DLC** | AWS catalog → "HuggingFace vLLM-Omni Inference" |
|
||||
| Same family, when NO `huggingface-vllm` tag is compatible (verified) or the region has none | AWS vLLM DLC (fallback — never for version freshness) | AWS catalog → "vLLM" |
|
||||
| Same family, alternative serving stack | DJL-LMI | AWS catalog → "DJL Inference" |
|
||||
| HuggingFace embeddings + encoder cross-encoder rerankers | **TEI DLC** | AWS catalog → "HuggingFace Text Embeddings Inference" |
|
||||
| Generative rerankers (causal-LM, e.g. Qwen3-Reranker) | **HuggingFace vLLM DLC** | Same as text-generation LLMs — see "Generative rerankers" below |
|
||||
| HuggingFace classifiers, NER, QA, summarization | HF Inference Toolkit (CPU; GPU tags broken as of July 2026 — see SKILL.md "Known-broken images") | AWS catalog → "HuggingFace PyTorch Inference" |
|
||||
| HuggingFace-curated SGLang build | HuggingFace SGLang | AWS catalog → "HuggingFace SGLang Inference" |
|
||||
| SGLang (without HF wrapper) | SGLang | AWS catalog → "SGLang" |
|
||||
| Amazon Nova (Lite, Micro, Pro) | SageMaker JumpStart | Use JumpStart deployment, not raw endpoint creation |
|
||||
| Stable Diffusion / diffusion / image generation | **DJL Inference** | AWS catalog → "DJL Inference" — not the HF Inference Toolkit (GPU tags broken, see below); StabilityAI DLC exists but is stale |
|
||||
| Inferentia / Trainium hardware | NeuronX variants | AWS catalog → search for "NeuronX" |
|
||||
| Custom inference code | BYOC | User provides URI |
|
||||
|
||||
## Why vLLM, not TGI
|
||||
|
||||
Text Generation Inference (TGI) was the long-standing default for HuggingFace LLMs. As of late 2025 / early 2026, **TGI is archived** — no more major updates. Models released after the archive (Qwen3 most famously) fail health checks on TGI. Use vLLM instead.
|
||||
|
||||
The SageMaker SDK v2 helper `get_huggingface_llm_image_uri` returned TGI URIs; the v3 SDK removed it entirely. Since [sagemaker-python-sdk PR #5960](https://github.com/aws/sagemaker-python-sdk/pull/5960) (merged June 2026), v3's `ModelBuilder` auto-routes HuggingFace models by task: `text-generation` → HuggingFace vLLM DLC, multimodal tasks (`image-text-to-text`, `any-to-any`, `audio-text-to-text`) → vLLM-Omni, `feature-extraction` / `sentence-similarity` / `text-ranking` → TEI, SGLang opt-in only. Our decision table matches, with one deliberate divergence: the SDK sends **all** `text-ranking` models to TEI, which fails for generative rerankers (see below). Either way, don't use TGI for new deployments.
|
||||
|
||||
## Generative rerankers → vLLM, not TEI
|
||||
|
||||
Two reranker architectures exist:
|
||||
|
||||
- **Encoder cross-encoders** (BAAI/bge-reranker-*, mixedbread rerankers): BERT-family + classification head, `config.json` `architectures` ends in `ForSequenceClassification`. TEI serves these natively via its `/rerank` route.
|
||||
- **Generative rerankers** (Qwen/Qwen3-Reranker-0.6B/4B/8B and similar): decoder-only causal LMs that emit a yes/no judgment token; relevance = P(yes)/(P(yes)+P(no)) from the logprobs. `config.json` `architectures` ends in `ForCausalLM`. These need **vLLM** — use the HuggingFace vLLM DLC, same as any text-generation LLM. TEI recognizes the Qwen3 architecture, starts loading it, then rejects the `classifier` model type — Qwen3 support in TEI is embeddings-only. That rejection costs a full endpoint-creation cycle (~20 min observed), so preflight with the model's `config.json` (`curl -s https://huggingface.co/<id>/raw/main/config.json`) before creating anything.
|
||||
|
||||
Deploy generative rerankers exactly like a vLLM LLM deployment (`SM_VLLM_*` env vars, AMI rule applies). The invocation pattern — raw completions API, `max_tokens=1`, logprobs scoring — is documented in `hf-cloud-sagemaker-production-defaults`.
|
||||
|
||||
## HuggingFace vLLM DLC (mandatory default for LLMs)
|
||||
|
||||
This is the default for **every** vLLM deployment, not a suggestion. The AWS vLLM repo further down is strictly a compatibility escape hatch.
|
||||
|
||||
URI pattern (from AWS catalog → "HuggingFace vLLM Inference"):
|
||||
```
|
||||
763104351884.dkr.ecr.<region>.amazonaws.com/huggingface-vllm:<version>-transformers<tv>-gpu-py<py>-cu<cuda>-ubuntu22.04
|
||||
```
|
||||
|
||||
Example: `763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04` (0.22.1 in the release pipeline as of July 2026 — read the newest tag from the catalog).
|
||||
|
||||
The image is built directly on the AWS vLLM DLC, so everything below about the AWS image applies to it too — same `SM_VLLM_*` env contract, same cu130 AMI requirement. On top of the base it carries current `transformers`, current `huggingface_hub` + `hf_xet` (current-generation Hub downloads; older images 403 against the XET CDN), ffmpeg for multimodal preprocessing, and HF performance defaults (expandable-segments allocator, LMCache CPU KV-offload). Model selection: `SM_VLLM_MODEL` as usual; if unset, the entrypoint auto-detects `/opt/ml/model` or falls back to `HF_MODEL_ID`.
|
||||
|
||||
For multimodal models (vision-language, audio, any-to-any) use the sibling `huggingface-vllm-omni` repo (catalog → "HuggingFace vLLM-Omni Inference") — same rules.
|
||||
|
||||
## AWS vLLM DLC (fallback only)
|
||||
|
||||
Use **only** when no `huggingface-vllm` tag is compatible with the model (a verified architecture/feature gap, not an assumed one) or the target region has no `huggingface-vllm` tag at all. This repo usually lists a **newer vLLM version** than `huggingface-vllm` — that alone is never a reason to pick it. If you are reading this section because the version number here is higher, go back to `huggingface-vllm`. URI pattern (from AWS catalog):
|
||||
```
|
||||
763104351884.dkr.ecr.<region>.amazonaws.com/vllm:<version>-gpu-py<py>-cu<cuda>-ubuntu22.04-sagemaker
|
||||
```
|
||||
|
||||
Example: `763104351884.dkr.ecr.eu-west-1.amazonaws.com/vllm:0.23.0-gpu-py312-cu130-ubuntu22.04-sagemaker`
|
||||
|
||||
**vLLM AMI requirement (both repos)**: images with `cu130` or higher require setting `InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1` on the ProductionVariant. Without it the container dies on startup with no CloudWatch logs created. See the "vLLM AMI requirement" lookup table in the SKILL.md to map a tag to the right AMI version.
|
||||
|
||||
For environment variable configuration of the vLLM DLCs, see the SKILL.md.
|
||||
|
||||
## TEI DLC
|
||||
|
||||
Listed on the AWS catalog under "HuggingFace Text Embeddings Inference". The catalog row uses account ID `683313688378` (different from the main `763104351884` used by most other DLCs). TEI is published from its own account namespace and the per-region account IDs vary — if the canonical `683313688378` returns an ECR pull error for a non-us-east-1 region, check the [Region Availability page](https://aws.github.io/deep-learning-containers/reference/region_availability/) for the correct mapping.
|
||||
|
||||
Two variants:
|
||||
|
||||
- Repo `tei` — GPU build
|
||||
- Repo `tei-cpu` — CPU build
|
||||
|
||||
Pick by instance type:
|
||||
- `ml.g*`, `ml.p*`, `ml.inf*` → `tei` (GPU)
|
||||
- `ml.c*`, `ml.m*`, `ml.t*` → `tei-cpu` (CPU)
|
||||
|
||||
CPU embeddings are dramatically cheaper than GPU and often fast enough — `ml.c6i.2xlarge` (~$0.20/hr) is a common starting point. GPU is needed for large embedding models (>1B params) or sustained high throughput.
|
||||
|
||||
### Supported architectures and staleness
|
||||
|
||||
TEI bakes architecture support into the image. The current upstream version supports BERT, CamemBERT, RoBERTa, XLM-RoBERTa, NomicBert, JinaBert, JinaCodeBert, Mistral, Qwen2/3, Gemma2/3, ModernBert. The AWS-published DLC sometimes lags upstream by months — if a recent architecture isn't supported, mirror the upstream image from `ghcr.io/huggingface/text-embeddings-inference:<version>` using `scripts/mirror_image.py` and pass the result to `deploy.py --image-uri` directly.
|
||||
|
||||
**Support is per (architecture, task), not per architecture.** Qwen3 appearing in the list means Qwen3 *embeddings*; TEI 1.8.x rejects Qwen3 with a classification head ("`classifier` model type is not supported"). Before picking TEI for a reranker, check the model's `config.json`: `architectures` ending in `ForSequenceClassification` on a BERT-family encoder is TEI territory; `ForCausalLM` means it's a generative reranker → vLLM (see above).
|
||||
|
||||
For environment variable configuration of TEI, see the SKILL.md.
|
||||
|
||||
## DJL Inference
|
||||
|
||||
Listed on the AWS catalog under "DJL Inference", published from the main `763104351884` account. Two reasons to reach for it:
|
||||
|
||||
1. **Diffusion / text-to-image models** — the recommended container for Stable Diffusion-class workloads (usually behind an async endpoint; see `hf-cloud-sagemaker-production-defaults`).
|
||||
2. **Fallback when an HF DLC fails with CUDA/NCCL errors** — DJL images bundle their own complete CUDA/NCCL stack and don't depend on host libraries, so they sidestep the packaging-defect class entirely.
|
||||
|
||||
Configuration goes in a `serving.properties` file packaged alongside the model artifacts (or the equivalent `OPTION_*` env vars) rather than `SM_VLLM_*`/`HF_MODEL_ID` — check the [DJL Serving docs](https://docs.djl.ai/master/docs/serving/index.html) for the engine matching your model type rather than guessing keys.
|
||||
|
||||
## HF Inference Toolkit: GPU tags broken (NCCL packaging defect)
|
||||
|
||||
As of July 2026, every recently-tested GPU tag of `huggingface-pytorch-inference` (PT 2.3–2.6, cu121/cu124, transformers 4.48–5.5.3) fails at `import torch` with:
|
||||
|
||||
```
|
||||
ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume
|
||||
```
|
||||
|
||||
torch in the image links NCCL 2.19+ symbols but the image ships an older NCCL. Verified on g5 **and** g6 — the defect is inside the container, so changing instance type, AMI, model, or inference code does not help, and neither does trying sibling tags. Worse, the MMS Java server keeps answering `/ping` while the Python worker crash-loops, so the endpoint can reach InService and silently serve nothing (async requests queue forever). Use DJL Inference or BYOC for GPU workloads until AWS ships fixed images; CPU tags are unaffected.
|
||||
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Mirror an ECR Public image to a private ECR repo. Cross-platform.
|
||||
|
||||
Needed when SageMaker runs in a VPC without NAT gateway: SageMaker can't pull
|
||||
from public.ecr.aws in that case, so the image must be in private ECR.
|
||||
|
||||
Requires: docker daemon, aws CLI with ecr/ecr-public permissions.
|
||||
Idempotent: skips push if the tag already exists in the private repo.
|
||||
|
||||
Runs `aws`/`docker` from the current shell, inheriting its AWS context — no
|
||||
Bash/WSL context-sharing problem on Windows.
|
||||
|
||||
Usage:
|
||||
python mirror_image.py <public-image-uri> <private-repo-name> [<tag-override>]
|
||||
|
||||
Prints the resulting private URI to stdout.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(f"[mirror_image] {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def require(cmd: str) -> str:
|
||||
exe = shutil.which(cmd)
|
||||
if not exe:
|
||||
log(f"ERROR: {cmd} not installed or not on PATH")
|
||||
sys.exit(1)
|
||||
return exe
|
||||
|
||||
|
||||
def run(args: list[str], **kwargs) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(args, capture_output=True, text=True, **kwargs)
|
||||
|
||||
|
||||
def resolve_region() -> str:
|
||||
for var in ("AWS_REGION", "AWS_DEFAULT_REGION"):
|
||||
if os.environ.get(var):
|
||||
return os.environ[var]
|
||||
proc = run([shutil.which("aws"), "configure", "get", "region"])
|
||||
return proc.stdout.strip() if proc.returncode == 0 else ""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) < 3:
|
||||
log(f"Usage: {os.path.basename(sys.argv[0])} <public-image-uri> <private-repo-name> [<tag-override>]")
|
||||
return 64
|
||||
|
||||
public_uri = sys.argv[1]
|
||||
private_repo = sys.argv[2]
|
||||
tag_override = sys.argv[3] if len(sys.argv) > 3 else ""
|
||||
|
||||
aws = require("aws")
|
||||
docker = require("docker")
|
||||
if run([docker, "info"]).returncode != 0:
|
||||
log("ERROR: docker daemon not running")
|
||||
return 1
|
||||
|
||||
# Extract tag
|
||||
if tag_override:
|
||||
tag = tag_override
|
||||
elif ":" in public_uri.rsplit("/", 1)[-1]:
|
||||
tag = public_uri.rsplit(":", 1)[-1]
|
||||
else:
|
||||
log("ERROR: public URI has no tag — refusing implicit ':latest'")
|
||||
return 1
|
||||
|
||||
acct = run([aws, "sts", "get-caller-identity", "--query", "Account", "--output", "text"])
|
||||
if acct.returncode != 0:
|
||||
log("ERROR: 'aws sts get-caller-identity' failed. Configure AWS credentials.")
|
||||
return 1
|
||||
account_id = acct.stdout.strip()
|
||||
|
||||
region = resolve_region()
|
||||
if not region:
|
||||
log("ERROR: no AWS region. Set AWS_REGION or configure profile.")
|
||||
return 1
|
||||
|
||||
registry = f"{account_id}.dkr.ecr.{region}.amazonaws.com"
|
||||
private_uri = f"{registry}/{private_repo}:{tag}"
|
||||
|
||||
log(f"Public : {public_uri}")
|
||||
log(f"Private: {private_uri}")
|
||||
|
||||
reg = ["--region", region]
|
||||
|
||||
# Create private repo if missing
|
||||
if run([aws, "ecr", "describe-repositories", "--repository-names", private_repo, *reg]).returncode != 0:
|
||||
log(f"Creating private ECR repo: {private_repo}")
|
||||
run(
|
||||
[aws, "ecr", "create-repository", "--repository-name", private_repo,
|
||||
"--image-scanning-configuration", "scanOnPush=true", *reg]
|
||||
)
|
||||
|
||||
# Skip if tag already exists in private
|
||||
if run(
|
||||
[aws, "ecr", "describe-images", "--repository-name", private_repo,
|
||||
"--image-ids", f"imageTag={tag}", *reg]
|
||||
).returncode == 0:
|
||||
log(f"Tag '{tag}' already in private ECR — skipping pull/push")
|
||||
print(private_uri)
|
||||
return 0
|
||||
|
||||
# Auth. ECR Public auth always uses us-east-1.
|
||||
def docker_login(password: str, registry_host: str) -> bool:
|
||||
login = subprocess.run(
|
||||
[docker, "login", "--username", "AWS", "--password-stdin", registry_host],
|
||||
input=password, capture_output=True, text=True,
|
||||
)
|
||||
return login.returncode == 0
|
||||
|
||||
pub_pw = run([aws, "ecr-public", "get-login-password", "--region", "us-east-1"])
|
||||
if pub_pw.returncode != 0 or not docker_login(pub_pw.stdout, "public.ecr.aws"):
|
||||
log("ERROR: failed to authenticate to public.ecr.aws")
|
||||
return 1
|
||||
|
||||
priv_pw = run([aws, "ecr", "get-login-password", *reg])
|
||||
if priv_pw.returncode != 0 or not docker_login(priv_pw.stdout, registry):
|
||||
log(f"ERROR: failed to authenticate to {registry}")
|
||||
return 1
|
||||
|
||||
log(f"Pulling {public_uri} (may take several minutes)...")
|
||||
if subprocess.run([docker, "pull", public_uri]).returncode != 0:
|
||||
log("ERROR: docker pull failed")
|
||||
return 1
|
||||
subprocess.run([docker, "tag", public_uri, private_uri])
|
||||
log("Pushing to private ECR...")
|
||||
if subprocess.run([docker, "push", private_uri]).returncode != 0:
|
||||
log("ERROR: docker push failed")
|
||||
return 1
|
||||
|
||||
log("Done.")
|
||||
print(private_uri)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
name: hf-mem
|
||||
description: Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
`hf_mem` estimates the required memory for inference, including model weights and an optional KV cache, for Safetensors and GGUF for models on the Hugging Face Hub using HTTP Range requests i.e., without downloading or loading any weights locally.
|
||||
|
||||
## When to use?
|
||||
|
||||
- User asks how much VRAM or memory a model needs to run
|
||||
- User wants to know if a model fits on their GPU or a given instance
|
||||
- User references a Hugging Face model ID or URL and asks about inference requirements
|
||||
|
||||
## What are the requirements?
|
||||
|
||||
- `uv` installed (for `uvx`)
|
||||
- `HF_TOKEN` env var or `--hf-token` flag (for gated or private models only)
|
||||
|
||||
## How to run?
|
||||
|
||||
Run with `--model-id` pointing to the Hugging Face Hub repository which will check that it either contains Safetensors (via `model.safetensors`, `model.safetensors.index.json` if sharded, or `model_index.json` for Diffusers) or GGUF model weights within.
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id <model-id> --json-output
|
||||
```
|
||||
|
||||
If the repository contains GGUF model weights in multiple precisions / quantizations, the estimations will be on a per-file basis, whereas for inference you won't load all of those but rather only a single precision. This being said, for GGUF you might as well need to provide `--gguf-file` to target the specific file (or path if sharded) you want to run.
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id <model-id> --gguf-file <file-or-path> --json-output
|
||||
```
|
||||
|
||||
Additionally, `hf-mem` comes with an `--experimental` flag that will also calculate the KV cache memory requirements too, useful for large-language models, meaning it applies to LLMs (`...ForCausalLM`), VLMs (`...ForConditionalGeneration`), and GGUF models.
|
||||
|
||||
As per the context window, it will be read from the default or overridden with `--max-model-len` a la vLLM. And, same goes for the KV cache precision, which will default to the model precision unless manually set via `--kv-cache-dtype` a la vLLM too.
|
||||
|
||||
For Safetensors use as:
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id <model-id> --experimental [--max-model-len N] [--batch-size N] [--kv-cache-dtype auto|bfloat16|fp8|fp8_ds_mla|fp8_e4m3|fp8_e5m2|fp8_inc] --json-output
|
||||
```
|
||||
|
||||
And, for GGUF use as:
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id <model-id> --gguf-file <file-or-path> --experimental [--max-model-len N] [--batch-size N] [--kv-cache-dtype auto|F32|F16|Q4_0|Q4_1|Q5_0|Q5_1|Q8_0|Q8_1|Q2_K|Q3_K|Q4_K|Q5_K|Q6_K|Q8_K|IQ2_XXS|IQ2_XS|IQ3_XXS|IQ1_S|IQ4_NL|IQ3_S|IQ2_S|IQ4_XS|I8|I16|I32|I64|F64|IQ1_M|BF16|TQ1_0|TQ2_0|MXFP4] --json-output
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
For Transformers with Safetensors weights:
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id MiniMaxAI/MiniMax-M2 --json-output
|
||||
```
|
||||
|
||||
For Diffusers with Safetensors weights:
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id Qwen/Qwen-Image --json-output
|
||||
```
|
||||
|
||||
For Sentence Transformers with Safetensors weights:
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id google/embeddinggemma-300m --json-output
|
||||
```
|
||||
|
||||
With `--experimental` to include the KV cache estimation for LLMs and VLMs:
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id mistralai/Mistral-7B-v0.1 --experimental --json-output
|
||||
```
|
||||
|
||||
And, for LLMs or VLMs with GGUF weights:
|
||||
|
||||
```bash
|
||||
uvx hf-mem --model-id unsloth/Qwen3.5-397B-A17B-GGUF --gguf-file Q4_K_M --experimental --json-output
|
||||
```
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
name: huggingface-best
|
||||
description: "Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: \"best model for X\", \"what model should I use for\", \"top models for [task]\", \"which model runs on my laptop/machine/device\", \"recommend a model for\", \"what LLM should I use for\", \"compare models for\", \"what's state of the art for\", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks."
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# HuggingFace Best Model Finder
|
||||
|
||||
Finds the best models for a task by querying official HF benchmark leaderboards, enriching
|
||||
results with model size data, filtering for what fits on the user's device, and returning a
|
||||
comparison table with benchmark scores.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Parse the request
|
||||
|
||||
Extract from the user's message:
|
||||
- **Task**: what they want the model to do (coding, math/reasoning, chat, OCR, RAG/retrieval, speech recognition, image classification, multimodal, agents, etc.)
|
||||
- **Device**: hardware constraints (MacBook M-series 8/16/32/64GB unified memory, RTX GPU with VRAM amount, CPU-only, cloud/no constraint, etc.)
|
||||
|
||||
If device is not mentioned, skip filtering entirely and return the highest-performing models regardless of size. If the task is genuinely ambiguous, ask one clarifying question.
|
||||
|
||||
### Device → max parameter budget
|
||||
|
||||
When a device is specified, extract its available memory (unified RAM for Apple Silicon, VRAM for discrete GPUs) and apply:
|
||||
|
||||
- **fp16 max params (B)** ≈ memory (GB) ÷ 2
|
||||
- **Q4 max params (B)** ≈ memory (GB) × 2
|
||||
|
||||
Examples: 16GB → 8B fp16 / 32B Q4 — 24GB VRAM → 12B fp16 / 48B Q4 — 8GB → 4B fp16 / 16B Q4
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Find relevant benchmark datasets
|
||||
|
||||
Fetch the full list of official HF benchmarks. All subsequent calls in this skill reuse the same auth token, exported once:
|
||||
|
||||
```bash
|
||||
export HF_AUTH="Bearer $(cat ~/.cache/huggingface/token)"
|
||||
curl -s -H "Authorization: $HF_AUTH" \
|
||||
"https://huggingface.co/api/datasets?filter=benchmark:official&limit=500" | jq '[.[] | {id, tags, description}]'
|
||||
```
|
||||
|
||||
Read the returned list and select the datasets most relevant to the user's task — match on dataset id, tags, and description. Use your judgment; don't limit yourself to 2-3. Aim for comprehensive coverage: if 5 benchmarks clearly cover the task, use all 5.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Fetch top models from leaderboards
|
||||
|
||||
For each selected benchmark dataset:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: $HF_AUTH" \
|
||||
"https://huggingface.co/api/datasets/<namespace>/<repo>/leaderboard" | jq '[.[:15] | .[] | {rank, modelId, value, verified}]'
|
||||
```
|
||||
|
||||
Collect model IDs and scores across all benchmarks. If a leaderboard returns an error (404, 401, etc.), skip it and note it in the output.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Enrich with model metadata
|
||||
|
||||
For the top 10-15 candidate model IDs, get model infos.
|
||||
|
||||
```bash
|
||||
# REST API
|
||||
curl -s -H "Authorization: $HF_AUTH" \
|
||||
"https://huggingface.co/api/models/org/model1" | jq '{safetensors, tags, cardData}'
|
||||
|
||||
# CLI (hf-cli)
|
||||
hf models info org/model1 --json | jq '{safetensors, tags, cardData}'
|
||||
```
|
||||
|
||||
Extract from each response:
|
||||
- **Parameters**: `safetensors.total` → convert to B (e.g., 7_241_748_480 → "7.2B")
|
||||
- **License**: from model card tags (look for `license:apache-2.0`, `license:mit`, etc.)
|
||||
- If `safetensors` is absent, parse size from the model name (look for "7b", "8b", "13b", "70b", "72b", etc.)
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Filter and rank
|
||||
|
||||
**If a device was specified:**
|
||||
1. Remove models exceeding the fp16 parameter budget for the device
|
||||
2. Flag models that fit only with Q4 quantization (multiply budget by ~4 for Q4 capacity)
|
||||
3. If a highly-ranked model is slightly over budget, keep it with a "needs Q4" note — don't silently drop it
|
||||
|
||||
**If no device was mentioned:** skip all size filtering — just rank by benchmark score.
|
||||
|
||||
Then: rank by benchmark score (descending), keep top 5-8 models.
|
||||
|
||||
Include proprietary models (GPT-4, Claude, Gemini) if they appear on leaderboards, but flag them as "API only / not self-hostable". If the user explicitly asked for local/open models only, exclude them.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Output
|
||||
|
||||
### Comparison table
|
||||
|
||||
```markdown
|
||||
| # | Model | Params | [Benchmark 1] | [Benchmark 2] | License | On device |
|
||||
|---|-------|--------|--------------|--------------|---------|-----------|
|
||||
| ⭐1 | [org/name](https://huggingface.co/org/name) | 7B | 85.2% | — | Apache 2.0 | Yes (fp16) |
|
||||
| 2 | [org/name](https://huggingface.co/org/name) | 13B | 83.1% | 71.5% | MIT | Q4 only |
|
||||
| 3 | [org/name](https://huggingface.co/org/name) | 70B | 90.0% | 81.0% | Llama | Too large |
|
||||
```
|
||||
|
||||
- Link model names to `https://huggingface.co/<model_id>`
|
||||
- Use `—` for benchmarks where the model wasn't evaluated
|
||||
- Star the top recommended pick with ⭐
|
||||
- "On device" values: `Yes (fp16)`, `Q4 only`, `Too large`, `API only`
|
||||
|
||||
### Follow-up
|
||||
|
||||
After presenting the table, ask the user: "Would you like to run **[top recommended model]**?"
|
||||
|
||||
If they say yes, ask whether they'd prefer to:
|
||||
- **Run locally** — ask about their device if not already known, then give appropriate setup instructions
|
||||
- **Run on HF Jobs** — point them to the HF Jobs guide: https://huggingface.co/docs/huggingface_hub/en/guides/jobs
|
||||
|
||||
---
|
||||
|
||||
## Error handling
|
||||
|
||||
- **Leaderboard not found**: skip, note "leaderboard unavailable" in output
|
||||
- **Model missing from hub_repo_details**: fall back to parsing size from model name
|
||||
- **No benchmarks found for task**: use the curated fallback table above, or try `hub_repo_search` with `filters=["<task>"]` sorted by `trendingScore`
|
||||
- **All leaderboards fail**: fall back to `hub_repo_search` for popular models tagged with the task, note that results are by popularity rather than benchmark score
|
||||
@@ -0,0 +1,208 @@
|
||||
---
|
||||
name: huggingface-community-evals
|
||||
description: Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Overview
|
||||
|
||||
This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**.
|
||||
|
||||
It covers:
|
||||
- `inspect-ai` with local inference
|
||||
- `lighteval` with local inference
|
||||
- choosing between `vllm`, Hugging Face Transformers, and `accelerate`
|
||||
- smoke tests, task selection, and backend fallback strategy
|
||||
|
||||
It does **not** cover:
|
||||
- Hugging Face Jobs orchestration
|
||||
- model-card or `model-index` edits
|
||||
- README table extraction
|
||||
- Artificial Analysis imports
|
||||
- `.eval_results` generation or publishing
|
||||
- PR creation or community-evals automation
|
||||
|
||||
If the user wants to **run the same eval remotely on Hugging Face Jobs**, submit the same script via `hf jobs uv run` (CLI) or the `hf_jobs()` MCP tool if configured, for remote GPU execution.
|
||||
|
||||
If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand that publishing step off to the project's own community-evals workflow.
|
||||
|
||||
> All paths below are relative to the directory containing this `SKILL.md`.
|
||||
|
||||
# When To Use Which Script
|
||||
|
||||
| Use case | Script |
|
||||
|---|---|
|
||||
| Local `inspect-ai` eval on a Hub model via inference providers | `scripts/inspect_eval_uv.py` |
|
||||
| Local GPU eval with `inspect-ai` using `vllm` or Transformers | `scripts/inspect_vllm_uv.py` |
|
||||
| Local GPU eval with `lighteval` using `vllm` or `accelerate` | `scripts/lighteval_vllm_uv.py` |
|
||||
| Extra command patterns | `examples/USAGE_EXAMPLES.md` |
|
||||
|
||||
# Prerequisites
|
||||
|
||||
- Prefer `uv run` for local execution.
|
||||
- Set `HF_TOKEN` for gated/private models.
|
||||
- For local GPU runs, verify GPU access before starting:
|
||||
|
||||
```bash
|
||||
uv --version
|
||||
printenv HF_TOKEN >/dev/null
|
||||
nvidia-smi
|
||||
```
|
||||
|
||||
If `nvidia-smi` is unavailable, either:
|
||||
- use `scripts/inspect_eval_uv.py` for lighter provider-backed evaluation, or
|
||||
- submit it to Hugging Face Jobs (`hf jobs uv run` or `hf_jobs()`) if the user wants remote compute.
|
||||
|
||||
# Core Workflow
|
||||
|
||||
1. Choose the evaluation framework.
|
||||
- Use `inspect-ai` when you want explicit task control and inspect-native flows.
|
||||
- Use `lighteval` when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks.
|
||||
2. Choose the inference backend.
|
||||
- Prefer `vllm` for throughput on supported architectures.
|
||||
- Use Hugging Face Transformers (`--backend hf`) or `accelerate` as compatibility fallbacks.
|
||||
3. Start with a smoke test.
|
||||
- `inspect-ai`: add `--limit 10` or similar.
|
||||
- `lighteval`: add `--max-samples 10`.
|
||||
4. Scale up only after the smoke test passes.
|
||||
5. If the user wants remote execution, submit the same script + args to Hugging Face Jobs (`hf jobs uv run` or `hf_jobs()`).
|
||||
|
||||
# Quick Start
|
||||
|
||||
## Option A: inspect-ai with local inference providers path
|
||||
|
||||
Best when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead.
|
||||
|
||||
```bash
|
||||
uv run scripts/inspect_eval_uv.py \
|
||||
--model meta-llama/Llama-3.2-1B \
|
||||
--task mmlu \
|
||||
--limit 20
|
||||
```
|
||||
|
||||
Use this path when:
|
||||
- you want a quick local smoke test
|
||||
- you do not need direct GPU control
|
||||
- the task already exists in `inspect-evals`
|
||||
|
||||
## Option B: inspect-ai on Local GPU
|
||||
|
||||
Best when you need to load the Hub model directly, use `vllm`, or fall back to Transformers for unsupported architectures.
|
||||
|
||||
Local GPU:
|
||||
|
||||
```bash
|
||||
uv run scripts/inspect_vllm_uv.py \
|
||||
--model meta-llama/Llama-3.2-1B \
|
||||
--task gsm8k \
|
||||
--limit 20
|
||||
```
|
||||
|
||||
Transformers fallback:
|
||||
|
||||
```bash
|
||||
uv run scripts/inspect_vllm_uv.py \
|
||||
--model microsoft/phi-2 \
|
||||
--task mmlu \
|
||||
--backend hf \
|
||||
--trust-remote-code \
|
||||
--limit 20
|
||||
```
|
||||
|
||||
## Option C: lighteval on Local GPU
|
||||
|
||||
Best when the task is naturally expressed as a `lighteval` task string, especially Open LLM Leaderboard style benchmarks.
|
||||
|
||||
Local GPU:
|
||||
|
||||
```bash
|
||||
uv run scripts/lighteval_vllm_uv.py \
|
||||
--model meta-llama/Llama-3.2-3B-Instruct \
|
||||
--tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
|
||||
--max-samples 20 \
|
||||
--use-chat-template
|
||||
```
|
||||
|
||||
`accelerate` fallback:
|
||||
|
||||
```bash
|
||||
uv run scripts/lighteval_vllm_uv.py \
|
||||
--model microsoft/phi-2 \
|
||||
--tasks "leaderboard|mmlu|5" \
|
||||
--backend accelerate \
|
||||
--trust-remote-code \
|
||||
--max-samples 20
|
||||
```
|
||||
|
||||
# Remote Execution Boundary
|
||||
|
||||
This skill intentionally stops at **local execution and backend selection**.
|
||||
|
||||
If the user wants to:
|
||||
- run these scripts on Hugging Face Jobs
|
||||
- pick remote hardware
|
||||
- pass secrets to remote jobs
|
||||
- schedule recurring runs
|
||||
- inspect / cancel / monitor jobs
|
||||
|
||||
then submit one of these scripts plus the chosen arguments to Hugging Face Jobs (`hf jobs uv run` or `hf_jobs()`).
|
||||
|
||||
# Task Selection
|
||||
|
||||
`inspect-ai` examples:
|
||||
- `mmlu`
|
||||
- `gsm8k`
|
||||
- `hellaswag`
|
||||
- `arc_challenge`
|
||||
- `truthfulqa`
|
||||
- `winogrande`
|
||||
- `humaneval`
|
||||
|
||||
`lighteval` task strings use `suite|task|num_fewshot`:
|
||||
- `leaderboard|mmlu|5`
|
||||
- `leaderboard|gsm8k|5`
|
||||
- `leaderboard|arc_challenge|25`
|
||||
- `lighteval|hellaswag|0`
|
||||
|
||||
Multiple `lighteval` tasks can be comma-separated in `--tasks`.
|
||||
|
||||
# Backend Selection
|
||||
|
||||
- Prefer `inspect_vllm_uv.py --backend vllm` for fast GPU inference on supported architectures.
|
||||
- Use `inspect_vllm_uv.py --backend hf` when `vllm` does not support the model.
|
||||
- Prefer `lighteval_vllm_uv.py --backend vllm` for throughput on supported models.
|
||||
- Use `lighteval_vllm_uv.py --backend accelerate` as the compatibility fallback.
|
||||
- Use `inspect_eval_uv.py` when Inference Providers already cover the model and you do not need direct GPU control.
|
||||
|
||||
# Hardware Guidance
|
||||
|
||||
| Model size | Suggested local hardware |
|
||||
|---|---|
|
||||
| `< 3B` | consumer GPU / Apple Silicon / small dev GPU |
|
||||
| `3B - 13B` | stronger local GPU |
|
||||
| `13B+` | high-memory local GPU or Hugging Face Jobs |
|
||||
|
||||
For smoke tests, prefer cheaper local runs plus `--limit` or `--max-samples`.
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
- CUDA or vLLM OOM:
|
||||
- reduce `--batch-size`
|
||||
- reduce `--gpu-memory-utilization`
|
||||
- switch to a smaller model for the smoke test
|
||||
- if necessary, submit the run to Hugging Face Jobs instead
|
||||
- Model unsupported by `vllm`:
|
||||
- switch to `--backend hf` for `inspect-ai`
|
||||
- switch to `--backend accelerate` for `lighteval`
|
||||
- Gated/private repo access fails:
|
||||
- verify `HF_TOKEN`
|
||||
- Custom model code required:
|
||||
- add `--trust-remote-code`
|
||||
|
||||
# Examples
|
||||
|
||||
See:
|
||||
- `examples/USAGE_EXAMPLES.md` for local command patterns
|
||||
- `scripts/inspect_eval_uv.py`
|
||||
- `scripts/inspect_vllm_uv.py`
|
||||
- `scripts/lighteval_vllm_uv.py`
|
||||
@@ -0,0 +1,3 @@
|
||||
# Hugging Face Token (required for gated/private models)
|
||||
# Get your token at: https://huggingface.co/settings/tokens
|
||||
HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
@@ -0,0 +1,101 @@
|
||||
# Usage Examples
|
||||
|
||||
This document provides practical examples for **running evaluations locally** against Hugging Face Hub models.
|
||||
|
||||
## What this skill covers
|
||||
|
||||
- `inspect-ai` local runs
|
||||
- `inspect-ai` with `vllm` or Transformers backends
|
||||
- `lighteval` local runs with `vllm` or `accelerate`
|
||||
- smoke tests and backend fallback patterns
|
||||
|
||||
## What this skill does NOT cover
|
||||
|
||||
- `model-index`
|
||||
- `.eval_results`
|
||||
- community eval publication workflows
|
||||
- model-card PR creation
|
||||
- Hugging Face Jobs orchestration
|
||||
|
||||
If you want to run these same scripts remotely, use the `hugging-face-jobs` skill and pass one of the scripts in `scripts/`.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
cd skills/hugging-face-evaluation
|
||||
export HF_TOKEN=hf_xxx
|
||||
uv --version
|
||||
```
|
||||
|
||||
For local GPU runs:
|
||||
|
||||
```bash
|
||||
nvidia-smi
|
||||
```
|
||||
|
||||
## inspect-ai examples
|
||||
|
||||
### Quick smoke test
|
||||
|
||||
```bash
|
||||
uv run scripts/inspect_eval_uv.py \
|
||||
--model meta-llama/Llama-3.2-1B \
|
||||
--task mmlu \
|
||||
--limit 10
|
||||
```
|
||||
|
||||
### Local GPU with vLLM
|
||||
|
||||
```bash
|
||||
uv run scripts/inspect_vllm_uv.py \
|
||||
--model meta-llama/Llama-3.2-8B-Instruct \
|
||||
--task gsm8k \
|
||||
--limit 20
|
||||
```
|
||||
|
||||
### Transformers fallback
|
||||
|
||||
```bash
|
||||
uv run scripts/inspect_vllm_uv.py \
|
||||
--model microsoft/phi-2 \
|
||||
--task mmlu \
|
||||
--backend hf \
|
||||
--trust-remote-code \
|
||||
--limit 20
|
||||
```
|
||||
|
||||
## lighteval examples
|
||||
|
||||
### Single task
|
||||
|
||||
```bash
|
||||
uv run scripts/lighteval_vllm_uv.py \
|
||||
--model meta-llama/Llama-3.2-3B-Instruct \
|
||||
--tasks "leaderboard|mmlu|5" \
|
||||
--max-samples 20
|
||||
```
|
||||
|
||||
### Multiple tasks
|
||||
|
||||
```bash
|
||||
uv run scripts/lighteval_vllm_uv.py \
|
||||
--model meta-llama/Llama-3.2-3B-Instruct \
|
||||
--tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
|
||||
--max-samples 20 \
|
||||
--use-chat-template
|
||||
```
|
||||
|
||||
### accelerate fallback
|
||||
|
||||
```bash
|
||||
uv run scripts/lighteval_vllm_uv.py \
|
||||
--model microsoft/phi-2 \
|
||||
--tasks "leaderboard|mmlu|5" \
|
||||
--backend accelerate \
|
||||
--trust-remote-code \
|
||||
--max-samples 20
|
||||
```
|
||||
|
||||
## Hand-off to Hugging Face Jobs
|
||||
|
||||
When local hardware is not enough, switch to the `hugging-face-jobs` skill and run one of these scripts remotely. Keep the script path and args; move the orchestration there.
|
||||
@@ -0,0 +1,104 @@
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "inspect-ai>=0.3.0",
|
||||
# "inspect-evals",
|
||||
# "openai",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
"""
|
||||
Entry point script for running inspect-ai evaluations against Hugging Face inference providers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def _inspect_evals_tasks_root() -> Optional[Path]:
|
||||
"""Return the installed inspect_evals package path if available."""
|
||||
try:
|
||||
import inspect_evals
|
||||
|
||||
return Path(inspect_evals.__file__).parent
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_task(task: str) -> str:
|
||||
"""Allow lighteval-style `suite|task|shots` strings by keeping the task name."""
|
||||
if "|" in task:
|
||||
parts = task.split("|")
|
||||
if len(parts) >= 2 and parts[1]:
|
||||
return parts[1]
|
||||
return task
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Inspect-ai job runner")
|
||||
parser.add_argument("--model", required=True, help="Model ID on Hugging Face Hub")
|
||||
parser.add_argument("--task", required=True, help="inspect-ai task to execute")
|
||||
parser.add_argument("--limit", type=int, default=None, help="Limit number of samples to evaluate")
|
||||
parser.add_argument(
|
||||
"--tasks-root",
|
||||
default=None,
|
||||
help="Optional path to inspect task files. Defaults to the installed inspect_evals package.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sandbox",
|
||||
default="local",
|
||||
help="Sandbox backend to use (default: local for HF jobs without Docker).",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Ensure downstream libraries can read the token passed as a secret
|
||||
hf_token = os.getenv("HF_TOKEN")
|
||||
if hf_token:
|
||||
os.environ.setdefault("HUGGING_FACE_HUB_TOKEN", hf_token)
|
||||
os.environ.setdefault("HF_HUB_TOKEN", hf_token)
|
||||
|
||||
task = _normalize_task(args.task)
|
||||
tasks_root = Path(args.tasks_root) if args.tasks_root else _inspect_evals_tasks_root()
|
||||
if tasks_root and not tasks_root.exists():
|
||||
tasks_root = None
|
||||
|
||||
cmd = [
|
||||
"inspect",
|
||||
"eval",
|
||||
task,
|
||||
"--model",
|
||||
f"hf-inference-providers/{args.model}",
|
||||
"--log-level",
|
||||
"info",
|
||||
# Reduce batch size to avoid OOM errors (default is 32)
|
||||
"--max-connections",
|
||||
"1",
|
||||
# Set a small positive temperature (HF doesn't allow temperature=0)
|
||||
"--temperature",
|
||||
"0.001",
|
||||
]
|
||||
|
||||
if args.sandbox:
|
||||
cmd.extend(["--sandbox", args.sandbox])
|
||||
|
||||
if args.limit:
|
||||
cmd.extend(["--limit", str(args.limit)])
|
||||
|
||||
try:
|
||||
subprocess.run(cmd, check=True, cwd=tasks_root)
|
||||
print("Evaluation complete.")
|
||||
except subprocess.CalledProcessError as exc:
|
||||
location = f" (cwd={tasks_root})" if tasks_root else ""
|
||||
print(f"Evaluation failed with exit code {exc.returncode}{location}", file=sys.stderr)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,306 @@
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "inspect-ai>=0.3.0",
|
||||
# "inspect-evals",
|
||||
# "vllm>=0.4.0",
|
||||
# "torch>=2.0.0",
|
||||
# "transformers>=4.40.0",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
"""
|
||||
Entry point script for running inspect-ai evaluations with vLLM or HuggingFace Transformers backend.
|
||||
|
||||
This script runs evaluations on custom HuggingFace models using local GPU inference,
|
||||
separate from inference provider scripts (which use external APIs).
|
||||
|
||||
Usage (standalone):
|
||||
uv run scripts/inspect_vllm_uv.py --model "meta-llama/Llama-3.2-1B" --task "mmlu"
|
||||
|
||||
Model backends:
|
||||
- vllm: Fast inference with vLLM (recommended for large models)
|
||||
- hf: HuggingFace Transformers backend (broader model compatibility)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def setup_environment() -> None:
|
||||
"""Configure environment variables for HuggingFace authentication."""
|
||||
hf_token = os.getenv("HF_TOKEN")
|
||||
if hf_token:
|
||||
os.environ.setdefault("HUGGING_FACE_HUB_TOKEN", hf_token)
|
||||
os.environ.setdefault("HF_HUB_TOKEN", hf_token)
|
||||
|
||||
|
||||
def run_inspect_vllm(
|
||||
model_id: str,
|
||||
task: str,
|
||||
limit: Optional[int] = None,
|
||||
max_connections: int = 4,
|
||||
temperature: float = 0.0,
|
||||
tensor_parallel_size: int = 1,
|
||||
gpu_memory_utilization: float = 0.8,
|
||||
dtype: str = "auto",
|
||||
trust_remote_code: bool = False,
|
||||
log_level: str = "info",
|
||||
) -> None:
|
||||
"""
|
||||
Run inspect-ai evaluation with vLLM backend.
|
||||
|
||||
Args:
|
||||
model_id: HuggingFace model ID
|
||||
task: inspect-ai task to execute (e.g., "mmlu", "gsm8k")
|
||||
limit: Limit number of samples to evaluate
|
||||
max_connections: Maximum concurrent connections
|
||||
temperature: Sampling temperature
|
||||
tensor_parallel_size: Number of GPUs for tensor parallelism
|
||||
gpu_memory_utilization: GPU memory fraction
|
||||
dtype: Data type (auto, float16, bfloat16)
|
||||
trust_remote_code: Allow remote code execution
|
||||
log_level: Logging level
|
||||
"""
|
||||
setup_environment()
|
||||
|
||||
model_spec = f"vllm/{model_id}"
|
||||
cmd = [
|
||||
"inspect",
|
||||
"eval",
|
||||
task,
|
||||
"--model",
|
||||
model_spec,
|
||||
"--log-level",
|
||||
log_level,
|
||||
"--max-connections",
|
||||
str(max_connections),
|
||||
]
|
||||
|
||||
# vLLM supports temperature=0 unlike HF inference providers
|
||||
cmd.extend(["--temperature", str(temperature)])
|
||||
|
||||
# Older inspect-ai CLI versions do not support --model-args; rely on defaults
|
||||
# and let vLLM choose sensible settings for small models.
|
||||
if tensor_parallel_size != 1:
|
||||
cmd.extend(["--tensor-parallel-size", str(tensor_parallel_size)])
|
||||
if gpu_memory_utilization != 0.8:
|
||||
cmd.extend(["--gpu-memory-utilization", str(gpu_memory_utilization)])
|
||||
if dtype != "auto":
|
||||
cmd.extend(["--dtype", dtype])
|
||||
if trust_remote_code:
|
||||
cmd.append("--trust-remote-code")
|
||||
|
||||
if limit:
|
||||
cmd.extend(["--limit", str(limit)])
|
||||
|
||||
print(f"Running: {' '.join(cmd)}")
|
||||
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print("Evaluation complete.")
|
||||
except subprocess.CalledProcessError as exc:
|
||||
print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr)
|
||||
sys.exit(exc.returncode)
|
||||
|
||||
|
||||
def run_inspect_hf(
|
||||
model_id: str,
|
||||
task: str,
|
||||
limit: Optional[int] = None,
|
||||
max_connections: int = 1,
|
||||
temperature: float = 0.001,
|
||||
device: str = "auto",
|
||||
dtype: str = "auto",
|
||||
trust_remote_code: bool = False,
|
||||
log_level: str = "info",
|
||||
) -> None:
|
||||
"""
|
||||
Run inspect-ai evaluation with HuggingFace Transformers backend.
|
||||
|
||||
Use this when vLLM doesn't support the model architecture.
|
||||
|
||||
Args:
|
||||
model_id: HuggingFace model ID
|
||||
task: inspect-ai task to execute
|
||||
limit: Limit number of samples
|
||||
max_connections: Maximum concurrent connections (keep low for memory)
|
||||
temperature: Sampling temperature
|
||||
device: Device to use (auto, cuda, cpu)
|
||||
dtype: Data type
|
||||
trust_remote_code: Allow remote code execution
|
||||
log_level: Logging level
|
||||
"""
|
||||
setup_environment()
|
||||
|
||||
model_spec = f"hf/{model_id}"
|
||||
|
||||
cmd = [
|
||||
"inspect",
|
||||
"eval",
|
||||
task,
|
||||
"--model",
|
||||
model_spec,
|
||||
"--log-level",
|
||||
log_level,
|
||||
"--max-connections",
|
||||
str(max_connections),
|
||||
"--temperature",
|
||||
str(temperature),
|
||||
]
|
||||
|
||||
if device != "auto":
|
||||
cmd.extend(["--device", device])
|
||||
if dtype != "auto":
|
||||
cmd.extend(["--dtype", dtype])
|
||||
if trust_remote_code:
|
||||
cmd.append("--trust-remote-code")
|
||||
|
||||
if limit:
|
||||
cmd.extend(["--limit", str(limit)])
|
||||
|
||||
print(f"Running: {' '.join(cmd)}")
|
||||
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print("Evaluation complete.")
|
||||
except subprocess.CalledProcessError as exc:
|
||||
print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr)
|
||||
sys.exit(exc.returncode)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run inspect-ai evaluations with vLLM or HuggingFace Transformers on custom models",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Run MMLU with vLLM backend
|
||||
uv run scripts/inspect_vllm_uv.py --model meta-llama/Llama-3.2-1B --task mmlu
|
||||
|
||||
# Run with HuggingFace Transformers backend
|
||||
uv run scripts/inspect_vllm_uv.py --model meta-llama/Llama-3.2-1B --task mmlu --backend hf
|
||||
|
||||
# Run with limited samples for testing
|
||||
uv run scripts/inspect_vllm_uv.py --model meta-llama/Llama-3.2-1B --task mmlu --limit 10
|
||||
|
||||
# Run on multiple GPUs with tensor parallelism
|
||||
uv run scripts/inspect_vllm_uv.py --model meta-llama/Llama-3.2-70B --task mmlu --tensor-parallel-size 4
|
||||
|
||||
Available tasks (from inspect-evals):
|
||||
- mmlu: Massive Multitask Language Understanding
|
||||
- gsm8k: Grade School Math
|
||||
- hellaswag: Common sense reasoning
|
||||
- arc_challenge: AI2 Reasoning Challenge
|
||||
- truthfulqa: TruthfulQA benchmark
|
||||
- winogrande: Winograd Schema Challenge
|
||||
- humaneval: Code generation (HumanEval)
|
||||
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
required=True,
|
||||
help="HuggingFace model ID (e.g., meta-llama/Llama-3.2-1B)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--task",
|
||||
required=True,
|
||||
help="inspect-ai task to execute (e.g., mmlu, gsm8k)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--backend",
|
||||
choices=["vllm", "hf"],
|
||||
default="vllm",
|
||||
help="Model backend (default: vllm)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Limit number of samples to evaluate",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-connections",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Maximum concurrent connections (default: 4 for vllm, 1 for hf)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--temperature",
|
||||
type=float,
|
||||
default=None,
|
||||
help="Sampling temperature (default: 0.0 for vllm, 0.001 for hf)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tensor-parallel-size",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of GPUs for tensor parallelism (vLLM only, default: 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--gpu-memory-utilization",
|
||||
type=float,
|
||||
default=0.8,
|
||||
help="GPU memory fraction to use (vLLM only, default: 0.8)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dtype",
|
||||
default="auto",
|
||||
choices=["auto", "float16", "bfloat16", "float32"],
|
||||
help="Data type for model weights (default: auto)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--device",
|
||||
default="auto",
|
||||
help="Device for HF backend (auto, cuda, cpu)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--trust-remote-code",
|
||||
action="store_true",
|
||||
help="Allow executing remote code from model repository",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-level",
|
||||
default="info",
|
||||
choices=["debug", "info", "warning", "error"],
|
||||
help="Logging level (default: info)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.backend == "vllm":
|
||||
run_inspect_vllm(
|
||||
model_id=args.model,
|
||||
task=args.task,
|
||||
limit=args.limit,
|
||||
max_connections=args.max_connections or 4,
|
||||
temperature=args.temperature if args.temperature is not None else 0.0,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
gpu_memory_utilization=args.gpu_memory_utilization,
|
||||
dtype=args.dtype,
|
||||
trust_remote_code=args.trust_remote_code,
|
||||
log_level=args.log_level,
|
||||
)
|
||||
else:
|
||||
run_inspect_hf(
|
||||
model_id=args.model,
|
||||
task=args.task,
|
||||
limit=args.limit,
|
||||
max_connections=args.max_connections or 1,
|
||||
temperature=args.temperature if args.temperature is not None else 0.001,
|
||||
device=args.device,
|
||||
dtype=args.dtype,
|
||||
trust_remote_code=args.trust_remote_code,
|
||||
log_level=args.log_level,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,297 @@
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "lighteval[accelerate,vllm]>=0.6.0",
|
||||
# "torch>=2.0.0",
|
||||
# "transformers>=4.40.0",
|
||||
# "accelerate>=0.30.0",
|
||||
# "vllm>=0.4.0",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
"""
|
||||
Entry point script for running lighteval evaluations with local GPU backends.
|
||||
|
||||
This script runs evaluations using vLLM or accelerate on custom HuggingFace models.
|
||||
It is separate from inference provider scripts and evaluates models directly on local hardware.
|
||||
|
||||
Usage (standalone):
|
||||
uv run scripts/lighteval_vllm_uv.py --model "meta-llama/Llama-3.2-1B" --tasks "leaderboard|mmlu|5"
|
||||
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def setup_environment() -> None:
|
||||
"""Configure environment variables for HuggingFace authentication."""
|
||||
hf_token = os.getenv("HF_TOKEN")
|
||||
if hf_token:
|
||||
os.environ.setdefault("HUGGING_FACE_HUB_TOKEN", hf_token)
|
||||
os.environ.setdefault("HF_HUB_TOKEN", hf_token)
|
||||
|
||||
|
||||
def run_lighteval_vllm(
|
||||
model_id: str,
|
||||
tasks: str,
|
||||
output_dir: Optional[str] = None,
|
||||
max_samples: Optional[int] = None,
|
||||
batch_size: int = 1,
|
||||
tensor_parallel_size: int = 1,
|
||||
gpu_memory_utilization: float = 0.8,
|
||||
dtype: str = "auto",
|
||||
trust_remote_code: bool = False,
|
||||
use_chat_template: bool = False,
|
||||
system_prompt: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Run lighteval with vLLM backend for efficient GPU inference.
|
||||
|
||||
Args:
|
||||
model_id: HuggingFace model ID (e.g., "meta-llama/Llama-3.2-1B")
|
||||
tasks: Task specification (e.g., "leaderboard|mmlu|5" or "lighteval|hellaswag|0")
|
||||
output_dir: Directory for evaluation results
|
||||
max_samples: Limit number of samples per task
|
||||
batch_size: Batch size for evaluation
|
||||
tensor_parallel_size: Number of GPUs for tensor parallelism
|
||||
gpu_memory_utilization: GPU memory fraction to use (0.0-1.0)
|
||||
dtype: Data type for model weights (auto, float16, bfloat16)
|
||||
trust_remote_code: Allow executing remote code from model repo
|
||||
use_chat_template: Apply chat template for conversational models
|
||||
system_prompt: System prompt for chat models
|
||||
"""
|
||||
setup_environment()
|
||||
|
||||
# Build lighteval vllm command
|
||||
cmd = [
|
||||
"lighteval",
|
||||
"vllm",
|
||||
model_id,
|
||||
tasks,
|
||||
"--batch-size", str(batch_size),
|
||||
"--tensor-parallel-size", str(tensor_parallel_size),
|
||||
"--gpu-memory-utilization", str(gpu_memory_utilization),
|
||||
"--dtype", dtype,
|
||||
]
|
||||
|
||||
if output_dir:
|
||||
cmd.extend(["--output-dir", output_dir])
|
||||
|
||||
if max_samples:
|
||||
cmd.extend(["--max-samples", str(max_samples)])
|
||||
|
||||
if trust_remote_code:
|
||||
cmd.append("--trust-remote-code")
|
||||
|
||||
if use_chat_template:
|
||||
cmd.append("--use-chat-template")
|
||||
|
||||
if system_prompt:
|
||||
cmd.extend(["--system-prompt", system_prompt])
|
||||
|
||||
print(f"Running: {' '.join(cmd)}")
|
||||
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print("Evaluation complete.")
|
||||
except subprocess.CalledProcessError as exc:
|
||||
print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr)
|
||||
sys.exit(exc.returncode)
|
||||
|
||||
|
||||
def run_lighteval_accelerate(
|
||||
model_id: str,
|
||||
tasks: str,
|
||||
output_dir: Optional[str] = None,
|
||||
max_samples: Optional[int] = None,
|
||||
batch_size: int = 1,
|
||||
dtype: str = "bfloat16",
|
||||
trust_remote_code: bool = False,
|
||||
use_chat_template: bool = False,
|
||||
system_prompt: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Run lighteval with accelerate backend for multi-GPU distributed inference.
|
||||
|
||||
Use this backend when vLLM is not available or for models not supported by vLLM.
|
||||
|
||||
Args:
|
||||
model_id: HuggingFace model ID
|
||||
tasks: Task specification
|
||||
output_dir: Directory for evaluation results
|
||||
max_samples: Limit number of samples per task
|
||||
batch_size: Batch size for evaluation
|
||||
dtype: Data type for model weights
|
||||
trust_remote_code: Allow executing remote code
|
||||
use_chat_template: Apply chat template
|
||||
system_prompt: System prompt for chat models
|
||||
"""
|
||||
setup_environment()
|
||||
|
||||
# Build lighteval accelerate command
|
||||
cmd = [
|
||||
"lighteval",
|
||||
"accelerate",
|
||||
model_id,
|
||||
tasks,
|
||||
"--batch-size", str(batch_size),
|
||||
"--dtype", dtype,
|
||||
]
|
||||
|
||||
if output_dir:
|
||||
cmd.extend(["--output-dir", output_dir])
|
||||
|
||||
if max_samples:
|
||||
cmd.extend(["--max-samples", str(max_samples)])
|
||||
|
||||
if trust_remote_code:
|
||||
cmd.append("--trust-remote-code")
|
||||
|
||||
if use_chat_template:
|
||||
cmd.append("--use-chat-template")
|
||||
|
||||
if system_prompt:
|
||||
cmd.extend(["--system-prompt", system_prompt])
|
||||
|
||||
print(f"Running: {' '.join(cmd)}")
|
||||
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print("Evaluation complete.")
|
||||
except subprocess.CalledProcessError as exc:
|
||||
print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr)
|
||||
sys.exit(exc.returncode)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run lighteval evaluations with vLLM or accelerate backend on custom HuggingFace models",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Run MMLU evaluation with vLLM
|
||||
uv run scripts/lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B --tasks "leaderboard|mmlu|5"
|
||||
|
||||
# Run with accelerate backend instead of vLLM
|
||||
uv run scripts/lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B --tasks "leaderboard|mmlu|5" --backend accelerate
|
||||
|
||||
# Run with chat template for instruction-tuned models
|
||||
uv run scripts/lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B-Instruct --tasks "leaderboard|mmlu|5" --use-chat-template
|
||||
|
||||
# Run with limited samples for testing
|
||||
uv run scripts/lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B --tasks "leaderboard|mmlu|5" --max-samples 10
|
||||
|
||||
Task format:
|
||||
Tasks use the format: "suite|task|num_fewshot"
|
||||
- leaderboard|mmlu|5 (MMLU with 5-shot)
|
||||
- lighteval|hellaswag|0 (HellaSwag zero-shot)
|
||||
- leaderboard|gsm8k|5 (GSM8K with 5-shot)
|
||||
- Multiple tasks: "leaderboard|mmlu|5,leaderboard|gsm8k|5"
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
required=True,
|
||||
help="HuggingFace model ID (e.g., meta-llama/Llama-3.2-1B)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tasks",
|
||||
required=True,
|
||||
help="Task specification (e.g., 'leaderboard|mmlu|5')",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--backend",
|
||||
choices=["vllm", "accelerate"],
|
||||
default="vllm",
|
||||
help="Inference backend to use (default: vllm)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
default=None,
|
||||
help="Directory for evaluation results",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-samples",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Limit number of samples per task (useful for testing)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch-size",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Batch size for evaluation (default: 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tensor-parallel-size",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of GPUs for tensor parallelism (vLLM only, default: 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--gpu-memory-utilization",
|
||||
type=float,
|
||||
default=0.8,
|
||||
help="GPU memory fraction to use (vLLM only, default: 0.8)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dtype",
|
||||
default="auto",
|
||||
choices=["auto", "float16", "bfloat16", "float32"],
|
||||
help="Data type for model weights (default: auto)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--trust-remote-code",
|
||||
action="store_true",
|
||||
help="Allow executing remote code from model repository",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--use-chat-template",
|
||||
action="store_true",
|
||||
help="Apply chat template for instruction-tuned/chat models",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--system-prompt",
|
||||
default=None,
|
||||
help="System prompt for chat models",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.backend == "vllm":
|
||||
run_lighteval_vllm(
|
||||
model_id=args.model,
|
||||
tasks=args.tasks,
|
||||
output_dir=args.output_dir,
|
||||
max_samples=args.max_samples,
|
||||
batch_size=args.batch_size,
|
||||
tensor_parallel_size=args.tensor_parallel_size,
|
||||
gpu_memory_utilization=args.gpu_memory_utilization,
|
||||
dtype=args.dtype,
|
||||
trust_remote_code=args.trust_remote_code,
|
||||
use_chat_template=args.use_chat_template,
|
||||
system_prompt=args.system_prompt,
|
||||
)
|
||||
else:
|
||||
run_lighteval_accelerate(
|
||||
model_id=args.model,
|
||||
tasks=args.tasks,
|
||||
output_dir=args.output_dir,
|
||||
max_samples=args.max_samples,
|
||||
batch_size=args.batch_size,
|
||||
dtype=args.dtype if args.dtype != "auto" else "bfloat16",
|
||||
trust_remote_code=args.trust_remote_code,
|
||||
use_chat_template=args.use_chat_template,
|
||||
system_prompt=args.system_prompt,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: huggingface-datasets
|
||||
description: Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Hugging Face Dataset Viewer
|
||||
|
||||
Use this skill to execute read-only Dataset Viewer API calls for dataset exploration and extraction.
|
||||
|
||||
## Core workflow
|
||||
|
||||
1. Optionally validate dataset availability with `/is-valid`.
|
||||
2. Resolve `config` + `split` with `/splits`.
|
||||
3. Preview with `/first-rows`.
|
||||
4. Paginate content with `/rows` using `offset` and `length` (max 100).
|
||||
5. Use `/search` for text matching and `/filter` for row predicates.
|
||||
6. Retrieve parquet links via `/parquet` and totals/metadata via `/size` and `/statistics`.
|
||||
|
||||
## Defaults
|
||||
|
||||
- Base URL: `https://datasets-server.huggingface.co`
|
||||
- Default API method: `GET`
|
||||
- Query params should be URL-encoded.
|
||||
- `offset` is 0-based.
|
||||
- `length` max is usually `100` for row-like endpoints.
|
||||
- Gated/private datasets require `Authorization: Bearer <HF_TOKEN>`.
|
||||
|
||||
## Dataset Viewer
|
||||
|
||||
- `Validate dataset`: `/is-valid?dataset=<namespace/repo>`
|
||||
- `List subsets and splits`: `/splits?dataset=<namespace/repo>`
|
||||
- `Preview first rows`: `/first-rows?dataset=<namespace/repo>&config=<config>&split=<split>`
|
||||
- `Paginate rows`: `/rows?dataset=<namespace/repo>&config=<config>&split=<split>&offset=<int>&length=<int>`
|
||||
- `Search text`: `/search?dataset=<namespace/repo>&config=<config>&split=<split>&query=<text>&offset=<int>&length=<int>`
|
||||
- `Filter with predicates`: `/filter?dataset=<namespace/repo>&config=<config>&split=<split>&where=<predicate>&orderby=<sort>&offset=<int>&length=<int>`
|
||||
- `List parquet shards`: `/parquet?dataset=<namespace/repo>`
|
||||
- `Get size totals`: `/size?dataset=<namespace/repo>`
|
||||
- `Get column statistics`: `/statistics?dataset=<namespace/repo>&config=<config>&split=<split>`
|
||||
- `Get Croissant metadata (if available)`: `/croissant?dataset=<namespace/repo>`
|
||||
|
||||
Pagination pattern:
|
||||
|
||||
```bash
|
||||
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100"
|
||||
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100"
|
||||
```
|
||||
|
||||
When pagination is partial, use response fields such as `num_rows_total`, `num_rows_per_page`, and `partial` to drive continuation logic.
|
||||
|
||||
Search/filter notes:
|
||||
|
||||
- `/search` matches string columns (full-text style behavior is internal to the API).
|
||||
- `/filter` requires predicate syntax in `where` and optional sort in `orderby`.
|
||||
- Keep filtering and searches read-only and side-effect free.
|
||||
|
||||
For CLI-based parquet URL discovery or SQL, use the `hf-cli` skill with `hf datasets parquet` and `hf datasets sql`.
|
||||
|
||||
## Creating and Uploading Datasets
|
||||
|
||||
Use one of these flows depending on dependency constraints.
|
||||
|
||||
Zero local dependencies (Hub UI):
|
||||
|
||||
- Create dataset repo in browser: `https://huggingface.co/new-dataset`
|
||||
- Upload parquet files in the repo "Files and versions" page.
|
||||
- Verify shards appear in Dataset Viewer:
|
||||
|
||||
```bash
|
||||
curl -s "https://datasets-server.huggingface.co/parquet?dataset=<namespace>/<repo>"
|
||||
```
|
||||
|
||||
Low dependency CLI flow (`npx @huggingface/hub` / `hfjs`):
|
||||
|
||||
- Set auth token:
|
||||
|
||||
```bash
|
||||
export HF_TOKEN=<your_hf_token>
|
||||
```
|
||||
|
||||
- Upload parquet folder to a dataset repo (auto-creates repo if missing):
|
||||
|
||||
```bash
|
||||
npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data
|
||||
```
|
||||
|
||||
- Upload as private repo on creation:
|
||||
|
||||
```bash
|
||||
npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data --private
|
||||
```
|
||||
|
||||
After upload, call `/parquet` to discover `<config>/<split>/<shard>` values for querying with `@~parquet`.
|
||||
|
||||
## Agent Traces
|
||||
|
||||
The Hub supports raw agent session traces from Claude Code, Codex, and Pi Agent. Upload them to Hugging Face Datasets as original JSONL files and the Hub can auto-detect the trace format, tag the dataset as `Traces`, and enable the trace viewer for browsing sessions, turns, tool calls, and model responses. Common local session directories:
|
||||
|
||||
- Claude Code: `~/.claude/projects`
|
||||
- Codex: `~/.codex/sessions`
|
||||
- Pi: `~/.pi/agent/sessions`
|
||||
|
||||
Default to private dataset repos because traces can contain prompts, file paths, tool outputs, secrets, or PII. Preserve the raw `.jsonl` files and nest them by project/cwd instead of uploading every session at the dataset root.
|
||||
|
||||
```bash
|
||||
hf repos create <namespace>/<repo> --type dataset --private --exist-ok
|
||||
hf upload <namespace>/<repo> ~/.codex/sessions codex/<project-or-cwd> --type dataset
|
||||
```
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
name: huggingface-gradio
|
||||
description: Build Gradio web UIs and demos in Python. Use when creating or editing Gradio apps, components, event listeners, layouts, or chatbots.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Gradio
|
||||
|
||||
Gradio is a Python library for building interactive web UIs and ML demos. This skill covers the core API, patterns, and examples.
|
||||
|
||||
## Guides
|
||||
|
||||
Detailed guides on specific topics (read these when relevant):
|
||||
|
||||
- [Quickstart](https://www.gradio.app/guides/quickstart)
|
||||
- [The Interface Class](https://www.gradio.app/guides/the-interface-class)
|
||||
- [Blocks and Event Listeners](https://www.gradio.app/guides/blocks-and-event-listeners)
|
||||
- [Controlling Layout](https://www.gradio.app/guides/controlling-layout)
|
||||
- [More Blocks Features](https://www.gradio.app/guides/more-blocks-features)
|
||||
- [Custom CSS and JS](https://www.gradio.app/guides/custom-CSS-and-JS)
|
||||
- [Streaming Outputs](https://www.gradio.app/guides/streaming-outputs)
|
||||
- [Streaming Inputs](https://www.gradio.app/guides/streaming-inputs)
|
||||
- [Sharing Your App](https://www.gradio.app/guides/sharing-your-app)
|
||||
- [Custom HTML Components](https://www.gradio.app/guides/custom-HTML-components)
|
||||
- [Getting Started with the Python Client](https://www.gradio.app/guides/getting-started-with-the-python-client)
|
||||
- [Getting Started with the JS Client](https://www.gradio.app/guides/getting-started-with-the-js-client)
|
||||
|
||||
## Core Patterns
|
||||
|
||||
**Interface** (high-level): wraps a function with input/output components.
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
def greet(name):
|
||||
return f"Hello {name}!"
|
||||
|
||||
gr.Interface(fn=greet, inputs="text", outputs="text").launch()
|
||||
```
|
||||
|
||||
**Blocks** (low-level): flexible layout with explicit event wiring.
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
name = gr.Textbox(label="Name")
|
||||
output = gr.Textbox(label="Greeting")
|
||||
btn = gr.Button("Greet")
|
||||
btn.click(fn=lambda n: f"Hello {n}!", inputs=name, outputs=output)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
**ChatInterface**: high-level wrapper for chatbot UIs.
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
def respond(message, history):
|
||||
return f"You said: {message}"
|
||||
|
||||
gr.ChatInterface(fn=respond).launch()
|
||||
```
|
||||
|
||||
## Key Component Signatures
|
||||
|
||||
### `Textbox(value: str | I18nData | Callable | None = None, type: Literal['text', 'password', 'email'] = "text", lines: int = 1, max_lines: int | None = None, placeholder: str | I18nData | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, autofocus: bool = False, autoscroll: bool = True, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", text_align: Literal['left', 'right'] | None = None, rtl: bool = False, buttons: list[Literal['copy'] | Button] | None = None, max_length: int | None = None, submit_btn: str | bool | None = False, stop_btn: str | bool | None = False, html_attributes: InputHTMLAttributes | None = None)`
|
||||
Creates a textarea for user to enter string input or display string output..
|
||||
|
||||
### `Number(value: float | Callable | None = None, label: str | I18nData | None = None, placeholder: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None, precision: int | None = None, minimum: float | None = None, maximum: float | None = None, step: float = 1)`
|
||||
Creates a numeric field for user to enter numbers as input or display numeric output..
|
||||
|
||||
### `Slider(minimum: float = 0, maximum: float = 100, value: float | Callable | None = None, step: float | None = None, precision: int | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", randomize: bool = False, buttons: list[Literal['reset']] | None = None)`
|
||||
Creates a slider that ranges from {minimum} to {maximum} with a step size of {step}..
|
||||
|
||||
### `Checkbox(value: bool | Callable = False, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None)`
|
||||
Creates a checkbox that can be set to `True` or `False`.
|
||||
|
||||
### `Dropdown(choices: Sequence[str | int | float | tuple[str, str | int | float]] | None = None, value: str | int | float | Sequence[str | int | float] | Callable | DefaultValue | None = DefaultValue(), type: Literal['value', 'index'] = "value", multiselect: bool | None = None, allow_custom_value: bool = False, max_choices: int | None = None, filterable: bool = True, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", buttons: list[Button] | None = None)`
|
||||
Creates a dropdown of choices from which a single entry or multiple entries can be selected (as an input component) or displayed (as an output component)..
|
||||
|
||||
### `Radio(choices: Sequence[str | int | float | tuple[str, str | int | float]] | None = None, value: str | int | float | Callable | None = None, type: Literal['value', 'index'] = "value", label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", rtl: bool = False, buttons: list[Button] | None = None)`
|
||||
Creates a set of (string or numeric type) radio buttons of which only one can be selected..
|
||||
|
||||
### `Image(value: str | PIL.Image.Image | np.ndarray | Callable | None = None, format: str = "webp", height: int | str | None = None, width: int | str | None = None, image_mode: Literal['1', 'L', 'P', 'RGB', 'RGBA', 'CMYK', 'YCbCr', 'LAB', 'HSV', 'I', 'F'] | None = "RGB", sources: list[Literal['upload', 'webcam', 'clipboard']] | Literal['upload', 'webcam', 'clipboard'] | None = None, type: Literal['numpy', 'pil', 'filepath'] = "numpy", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, buttons: list[Literal['download', 'share', 'fullscreen'] | Button] | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", webcam_options: WebcamOptions | None = None, placeholder: str | None = None, watermark: WatermarkOptions | None = None)`
|
||||
Creates an image component that can be used to upload images (as an input) or display images (as an output)..
|
||||
|
||||
### `Audio(value: str | Path | tuple[int, np.ndarray] | Callable | None = None, sources: list[Literal['upload', 'microphone']] | Literal['upload', 'microphone'] | None = None, type: Literal['numpy', 'filepath'] = "numpy", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", format: Literal['wav', 'mp3'] | None = None, autoplay: bool = False, editable: bool = True, buttons: list[Literal['download', 'share'] | Button] | None = None, waveform_options: WaveformOptions | dict | None = None, loop: bool = False, recording: bool = False, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)`
|
||||
Creates an audio component that can be used to upload/record audio (as an input) or display audio (as an output)..
|
||||
|
||||
### `Video(value: str | Path | Callable | None = None, format: str | None = None, sources: list[Literal['upload', 'webcam']] | Literal['upload', 'webcam'] | None = None, height: int | str | None = None, width: int | str | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", webcam_options: WebcamOptions | None = None, include_audio: bool | None = None, autoplay: bool = False, buttons: list[Literal['download', 'share'] | Button] | None = None, loop: bool = False, streaming: bool = False, watermark: WatermarkOptions | None = None, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)`
|
||||
Creates a video component that can be used to upload/record videos (as an input) or display videos (as an output).
|
||||
|
||||
### `File(value: str | list[str] | Callable | None = None, file_count: Literal['single', 'multiple', 'directory'] = "single", file_types: list[str] | None = None, type: Literal['filepath', 'binary'] = "filepath", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, height: int | str | float | None = None, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", allow_reordering: bool = False, buttons: list[Button] | None = None)`
|
||||
Creates a file component that allows uploading one or more generic files (when used as an input) or displaying generic files or URLs for download (as output).
|
||||
|
||||
### `Chatbot(value: list[MessageDict | Message] | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, autoscroll: bool = True, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", height: int | str | None = 400, resizable: bool = False, max_height: int | str | None = None, min_height: int | str | None = None, editable: Literal['user', 'all'] | None = None, latex_delimiters: list[dict[str, str | bool]] | None = None, rtl: bool = False, buttons: list[Literal['share', 'copy', 'copy_all'] | Button] | None = None, watermark: str | None = None, avatar_images: tuple[str | Path | None, str | Path | None] | None = None, sanitize_html: bool = True, render_markdown: bool = True, feedback_options: list[str] | tuple[str, ...] | None = ('Like', 'Dislike'), feedback_value: Sequence[str | None] | None = None, line_breaks: bool = True, layout: Literal['panel', 'bubble'] | None = None, placeholder: str | None = None, examples: list[ExampleMessage] | None = None, allow_file_downloads: <class 'inspect._empty'> = True, group_consecutive_messages: bool = True, allow_tags: list[str] | bool = True, reasoning_tags: list[tuple[str, str]] | None = None, like_user_message: bool = False)`
|
||||
Creates a chatbot that displays user-submitted messages and responses.
|
||||
|
||||
### `Button(value: str | I18nData | Callable = "Run", every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, variant: Literal['primary', 'secondary', 'stop', 'huggingface'] = "secondary", size: Literal['sm', 'md', 'lg'] = "lg", icon: str | Path | None = None, link: str | None = None, link_target: Literal['_self', '_blank', '_parent', '_top'] = "_self", visible: bool | Literal['hidden'] = True, interactive: bool = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", scale: int | None = None, min_width: int | None = None)`
|
||||
Creates a button that can be assigned arbitrary .click() events.
|
||||
|
||||
### `Markdown(value: str | I18nData | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, rtl: bool = False, latex_delimiters: list[dict[str, str | bool]] | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", sanitize_html: bool = True, line_breaks: bool = False, header_links: bool = False, height: int | str | None = None, max_height: int | str | None = None, min_height: int | str | None = None, buttons: list[Literal['copy']] | None = None, container: bool = False, padding: bool = False)`
|
||||
Used to render arbitrary Markdown output.
|
||||
|
||||
### `HTML(value: Any | Callable | None = None, label: str | I18nData | None = None, html_template: str = "${value}", css_template: str = "", js_on_load: str | None = "element.addEventListener('click', function() { trigger('click') });", apply_default_css: bool = True, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool = False, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = "value", min_height: int | None = None, max_height: int | None = None, container: bool = False, padding: bool = False, autoscroll: bool = False, buttons: list[Button] | None = None, server_functions: list[Callable] | None = None, props: Any)`
|
||||
Creates a component with arbitrary HTML.
|
||||
|
||||
|
||||
## Custom HTML Components
|
||||
|
||||
If a task requires significant customization of an existing component or a component that doesn't exist in Gradio, you can create one with `gr.HTML`. It supports `html_template` (with `${}` JS expressions and `{{}}` Handlebars syntax), `css_template` for scoped styles, and `js_on_load` for interactivity — where `props.value` updates the component value and `trigger('event_name')` fires Gradio events. For reuse, subclass `gr.HTML` and define `api_info()` for API/MCP support. See the [full guide](https://www.gradio.app/guides/custom-HTML-components).
|
||||
|
||||
Here's an example that shows how to create and use these kinds of components:
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
class StarRating(gr.HTML):
|
||||
def __init__(self, label, value=0, **kwargs):
|
||||
html_template = """
|
||||
<h2>${label} rating:</h2>
|
||||
${Array.from({length: 5}, (_, i) => `<img class='${i < value ? '' : 'faded'}' src='https://upload.wikimedia.org/wikipedia/commons/d/df/Award-star-gold-3d.svg'>`).join('')}
|
||||
"""
|
||||
css_template = """
|
||||
img { height: 50px; display: inline-block; cursor: pointer; }
|
||||
.faded { filter: grayscale(100%); opacity: 0.3; }
|
||||
"""
|
||||
js_on_load = """
|
||||
const imgs = element.querySelectorAll('img');
|
||||
imgs.forEach((img, index) => {
|
||||
img.addEventListener('click', () => {
|
||||
props.value = index + 1;
|
||||
});
|
||||
});
|
||||
"""
|
||||
super().__init__(value=value, label=label, html_template=html_template, css_template=css_template, js_on_load=js_on_load, **kwargs)
|
||||
|
||||
def api_info(self):
|
||||
return {"type": "integer", "minimum": 0, "maximum": 5}
|
||||
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
gr.Markdown("# Restaurant Review")
|
||||
food_rating = StarRating(label="Food", value=3)
|
||||
service_rating = StarRating(label="Service", value=3)
|
||||
ambience_rating = StarRating(label="Ambience", value=3)
|
||||
average_btn = gr.Button("Calculate Average Rating")
|
||||
rating_output = StarRating(label="Average", value=3)
|
||||
def calculate_average(food, service, ambience):
|
||||
return round((food + service + ambience) / 3)
|
||||
average_btn.click(
|
||||
fn=calculate_average,
|
||||
inputs=[food_rating, service_rating, ambience_rating],
|
||||
outputs=rating_output
|
||||
)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Event Listeners
|
||||
|
||||
All event listeners share the same signature:
|
||||
|
||||
```python
|
||||
component.event_name(
|
||||
fn: Callable | None | Literal["decorator"] = "decorator",
|
||||
inputs: Component | Sequence[Component] | set[Component] | None = None,
|
||||
outputs: Component | Sequence[Component] | set[Component] | None = None,
|
||||
api_name: str | None = None,
|
||||
api_description: str | None | Literal[False] = None,
|
||||
scroll_to_output: bool = False,
|
||||
show_progress: Literal["full", "minimal", "hidden"] = "full",
|
||||
show_progress_on: Component | Sequence[Component] | None = None,
|
||||
queue: bool = True,
|
||||
batch: bool = False,
|
||||
max_batch_size: int = 4,
|
||||
preprocess: bool = True,
|
||||
postprocess: bool = True,
|
||||
cancels: dict[str, Any] | list[dict[str, Any]] | None = None,
|
||||
trigger_mode: Literal["once", "multiple", "always_last"] | None = None,
|
||||
js: str | Literal[True] | None = None,
|
||||
concurrency_limit: int | None | Literal["default"] = "default",
|
||||
concurrency_id: str | None = None,
|
||||
api_visibility: Literal["public", "private", "undocumented"] = "public",
|
||||
time_limit: int | None = None,
|
||||
stream_every: float = 0.5,
|
||||
key: int | str | tuple[int | str, ...] | None = None,
|
||||
validator: Callable | None = None,
|
||||
) -> Dependency
|
||||
```
|
||||
|
||||
Supported events per component:
|
||||
|
||||
- **AnnotatedImage**: select
|
||||
- **Audio**: stream, change, clear, play, pause, stop, pause, start_recording, pause_recording, stop_recording, upload, input
|
||||
- **BarPlot**: select, double_click
|
||||
- **BrowserState**: change
|
||||
- **Button**: click
|
||||
- **Chatbot**: change, select, like, retry, undo, example_select, option_select, clear, copy, edit
|
||||
- **Checkbox**: change, input, select
|
||||
- **CheckboxGroup**: change, input, select
|
||||
- **ClearButton**: click
|
||||
- **Code**: change, input, focus, blur
|
||||
- **ColorPicker**: change, input, submit, focus, blur
|
||||
- **Dataframe**: change, input, select, edit
|
||||
- **Dataset**: click, select
|
||||
- **DateTime**: change, submit
|
||||
- **DeepLinkButton**: click
|
||||
- **Dialogue**: change, input, submit
|
||||
- **DownloadButton**: click
|
||||
- **Dropdown**: change, input, select, focus, blur, key_up
|
||||
- **DuplicateButton**: click
|
||||
- **File**: change, select, clear, upload, delete, download
|
||||
- **FileExplorer**: change, input, select
|
||||
- **Gallery**: select, upload, change, delete, preview_close, preview_open
|
||||
- **HTML**: change, input, click, double_click, submit, stop, edit, clear, play, pause, end, start_recording, pause_recording, stop_recording, focus, blur, upload, release, select, stream, like, example_select, option_select, load, key_up, apply, delete, tick, undo, retry, expand, collapse, download, copy
|
||||
- **HighlightedText**: change, select
|
||||
- **Image**: clear, change, stream, select, upload, input
|
||||
- **ImageEditor**: clear, change, input, select, upload, apply
|
||||
- **ImageSlider**: clear, change, stream, select, upload, input
|
||||
- **JSON**: change
|
||||
- **Label**: change, select
|
||||
- **LinePlot**: select, double_click
|
||||
- **LoginButton**: click
|
||||
- **Markdown**: change, copy
|
||||
- **Model3D**: change, upload, edit, clear
|
||||
- **MultimodalTextbox**: change, input, select, submit, focus, blur, stop
|
||||
- **Navbar**: change
|
||||
- **Number**: change, input, submit, focus, blur
|
||||
- **ParamViewer**: change, upload
|
||||
- **Plot**: change
|
||||
- **Radio**: select, change, input
|
||||
- **ScatterPlot**: select, double_click
|
||||
- **SimpleImage**: clear, change, upload
|
||||
- **Slider**: change, input, release
|
||||
- **State**: change
|
||||
- **Textbox**: change, input, select, submit, focus, blur, stop, copy
|
||||
- **Timer**: tick
|
||||
- **UploadButton**: click, upload
|
||||
- **Video**: change, clear, start_recording, stop_recording, stop, play, pause, end, upload, input
|
||||
|
||||
## Prediction CLI
|
||||
|
||||
The `gradio` CLI includes `info` and `predict` commands for interacting with Gradio apps programmatically. These are especially useful for coding agents that need to use Spaces in their workflows.
|
||||
|
||||
### `gradio info` — Discover endpoints and parameters
|
||||
|
||||
```bash
|
||||
gradio info <space_id_or_url>
|
||||
```
|
||||
|
||||
Returns a JSON payload describing all endpoints, their parameters (with types and defaults), and return values.
|
||||
|
||||
```bash
|
||||
gradio info gradio/calculator
|
||||
# {
|
||||
# "/predict": {
|
||||
# "parameters": [
|
||||
# {"name": "num1", "required": true, "default": null, "type": {"type": "number"}},
|
||||
# {"name": "operation", "required": true, "default": null, "type": {"enum": ["add", "subtract", "multiply", "divide"], "type": "string"}},
|
||||
# {"name": "num2", "required": true, "default": null, "type": {"type": "number"}}
|
||||
# ],
|
||||
# "returns": [{"name": "output", "type": {"type": "number"}}],
|
||||
# "description": ""
|
||||
# }
|
||||
# }
|
||||
```
|
||||
|
||||
File-type parameters show `"type": "filepath"` with instructions to include `"meta": {"_type": "gradio.FileData"}` — this signals the file will be uploaded to the remote server.
|
||||
|
||||
### `gradio predict` — Send predictions
|
||||
|
||||
```bash
|
||||
gradio predict <space_id_or_url> <endpoint> <json_payload>
|
||||
```
|
||||
|
||||
Returns a JSON object with named output keys.
|
||||
|
||||
```bash
|
||||
# Simple numeric prediction
|
||||
gradio predict gradio/calculator /predict '{"num1": 5, "operation": "multiply", "num2": 3}'
|
||||
# {"output": 15}
|
||||
|
||||
# Image generation
|
||||
gradio predict black-forest-labs/FLUX.2-dev /infer '{"prompt": "A majestic dragon"}'
|
||||
# {"Result": "/tmp/gradio/.../image.webp", "Seed": 1117868604}
|
||||
|
||||
# File upload (must include meta key)
|
||||
gradio predict gradio/image_mod /predict '{"image": {"path": "/path/to/image.png", "meta": {"_type": "gradio.FileData"}}}'
|
||||
# {"output": "/tmp/gradio/.../output.png"}
|
||||
```
|
||||
|
||||
Both commands accept `--token` for accessing private Spaces.
|
||||
|
||||
## Additional Reference
|
||||
|
||||
- [End-to-End Examples](examples.md) — complete working apps
|
||||
@@ -0,0 +1,613 @@
|
||||
# Gradio End-to-End Examples
|
||||
|
||||
Complete working Gradio apps for reference.
|
||||
|
||||
## Blocks Essay Simple
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
def change_textbox(choice):
|
||||
if choice == "short":
|
||||
return gr.Textbox(lines=2, visible=True)
|
||||
elif choice == "long":
|
||||
return gr.Textbox(lines=8, visible=True, value="Lorem ipsum dolor sit amet")
|
||||
else:
|
||||
return gr.Textbox(visible=False)
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
radio = gr.Radio(
|
||||
["short", "long", "none"], label="What kind of essay would you like to write?"
|
||||
)
|
||||
text = gr.Textbox(lines=2, interactive=True, buttons=["copy"])
|
||||
radio.change(fn=change_textbox, inputs=radio, outputs=text)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Blocks Flipper
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import gradio as gr
|
||||
|
||||
def flip_text(x):
|
||||
return x[::-1]
|
||||
|
||||
def flip_image(x):
|
||||
return np.fliplr(x)
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
gr.Markdown("Flip text or image files using this demo.")
|
||||
with gr.Tab("Flip Text"):
|
||||
text_input = gr.Textbox()
|
||||
text_output = gr.Textbox()
|
||||
text_button = gr.Button("Flip")
|
||||
with gr.Tab("Flip Image"):
|
||||
with gr.Row():
|
||||
image_input = gr.Image()
|
||||
image_output = gr.Image()
|
||||
image_button = gr.Button("Flip")
|
||||
|
||||
with gr.Accordion("Open for More!", open=False):
|
||||
gr.Markdown("Look at me...")
|
||||
temp_slider = gr.Slider(
|
||||
0, 1,
|
||||
value=0.1,
|
||||
step=0.1,
|
||||
interactive=True,
|
||||
label="Slide me",
|
||||
)
|
||||
|
||||
text_button.click(flip_text, inputs=text_input, outputs=text_output)
|
||||
image_button.click(flip_image, inputs=image_input, outputs=image_output)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Blocks Form
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
name_box = gr.Textbox(label="Name")
|
||||
age_box = gr.Number(label="Age", minimum=0, maximum=100)
|
||||
symptoms_box = gr.CheckboxGroup(["Cough", "Fever", "Runny Nose"])
|
||||
submit_btn = gr.Button("Submit")
|
||||
|
||||
with gr.Column(visible=False) as output_col:
|
||||
diagnosis_box = gr.Textbox(label="Diagnosis")
|
||||
patient_summary_box = gr.Textbox(label="Patient Summary")
|
||||
|
||||
def submit(name, age, symptoms):
|
||||
return {
|
||||
submit_btn: gr.Button(visible=False),
|
||||
output_col: gr.Column(visible=True),
|
||||
diagnosis_box: "covid" if "Cough" in symptoms else "flu",
|
||||
patient_summary_box: f"{name}, {age} y/o",
|
||||
}
|
||||
|
||||
submit_btn.click(
|
||||
submit,
|
||||
[name_box, age_box, symptoms_box],
|
||||
[submit_btn, diagnosis_box, patient_summary_box, output_col],
|
||||
)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Blocks Hello
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
def welcome(name):
|
||||
return f"Welcome to Gradio, {name}!"
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
gr.Markdown(
|
||||
"""
|
||||
# Hello World!
|
||||
Start typing below to see the output.
|
||||
""")
|
||||
inp = gr.Textbox(placeholder="What is your name?")
|
||||
out = gr.Textbox()
|
||||
inp.change(welcome, inp, out)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Blocks Layout
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
demo = gr.Blocks()
|
||||
|
||||
with demo:
|
||||
with gr.Row():
|
||||
gr.Image(interactive=True, scale=2)
|
||||
gr.Image()
|
||||
with gr.Row():
|
||||
gr.Textbox(label="Text")
|
||||
gr.Number(label="Count", scale=2)
|
||||
gr.Radio(choices=["One", "Two"])
|
||||
with gr.Row():
|
||||
gr.Button("500", scale=0, min_width=500)
|
||||
gr.Button("A", scale=0)
|
||||
gr.Button("grow")
|
||||
with gr.Row():
|
||||
gr.Textbox()
|
||||
gr.Textbox()
|
||||
gr.Button()
|
||||
with gr.Row():
|
||||
with gr.Row():
|
||||
with gr.Column():
|
||||
gr.Textbox(label="Text")
|
||||
gr.Number(label="Count")
|
||||
gr.Radio(choices=["One", "Two"])
|
||||
gr.Image()
|
||||
with gr.Column():
|
||||
gr.Image(interactive=True)
|
||||
gr.Image()
|
||||
gr.Image()
|
||||
gr.Textbox(label="Text")
|
||||
gr.Number(label="Count")
|
||||
gr.Radio(choices=["One", "Two"])
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Calculator
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
def calculator(num1, operation, num2):
|
||||
if operation == "add":
|
||||
return num1 + num2
|
||||
elif operation == "subtract":
|
||||
return num1 - num2
|
||||
elif operation == "multiply":
|
||||
return num1 * num2
|
||||
elif operation == "divide":
|
||||
if num2 == 0:
|
||||
raise gr.Error("Cannot divide by zero!")
|
||||
return num1 / num2
|
||||
|
||||
demo = gr.Interface(
|
||||
calculator,
|
||||
[
|
||||
"number",
|
||||
gr.Radio(["add", "subtract", "multiply", "divide"]),
|
||||
"number"
|
||||
],
|
||||
"number",
|
||||
examples=[
|
||||
[45, "add", 3],
|
||||
[3.14, "divide", 2],
|
||||
[144, "multiply", 2.5],
|
||||
[0, "subtract", 1.2],
|
||||
],
|
||||
title="Toy Calculator",
|
||||
description="Here's a sample toy calculator.",
|
||||
api_name="predict"
|
||||
)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Chatbot Simple
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
import random
|
||||
import time
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
chatbot = gr.Chatbot()
|
||||
msg = gr.Textbox()
|
||||
clear = gr.ClearButton([msg, chatbot])
|
||||
|
||||
def respond(message, chat_history):
|
||||
bot_message = random.choice(["How are you?", "Today is a great day", "I'm very hungry"])
|
||||
chat_history.append({"role": "user", "content": message})
|
||||
chat_history.append({"role": "assistant", "content": bot_message})
|
||||
time.sleep(2)
|
||||
return "", chat_history
|
||||
|
||||
msg.submit(respond, [msg, chatbot], [msg, chatbot])
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Chatbot Streaming
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
import random
|
||||
import time
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
chatbot = gr.Chatbot()
|
||||
msg = gr.Textbox()
|
||||
clear = gr.Button("Clear")
|
||||
|
||||
def user(user_message, history: list):
|
||||
return "", history + [{"role": "user", "content": user_message}]
|
||||
|
||||
def bot(history: list):
|
||||
bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
|
||||
history.append({"role": "assistant", "content": ""})
|
||||
for character in bot_message:
|
||||
history[-1]['content'] += character
|
||||
time.sleep(0.05)
|
||||
yield history
|
||||
|
||||
msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
|
||||
bot, chatbot, chatbot
|
||||
)
|
||||
clear.click(lambda: None, None, chatbot, queue=False)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Custom Css
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
with gr.Column(elem_classes="cool-col"):
|
||||
gr.Markdown("### Gradio Demo with Custom CSS", elem_classes="darktest")
|
||||
gr.Markdown(
|
||||
elem_classes="markdown",
|
||||
value="Resize the browser window to see the CSS media query in action.",
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch(css_paths=["demo/custom_css/custom_css.css"])
|
||||
```
|
||||
|
||||
## Fake Diffusion
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
import numpy as np
|
||||
import time
|
||||
|
||||
def fake_diffusion(steps):
|
||||
rng = np.random.default_rng()
|
||||
for i in range(steps):
|
||||
time.sleep(1)
|
||||
image = rng.random(size=(600, 600, 3))
|
||||
yield image
|
||||
image = np.ones((1000,1000,3), np.uint8)
|
||||
image[:] = [255, 124, 0]
|
||||
yield image
|
||||
|
||||
demo = gr.Interface(fake_diffusion,
|
||||
inputs=gr.Slider(1, 10, 3, step=1),
|
||||
outputs="image",
|
||||
api_name="predict")
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Hello World
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
|
||||
def greet(name):
|
||||
return "Hello " + name + "!"
|
||||
|
||||
|
||||
demo = gr.Interface(fn=greet, inputs="textbox", outputs="textbox", api_name="predict")
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Image Editor
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
import time
|
||||
|
||||
|
||||
def sleep(im):
|
||||
time.sleep(5)
|
||||
return [im["background"], im["layers"][0], im["layers"][1], im["composite"]]
|
||||
|
||||
|
||||
def predict(im):
|
||||
return im["composite"]
|
||||
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
with gr.Row():
|
||||
im = gr.ImageEditor(
|
||||
type="numpy",
|
||||
)
|
||||
im_preview = gr.Image()
|
||||
n_upload = gr.Number(0, label="Number of upload events", step=1)
|
||||
n_change = gr.Number(0, label="Number of change events", step=1)
|
||||
n_input = gr.Number(0, label="Number of input events", step=1)
|
||||
|
||||
im.upload(lambda x: x + 1, outputs=n_upload, inputs=n_upload)
|
||||
im.change(lambda x: x + 1, outputs=n_change, inputs=n_change)
|
||||
im.input(lambda x: x + 1, outputs=n_input, inputs=n_input)
|
||||
im.change(predict, outputs=im_preview, inputs=im, show_progress="hidden")
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## On Listener Decorator
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
name = gr.Textbox(label="Name")
|
||||
output = gr.Textbox(label="Output Box")
|
||||
greet_btn = gr.Button("Greet")
|
||||
|
||||
@gr.on(triggers=[name.submit, greet_btn.click], inputs=name, outputs=output)
|
||||
def greet(name):
|
||||
return "Hello " + name + "!"
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Render Merge
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
import time
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
text_count = gr.Slider(1, 5, value=1, step=1, label="Textbox Count")
|
||||
|
||||
@gr.render(inputs=text_count)
|
||||
def render_count(count):
|
||||
boxes = []
|
||||
for i in range(count):
|
||||
box = gr.Textbox(label=f"Box {i}")
|
||||
boxes.append(box)
|
||||
|
||||
def merge(*args):
|
||||
time.sleep(0.2) # simulate a delay
|
||||
return " ".join(args)
|
||||
|
||||
merge_btn.click(merge, boxes, output)
|
||||
|
||||
def clear():
|
||||
time.sleep(0.2) # simulate a delay
|
||||
return [" "] * count
|
||||
|
||||
clear_btn.click(clear, None, boxes)
|
||||
|
||||
def countup():
|
||||
time.sleep(0.2) # simulate a delay
|
||||
return list(range(count))
|
||||
|
||||
count_btn.click(countup, None, boxes, queue=False)
|
||||
|
||||
with gr.Row():
|
||||
merge_btn = gr.Button("Merge")
|
||||
clear_btn = gr.Button("Clear")
|
||||
count_btn = gr.Button("Count")
|
||||
|
||||
output = gr.Textbox()
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Reverse Audio 2
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
import numpy as np
|
||||
|
||||
def reverse_audio(audio):
|
||||
sr, data = audio
|
||||
return (sr, np.flipud(data))
|
||||
|
||||
demo = gr.Interface(fn=reverse_audio,
|
||||
inputs="microphone",
|
||||
outputs="audio", api_name="predict")
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Sepia Filter
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import gradio as gr
|
||||
|
||||
def sepia(input_img):
|
||||
sepia_filter = np.array([
|
||||
[0.393, 0.769, 0.189],
|
||||
[0.349, 0.686, 0.168],
|
||||
[0.272, 0.534, 0.131]
|
||||
])
|
||||
sepia_img = input_img.dot(sepia_filter.T)
|
||||
sepia_img /= sepia_img.max()
|
||||
return sepia_img
|
||||
|
||||
demo = gr.Interface(sepia, gr.Image(), "image", api_name="predict")
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Sort Records
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
def sort_records(records):
|
||||
return records.sort("Quantity")
|
||||
|
||||
demo = gr.Interface(
|
||||
sort_records,
|
||||
gr.Dataframe(
|
||||
headers=["Item", "Quantity"],
|
||||
datatype=["str", "number"],
|
||||
row_count=3,
|
||||
column_count=2,
|
||||
column_limits=(2, 2),
|
||||
type="polars"
|
||||
),
|
||||
"dataframe",
|
||||
description="Sort by Quantity"
|
||||
)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Streaming Simple
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
with gr.Row():
|
||||
with gr.Column():
|
||||
input_img = gr.Image(label="Input", sources="webcam")
|
||||
with gr.Column():
|
||||
output_img = gr.Image(label="Output")
|
||||
input_img.stream(lambda s: s, input_img, output_img, time_limit=15, stream_every=0.1, concurrency_limit=30)
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Tabbed Interface Lite
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
hello_world = gr.Interface(lambda name: "Hello " + name, "text", "text", api_name="predict")
|
||||
bye_world = gr.Interface(lambda name: "Bye " + name, "text", "text", api_name="predict")
|
||||
chat = gr.ChatInterface(lambda *args: "Hello " + args[0], api_name="chat")
|
||||
|
||||
demo = gr.TabbedInterface([hello_world, bye_world, chat], ["Hello World", "Bye World", "Chat"])
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Tax Calculator
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
def tax_calculator(income, marital_status, assets):
|
||||
tax_brackets = [(10, 0), (25, 8), (60, 12), (120, 20), (250, 30)]
|
||||
total_deductible = sum(cost for cost, deductible in zip(assets["Cost"], assets["Deductible"]) if deductible)
|
||||
taxable_income = income - total_deductible
|
||||
|
||||
total_tax = 0
|
||||
for bracket, rate in tax_brackets:
|
||||
if taxable_income > bracket:
|
||||
total_tax += (taxable_income - bracket) * rate / 100
|
||||
|
||||
if marital_status == "Married":
|
||||
total_tax *= 0.75
|
||||
elif marital_status == "Divorced":
|
||||
total_tax *= 0.8
|
||||
|
||||
return round(total_tax)
|
||||
|
||||
demo = gr.Interface(
|
||||
tax_calculator,
|
||||
[
|
||||
"number",
|
||||
gr.Radio(["Single", "Married", "Divorced"]),
|
||||
gr.Dataframe(
|
||||
headers=["Item", "Cost", "Deductible"],
|
||||
datatype=["str", "number", "bool"],
|
||||
label="Assets Purchased this Year",
|
||||
),
|
||||
],
|
||||
gr.Number(label="Tax due"),
|
||||
examples=[
|
||||
[10000, "Married", [["Suit", 5000, True], ["Laptop (for work)", 800, False], ["Car", 1800, True]]],
|
||||
[80000, "Single", [["Suit", 800, True], ["Watch", 1800, True], ["Food", 800, True]]],
|
||||
],
|
||||
live=True,
|
||||
api_name="predict"
|
||||
)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Timer Simple
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
import random
|
||||
import time
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
timer = gr.Timer(1)
|
||||
timestamp = gr.Number(label="Time")
|
||||
timer.tick(lambda: round(time.time()), outputs=timestamp, api_name="timestamp")
|
||||
|
||||
number = gr.Number(lambda: random.randint(1, 10), every=timer, label="Random Number")
|
||||
with gr.Row():
|
||||
gr.Button("Start").click(lambda: gr.Timer(active=True), None, timer)
|
||||
gr.Button("Stop").click(lambda: gr.Timer(active=False), None, timer)
|
||||
gr.Button("Go Fast").click(lambda: 0.2, None, timer)
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Variable Outputs
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
max_textboxes = 10
|
||||
|
||||
def variable_outputs(k):
|
||||
k = int(k)
|
||||
return [gr.Textbox(visible=True)]*k + [gr.Textbox(visible=False)]*(max_textboxes-k)
|
||||
|
||||
with gr.Blocks() as demo:
|
||||
s = gr.Slider(1, max_textboxes, value=max_textboxes, step=1, label="How many textboxes to show:")
|
||||
textboxes = []
|
||||
for i in range(max_textboxes):
|
||||
t = gr.Textbox(f"Textbox {i}")
|
||||
textboxes.append(t)
|
||||
|
||||
s.change(variable_outputs, s, textboxes)
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
## Video Identity
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
from gradio.media import get_video
|
||||
|
||||
def video_identity(video):
|
||||
return video
|
||||
|
||||
# get_video() returns file paths to sample media included with Gradio
|
||||
demo = gr.Interface(video_identity,
|
||||
gr.Video(),
|
||||
"playable_video",
|
||||
examples=[
|
||||
get_video("world.mp4")
|
||||
],
|
||||
cache_examples=True,
|
||||
api_name="predict",)
|
||||
|
||||
demo.launch()
|
||||
```
|
||||
@@ -0,0 +1,204 @@
|
||||
---
|
||||
name: huggingface-llm-trainer
|
||||
description: Train or fine-tune language and vision models using TRL (Transformer Reinforcement Learning) or Unsloth with Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes guidance on the TRL Jobs package, UV scripts with PEP 723 format, dataset preparation and validation, hardware selection, cost estimation, Trackio monitoring, Hub authentication, model selection/leaderboards and model persistence. Use for tasks involving cloud GPU training, GGUF conversion, or when users mention training on Hugging Face Jobs without local GPU setup.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# TRL Training on Hugging Face Jobs
|
||||
|
||||
## Overview
|
||||
|
||||
Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required — models train on cloud GPUs and results are automatically saved to the Hugging Face Hub.
|
||||
|
||||
**TRL provides multiple training methods:**
|
||||
- **SFT** (Supervised Fine-Tuning) — standard instruction tuning
|
||||
- **DPO** (Direct Preference Optimization) — alignment from preference data
|
||||
- **GRPO** (Group Relative Policy Optimization) — online RL training
|
||||
- **Reward Modeling** — train reward models for RLHF
|
||||
|
||||
See `references/training_methods.md` for method overviews and selection guidance.
|
||||
|
||||
### When to Use Unsloth
|
||||
|
||||
Use **Unsloth** (`references/unsloth.md`) instead of standard TRL when GPU memory is limited (~60% less VRAM), speed matters (~2x faster), training large models (>13B), or training Vision-Language Models (Unsloth has `FastVisionModel` support). See `scripts/unsloth_sft_example.py` for a production-ready training script.
|
||||
|
||||
## Key Directives
|
||||
|
||||
1. **Submit jobs via `hf jobs uv run` (CLI) or the `hf_jobs()` MCP tool if the Hugging Face MCP server is configured** — pass the training script inline, don't save to a local file unless the user explicitly requests it. If the user asks to "train a model" or "fine-tune", create the training script AND submit the job immediately.
|
||||
2. **Always include Trackio** for real-time monitoring — use `scripts/` templates.
|
||||
3. **Provide job details after submission**: job ID, monitoring URL, estimated time; note the user can request status checks later.
|
||||
4. **Use example scripts as templates**: `scripts/train_sft_example.py`, `scripts/train_dpo_example.py`, etc.
|
||||
|
||||
## Local Script Execution
|
||||
|
||||
Repository scripts use PEP 723 inline dependencies. Run them with `uv run`:
|
||||
```bash
|
||||
uv run scripts/estimate_cost.py --help
|
||||
uv run scripts/dataset_inspector.py --help
|
||||
```
|
||||
|
||||
## Prerequisites Checklist
|
||||
|
||||
**Account & Authentication:**
|
||||
- Hugging Face account with Pro/Team/Enterprise plan (Jobs require a paid plan); authenticated login.
|
||||
- **HF_TOKEN for Hub push is CRITICAL** — the training environment is ephemeral, so results are lost unless pushed to the Hub. Token must have write permissions. Pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in the job config.
|
||||
|
||||
**Dataset Requirements:**
|
||||
- Must exist on the Hub or be loadable via `datasets.load_dataset()`.
|
||||
- Format must match the training method (SFT: messages/text/prompt-completion; DPO: chosen/rejected; GRPO: prompt-only). **Always validate unknown datasets first** (see Dataset Validation below).
|
||||
- Size appropriate for hardware (demo: 50-100 examples on t4-small; production: 1K-10K+ on a10g-large/a100-large).
|
||||
|
||||
**Critical Settings:**
|
||||
- Timeout must exceed expected training time — default 30min is too short; minimum recommended 1-2 hours. The job fails and loses all progress if the timeout is exceeded.
|
||||
- Hub push must be enabled: `push_to_hub=True`, `hub_model_id="username/model-name"`, `secrets={"HF_TOKEN": "$HF_TOKEN"}`.
|
||||
|
||||
## Asynchronous Jobs
|
||||
|
||||
Training jobs run in the background and can take hours. After submitting: report the job ID, monitoring URL, and estimated time; wait for the user to request status checks rather than polling. Initial logs can take 30-60 seconds to appear.
|
||||
|
||||
## Quick Start
|
||||
|
||||
**Sequence length:** TRL config classes use `max_length` (not `max_seq_length`). Default is `max_length=1024` (truncates from right) — override higher for longer context, lower under memory constraints, or `None` for vision models (to avoid cutting image tokens).
|
||||
|
||||
### Approach 1: UV Scripts (default choice)
|
||||
|
||||
UV scripts use PEP 723 inline dependencies for clean, self-contained training:
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": """
|
||||
# /// script
|
||||
# dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"]
|
||||
# ///
|
||||
from datasets import load_dataset
|
||||
from peft import LoraConfig
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
import trackio
|
||||
|
||||
dataset = load_dataset("trl-lib/Capybara", split="train")
|
||||
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset_split["train"],
|
||||
eval_dataset=dataset_split["test"],
|
||||
peft_config=LoraConfig(r=16, lora_alpha=32),
|
||||
args=SFTConfig(
|
||||
output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model",
|
||||
num_train_epochs=3, eval_strategy="steps", eval_steps=50,
|
||||
report_to="trackio", project="my_project", run_name="my_run",
|
||||
),
|
||||
)
|
||||
trainer.train()
|
||||
trainer.push_to_hub()
|
||||
""",
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "2h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"},
|
||||
})
|
||||
```
|
||||
|
||||
The `script` parameter accepts inline code or a publicly-accessible/Hub/GitHub/Gist URL — **local file paths do not work** (jobs run in isolated containers with no access to the local filesystem). To use a local script, upload it to the Hub first (`hf upload ...`) and reference its resolved URL.
|
||||
|
||||
### Approach 2: TRL Maintained Scripts
|
||||
|
||||
Run TRL's battle-tested example scripts directly from a URL, passing CLI-style `script_args` (`--model_name_or_path`, `--dataset_name`, `--output_dir`, `--push_to_hub`, `--hub_model_id`). Available at https://github.com/huggingface/trl/tree/main/examples/scripts.
|
||||
|
||||
### Approach 3: HF Jobs CLI
|
||||
|
||||
When no `hf_jobs`-style tool is available, use the `hf jobs` CLI directly. **Flags must come before the script URL**, the subcommand order is `hf jobs uv run` (not `run uv`), and use `--secrets` (plural):
|
||||
|
||||
```bash
|
||||
hf jobs uv run \
|
||||
--flavor a10g-large --timeout 2h --secrets HF_TOKEN \
|
||||
"https://huggingface.co/user/repo/resolve/main/train.py"
|
||||
```
|
||||
|
||||
Check status: `hf jobs ps`, `hf jobs logs <job-id>`, `hf jobs inspect <job-id>`, `hf jobs cancel <job-id>`.
|
||||
|
||||
### Approach 4: TRL Jobs Package
|
||||
|
||||
`uvx trl-jobs sft --model_name Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara` gives pre-configured defaults, automatic Trackio integration, and automatic Hub push — best for terminal-only, quick local experimentation. Repository: https://github.com/huggingface/trl-jobs.
|
||||
|
||||
## Hardware Selection
|
||||
|
||||
| Model Size | Recommended Hardware | Cost (approx/hr) |
|
||||
|------------|---------------------|------------------|
|
||||
| <1B params | `t4-small` | ~$0.75 |
|
||||
| 1-3B params | `t4-medium`, `l4x1` | ~$1.50-2.50 |
|
||||
| 3-7B params | `a10g-small`, `a10g-large` | ~$3.50-5.00 |
|
||||
| 7-13B params | `a10g-large`, `a100-large` (LoRA) | ~$5-10 |
|
||||
| 13B+ params | `a100-large`, `a10g-largex2` (LoRA) | ~$10-20 |
|
||||
|
||||
Use LoRA/PEFT for models >7B; multi-GPU is handled automatically by TRL/Accelerate. See `references/hardware_guide.md` for full specs.
|
||||
|
||||
## Saving Results to the Hub
|
||||
|
||||
**The Jobs environment is ephemeral — everything is deleted when the job ends.** Set `push_to_hub=True` and `hub_model_id="username/model-name"` in the training config, and pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in the job submission. See `references/hub_saving.md` for troubleshooting.
|
||||
|
||||
## Timeout Management
|
||||
|
||||
Default is 30 minutes — too short for real training. Set explicitly (`"timeout": "2h"`, formats: `"90m"`, `"2h"`, seconds as integer) with a 20-30% buffer for loading/checkpointing/Hub push. Guideline: quick demo 10-30min, development 1-2h, production (3-7B) 4-6h. On timeout the job is killed immediately and unsaved progress is lost.
|
||||
|
||||
## Choosing a Base Model
|
||||
|
||||
Use `scripts/hf_benchmarks.py` to find top-performing models for a task, keeping size/hardware constraints in mind: `uv run scripts/hf_benchmarks.py search --query ocr` then `uv run scripts/hf_benchmarks.py leaderboard <benchmark-id>`.
|
||||
|
||||
## Cost Estimation
|
||||
|
||||
Offer to estimate cost when parameters are known (hardware, dataset size, epochs), with `scripts/estimate_cost.py`:
|
||||
```bash
|
||||
uv run scripts/estimate_cost.py --model meta-llama/Llama-2-7b-hf --dataset trl-lib/Capybara --hardware a10g-large --dataset-size 16000 --epochs 3
|
||||
```
|
||||
|
||||
## Example Training Scripts
|
||||
|
||||
Production-ready templates: `scripts/train_sft_example.py`, `scripts/train_dpo_example.py`, `scripts/train_grpo_example.py`, `scripts/unsloth_sft_example.py` (Unsloth, faster/less VRAM). Pass their content inline or use as templates.
|
||||
|
||||
## Monitoring with Trackio
|
||||
|
||||
Add `trackio` to dependencies and configure `report_to="trackio"`, `run_name="meaningful_name"`. Defaults: space ID `{username}/trackio`, minimal config (hyperparameters + model/dataset info), a Project Name to group runs. Apply the user's preferences instead when specified. See `references/trackio_guide.md` for grouping runs across experiments.
|
||||
|
||||
## Dataset Validation
|
||||
|
||||
**Validate BEFORE launching GPU training** — 50%+ of training failures are format mismatches, and DPO is especially strict about column names (`prompt`, `chosen`, `rejected`). Validation on CPU costs ~$0.01 and takes <1 minute vs. wasting $1-10 and 30-60 minutes on a failed GPU job.
|
||||
|
||||
Always validate unknown/custom datasets and any DPO dataset; skip validation only for well-known TRL datasets (`trl-lib/ultrachat_200k`, `trl-lib/Capybara`, etc.). Use the Hub-hosted dataset inspector script (`--dataset name --split train`); output markers are `✓ READY`, `✗ NEEDS MAPPING` (includes copy-paste mapping code), or `✗ INCOMPATIBLE`.
|
||||
|
||||
## Converting Models to GGUF
|
||||
|
||||
Convert trained models to GGUF for llama.cpp/Ollama/LM Studio/local inference — supports 4/5/8-bit quantization, typically 2-8GB for 7B models vs. 14GB unquantized. See `references/gguf_conversion.md` for the complete conversion script, quantization options, and troubleshooting.
|
||||
|
||||
## Common Training Patterns
|
||||
|
||||
See `references/training_patterns.md`: quick demo, production with checkpoints, multi-GPU, DPO, GRPO.
|
||||
|
||||
## Common Failure Modes
|
||||
|
||||
- **Out of memory**: reduce `per_device_train_batch_size` (increase `gradient_accumulation_steps` to compensate, target effective batch size ~128), enable `gradient_checkpointing=True`, or upgrade hardware.
|
||||
- **Dataset misformatted**: validate first with the dataset inspector, apply the suggested mapping code.
|
||||
- **Job timeout**: check actual runtime via logs, increase timeout with a 30% buffer, or reduce `num_train_epochs`/dataset size; save checkpoints (`save_strategy="steps"`, `hub_strategy="every_save"`) so partial progress survives.
|
||||
- **Hub push failures**: confirm `secrets={"HF_TOKEN": "$HF_TOKEN"}`, `push_to_hub=True`, `hub_model_id`, write permissions, and that the target repo exists (or `hub_private_repo=True`).
|
||||
- **Missing dependencies**: add them to the PEP 723 header.
|
||||
|
||||
See `references/troubleshooting.md` for the complete guide.
|
||||
|
||||
## Resources
|
||||
|
||||
**References:** `references/training_methods.md`, `training_patterns.md`, `unsloth.md`, `gguf_conversion.md`, `trackio_guide.md`, `hardware_guide.md`, `hub_saving.md`, `troubleshooting.md`, `local_training_macos.md`.
|
||||
|
||||
**Scripts:** `scripts/train_sft_example.py`, `train_dpo_example.py`, `train_grpo_example.py`, `unsloth_sft_example.py`, `estimate_cost.py`, `convert_to_gguf.py`, `hf_benchmarks.py`.
|
||||
|
||||
**External:** [TRL docs](https://huggingface.co/docs/trl), [TRL Jobs training guide](https://huggingface.co/docs/trl/en/jobs_training), [TRL Jobs package](https://github.com/huggingface/trl-jobs), [HF Jobs docs](https://huggingface.co/docs/huggingface_hub/guides/jobs), [UV scripts guide](https://docs.astral.sh/uv/guides/scripts/).
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Submit scripts inline — no file saving required unless the user asks.
|
||||
2. Jobs are asynchronous — don't poll; let the user check status when ready.
|
||||
3. Always set a timeout above the default 30min (1-2h minimum).
|
||||
4. Always enable Hub push — the environment is ephemeral.
|
||||
5. Include Trackio for real-time monitoring.
|
||||
6. Offer cost estimation when parameters are known.
|
||||
7. Default to UV scripts (Approach 1) or the TRL maintained scripts (Approach 2); fall back to the `hf jobs` CLI (Approach 3) when no job-submission tool is available.
|
||||
8. Validate dataset format before training, especially for DPO.
|
||||
9. Choose hardware for model size; use LoRA for models >7B.
|
||||
@@ -0,0 +1,296 @@
|
||||
# GGUF Conversion Guide
|
||||
|
||||
After training models with TRL on Hugging Face Jobs, convert them to **GGUF format** for use with llama.cpp, Ollama, LM Studio, and other local inference tools.
|
||||
|
||||
**This guide provides production-ready, tested code based on successful conversions.** All critical dependencies and build steps are included.
|
||||
|
||||
## What is GGUF?
|
||||
|
||||
**GGUF** (GPT-Generated Unified Format):
|
||||
- Optimized format for CPU/GPU inference with llama.cpp
|
||||
- Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size
|
||||
- Compatible with: Ollama, LM Studio, Jan, GPT4All, llama.cpp
|
||||
- Typically 2-8GB for 7B models (vs 14GB unquantized)
|
||||
|
||||
## When to Convert to GGUF
|
||||
|
||||
**Convert when:**
|
||||
- Running models locally with Ollama or LM Studio
|
||||
- Using CPU-optimized inference
|
||||
- Reducing model size with quantization
|
||||
- Deploying to edge devices
|
||||
- Sharing models for local-first use
|
||||
|
||||
## Critical Success Factors
|
||||
|
||||
Based on production testing, these are **essential** for reliable conversion:
|
||||
|
||||
### 1. ✅ Install Build Tools FIRST
|
||||
**Before cloning llama.cpp**, install build dependencies:
|
||||
```python
|
||||
subprocess.run(["apt-get", "update", "-qq"], check=True, capture_output=True)
|
||||
subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True, capture_output=True)
|
||||
```
|
||||
|
||||
**Why:** The quantization tool requires gcc and cmake. Installing after cloning doesn't help.
|
||||
|
||||
### 2. ✅ Use CMake (Not Make)
|
||||
**Build the quantize tool with CMake:**
|
||||
```python
|
||||
# Create build directory
|
||||
os.makedirs("/tmp/llama.cpp/build", exist_ok=True)
|
||||
|
||||
# Configure
|
||||
subprocess.run([
|
||||
"cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
|
||||
"-DGGML_CUDA=OFF" # Faster build, CUDA not needed for quantization
|
||||
], check=True, capture_output=True, text=True)
|
||||
|
||||
# Build
|
||||
subprocess.run([
|
||||
"cmake", "--build", "/tmp/llama.cpp/build",
|
||||
"--target", "llama-quantize", "-j", "4"
|
||||
], check=True, capture_output=True, text=True)
|
||||
|
||||
# Binary path
|
||||
quantize_bin = "/tmp/llama.cpp/build/bin/llama-quantize"
|
||||
```
|
||||
|
||||
**Why:** CMake is more reliable than `make` and produces consistent binary paths.
|
||||
|
||||
### 3. ✅ Include All Dependencies
|
||||
**PEP 723 header must include:**
|
||||
```python
|
||||
# /// script
|
||||
# dependencies = [
|
||||
# "transformers>=4.36.0",
|
||||
# "peft>=0.7.0",
|
||||
# "torch>=2.0.0",
|
||||
# "accelerate>=0.24.0",
|
||||
# "huggingface_hub>=0.20.0",
|
||||
# "sentencepiece>=0.1.99", # Required for tokenizer
|
||||
# "protobuf>=3.20.0", # Required for tokenizer
|
||||
# "numpy",
|
||||
# "gguf",
|
||||
# ]
|
||||
# ///
|
||||
```
|
||||
|
||||
**Why:** `sentencepiece` and `protobuf` are critical for tokenizer conversion. Missing them causes silent failures.
|
||||
|
||||
### 4. ✅ Verify Names Before Use
|
||||
**Always verify repos exist:**
|
||||
```python
|
||||
# Before submitting job, verify:
|
||||
hub_repo_details([ADAPTER_MODEL], repo_type="model")
|
||||
hub_repo_details([BASE_MODEL], repo_type="model")
|
||||
```
|
||||
|
||||
**Why:** Non-existent dataset/model names cause job failures that could be caught in seconds.
|
||||
|
||||
## Complete Conversion Script
|
||||
|
||||
See `scripts/convert_to_gguf.py` for the complete, production-ready script.
|
||||
|
||||
**Key features:**
|
||||
- ✅ All dependencies in PEP 723 header
|
||||
- ✅ Build tools installed automatically
|
||||
- ✅ CMake build process (reliable)
|
||||
- ✅ Comprehensive error handling
|
||||
- ✅ Environment variable configuration
|
||||
- ✅ Automatic README generation
|
||||
|
||||
## Quick Conversion Job
|
||||
|
||||
```python
|
||||
# Before submitting: VERIFY MODELS EXIST
|
||||
hub_repo_details(["username/my-finetuned-model"], repo_type="model")
|
||||
hub_repo_details(["Qwen/Qwen2.5-0.5B"], repo_type="model")
|
||||
|
||||
# Submit conversion job
|
||||
hf_jobs("uv", {
|
||||
"script": open("trl/scripts/convert_to_gguf.py").read(), # Or inline the script
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "45m",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"},
|
||||
"env": {
|
||||
"ADAPTER_MODEL": "username/my-finetuned-model",
|
||||
"BASE_MODEL": "Qwen/Qwen2.5-0.5B",
|
||||
"OUTPUT_REPO": "username/my-model-gguf",
|
||||
"HF_USERNAME": "username" # Optional, for README
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Conversion Process
|
||||
|
||||
The script performs these steps:
|
||||
|
||||
1. **Load and Merge** - Load base model and LoRA adapter, merge them
|
||||
2. **Install Build Tools** - Install gcc, cmake (CRITICAL: before cloning llama.cpp)
|
||||
3. **Setup llama.cpp** - Clone repo, install Python dependencies
|
||||
4. **Convert to GGUF** - Create FP16 GGUF using llama.cpp converter
|
||||
5. **Build Quantize Tool** - Use CMake to build `llama-quantize`
|
||||
6. **Quantize** - Create Q4_K_M, Q5_K_M, Q8_0 versions
|
||||
7. **Upload** - Upload all versions + README to Hub
|
||||
|
||||
## Quantization Options
|
||||
|
||||
Common quantization formats (from smallest to largest):
|
||||
|
||||
| Format | Size | Quality | Use Case |
|
||||
|--------|------|---------|----------|
|
||||
| **Q4_K_M** | ~300MB | Good | **Recommended** - best balance of size/quality |
|
||||
| **Q5_K_M** | ~350MB | Better | Higher quality, slightly larger |
|
||||
| **Q8_0** | ~500MB | Very High | Near-original quality |
|
||||
| **F16** | ~1GB | Original | Full precision, largest file |
|
||||
|
||||
**Recommendation:** Create Q4_K_M, Q5_K_M, and Q8_0 versions to give users options.
|
||||
|
||||
## Hardware Requirements
|
||||
|
||||
**For conversion:**
|
||||
- Small models (<1B): CPU-basic works, but slow
|
||||
- Medium models (1-7B): a10g-large recommended
|
||||
- Large models (7B+): a10g-large or a100-large
|
||||
|
||||
**Time estimates:**
|
||||
- 0.5B model: ~15-25 minutes on A10G
|
||||
- 3B model: ~30-45 minutes on A10G
|
||||
- 7B model: ~45-60 minutes on A10G
|
||||
|
||||
## Using GGUF Models
|
||||
|
||||
**GGUF models work on both CPU and GPU.** They're optimized for CPU inference but can also leverage GPU acceleration when available.
|
||||
|
||||
### With Ollama (auto-detects GPU)
|
||||
```bash
|
||||
# Download GGUF
|
||||
hf download username/my-model-gguf model-q4_k_m.gguf
|
||||
|
||||
# Create Modelfile
|
||||
echo "FROM ./model-q4_k_m.gguf" > Modelfile
|
||||
|
||||
# Create and run (uses GPU automatically if available)
|
||||
ollama create my-model -f Modelfile
|
||||
ollama run my-model
|
||||
```
|
||||
|
||||
### With llama.cpp
|
||||
```bash
|
||||
# CPU only
|
||||
./llama-cli -m model-q4_k_m.gguf -p "Your prompt"
|
||||
|
||||
# With GPU acceleration (offload 32 layers to GPU)
|
||||
./llama-cli -m model-q4_k_m.gguf -ngl 32 -p "Your prompt"
|
||||
```
|
||||
|
||||
### With LM Studio
|
||||
1. Download the `.gguf` file
|
||||
2. Import into LM Studio
|
||||
3. Start chatting
|
||||
|
||||
## Best Practices
|
||||
|
||||
### ✅ DO:
|
||||
1. **Verify repos exist** before submitting jobs (use `hub_repo_details`)
|
||||
2. **Install build tools FIRST** before cloning llama.cpp
|
||||
3. **Use CMake** for building quantize tool (not make)
|
||||
4. **Include all dependencies** in PEP 723 header (especially sentencepiece, protobuf)
|
||||
5. **Create multiple quantizations** - Give users choice
|
||||
6. **Test on known models** before production use
|
||||
7. **Use A10G GPU** for faster conversion
|
||||
|
||||
### ❌ DON'T:
|
||||
1. **Assume repos exist** - Always verify with hub tools
|
||||
2. **Use make** instead of CMake - Less reliable
|
||||
3. **Remove dependencies** to "simplify" - They're all needed
|
||||
4. **Skip build tools** - Quantization will fail silently
|
||||
5. **Use default paths** - CMake puts binaries in build/bin/
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Out of memory during merge
|
||||
**Fix:**
|
||||
- Use larger GPU (a10g-large or a100-large)
|
||||
- Ensure `device_map="auto"` for automatic placement
|
||||
- Use `dtype=torch.float16` or `torch.bfloat16`
|
||||
|
||||
### Conversion fails with architecture error
|
||||
**Fix:**
|
||||
- Ensure llama.cpp supports the model architecture
|
||||
- Check for standard architecture (Qwen, Llama, Mistral, etc.)
|
||||
- Update llama.cpp to latest: `git clone --depth 1 https://github.com/ggerganov/llama.cpp.git`
|
||||
- Check llama.cpp documentation for model support
|
||||
|
||||
### Quantization fails
|
||||
**Fix:**
|
||||
- Verify build tools installed: `apt-get install build-essential cmake`
|
||||
- Use CMake (not make) to build quantize tool
|
||||
- Check binary path: `/tmp/llama.cpp/build/bin/llama-quantize`
|
||||
- Verify FP16 GGUF exists before quantizing
|
||||
|
||||
### Missing sentencepiece error
|
||||
**Fix:**
|
||||
- Add to PEP 723 header: `"sentencepiece>=0.1.99", "protobuf>=3.20.0"`
|
||||
- Don't remove dependencies to "simplify" - all are required
|
||||
|
||||
### Upload fails or times out
|
||||
**Fix:**
|
||||
- Large models (>2GB) need longer timeout: `"timeout": "1h"`
|
||||
- Upload quantized versions separately if needed
|
||||
- Check network/Hub status
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
These are from production testing and real failures:
|
||||
|
||||
### 1. Always Verify Before Use
|
||||
**Lesson:** Don't assume repos/datasets exist. Check first.
|
||||
```python
|
||||
# BEFORE submitting job
|
||||
hub_repo_details(["trl-lib/argilla-dpo-mix-7k"], repo_type="dataset") # Would catch error
|
||||
```
|
||||
**Prevented failures:** Non-existent dataset names, typos in model names
|
||||
|
||||
### 2. Prioritize Reliability Over Performance
|
||||
**Lesson:** Default to what's most likely to succeed.
|
||||
- Use CMake (not make) - more reliable
|
||||
- Disable CUDA in build - faster, not needed
|
||||
- Include all dependencies - don't "simplify"
|
||||
|
||||
**Prevented failures:** Build failures, missing binaries
|
||||
|
||||
### 3. Create Atomic, Self-Contained Scripts
|
||||
**Lesson:** Don't remove dependencies or steps. Scripts should work as a unit.
|
||||
- All dependencies in PEP 723 header
|
||||
- All build steps included
|
||||
- Clear error messages
|
||||
|
||||
**Prevented failures:** Missing tokenizer libraries, build tool failures
|
||||
|
||||
## References
|
||||
|
||||
**In this skill:**
|
||||
- `scripts/convert_to_gguf.py` - Complete, production-ready script
|
||||
|
||||
**External:**
|
||||
- [llama.cpp Repository](https://github.com/ggerganov/llama.cpp)
|
||||
- [GGUF Specification](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md)
|
||||
- [Ollama Documentation](https://ollama.ai)
|
||||
- [LM Studio](https://lmstudio.ai)
|
||||
|
||||
## Summary
|
||||
|
||||
**Critical checklist for GGUF conversion:**
|
||||
- [ ] Verify adapter and base models exist on Hub
|
||||
- [ ] Use production script from `scripts/convert_to_gguf.py`
|
||||
- [ ] All dependencies in PEP 723 header (including sentencepiece, protobuf)
|
||||
- [ ] Build tools installed before cloning llama.cpp
|
||||
- [ ] CMake used for building quantize tool (not make)
|
||||
- [ ] Correct binary path: `/tmp/llama.cpp/build/bin/llama-quantize`
|
||||
- [ ] A10G GPU selected for reasonable conversion time
|
||||
- [ ] Timeout set to 45m minimum
|
||||
- [ ] HF_TOKEN in secrets for Hub upload
|
||||
|
||||
**The script in `scripts/convert_to_gguf.py` incorporates all these lessons and has been tested successfully in production.**
|
||||
@@ -0,0 +1,283 @@
|
||||
# Hardware Selection Guide
|
||||
|
||||
Choosing the right hardware (flavor) is critical for cost-effective training.
|
||||
|
||||
## Available Hardware
|
||||
|
||||
### CPU
|
||||
- `cpu-basic` - Basic CPU, testing only
|
||||
- `cpu-upgrade` - Enhanced CPU
|
||||
|
||||
**Use cases:** Dataset validation, preprocessing, testing scripts
|
||||
**Not recommended for training:** Too slow for any meaningful training
|
||||
|
||||
### GPU Options
|
||||
|
||||
| Flavor | GPU | Memory | Use Case | Cost/hour |
|
||||
|--------|-----|--------|----------|-----------|
|
||||
| `t4-small` | NVIDIA T4 | 16GB | <1B models, demos | ~$0.50-1 |
|
||||
| `t4-medium` | NVIDIA T4 | 16GB | 1-3B models, development | ~$1-2 |
|
||||
| `l4x1` | NVIDIA L4 | 24GB | 3-7B models, efficient training | ~$2-3 |
|
||||
| `l4x4` | 4x NVIDIA L4 | 96GB | Multi-GPU training | ~$8-12 |
|
||||
| `a10g-small` | NVIDIA A10G | 24GB | 3-7B models, production | ~$3-4 |
|
||||
| `a10g-large` | NVIDIA A10G | 24GB | 7-13B models | ~$4-6 |
|
||||
| `a10g-largex2` | 2x NVIDIA A10G | 48GB | Multi-GPU, large models | ~$8-12 |
|
||||
| `a10g-largex4` | 4x NVIDIA A10G | 96GB | Multi-GPU, very large models | ~$16-24 |
|
||||
| `a100-large` | NVIDIA A100 | 40GB | 13B+ models, fast training | ~$8-12 |
|
||||
|
||||
### TPU Options
|
||||
|
||||
| Flavor | Type | Use Case |
|
||||
|--------|------|----------|
|
||||
| `v5e-1x1` | TPU v5e | Small TPU workloads |
|
||||
| `v5e-2x2` | 4x TPU v5e | Medium TPU workloads |
|
||||
| `v5e-2x4` | 8x TPU v5e | Large TPU workloads |
|
||||
|
||||
**Note:** TPUs require TPU-optimized code. Most TRL training uses GPUs.
|
||||
|
||||
## Selection Guidelines
|
||||
|
||||
### By Model Size
|
||||
|
||||
**Tiny Models (<1B parameters)**
|
||||
- **Recommended:** `t4-small`
|
||||
- **Example:** Qwen2.5-0.5B, TinyLlama
|
||||
- **Batch size:** 4-8
|
||||
- **Training time:** 1-2 hours for 1K examples
|
||||
|
||||
**Small Models (1-3B parameters)**
|
||||
- **Recommended:** `t4-medium` or `a10g-small`
|
||||
- **Example:** Qwen2.5-1.5B, Phi-2
|
||||
- **Batch size:** 2-4
|
||||
- **Training time:** 2-4 hours for 10K examples
|
||||
|
||||
**Medium Models (3-7B parameters)**
|
||||
- **Recommended:** `a10g-small` or `a10g-large`
|
||||
- **Example:** Qwen2.5-7B, Mistral-7B
|
||||
- **Batch size:** 1-2 (or LoRA with 4-8)
|
||||
- **Training time:** 4-8 hours for 10K examples
|
||||
|
||||
**Large Models (7-13B parameters)**
|
||||
- **Recommended:** `a10g-large` or `a100-large`
|
||||
- **Example:** Llama-3-8B, Mixtral-8x7B (with LoRA)
|
||||
- **Batch size:** 1 (full fine-tuning) or 2-4 (LoRA)
|
||||
- **Training time:** 6-12 hours for 10K examples
|
||||
- **Note:** Always use LoRA/PEFT
|
||||
|
||||
**Very Large Models (13B+ parameters)**
|
||||
- **Recommended:** `a100-large` with LoRA
|
||||
- **Example:** Llama-3-13B, Llama-3-70B (LoRA only)
|
||||
- **Batch size:** 1-2 with LoRA
|
||||
- **Training time:** 8-24 hours for 10K examples
|
||||
- **Note:** Full fine-tuning not feasible, use LoRA/PEFT
|
||||
|
||||
### By Budget
|
||||
|
||||
**Minimal Budget (<$5 total)**
|
||||
- Use `t4-small`
|
||||
- Train on subset of data (100-500 examples)
|
||||
- Limit to 1-2 epochs
|
||||
- Use small model (<1B)
|
||||
|
||||
**Small Budget ($5-20)**
|
||||
- Use `t4-medium` or `a10g-small`
|
||||
- Train on 1K-5K examples
|
||||
- 2-3 epochs
|
||||
- Model up to 3B parameters
|
||||
|
||||
**Medium Budget ($20-50)**
|
||||
- Use `a10g-small` or `a10g-large`
|
||||
- Train on 5K-20K examples
|
||||
- 3-5 epochs
|
||||
- Model up to 7B parameters
|
||||
|
||||
**Large Budget ($50-200)**
|
||||
- Use `a10g-large` or `a100-large`
|
||||
- Full dataset training
|
||||
- Multiple epochs
|
||||
- Model up to 13B parameters with LoRA
|
||||
|
||||
### By Training Type
|
||||
|
||||
**Quick Demo/Experiment**
|
||||
- `t4-small`
|
||||
- 50-100 examples
|
||||
- 5-10 steps
|
||||
- ~10-15 minutes
|
||||
|
||||
**Development/Iteration**
|
||||
- `t4-medium` or `a10g-small`
|
||||
- 1K examples
|
||||
- 1 epoch
|
||||
- ~30-60 minutes
|
||||
|
||||
**Production Training**
|
||||
- `a10g-large` or `a100-large`
|
||||
- Full dataset
|
||||
- 3-5 epochs
|
||||
- 4-12 hours
|
||||
|
||||
**Research/Experimentation**
|
||||
- `a100-large`
|
||||
- Multiple runs
|
||||
- Various hyperparameters
|
||||
- Budget for 20-50 hours
|
||||
|
||||
## Memory Considerations
|
||||
|
||||
### Estimating Memory Requirements
|
||||
|
||||
**Full fine-tuning:**
|
||||
```
|
||||
Memory (GB) ≈ (Model params in billions) × 20
|
||||
```
|
||||
|
||||
**LoRA fine-tuning:**
|
||||
```
|
||||
Memory (GB) ≈ (Model params in billions) × 4
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
- Qwen2.5-0.5B full: ~10GB ✅ fits t4-small
|
||||
- Qwen2.5-1.5B full: ~30GB ❌ exceeds most GPUs
|
||||
- Qwen2.5-1.5B LoRA: ~6GB ✅ fits t4-small
|
||||
- Qwen2.5-7B full: ~140GB ❌ not feasible
|
||||
- Qwen2.5-7B LoRA: ~28GB ✅ fits a10g-large
|
||||
|
||||
### Memory Optimization
|
||||
|
||||
If hitting memory limits:
|
||||
|
||||
1. **Use LoRA/PEFT**
|
||||
```python
|
||||
peft_config=LoraConfig(r=16, lora_alpha=32)
|
||||
```
|
||||
|
||||
2. **Reduce batch size**
|
||||
```python
|
||||
per_device_train_batch_size=1
|
||||
```
|
||||
|
||||
3. **Increase gradient accumulation**
|
||||
```python
|
||||
gradient_accumulation_steps=8 # Effective batch size = 1×8
|
||||
```
|
||||
|
||||
4. **Enable gradient checkpointing**
|
||||
```python
|
||||
gradient_checkpointing=True
|
||||
```
|
||||
|
||||
5. **Use mixed precision**
|
||||
```python
|
||||
bf16=True # or fp16=True
|
||||
```
|
||||
|
||||
6. **Upgrade to larger GPU**
|
||||
- t4 → a10g → a100
|
||||
|
||||
## Cost Estimation
|
||||
|
||||
### Formula
|
||||
|
||||
```
|
||||
Total Cost = (Hours of training) × (Cost per hour)
|
||||
```
|
||||
|
||||
### Example Calculations
|
||||
|
||||
**Quick demo:**
|
||||
- Hardware: t4-small ($0.75/hour)
|
||||
- Time: 15 minutes (0.25 hours)
|
||||
- Cost: $0.19
|
||||
|
||||
**Development training:**
|
||||
- Hardware: a10g-small ($3.50/hour)
|
||||
- Time: 2 hours
|
||||
- Cost: $7.00
|
||||
|
||||
**Production training:**
|
||||
- Hardware: a10g-large ($5/hour)
|
||||
- Time: 6 hours
|
||||
- Cost: $30.00
|
||||
|
||||
**Large model with LoRA:**
|
||||
- Hardware: a100-large ($10/hour)
|
||||
- Time: 8 hours
|
||||
- Cost: $80.00
|
||||
|
||||
### Cost Optimization Tips
|
||||
|
||||
1. **Start small:** Test on t4-small with subset
|
||||
2. **Use LoRA:** 4-5x cheaper than full fine-tuning
|
||||
3. **Optimize hyperparameters:** Fewer epochs if possible
|
||||
4. **Set appropriate timeout:** Don't waste compute on stalled jobs
|
||||
5. **Use checkpointing:** Resume if job fails
|
||||
6. **Monitor costs:** Check running jobs regularly
|
||||
|
||||
## Multi-GPU Training
|
||||
|
||||
TRL automatically handles multi-GPU training with Accelerate when using multi-GPU flavors.
|
||||
|
||||
**Multi-GPU flavors:**
|
||||
- `l4x4` - 4x L4 GPUs
|
||||
- `a10g-largex2` - 2x A10G GPUs
|
||||
- `a10g-largex4` - 4x A10G GPUs
|
||||
|
||||
**When to use:**
|
||||
- Models >13B parameters
|
||||
- Need faster training (linear speedup)
|
||||
- Large datasets (>50K examples)
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "train.py",
|
||||
"flavor": "a10g-largex2", # 2 GPUs
|
||||
"timeout": "4h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
No code changes needed—TRL/Accelerate handles distribution automatically.
|
||||
|
||||
## Choosing Between Options
|
||||
|
||||
### a10g vs a100
|
||||
|
||||
**Choose a10g when:**
|
||||
- Model <13B parameters
|
||||
- Budget conscious
|
||||
- Training time not critical
|
||||
|
||||
**Choose a100 when:**
|
||||
- Model 13B+ parameters
|
||||
- Need fastest training
|
||||
- Memory requirements high
|
||||
- Budget allows
|
||||
|
||||
### Single vs Multi-GPU
|
||||
|
||||
**Choose single GPU when:**
|
||||
- Model <7B parameters
|
||||
- Budget constrained
|
||||
- Simpler debugging
|
||||
|
||||
**Choose multi-GPU when:**
|
||||
- Model >13B parameters
|
||||
- Need faster training
|
||||
- Large batch sizes required
|
||||
- Cost-effective for large jobs
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```python
|
||||
# Model size → Hardware selection
|
||||
HARDWARE_MAP = {
|
||||
"<1B": "t4-small",
|
||||
"1-3B": "a10g-small",
|
||||
"3-7B": "a10g-large",
|
||||
"7-13B": "a10g-large (LoRA) or a100-large",
|
||||
">13B": "a100-large (LoRA required)"
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,364 @@
|
||||
# Saving Training Results to Hugging Face Hub
|
||||
|
||||
**⚠️ CRITICAL:** Training environments are ephemeral. ALL results are lost when a job completes unless pushed to the Hub.
|
||||
|
||||
## Why Hub Push is Required
|
||||
|
||||
When running on Hugging Face Jobs:
|
||||
- Environment is temporary
|
||||
- All files deleted on job completion
|
||||
- No local disk persistence
|
||||
- Cannot access results after job ends
|
||||
|
||||
**Without Hub push, training is completely wasted.**
|
||||
|
||||
## Required Configuration
|
||||
|
||||
### 1. Training Configuration
|
||||
|
||||
In your SFTConfig or trainer config:
|
||||
|
||||
```python
|
||||
SFTConfig(
|
||||
push_to_hub=True, # Enable Hub push
|
||||
hub_model_id="username/model-name", # Target repository
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Job Configuration
|
||||
|
||||
When submitting the job:
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "train.py",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"} # Provide authentication
|
||||
})
|
||||
```
|
||||
|
||||
**The `$HF_TOKEN` placeholder is automatically replaced with your Hugging Face token.**
|
||||
|
||||
## Complete Example
|
||||
|
||||
```python
|
||||
# train.py
|
||||
# /// script
|
||||
# dependencies = ["trl"]
|
||||
# ///
|
||||
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
from datasets import load_dataset
|
||||
|
||||
dataset = load_dataset("trl-lib/Capybara", split="train")
|
||||
|
||||
# Configure with Hub push
|
||||
config = SFTConfig(
|
||||
output_dir="my-model",
|
||||
num_train_epochs=3,
|
||||
|
||||
# ✅ CRITICAL: Hub push configuration
|
||||
push_to_hub=True,
|
||||
hub_model_id="myusername/my-trained-model",
|
||||
|
||||
# Optional: Push strategy
|
||||
push_to_hub_model_id="myusername/my-trained-model",
|
||||
push_to_hub_organization=None,
|
||||
push_to_hub_token=None, # Uses environment token
|
||||
)
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset,
|
||||
args=config,
|
||||
)
|
||||
|
||||
trainer.train()
|
||||
|
||||
# ✅ Push final model
|
||||
trainer.push_to_hub()
|
||||
|
||||
print("✅ Model saved to: https://huggingface.co/myusername/my-trained-model")
|
||||
```
|
||||
|
||||
**Submit with authentication:**
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "train.py",
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "2h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Required!
|
||||
})
|
||||
```
|
||||
|
||||
## What Gets Saved
|
||||
|
||||
When `push_to_hub=True`:
|
||||
|
||||
1. **Model weights** - Final trained parameters
|
||||
2. **Tokenizer** - Associated tokenizer
|
||||
3. **Configuration** - Model config (config.json)
|
||||
4. **Training arguments** - Hyperparameters used
|
||||
5. **Model card** - Auto-generated documentation
|
||||
6. **Checkpoints** - If `save_strategy="steps"` enabled
|
||||
|
||||
## Checkpoint Saving
|
||||
|
||||
Save intermediate checkpoints during training:
|
||||
|
||||
```python
|
||||
SFTConfig(
|
||||
output_dir="my-model",
|
||||
push_to_hub=True,
|
||||
hub_model_id="username/my-model",
|
||||
|
||||
# Checkpoint configuration
|
||||
save_strategy="steps",
|
||||
save_steps=100, # Save every 100 steps
|
||||
save_total_limit=3, # Keep only last 3 checkpoints
|
||||
)
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Resume training if job fails
|
||||
- Compare checkpoint performance
|
||||
- Use intermediate models
|
||||
|
||||
**Checkpoints are pushed to:** `username/my-model` (same repo)
|
||||
|
||||
## Authentication Methods
|
||||
|
||||
### Method 1: Automatic Token (Recommended)
|
||||
|
||||
```python
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
```
|
||||
|
||||
Uses your logged-in Hugging Face token automatically.
|
||||
|
||||
### Method 2: Explicit Token
|
||||
|
||||
```python
|
||||
"secrets": {"HF_TOKEN": "hf_abc123..."}
|
||||
```
|
||||
|
||||
Provide token explicitly (not recommended for security).
|
||||
|
||||
### Method 3: Environment Variable
|
||||
|
||||
```python
|
||||
"env": {"HF_TOKEN": "hf_abc123..."}
|
||||
```
|
||||
|
||||
Pass as regular environment variable (less secure than secrets).
|
||||
|
||||
**Always prefer Method 1** for security and convenience.
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Before submitting any training job, verify:
|
||||
|
||||
- [ ] `push_to_hub=True` in training config
|
||||
- [ ] `hub_model_id` is specified (format: `username/model-name`)
|
||||
- [ ] `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config
|
||||
- [ ] Repository name doesn't conflict with existing repos
|
||||
- [ ] You have write access to the target namespace
|
||||
|
||||
## Repository Setup
|
||||
|
||||
### Automatic Creation
|
||||
|
||||
If repository doesn't exist, it's created automatically when first pushing.
|
||||
|
||||
### Manual Creation
|
||||
|
||||
Create repository before training:
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi
|
||||
|
||||
api = HfApi()
|
||||
api.create_repo(
|
||||
repo_id="username/model-name",
|
||||
repo_type="model",
|
||||
private=False, # or True for private repo
|
||||
)
|
||||
```
|
||||
|
||||
### Repository Naming
|
||||
|
||||
**Valid names:**
|
||||
- `username/my-model`
|
||||
- `username/model-name`
|
||||
- `organization/model-name`
|
||||
|
||||
**Invalid names:**
|
||||
- `model-name` (missing username)
|
||||
- `username/model name` (spaces not allowed)
|
||||
- `username/MODEL` (uppercase discouraged)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Error: 401 Unauthorized
|
||||
|
||||
**Cause:** HF_TOKEN not provided or invalid
|
||||
|
||||
**Solutions:**
|
||||
1. Verify `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config
|
||||
2. Check you're logged in: `hf auth whoami`
|
||||
3. Re-login: `hf auth login`
|
||||
|
||||
### Error: 403 Forbidden
|
||||
|
||||
**Cause:** No write access to repository
|
||||
|
||||
**Solutions:**
|
||||
1. Check repository namespace matches your username
|
||||
2. Verify you're a member of organization (if using org namespace)
|
||||
3. Check repository isn't private (if accessing org repo)
|
||||
|
||||
### Error: Repository not found
|
||||
|
||||
**Cause:** Repository doesn't exist and auto-creation failed
|
||||
|
||||
**Solutions:**
|
||||
1. Manually create repository first
|
||||
2. Check repository name format
|
||||
3. Verify namespace exists
|
||||
|
||||
### Error: Push failed during training
|
||||
|
||||
**Cause:** Network issues or Hub unavailable
|
||||
|
||||
**Solutions:**
|
||||
1. Training continues but final push fails
|
||||
2. Checkpoints may be saved
|
||||
3. Re-run push manually after job completes
|
||||
|
||||
### Issue: Model saved but not visible
|
||||
|
||||
**Possible causes:**
|
||||
1. Repository is private—check https://huggingface.co/username
|
||||
2. Wrong namespace—verify `hub_model_id` matches login
|
||||
3. Push still in progress—wait a few minutes
|
||||
|
||||
## Manual Push After Training
|
||||
|
||||
If training completes but push fails, push manually:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
# Load from local checkpoint
|
||||
model = AutoModel.from_pretrained("./output_dir")
|
||||
tokenizer = AutoTokenizer.from_pretrained("./output_dir")
|
||||
|
||||
# Push to Hub
|
||||
model.push_to_hub("username/model-name", token="hf_abc123...")
|
||||
tokenizer.push_to_hub("username/model-name", token="hf_abc123...")
|
||||
```
|
||||
|
||||
**Note:** Only possible if job hasn't completed (files still exist).
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always enable `push_to_hub=True`**
|
||||
2. **Use checkpoint saving** for long training runs
|
||||
3. **Verify Hub push** in logs before job completes
|
||||
4. **Set appropriate `save_total_limit`** to avoid excessive checkpoints
|
||||
5. **Use descriptive repo names** (e.g., `qwen-capybara-sft` not `model1`)
|
||||
6. **Add model card** with training details
|
||||
7. **Tag models** with relevant tags (e.g., `text-generation`, `fine-tuned`)
|
||||
|
||||
## Monitoring Push Progress
|
||||
|
||||
Check logs for push progress:
|
||||
|
||||
```python
|
||||
hf_jobs("logs", {"job_id": "your-job-id"})
|
||||
```
|
||||
|
||||
**Look for:**
|
||||
```
|
||||
Pushing model to username/model-name...
|
||||
Upload file pytorch_model.bin: 100%
|
||||
✅ Model pushed successfully
|
||||
```
|
||||
|
||||
## Example: Full Production Setup
|
||||
|
||||
```python
|
||||
# production_train.py
|
||||
# /// script
|
||||
# dependencies = ["trl>=0.12.0", "peft>=0.7.0"]
|
||||
# ///
|
||||
|
||||
from datasets import load_dataset
|
||||
from peft import LoraConfig
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
import os
|
||||
|
||||
# Verify token is available
|
||||
assert "HF_TOKEN" in os.environ, "HF_TOKEN not found in environment!"
|
||||
|
||||
# Load dataset
|
||||
dataset = load_dataset("trl-lib/Capybara", split="train")
|
||||
print(f"✅ Dataset loaded: {len(dataset)} examples")
|
||||
|
||||
# Configure with comprehensive Hub settings
|
||||
config = SFTConfig(
|
||||
output_dir="qwen-capybara-sft",
|
||||
|
||||
# Hub configuration
|
||||
push_to_hub=True,
|
||||
hub_model_id="myusername/qwen-capybara-sft",
|
||||
hub_strategy="checkpoint", # Push checkpoints
|
||||
|
||||
# Checkpoint configuration
|
||||
save_strategy="steps",
|
||||
save_steps=100,
|
||||
save_total_limit=3,
|
||||
|
||||
# Training settings
|
||||
num_train_epochs=3,
|
||||
per_device_train_batch_size=4,
|
||||
|
||||
# Logging
|
||||
logging_steps=10,
|
||||
logging_first_step=True,
|
||||
)
|
||||
|
||||
# Train with LoRA
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset,
|
||||
args=config,
|
||||
peft_config=LoraConfig(r=16, lora_alpha=32),
|
||||
)
|
||||
|
||||
print("🚀 Starting training...")
|
||||
trainer.train()
|
||||
|
||||
print("💾 Pushing final model to Hub...")
|
||||
trainer.push_to_hub()
|
||||
|
||||
print("✅ Training complete!")
|
||||
print(f"Model available at: https://huggingface.co/myusername/qwen-capybara-sft")
|
||||
```
|
||||
|
||||
**Submit:**
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "production_train.py",
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "6h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
## Key Takeaway
|
||||
|
||||
**Without `push_to_hub=True` and `secrets={"HF_TOKEN": "$HF_TOKEN"}`, all training results are permanently lost.**
|
||||
|
||||
Always verify both are configured before submitting any training job.
|
||||
@@ -0,0 +1,231 @@
|
||||
# Local Training on macOS (Apple Silicon)
|
||||
|
||||
Run small LoRA fine-tuning jobs locally on Mac for smoke tests and quick iteration before submitting to HF Jobs.
|
||||
|
||||
## When to Use Local Mac vs HF Jobs
|
||||
|
||||
| Local Mac | HF Jobs / Cloud GPU |
|
||||
|-----------|-------------------|
|
||||
| Model ≤3B, text-only | Model 7B+ |
|
||||
| LoRA/PEFT only | QLoRA 4-bit (CUDA/bitsandbytes) |
|
||||
| Short context (≤1024) | Long context / full fine-tuning |
|
||||
| Smoke tests, dataset validation | Production runs, VLMs |
|
||||
|
||||
**Typical workflow:** local smoke test → HF Jobs with same config → export/quantize ([gguf_conversion.md](gguf_conversion.md))
|
||||
|
||||
## Recommended Defaults
|
||||
|
||||
| Setting | Value | Notes |
|
||||
|---------|-------|-------|
|
||||
| Model size | 0.5B–1.5B first run | Scale up after verifying |
|
||||
| Max seq length | 512–1024 | Lower = less memory |
|
||||
| Batch size | 1 | Scale via gradient accumulation |
|
||||
| Gradient accumulation | 8–16 | Effective batch = 8–16 |
|
||||
| LoRA rank (r) | 8–16 | alpha = 2×r |
|
||||
| Dtype | float32 | fp16 causes NaN on MPS; bf16 only on M1 Pro+ and M2/M3/M4 |
|
||||
|
||||
### Memory by hardware
|
||||
|
||||
| Unified RAM | Max Model Size |
|
||||
|-------------|---------------|
|
||||
| 16 GB | ~0.5B–1.5B |
|
||||
| 32 GB | ~1.5B–3B |
|
||||
| 64 GB | ~3B (short context) |
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
xcode-select --install
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -U "torch>=2.2" "transformers>=4.40" "trl>=0.12" "peft>=0.10" \
|
||||
datasets accelerate safetensors huggingface_hub
|
||||
```
|
||||
|
||||
Verify MPS:
|
||||
```bash
|
||||
python -c "import torch; print(torch.__version__, '| MPS:', torch.backends.mps.is_available())"
|
||||
```
|
||||
|
||||
Optional — configure Accelerate for local Mac (no distributed, no mixed precision, MPS device):
|
||||
```bash
|
||||
accelerate config
|
||||
```
|
||||
|
||||
## Training Script
|
||||
|
||||
<details>
|
||||
<summary><strong>train_lora_sft.py</strong></summary>
|
||||
|
||||
```python
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
|
||||
from peft import LoraConfig
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
|
||||
set_seed(42)
|
||||
|
||||
@dataclass
|
||||
class Cfg:
|
||||
model_id: str = os.environ.get("MODEL_ID", "Qwen/Qwen2.5-0.5B-Instruct")
|
||||
dataset_id: str = os.environ.get("DATASET_ID", "HuggingFaceH4/ultrachat_200k")
|
||||
dataset_split: str = os.environ.get("DATASET_SPLIT", "train_sft[:500]")
|
||||
data_files: Optional[str] = os.environ.get("DATA_FILES", None)
|
||||
text_field: str = os.environ.get("TEXT_FIELD", "")
|
||||
messages_field: str = os.environ.get("MESSAGES_FIELD", "messages")
|
||||
out_dir: str = os.environ.get("OUT_DIR", "outputs/local-lora")
|
||||
max_seq_length: int = int(os.environ.get("MAX_SEQ_LENGTH", "512"))
|
||||
max_steps: int = int(os.environ.get("MAX_STEPS", "-1"))
|
||||
|
||||
cfg = Cfg()
|
||||
device = "mps" if torch.backends.mps.is_available() else "cpu"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(cfg.model_id, use_fast=True)
|
||||
if tokenizer.pad_token is None:
|
||||
tokenizer.pad_token = tokenizer.eos_token
|
||||
tokenizer.padding_side = "right"
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(cfg.model_id, torch_dtype=torch.float32)
|
||||
model.to(device)
|
||||
model.config.use_cache = False
|
||||
|
||||
if cfg.data_files:
|
||||
ds = load_dataset("json", data_files=cfg.data_files, split="train")
|
||||
else:
|
||||
ds = load_dataset(cfg.dataset_id, split=cfg.dataset_split)
|
||||
|
||||
def format_example(ex):
|
||||
if cfg.text_field and isinstance(ex.get(cfg.text_field), str):
|
||||
ex["text"] = ex[cfg.text_field]
|
||||
return ex
|
||||
msgs = ex.get(cfg.messages_field)
|
||||
if isinstance(msgs, list):
|
||||
if hasattr(tokenizer, "apply_chat_template"):
|
||||
try:
|
||||
ex["text"] = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)
|
||||
return ex
|
||||
except Exception:
|
||||
pass
|
||||
ex["text"] = "\n".join([str(m) for m in msgs])
|
||||
return ex
|
||||
ex["text"] = str(ex)
|
||||
return ex
|
||||
|
||||
ds = ds.map(format_example)
|
||||
ds = ds.remove_columns([c for c in ds.column_names if c != "text"])
|
||||
|
||||
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
|
||||
task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
|
||||
|
||||
sft_kwargs = dict(
|
||||
output_dir=cfg.out_dir, per_device_train_batch_size=1, gradient_accumulation_steps=8,
|
||||
learning_rate=2e-4, logging_steps=10, save_steps=200, save_total_limit=2,
|
||||
gradient_checkpointing=True, report_to="none", fp16=False, bf16=False,
|
||||
max_seq_length=cfg.max_seq_length, dataset_text_field="text",
|
||||
)
|
||||
if cfg.max_steps > 0:
|
||||
sft_kwargs["max_steps"] = cfg.max_steps
|
||||
else:
|
||||
sft_kwargs["num_train_epochs"] = 1
|
||||
|
||||
trainer = SFTTrainer(model=model, train_dataset=ds, peft_config=lora,
|
||||
args=SFTConfig(**sft_kwargs), processing_class=tokenizer)
|
||||
trainer.train()
|
||||
trainer.save_model(cfg.out_dir)
|
||||
print(f"✅ Saved to: {cfg.out_dir}")
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
python train_lora_sft.py
|
||||
```
|
||||
|
||||
**Env overrides:**
|
||||
|
||||
```bash
|
||||
MODEL_ID="Qwen/Qwen2.5-1.5B-Instruct" python train_lora_sft.py # different model
|
||||
MAX_STEPS=50 python train_lora_sft.py # quick 50-step test
|
||||
DATA_FILES="my_data.jsonl" python train_lora_sft.py # local JSONL file
|
||||
PYTORCH_ENABLE_MPS_FALLBACK=1 python train_lora_sft.py # MPS op fallback to CPU
|
||||
PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 python train_lora_sft.py # disable MPS memory limit (use with caution)
|
||||
```
|
||||
|
||||
**Local JSONL format** — chat messages or plain text:
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi!"}]}
|
||||
```
|
||||
```jsonl
|
||||
{"text": "User: Hello\nAssistant: Hi!"}
|
||||
```
|
||||
For plain text: `DATA_FILES="file.jsonl" TEXT_FIELD="text" MESSAGES_FIELD="" python train_lora_sft.py`
|
||||
|
||||
### Verify Success
|
||||
|
||||
- Loss decreases over steps
|
||||
- `outputs/local-lora/` contains `adapter_config.json` + `*.safetensors`
|
||||
|
||||
## Quick Evaluation
|
||||
|
||||
<details>
|
||||
<summary><strong>eval_generate.py</strong></summary>
|
||||
|
||||
```python
|
||||
import os, torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
from peft import PeftModel
|
||||
|
||||
BASE = os.environ.get("MODEL_ID", "Qwen/Qwen2.5-0.5B-Instruct")
|
||||
ADAPTER = os.environ.get("ADAPTER_DIR", "outputs/local-lora")
|
||||
device = "mps" if torch.backends.mps.is_available() else "cpu"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(BASE, use_fast=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float32)
|
||||
model.to(device)
|
||||
model = PeftModel.from_pretrained(model, ADAPTER)
|
||||
|
||||
prompt = os.environ.get("PROMPT", "Explain gradient accumulation in 3 bullet points.")
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.9)
|
||||
print(tokenizer.decode(out[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Troubleshooting (macOS-Specific)
|
||||
|
||||
For general training issues, see [troubleshooting.md](troubleshooting.md).
|
||||
|
||||
| Problem | Fix |
|
||||
|---------|-----|
|
||||
| MPS unsupported op / crash | `PYTORCH_ENABLE_MPS_FALLBACK=1` |
|
||||
| OOM / system instability | Reduce `MAX_SEQ_LENGTH`, use smaller model, set `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0` (caution) |
|
||||
| fp16 NaN / loss explosion | Keep `fp16=False` (default), lower learning rate |
|
||||
| LoRA "module not found" | Print `model.named_modules()` to find correct target names |
|
||||
| TRL TypeError on args | Check TRL version; script uses `SFTConfig` + `processing_class` (TRL ≥0.12) |
|
||||
| Intel Mac | No MPS — use HF Jobs instead |
|
||||
|
||||
**Common LoRA target modules by architecture:**
|
||||
|
||||
| Architecture | target_modules |
|
||||
|-------------|---------------|
|
||||
| Llama/Qwen/Mistral | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
|
||||
| GPT-2/GPT-J | `c_attn`, `c_proj` |
|
||||
| BLOOM | `query_key_value`, `dense` |
|
||||
|
||||
## MLX Alternative
|
||||
|
||||
[MLX](https://github.com/ml-explore/mlx) offers tighter Apple Silicon integration but has a smaller ecosystem and less mature training APIs. For this skill's workflow (local validation → HF Jobs), PyTorch + MPS is recommended for consistency. See [mlx-lm](https://github.com/ml-explore/mlx-lm) for MLX-based fine-tuning.
|
||||
|
||||
## See Also
|
||||
|
||||
- [troubleshooting.md](troubleshooting.md) — General TRL troubleshooting
|
||||
- [hardware_guide.md](hardware_guide.md) — GPU selection for HF Jobs
|
||||
- [gguf_conversion.md](gguf_conversion.md) — Export for on-device inference
|
||||
- [training_methods.md](training_methods.md) — SFT, DPO, GRPO overview
|
||||
@@ -0,0 +1,371 @@
|
||||
# Reliability Principles for Training Jobs
|
||||
|
||||
These principles are derived from real production failures and successful fixes. Following them prevents common failure modes and ensures reliable job execution.
|
||||
|
||||
## Principle 1: Always Verify Before Use
|
||||
|
||||
**Rule:** Never assume repos, datasets, or resources exist. Verify with tools first.
|
||||
|
||||
### What It Prevents
|
||||
|
||||
- **Non-existent datasets** - Jobs fail immediately when dataset doesn't exist
|
||||
- **Typos in names** - Simple mistakes like "argilla-dpo-mix-7k" vs "ultrafeedback_binarized"
|
||||
- **Incorrect paths** - Old or moved repos, renamed files
|
||||
- **Missing dependencies** - Undocumented requirements
|
||||
|
||||
### How to Apply
|
||||
|
||||
**Before submitting ANY job:**
|
||||
|
||||
```python
|
||||
# Verify dataset exists
|
||||
dataset_search({"query": "dataset-name", "author": "author-name", "limit": 5})
|
||||
hub_repo_details(["author/dataset-name"], repo_type="dataset")
|
||||
|
||||
# Verify model exists
|
||||
hub_repo_details(["org/model-name"], repo_type="model")
|
||||
|
||||
# Check script/file paths (for URL-based scripts)
|
||||
# Verify before using: https://github.com/user/repo/blob/main/script.py
|
||||
```
|
||||
|
||||
**Examples that would have caught errors:**
|
||||
|
||||
```python
|
||||
# ❌ WRONG: Assumed dataset exists
|
||||
hf_jobs("uv", {
|
||||
"script": """...""",
|
||||
"env": {"DATASET": "trl-lib/argilla-dpo-mix-7k"} # Doesn't exist!
|
||||
})
|
||||
|
||||
# ✅ CORRECT: Verify first
|
||||
dataset_search({"query": "argilla dpo", "author": "trl-lib"})
|
||||
# Would show: "trl-lib/ultrafeedback_binarized" is the correct name
|
||||
|
||||
hub_repo_details(["trl-lib/ultrafeedback_binarized"], repo_type="dataset")
|
||||
# Confirms it exists before using
|
||||
```
|
||||
|
||||
### Implementation Checklist
|
||||
|
||||
- [ ] Check dataset exists before training
|
||||
- [ ] Verify base model exists before fine-tuning
|
||||
- [ ] Confirm adapter model exists before GGUF conversion
|
||||
- [ ] Test script URLs are valid before submitting
|
||||
- [ ] Validate file paths in repositories
|
||||
- [ ] Check for recent updates/renames of resources
|
||||
|
||||
**Time cost:** 5-10 seconds
|
||||
**Time saved:** Hours of failed job time + debugging
|
||||
|
||||
---
|
||||
|
||||
## Principle 2: Prioritize Reliability Over Performance
|
||||
|
||||
**Rule:** Default to what is most likely to succeed, not what is theoretically fastest.
|
||||
|
||||
### What It Prevents
|
||||
|
||||
- **Hardware incompatibilities** - Features that fail on certain GPUs
|
||||
- **Unstable optimizations** - Speed-ups that cause crashes
|
||||
- **Complex configurations** - More failure points
|
||||
- **Build system issues** - Unreliable compilation methods
|
||||
|
||||
### How to Apply
|
||||
|
||||
**Choose reliability:**
|
||||
|
||||
```python
|
||||
# ❌ RISKY: Aggressive optimization that may fail
|
||||
SFTConfig(
|
||||
torch_compile=True, # Can fail on T4, A10G GPUs
|
||||
optim="adamw_bnb_8bit", # Requires specific setup
|
||||
fp16=False, # May cause training instability
|
||||
...
|
||||
)
|
||||
|
||||
# ✅ SAFE: Proven defaults
|
||||
SFTConfig(
|
||||
# torch_compile=True, # Commented with note: "Enable on H100 for 20% speedup"
|
||||
optim="adamw_torch", # Standard, always works
|
||||
fp16=True, # Stable and fast
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
**For build processes:**
|
||||
|
||||
```python
|
||||
# ❌ UNRELIABLE: Uses make (platform-dependent)
|
||||
subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"], check=True)
|
||||
|
||||
# ✅ RELIABLE: Uses CMake (consistent, documented)
|
||||
subprocess.run([
|
||||
"cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
|
||||
"-DGGML_CUDA=OFF" # Disable CUDA for faster, more reliable build
|
||||
], check=True)
|
||||
|
||||
subprocess.run([
|
||||
"cmake", "--build", "/tmp/llama.cpp/build",
|
||||
"--target", "llama-quantize", "-j", "4"
|
||||
], check=True)
|
||||
```
|
||||
|
||||
### Real-World Example
|
||||
|
||||
**The `torch.compile` failure:**
|
||||
- Added for "20% speedup" on H100
|
||||
- **Failed fatally on T4-medium** with cryptic error
|
||||
- Misdiagnosed as dataset issue (cost hours)
|
||||
- **Fix:** Disable by default, add as optional comment
|
||||
|
||||
**Result:** Reliability > 20% performance gain
|
||||
|
||||
### Implementation Checklist
|
||||
|
||||
- [ ] Use proven, standard configurations by default
|
||||
- [ ] Comment out performance optimizations with hardware notes
|
||||
- [ ] Use stable build systems (CMake > make)
|
||||
- [ ] Test on target hardware before production
|
||||
- [ ] Document known incompatibilities
|
||||
- [ ] Provide "safe" and "fast" variants when needed
|
||||
|
||||
**Performance loss:** 10-20% in best case
|
||||
**Reliability gain:** 95%+ success rate vs 60-70%
|
||||
|
||||
---
|
||||
|
||||
## Principle 3: Create Atomic, Self-Contained Scripts
|
||||
|
||||
**Rule:** Scripts should work as complete, independent units. Don't remove parts to "simplify."
|
||||
|
||||
### What It Prevents
|
||||
|
||||
- **Missing dependencies** - Removed "unnecessary" packages that are actually required
|
||||
- **Incomplete processes** - Skipped steps that seem redundant
|
||||
- **Environment assumptions** - Scripts that need pre-setup
|
||||
- **Partial failures** - Some parts work, others fail silently
|
||||
|
||||
### How to Apply
|
||||
|
||||
**Complete dependency specifications:**
|
||||
|
||||
```python
|
||||
# ❌ INCOMPLETE: "Simplified" by removing dependencies
|
||||
# /// script
|
||||
# dependencies = [
|
||||
# "transformers",
|
||||
# "peft",
|
||||
# "torch",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
# ✅ COMPLETE: All dependencies explicit
|
||||
# /// script
|
||||
# dependencies = [
|
||||
# "transformers>=4.36.0",
|
||||
# "peft>=0.7.0",
|
||||
# "torch>=2.0.0",
|
||||
# "accelerate>=0.24.0",
|
||||
# "huggingface_hub>=0.20.0",
|
||||
# "sentencepiece>=0.1.99", # Required for tokenizers
|
||||
# "protobuf>=3.20.0", # Required for tokenizers
|
||||
# "numpy",
|
||||
# "gguf",
|
||||
# ]
|
||||
# ///
|
||||
```
|
||||
|
||||
**Complete build processes:**
|
||||
|
||||
```python
|
||||
# ❌ INCOMPLETE: Assumes build tools exist
|
||||
subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
|
||||
subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"]) # FAILS: no gcc/make
|
||||
|
||||
# ✅ COMPLETE: Installs all requirements
|
||||
subprocess.run(["apt-get", "update", "-qq"], check=True)
|
||||
subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True)
|
||||
subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
|
||||
# ... then build
|
||||
```
|
||||
|
||||
### Real-World Example
|
||||
|
||||
**The `sentencepiece` failure:**
|
||||
- Original script had it: worked fine
|
||||
- "Simplified" version removed it: "doesn't look necessary"
|
||||
- **GGUF conversion failed silently** - tokenizer couldn't convert
|
||||
- Hard to debug: no obvious error message
|
||||
- **Fix:** Restore all original dependencies
|
||||
|
||||
**Result:** Don't remove dependencies without thorough testing
|
||||
|
||||
### Implementation Checklist
|
||||
|
||||
- [ ] All dependencies in PEP 723 header with version pins
|
||||
- [ ] All system packages installed by script
|
||||
- [ ] No assumptions about pre-existing environment
|
||||
- [ ] No "optional" steps that are actually required
|
||||
- [ ] Test scripts in clean environment
|
||||
- [ ] Document why each dependency is needed
|
||||
|
||||
**Complexity:** Slightly longer scripts
|
||||
**Reliability:** Scripts "just work" every time
|
||||
|
||||
---
|
||||
|
||||
## Principle 4: Provide Clear Error Context
|
||||
|
||||
**Rule:** When things fail, make it obvious what went wrong and how to fix it.
|
||||
|
||||
### How to Apply
|
||||
|
||||
**Wrap subprocess calls:**
|
||||
|
||||
```python
|
||||
# ❌ UNCLEAR: Silent failure
|
||||
subprocess.run([...], check=True, capture_output=True)
|
||||
|
||||
# ✅ CLEAR: Shows what failed
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[...],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
print(result.stdout)
|
||||
if result.stderr:
|
||||
print("Warnings:", result.stderr)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ Command failed!")
|
||||
print("STDOUT:", e.stdout)
|
||||
print("STDERR:", e.stderr)
|
||||
raise
|
||||
```
|
||||
|
||||
**Validate inputs:**
|
||||
|
||||
```python
|
||||
# ❌ UNCLEAR: Fails later with cryptic error
|
||||
model = load_model(MODEL_NAME)
|
||||
|
||||
# ✅ CLEAR: Fails fast with clear message
|
||||
if not MODEL_NAME:
|
||||
raise ValueError("MODEL_NAME environment variable not set!")
|
||||
|
||||
print(f"Loading model: {MODEL_NAME}")
|
||||
try:
|
||||
model = load_model(MODEL_NAME)
|
||||
print(f"✅ Model loaded successfully")
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to load model: {MODEL_NAME}")
|
||||
print(f"Error: {e}")
|
||||
print("Hint: Check that model exists on Hub")
|
||||
raise
|
||||
```
|
||||
|
||||
### Implementation Checklist
|
||||
|
||||
- [ ] Wrap external calls with try/except
|
||||
- [ ] Print stdout/stderr on failure
|
||||
- [ ] Validate environment variables early
|
||||
- [ ] Add progress indicators (✅, ❌, 🔄)
|
||||
- [ ] Include hints for common failures
|
||||
- [ ] Log configuration at start
|
||||
|
||||
---
|
||||
|
||||
## Principle 5: Test the Happy Path on Known-Good Inputs
|
||||
|
||||
**Rule:** Before using new code in production, test with inputs you know work.
|
||||
|
||||
### How to Apply
|
||||
|
||||
**Known-good test inputs:**
|
||||
|
||||
```python
|
||||
# For training
|
||||
TEST_DATASET = "trl-lib/Capybara" # Small, well-formatted, widely used
|
||||
TEST_MODEL = "Qwen/Qwen2.5-0.5B" # Small, fast, reliable
|
||||
|
||||
# For GGUF conversion
|
||||
TEST_ADAPTER = "evalstate/qwen-capybara-medium" # Known working model
|
||||
TEST_BASE = "Qwen/Qwen2.5-0.5B" # Compatible base
|
||||
```
|
||||
|
||||
**Testing workflow:**
|
||||
|
||||
1. Test with known-good inputs first
|
||||
2. If that works, try production inputs
|
||||
3. If production fails, you know it's the inputs (not code)
|
||||
4. Isolate the difference
|
||||
|
||||
### Implementation Checklist
|
||||
|
||||
- [ ] Maintain list of known-good test models/datasets
|
||||
- [ ] Test new scripts with test inputs first
|
||||
- [ ] Document what makes inputs "good"
|
||||
- [ ] Keep test jobs cheap (small models, short timeouts)
|
||||
- [ ] Only move to production after test succeeds
|
||||
|
||||
**Time cost:** 5-10 minutes for test run
|
||||
**Debugging time saved:** Hours
|
||||
|
||||
---
|
||||
|
||||
## Summary: The Reliability Checklist
|
||||
|
||||
Before submitting ANY job:
|
||||
|
||||
### Pre-Flight Checks
|
||||
- [ ] **Verified** all repos/datasets exist (hub_repo_details)
|
||||
- [ ] **Tested** with known-good inputs if new code
|
||||
- [ ] **Using** proven hardware/configuration
|
||||
- [ ] **Included** all dependencies in PEP 723 header
|
||||
- [ ] **Installed** system requirements (build tools, etc.)
|
||||
- [ ] **Set** appropriate timeout (not default 30m)
|
||||
- [ ] **Configured** Hub push with HF_TOKEN
|
||||
- [ ] **Added** clear error handling
|
||||
|
||||
### Script Quality
|
||||
- [ ] Self-contained (no external setup needed)
|
||||
- [ ] Complete dependencies listed
|
||||
- [ ] Build tools installed by script
|
||||
- [ ] Progress indicators included
|
||||
- [ ] Error messages are clear
|
||||
- [ ] Configuration logged at start
|
||||
|
||||
### Job Configuration
|
||||
- [ ] Timeout > expected runtime + 30% buffer
|
||||
- [ ] Hardware appropriate for model size
|
||||
- [ ] Secrets include HF_TOKEN
|
||||
- [ ] Environment variables set correctly
|
||||
- [ ] Cost estimated and acceptable
|
||||
|
||||
**Following these principles transforms job success rate from ~60-70% to ~95%+**
|
||||
|
||||
---
|
||||
|
||||
## When Principles Conflict
|
||||
|
||||
Sometimes reliability and performance conflict. Here's how to choose:
|
||||
|
||||
| Scenario | Choose | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| Demo/test | Reliability | Fast failure is worse than slow success |
|
||||
| Production (first run) | Reliability | Prove it works before optimizing |
|
||||
| Production (proven) | Performance | Safe to optimize after validation |
|
||||
| Time-critical | Reliability | Failures cause more delay than slow runs |
|
||||
| Cost-critical | Balanced | Test with small model, then optimize |
|
||||
|
||||
**General rule:** Reliability first, optimize second.
|
||||
|
||||
---
|
||||
|
||||
## Further Reading
|
||||
|
||||
- `troubleshooting.md` - Common issues and fixes
|
||||
- `training_patterns.md` - Proven training configurations
|
||||
- `gguf_conversion.md` - Production GGUF workflow
|
||||
@@ -0,0 +1,191 @@
|
||||
# Trackio Integration for TRL Training
|
||||
|
||||
**Trackio** is an experiment tracking library that provides real-time metrics visualization for remote training on Hugging Face Jobs infrastructure.
|
||||
|
||||
⚠️ **IMPORTANT**: For Jobs training (remote cloud GPUs):
|
||||
- Training happens on ephemeral cloud runners (not your local machine)
|
||||
- Trackio syncs metrics to a Hugging Face Space for real-time monitoring
|
||||
- Without a Space, metrics are lost when the job completes
|
||||
- The Space dashboard persists your training metrics permanently
|
||||
- Auto-created Spaces are **public by default** — pass `private=True` to `trackio.init()` if the metrics should not be public
|
||||
|
||||
## Setting Up Trackio for Jobs
|
||||
|
||||
**Step 1: Add trackio dependency**
|
||||
```python
|
||||
# /// script
|
||||
# dependencies = [
|
||||
# "trl>=0.12.0",
|
||||
# "trackio", # Required!
|
||||
# ]
|
||||
# ///
|
||||
```
|
||||
|
||||
**Step 2: Create a Trackio Space (one-time setup)**
|
||||
|
||||
**Option A: Let Trackio auto-create (Recommended)**
|
||||
Pass a `space_id` to `trackio.init()` and Trackio will automatically create the Space if it doesn't exist.
|
||||
|
||||
**Option B: Create manually**
|
||||
- Create Space via Hub UI at https://huggingface.co/new-space
|
||||
- Select Gradio SDK
|
||||
- OR use command: `hf repos create my-trackio-dashboard --type space --space-sdk gradio`
|
||||
|
||||
**Step 3: Initialize Trackio with space_id**
|
||||
```python
|
||||
import trackio
|
||||
|
||||
trackio.init(
|
||||
project="my-training",
|
||||
space_id="username/trackio", # CRITICAL for Jobs! Replace 'username' with your HF username
|
||||
private=True, # Spaces are PUBLIC by default; omit for a shareable dashboard
|
||||
config={
|
||||
"model": "Qwen/Qwen2.5-0.5B",
|
||||
"dataset": "trl-lib/Capybara",
|
||||
"learning_rate": 2e-5,
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
**Step 4: Configure TRL to use Trackio**
|
||||
```python
|
||||
SFTConfig(
|
||||
report_to="trackio",
|
||||
# ... other config
|
||||
)
|
||||
```
|
||||
|
||||
**Step 5: Finish tracking**
|
||||
```python
|
||||
trainer.train()
|
||||
trackio.finish() # Ensures final metrics are synced
|
||||
```
|
||||
|
||||
## What Trackio Tracks
|
||||
|
||||
Trackio automatically logs:
|
||||
- ✅ Training loss
|
||||
- ✅ Learning rate
|
||||
- ✅ GPU utilization and memory (when an NVIDIA GPU is detected and `nvidia-ml-py` is installed — true for standard Jobs GPU flavors)
|
||||
- ✅ Training throughput
|
||||
- ✅ Custom metrics
|
||||
|
||||
## How It Works with Jobs
|
||||
|
||||
1. **Training runs** → Metrics stream to the running Space in small batches (sub-second)
|
||||
2. **Space unreachable or still building** → Metrics spill to an HF Bucket (auto-derived from `space_id`, or pinned with `bucket_id=`) every ~30 seconds
|
||||
3. **Space dashboard** → Stores its SQLite DB in the Bucket and ingests spilled metrics every ~15 seconds
|
||||
4. **Job completes** → `trackio.finish()` drains any pending metrics so everything is persisted
|
||||
|
||||
## Default Configuration Pattern
|
||||
|
||||
**Use sensible defaults for trackio configuration unless user requests otherwise.**
|
||||
|
||||
### Recommended Defaults
|
||||
|
||||
```python
|
||||
import trackio
|
||||
|
||||
trackio.init(
|
||||
project="qwen-capybara-sft",
|
||||
name="baseline-run", # Descriptive name user will recognize
|
||||
space_id="username/trackio", # Default space: {username}/trackio
|
||||
private=True, # Spaces are PUBLIC by default; omit for a shareable dashboard
|
||||
config={
|
||||
# Keep config minimal - hyperparameters and model/dataset info only
|
||||
"model": "Qwen/Qwen2.5-0.5B",
|
||||
"dataset": "trl-lib/Capybara",
|
||||
"learning_rate": 2e-5,
|
||||
"num_epochs": 3,
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
**Key principles:**
|
||||
- **Space ID**: Use `{username}/trackio` with "trackio" as default space name
|
||||
- **Run naming**: Unless otherwise specified, name the run in a way the user will recognize
|
||||
- **Config**: Keep minimal - don't automatically capture job metadata unless requested
|
||||
- **Grouping**: Optional - only use if user requests organizing related experiments
|
||||
|
||||
## Grouping Runs (Optional)
|
||||
|
||||
The `group` parameter helps organize related runs together in the dashboard sidebar. This is useful when user is running multiple experiments with different configurations but wants to compare them together:
|
||||
|
||||
```python
|
||||
# Example: Group runs by experiment type
|
||||
trackio.init(project="my-project", run_name="baseline-run-1", group="baseline")
|
||||
trackio.init(project="my-project", run_name="augmented-run-1", group="augmented")
|
||||
trackio.init(project="my-project", run_name="tuned-run-1", group="tuned")
|
||||
```
|
||||
|
||||
Runs with the same group name can be grouped together in the sidebar, making it easier to compare related experiments. You can group by any configuration parameter:
|
||||
|
||||
```python
|
||||
# Hyperparameter sweep - group by learning rate
|
||||
trackio.init(project="hyperparam-sweep", run_name="lr-0.001-run", group="lr_0.001")
|
||||
trackio.init(project="hyperparam-sweep", run_name="lr-0.01-run", group="lr_0.01")
|
||||
```
|
||||
|
||||
## Environment Variables for Jobs
|
||||
|
||||
You can configure trackio using environment variables instead of passing parameters to `trackio.init()`. This is useful for managing configuration across multiple jobs.
|
||||
|
||||
|
||||
|
||||
**`HF_TOKEN`**
|
||||
Required for creating Spaces and writing metrics to the HF Bucket (passed via `secrets`):
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "...",
|
||||
"secrets": {
|
||||
"HF_TOKEN": "$HF_TOKEN" # Enables Space creation and Hub push
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Example with Environment Variables
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": """
|
||||
# Training script - trackio config from environment
|
||||
import trackio
|
||||
from datetime import datetime
|
||||
|
||||
# Auto-generate run name
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M")
|
||||
run_name = f"sft_qwen25_{timestamp}"
|
||||
|
||||
# Project and space_id can come from environment variables
|
||||
trackio.init(run_name=run_name, group="SFT")
|
||||
|
||||
# ... training code ...
|
||||
trackio.finish()
|
||||
""",
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "2h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
**When to use environment variables:**
|
||||
- Managing multiple jobs with same configuration
|
||||
- Keeping training scripts portable across projects
|
||||
- Separating configuration from code
|
||||
|
||||
**When to use direct parameters:**
|
||||
- Single job with specific configuration
|
||||
- When clarity in code is preferred
|
||||
- When each job has different project/space
|
||||
|
||||
## Viewing the Dashboard
|
||||
|
||||
After starting training:
|
||||
1. Navigate to the Space: `https://huggingface.co/spaces/username/trackio`
|
||||
2. The Gradio dashboard shows all tracked experiments
|
||||
3. Filter by project, compare runs, view charts with smoothing
|
||||
|
||||
## Recommendation
|
||||
|
||||
- **Trackio**: Best for real-time monitoring during long training runs
|
||||
- **Weights & Biases**: Best for team collaboration, requires account
|
||||
@@ -0,0 +1,150 @@
|
||||
# TRL Training Methods Overview
|
||||
|
||||
TRL (Transformer Reinforcement Learning) provides multiple training methods for fine-tuning and aligning language models. This reference provides a brief overview of each method.
|
||||
|
||||
## Supervised Fine-Tuning (SFT)
|
||||
|
||||
**What it is:** Standard instruction tuning with supervised learning on demonstration data.
|
||||
|
||||
**When to use:**
|
||||
- Initial fine-tuning of base models on task-specific data
|
||||
- Teaching new capabilities or domains
|
||||
- Most common starting point for fine-tuning
|
||||
|
||||
**Dataset format:** Conversational format with "messages" field, OR text field, OR prompt/completion pairs
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset,
|
||||
args=SFTConfig(
|
||||
output_dir="my-model",
|
||||
push_to_hub=True,
|
||||
hub_model_id="username/my-model",
|
||||
eval_strategy="no", # Disable eval for simple example
|
||||
# max_length=1024 is the default - only set if you need different length
|
||||
)
|
||||
)
|
||||
trainer.train()
|
||||
```
|
||||
|
||||
**Note:** For production training with evaluation monitoring, see `scripts/train_sft_example.py`
|
||||
|
||||
**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/sft_trainer")`
|
||||
|
||||
## Direct Preference Optimization (DPO)
|
||||
|
||||
**What it is:** Alignment method that trains directly on preference pairs (chosen vs rejected responses) without requiring a reward model.
|
||||
|
||||
**When to use:**
|
||||
- Aligning models to human preferences
|
||||
- Improving response quality after SFT
|
||||
- Have paired preference data (chosen/rejected responses)
|
||||
|
||||
**Dataset format:** Preference pairs with "chosen" and "rejected" fields
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from trl import DPOTrainer, DPOConfig
|
||||
|
||||
trainer = DPOTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B-Instruct", # Use instruct model
|
||||
train_dataset=dataset,
|
||||
args=DPOConfig(
|
||||
output_dir="dpo-model",
|
||||
beta=0.1, # KL penalty coefficient
|
||||
eval_strategy="no", # Disable eval for simple example
|
||||
# max_length=1024 is the default - only set if you need different length
|
||||
)
|
||||
)
|
||||
trainer.train()
|
||||
```
|
||||
|
||||
**Note:** For production training with evaluation monitoring, see `scripts/train_dpo_example.py`
|
||||
|
||||
**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")`
|
||||
|
||||
## Group Relative Policy Optimization (GRPO)
|
||||
|
||||
**What it is:** Online RL method that optimizes relative to group performance, useful for tasks with verifiable rewards.
|
||||
|
||||
**When to use:**
|
||||
- Tasks with automatic reward signals (code execution, math verification)
|
||||
- Online learning scenarios
|
||||
- When DPO offline data is insufficient
|
||||
|
||||
**Dataset format:** Prompt-only format (model generates responses, reward computed online)
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# Use TRL maintained script
|
||||
hf_jobs("uv", {
|
||||
"script": "https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py",
|
||||
"script_args": [
|
||||
"--model_name_or_path", "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"--dataset_name", "trl-lib/math_shepherd",
|
||||
"--output_dir", "grpo-model"
|
||||
],
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "4h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/grpo_trainer")`
|
||||
|
||||
## Reward Modeling
|
||||
|
||||
**What it is:** Train a reward model to score responses, used as a component in RLHF pipelines.
|
||||
|
||||
**When to use:**
|
||||
- Building RLHF pipeline
|
||||
- Need automatic quality scoring
|
||||
- Creating reward signals for PPO training
|
||||
|
||||
**Dataset format:** Preference pairs with "chosen" and "rejected" responses
|
||||
|
||||
**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/reward_trainer")`
|
||||
|
||||
## Method Selection Guide
|
||||
|
||||
| Method | Complexity | Data Required | Use Case |
|
||||
|--------|-----------|---------------|----------|
|
||||
| **SFT** | Low | Demonstrations | Initial fine-tuning |
|
||||
| **DPO** | Medium | Paired preferences | Post-SFT alignment |
|
||||
| **GRPO** | Medium | Prompts + reward fn | Online RL with automatic rewards |
|
||||
| **Reward** | Medium | Paired preferences | Building RLHF pipeline |
|
||||
|
||||
## Recommended Pipeline
|
||||
|
||||
**For most use cases:**
|
||||
1. **Start with SFT** - Fine-tune base model on task data
|
||||
2. **Follow with DPO** - Align to preferences using paired data
|
||||
3. **Optional: GGUF conversion** - Deploy for local inference
|
||||
|
||||
**For advanced RL scenarios:**
|
||||
1. **Start with SFT** - Fine-tune base model
|
||||
2. **Train reward model** - On preference data
|
||||
|
||||
## Dataset Format Reference
|
||||
|
||||
For complete dataset format specifications, use:
|
||||
```python
|
||||
hf_doc_fetch("https://huggingface.co/docs/trl/dataset_formats")
|
||||
```
|
||||
|
||||
Or validate your dataset:
|
||||
```bash
|
||||
uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \
|
||||
--dataset your/dataset --split train
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- `references/training_patterns.md` - Common training patterns and examples
|
||||
- `scripts/train_sft_example.py` - Complete SFT template
|
||||
- `scripts/train_dpo_example.py` - Complete DPO template
|
||||
- [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Dataset format validation tool
|
||||
@@ -0,0 +1,203 @@
|
||||
# Common Training Patterns
|
||||
|
||||
This guide provides common training patterns and use cases for TRL on Hugging Face Jobs.
|
||||
|
||||
## Multi-GPU Training
|
||||
|
||||
Automatic distributed training across multiple GPUs. TRL/Accelerate handles distribution automatically:
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": """
|
||||
# Your training script here (same as single GPU)
|
||||
# No changes needed - Accelerate detects multiple GPUs
|
||||
""",
|
||||
"flavor": "a10g-largex2", # 2x A10G GPUs
|
||||
"timeout": "4h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
**Tips for multi-GPU:**
|
||||
- No code changes needed
|
||||
- Use `per_device_train_batch_size` (per GPU, not total)
|
||||
- Effective batch size = `per_device_train_batch_size` × `num_gpus` × `gradient_accumulation_steps`
|
||||
- Monitor GPU utilization to ensure both GPUs are being used
|
||||
|
||||
## DPO Training (Preference Learning)
|
||||
|
||||
Train with preference data for alignment:
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": """
|
||||
# /// script
|
||||
# dependencies = ["trl>=0.12.0", "trackio"]
|
||||
# ///
|
||||
|
||||
from datasets import load_dataset
|
||||
from trl import DPOTrainer, DPOConfig
|
||||
import trackio
|
||||
|
||||
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
|
||||
|
||||
# Create train/eval split
|
||||
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
|
||||
|
||||
config = DPOConfig(
|
||||
output_dir="dpo-model",
|
||||
push_to_hub=True,
|
||||
hub_model_id="username/dpo-model",
|
||||
num_train_epochs=1,
|
||||
beta=0.1, # KL penalty coefficient
|
||||
eval_strategy="steps",
|
||||
eval_steps=50,
|
||||
report_to="trackio",
|
||||
run_name="baseline_run", # use a meaningful run name
|
||||
# max_length=1024, # Default - only set if you need different sequence length
|
||||
)
|
||||
|
||||
trainer = DPOTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B-Instruct", # Use instruct model as base
|
||||
train_dataset=dataset_split["train"],
|
||||
eval_dataset=dataset_split["test"], # IMPORTANT: Provide eval_dataset when eval_strategy is enabled
|
||||
args=config,
|
||||
)
|
||||
|
||||
trainer.train()
|
||||
trainer.push_to_hub()
|
||||
trackio.finish()
|
||||
""",
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "3h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
**For DPO documentation:** Use `hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")`
|
||||
|
||||
## GRPO Training (Online RL)
|
||||
|
||||
Group Relative Policy Optimization for online reinforcement learning:
|
||||
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py",
|
||||
"script_args": [
|
||||
"--model_name_or_path", "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"--dataset_name", "trl-lib/math_shepherd",
|
||||
"--output_dir", "grpo-model",
|
||||
"--push_to_hub",
|
||||
"--hub_model_id", "username/grpo-model"
|
||||
],
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "4h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
**For GRPO documentation:** Use `hf_doc_fetch("https://huggingface.co/docs/trl/grpo_trainer")`
|
||||
|
||||
## Trackio Configuration
|
||||
|
||||
**Use sensible defaults for trackio setup.** See `references/trackio_guide.md` for complete documentation including grouping runs for experiments.
|
||||
|
||||
### Basic Pattern
|
||||
|
||||
```python
|
||||
import trackio
|
||||
|
||||
trackio.init(
|
||||
project="my-training",
|
||||
run_name="baseline-run", # Descriptive name user will recognize
|
||||
space_id="username/trackio", # Default space: {username}/trackio
|
||||
config={
|
||||
# Keep config minimal - hyperparameters and model/dataset info only
|
||||
"model": "Qwen/Qwen2.5-0.5B",
|
||||
"dataset": "trl-lib/Capybara",
|
||||
"learning_rate": 2e-5,
|
||||
}
|
||||
)
|
||||
|
||||
# Your training code...
|
||||
|
||||
trackio.finish()
|
||||
```
|
||||
|
||||
### Grouping for Experiments (Optional)
|
||||
|
||||
When user wants to compare related runs, use the `group` parameter:
|
||||
|
||||
```python
|
||||
# Hyperparameter sweep
|
||||
trackio.init(project="hyperparam-sweep", run_name="lr-0.001", group="lr_0.001")
|
||||
trackio.init(project="hyperparam-sweep", run_name="lr-0.01", group="lr_0.01")
|
||||
```
|
||||
|
||||
## Pattern Selection Guide
|
||||
|
||||
| Use Case | Pattern | Hardware | Time |
|
||||
|----------|---------|----------|------|
|
||||
| SFT training | `scripts/train_sft_example.py` | a10g-large | 2-6 hours |
|
||||
| Large dataset (>10K) | Multi-GPU | a10g-largex2 | 4-12 hours |
|
||||
| Preference learning | DPO Training | a10g-large | 2-4 hours |
|
||||
| Online RL | GRPO Training | a10g-large | 3-6 hours |
|
||||
|
||||
## Critical: Evaluation Dataset Requirements
|
||||
|
||||
**⚠️ IMPORTANT**: If you set `eval_strategy="steps"` or `eval_strategy="epoch"`, you **MUST** provide an `eval_dataset` to the trainer, or the training will hang.
|
||||
|
||||
### ✅ CORRECT - With eval dataset:
|
||||
```python
|
||||
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset_split["train"],
|
||||
eval_dataset=dataset_split["test"], # ← MUST provide when eval_strategy is enabled
|
||||
args=SFTConfig(eval_strategy="steps", ...),
|
||||
)
|
||||
```
|
||||
|
||||
### ❌ WRONG - Will hang:
|
||||
```python
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset,
|
||||
# NO eval_dataset but eval_strategy="steps" ← WILL HANG
|
||||
args=SFTConfig(eval_strategy="steps", ...),
|
||||
)
|
||||
```
|
||||
|
||||
### Option: Disable evaluation if no eval dataset
|
||||
```python
|
||||
config = SFTConfig(
|
||||
eval_strategy="no", # ← Explicitly disable evaluation
|
||||
# ... other config
|
||||
)
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset,
|
||||
# No eval_dataset needed
|
||||
args=config,
|
||||
)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use train/eval splits** - Create evaluation split for monitoring progress
|
||||
2. **Enable Trackio** - Monitor progress in real-time
|
||||
3. **Add 20-30% buffer to timeout** - Account for loading/saving overhead
|
||||
4. **Test with TRL official scripts first** - Use maintained examples before custom code
|
||||
5. **Always provide eval_dataset** - When using eval_strategy, or set to "no"
|
||||
6. **Use multi-GPU for large models** - 7B+ models benefit significantly
|
||||
|
||||
## See Also
|
||||
|
||||
- `scripts/train_sft_example.py` - Complete SFT template with Trackio and eval split
|
||||
- `scripts/train_dpo_example.py` - Complete DPO template
|
||||
- `scripts/train_grpo_example.py` - Complete GRPO template
|
||||
- `references/hardware_guide.md` - Detailed hardware specifications
|
||||
- `references/training_methods.md` - Overview of all TRL training methods
|
||||
- `references/troubleshooting.md` - Common issues and solutions
|
||||
@@ -0,0 +1,282 @@
|
||||
# Troubleshooting TRL Training Jobs
|
||||
|
||||
Common issues and solutions when training with TRL on Hugging Face Jobs.
|
||||
|
||||
## Training Hangs at "Starting training..." Step
|
||||
|
||||
**Problem:** Job starts but hangs at the training step - never progresses, never times out, just sits there.
|
||||
|
||||
**Root Cause:** Using `eval_strategy="steps"` or `eval_strategy="epoch"` without providing an `eval_dataset` to the trainer.
|
||||
|
||||
**Solution:**
|
||||
|
||||
**Option A: Provide eval_dataset (recommended)**
|
||||
```python
|
||||
# Create train/eval split
|
||||
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset_split["train"],
|
||||
eval_dataset=dataset_split["test"], # ← MUST provide when eval_strategy is enabled
|
||||
args=SFTConfig(
|
||||
eval_strategy="steps",
|
||||
eval_steps=50,
|
||||
...
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
**Option B: Disable evaluation**
|
||||
```python
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=dataset,
|
||||
# No eval_dataset
|
||||
args=SFTConfig(
|
||||
eval_strategy="no", # ← Explicitly disable
|
||||
...
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
**Prevention:**
|
||||
- Always create train/eval split for better monitoring
|
||||
- Use `dataset.train_test_split(test_size=0.1, seed=42)`
|
||||
- Check example scripts: `scripts/train_sft_example.py` includes proper eval setup
|
||||
|
||||
## Job Times Out
|
||||
|
||||
**Problem:** Job terminates before training completes, all progress lost.
|
||||
|
||||
**Solutions:**
|
||||
- Increase timeout parameter (e.g., `"timeout": "4h"`)
|
||||
- Reduce `num_train_epochs` or use smaller dataset slice
|
||||
- Use smaller model or enable LoRA/PEFT to speed up training
|
||||
- Add 20-30% buffer to estimated time for loading/saving overhead
|
||||
|
||||
**Prevention:**
|
||||
- Always start with a quick demo run to estimate timing
|
||||
- Use `scripts/estimate_cost.py` to get time estimates
|
||||
- Monitor first runs closely via Trackio or logs
|
||||
|
||||
## Model Not Saved to Hub
|
||||
|
||||
**Problem:** Training completes but model doesn't appear on Hub - all work lost.
|
||||
|
||||
**Check:**
|
||||
- [ ] `push_to_hub=True` in training config
|
||||
- [ ] `hub_model_id` specified with username (e.g., `"username/model-name"`)
|
||||
- [ ] `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job submission
|
||||
- [ ] User has write access to target repo
|
||||
- [ ] Token has write permissions (check at https://huggingface.co/settings/tokens)
|
||||
- [ ] Training script calls `trainer.push_to_hub()` at the end
|
||||
|
||||
**See:** `references/hub_saving.md` for detailed Hub authentication troubleshooting
|
||||
|
||||
## Out of Memory (OOM)
|
||||
|
||||
**Problem:** Job fails with CUDA out of memory error.
|
||||
|
||||
**Solutions (in order of preference):**
|
||||
1. **Reduce batch size:** Lower `per_device_train_batch_size` (try 4 → 2 → 1)
|
||||
2. **Increase gradient accumulation:** Raise `gradient_accumulation_steps` to maintain effective batch size
|
||||
3. **Disable evaluation:** Remove `eval_dataset` and `eval_strategy` (saves ~40% memory, good for demos)
|
||||
4. **Enable LoRA/PEFT:** Use `peft_config=LoraConfig(r=8, lora_alpha=16)` to train adapters only (smaller rank = less memory)
|
||||
5. **Use larger GPU:** Switch from `t4-small` → `l4x1` → `a10g-large` → `a100-large`
|
||||
6. **Enable gradient checkpointing:** Set `gradient_checkpointing=True` in config (slower but saves memory)
|
||||
7. **Use smaller model:** Try a smaller variant (e.g., 0.5B instead of 3B)
|
||||
|
||||
**Memory guidelines:**
|
||||
- T4 (16GB): <1B models with LoRA
|
||||
- A10G (24GB): 1-3B models with LoRA, <1B full fine-tune
|
||||
- A100 (40GB/80GB): 7B+ models with LoRA, 3B full fine-tune
|
||||
|
||||
## Parameter Naming Issues
|
||||
|
||||
**Problem:** `TypeError: SFTConfig.__init__() got an unexpected keyword argument 'max_seq_length'`
|
||||
|
||||
**Cause:** TRL config classes use `max_length`, not `max_seq_length`.
|
||||
|
||||
**Solution:**
|
||||
```python
|
||||
# ✅ CORRECT - TRL uses max_length
|
||||
SFTConfig(max_length=512)
|
||||
DPOConfig(max_length=512)
|
||||
|
||||
# ❌ WRONG - This will fail
|
||||
SFTConfig(max_seq_length=512)
|
||||
```
|
||||
|
||||
**Note:** Most TRL configs don't require explicit max_length - the default (1024) works well. Only set if you need a specific value.
|
||||
|
||||
## Dataset Format Error
|
||||
|
||||
**Problem:** Training fails with dataset format errors or missing fields.
|
||||
|
||||
**Solutions:**
|
||||
1. **Check format documentation:**
|
||||
```python
|
||||
hf_doc_fetch("https://huggingface.co/docs/trl/dataset_formats")
|
||||
```
|
||||
|
||||
2. **Validate dataset before training:**
|
||||
```bash
|
||||
uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \
|
||||
--dataset <dataset-name> --split train
|
||||
```
|
||||
Or via hf_jobs:
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py",
|
||||
"script_args": ["--dataset", "dataset-name", "--split", "train"]
|
||||
})
|
||||
```
|
||||
|
||||
3. **Verify field names:**
|
||||
- **SFT:** Needs "messages" field (conversational), OR "text" field, OR "prompt"/"completion"
|
||||
- **DPO:** Needs "chosen" and "rejected" fields
|
||||
- **GRPO:** Needs prompt-only format
|
||||
|
||||
4. **Check dataset split:**
|
||||
- Ensure split exists (e.g., `split="train"`)
|
||||
- Preview dataset: `load_dataset("name", split="train[:5]")`
|
||||
|
||||
## Import/Module Errors
|
||||
|
||||
**Problem:** Job fails with "ModuleNotFoundError" or import errors.
|
||||
|
||||
**Solutions:**
|
||||
1. **Add PEP 723 header with dependencies:**
|
||||
```python
|
||||
# /// script
|
||||
# dependencies = [
|
||||
# "trl>=0.12.0",
|
||||
# "peft>=0.7.0",
|
||||
# "transformers>=4.36.0",
|
||||
# ]
|
||||
# ///
|
||||
```
|
||||
|
||||
2. **Verify exact format:**
|
||||
- Must have `# ///` delimiters (with space after `#`)
|
||||
- Dependencies must be valid PyPI package names
|
||||
- Check spelling and version constraints
|
||||
|
||||
3. **Test locally first:**
|
||||
```bash
|
||||
uv run train.py # Tests if dependencies are correct
|
||||
```
|
||||
|
||||
## Authentication Errors
|
||||
|
||||
**Problem:** Job fails with authentication or permission errors when pushing to Hub.
|
||||
|
||||
**Solutions:**
|
||||
1. **Verify authentication:**
|
||||
```python
|
||||
mcp__huggingface__hf_whoami() # Check who's authenticated
|
||||
```
|
||||
|
||||
2. **Check token permissions:**
|
||||
- Go to https://huggingface.co/settings/tokens
|
||||
- Ensure token has "write" permission
|
||||
- Token must not be "read-only"
|
||||
|
||||
3. **Verify token in job:**
|
||||
```python
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"} # Must be in job config
|
||||
```
|
||||
|
||||
4. **Check repo permissions:**
|
||||
- User must have write access to target repo
|
||||
- If org repo, user must be member with write access
|
||||
- Repo must exist or user must have permission to create
|
||||
|
||||
## Job Stuck or Not Starting
|
||||
|
||||
**Problem:** Job shows "pending" or "starting" for extended period.
|
||||
|
||||
**Solutions:**
|
||||
- Check Jobs dashboard for status: https://huggingface.co/jobs
|
||||
- Verify hardware availability (some GPU types may have queues)
|
||||
- Try different hardware flavor if one is heavily utilized
|
||||
- Check for account billing issues (Jobs requires paid plan)
|
||||
|
||||
**Typical startup times:**
|
||||
- CPU jobs: 10-30 seconds
|
||||
- GPU jobs: 30-90 seconds
|
||||
- If >3 minutes: likely queued or stuck
|
||||
|
||||
## Training Loss Not Decreasing
|
||||
|
||||
**Problem:** Training runs but loss stays flat or doesn't improve.
|
||||
|
||||
**Solutions:**
|
||||
1. **Check learning rate:** May be too low (try 2e-5 to 5e-5) or too high (try 1e-6)
|
||||
2. **Verify dataset quality:** Inspect examples to ensure they're reasonable
|
||||
3. **Check model size:** Very small models may not have capacity for task
|
||||
4. **Increase training steps:** May need more epochs or larger dataset
|
||||
5. **Verify dataset format:** Wrong format may cause degraded training
|
||||
|
||||
## Logs Not Appearing
|
||||
|
||||
**Problem:** Cannot see training logs or progress.
|
||||
|
||||
**Solutions:**
|
||||
1. **Wait 30-60 seconds:** Initial logs can be delayed
|
||||
2. **Check logs via MCP tool:**
|
||||
```python
|
||||
hf_jobs("logs", {"job_id": "your-job-id"})
|
||||
```
|
||||
3. **Use Trackio for real-time monitoring:** See `references/trackio_guide.md`
|
||||
4. **Verify job is actually running:**
|
||||
```python
|
||||
hf_jobs("inspect", {"job_id": "your-job-id"})
|
||||
```
|
||||
|
||||
## Checkpoint/Resume Issues
|
||||
|
||||
**Problem:** Cannot resume from checkpoint or checkpoint not saved.
|
||||
|
||||
**Solutions:**
|
||||
1. **Enable checkpoint saving:**
|
||||
```python
|
||||
SFTConfig(
|
||||
save_strategy="steps",
|
||||
save_steps=100,
|
||||
hub_strategy="every_save", # Push each checkpoint
|
||||
)
|
||||
```
|
||||
|
||||
2. **Verify checkpoints pushed to Hub:** Check model repo for checkpoint folders
|
||||
|
||||
3. **Resume from checkpoint:**
|
||||
```python
|
||||
trainer = SFTTrainer(
|
||||
model="username/model-name", # Can be checkpoint path
|
||||
resume_from_checkpoint="username/model-name/checkpoint-1000",
|
||||
)
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
If issues persist:
|
||||
|
||||
1. **Check TRL documentation:**
|
||||
```python
|
||||
hf_doc_search("your issue", product="trl")
|
||||
```
|
||||
|
||||
2. **Check Jobs documentation:**
|
||||
```python
|
||||
hf_doc_fetch("https://huggingface.co/docs/huggingface_hub/guides/jobs")
|
||||
```
|
||||
|
||||
3. **Review related guides:**
|
||||
- `references/hub_saving.md` - Hub authentication issues
|
||||
- `references/hardware_guide.md` - Hardware selection and specs
|
||||
- `references/training_patterns.md` - Eval dataset requirements
|
||||
- SKILL.md "Working with Scripts" section - Script format and URL issues
|
||||
|
||||
4. **Ask in HF forums:** https://discuss.huggingface.co/
|
||||
@@ -0,0 +1,313 @@
|
||||
# Unsloth: Fast Fine-Tuning with Memory Optimization
|
||||
|
||||
**Unsloth** is a fine-tuning library that provides ~2x faster training and ~60% less VRAM usage for LLM training. It's particularly useful when working with limited GPU memory or when speed is critical.
|
||||
|
||||
- **GitHub**: [unslothai/unsloth](https://github.com/unslothai/unsloth)
|
||||
- **Docs**: [unsloth.ai/docs](https://unsloth.ai/docs)
|
||||
|
||||
## When to Use Unsloth
|
||||
|
||||
Use Unsloth if instructed to do so, or one of the following use cases applies:
|
||||
|
||||
| Use Case | Recommendation |
|
||||
|----------|----------------|
|
||||
| Standard text LLM fine-tuning | TRL is sufficient, but Unsloth is faster |
|
||||
| Limited GPU memory | **Use Unsloth** - 60% less VRAM |
|
||||
| Need maximum speed | **Use Unsloth** - 2x faster |
|
||||
| Large models (>13B) | **Use Unsloth** - memory efficiency critical |
|
||||
|
||||
## Supported Models
|
||||
|
||||
Unsloth supports many popular models including:
|
||||
- **Text LLMs**: Llama 3/3.1/3.2/3.3, Qwen 2.5/3, Mistral, Phi-4, Gemma 2/3, LFM2/2.5
|
||||
- **Vision LLMs**: Qwen3-VL, Gemma 3, Llama 3.2 Vision, Pixtral
|
||||
|
||||
Use Unsloth's pre-optimized model variants when available:
|
||||
```python
|
||||
# Unsloth-optimized models load faster and use less memory
|
||||
model_id = "unsloth/LFM2.5-1.2B-Instruct" # 4-bit quantized
|
||||
model_id = "unsloth/gemma-3-4b-pt" # Vision model
|
||||
model_id = "unsloth/Qwen3-VL-8B-Instruct" # Vision model
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```python
|
||||
# /// script
|
||||
# dependencies = [
|
||||
# "unsloth",
|
||||
# "trl",
|
||||
# "datasets",
|
||||
# "trackio",
|
||||
# ]
|
||||
# ///
|
||||
```
|
||||
|
||||
## Basic Usage: Text LLM
|
||||
|
||||
```python
|
||||
from unsloth import FastLanguageModel
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
from datasets import load_dataset
|
||||
|
||||
# Load model with Unsloth optimizations
|
||||
model, tokenizer = FastLanguageModel.from_pretrained(
|
||||
model_name="LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
max_seq_length=4096,
|
||||
)
|
||||
|
||||
# Add LoRA adapters
|
||||
model = FastLanguageModel.get_peft_model(
|
||||
model,
|
||||
r=16,
|
||||
lora_alpha=16,
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "out_proj", "in_proj", "w1", "w2", "w3"],
|
||||
lora_dropout=0,
|
||||
bias="none",
|
||||
use_gradient_checkpointing="unsloth",
|
||||
random_state=3407,
|
||||
)
|
||||
|
||||
# Load dataset
|
||||
dataset = load_dataset("trl-lib/Capybara", split="train")
|
||||
|
||||
# Train with TRL
|
||||
trainer = SFTTrainer(
|
||||
model=model,
|
||||
tokenizer=tokenizer,
|
||||
train_dataset=dataset,
|
||||
args=SFTConfig(
|
||||
output_dir="./output",
|
||||
per_device_train_batch_size=2,
|
||||
gradient_accumulation_steps=4,
|
||||
max_steps=500,
|
||||
learning_rate=2e-4,
|
||||
report_to="trackio",
|
||||
),
|
||||
)
|
||||
|
||||
trainer.train()
|
||||
```
|
||||
|
||||
## LFM2.5 Specific Settings
|
||||
|
||||
For LFM2.5 inference, use these recommended generation parameters:
|
||||
|
||||
**Instruct models:**
|
||||
```python
|
||||
temperature = 0.1
|
||||
top_k = 50
|
||||
top_p = 0.1
|
||||
repetition_penalty = 1.05
|
||||
```
|
||||
|
||||
**Thinking models:**
|
||||
```python
|
||||
temperature = 0.05
|
||||
top_k = 50
|
||||
repetition_penalty = 1.05
|
||||
```
|
||||
|
||||
## Vision-Language Models (VLMs)
|
||||
|
||||
Unsloth provides specialized support for VLMs with `FastVisionModel`:
|
||||
|
||||
```python
|
||||
from unsloth import FastVisionModel, get_chat_template
|
||||
from unsloth.trainer import UnslothVisionDataCollator
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
from datasets import load_dataset
|
||||
|
||||
# Load VLM with Unsloth
|
||||
model, processor = FastVisionModel.from_pretrained(
|
||||
"unsloth/gemma-3-4b-pt", # or "unsloth/Qwen3-VL-8B-Instruct"
|
||||
load_in_4bit=True,
|
||||
use_gradient_checkpointing="unsloth",
|
||||
)
|
||||
|
||||
# Add LoRA for all modalities
|
||||
model = FastVisionModel.get_peft_model(
|
||||
model,
|
||||
finetune_vision_layers=True, # Train vision encoder
|
||||
finetune_language_layers=True, # Train language model
|
||||
finetune_attention_modules=True, # Train attention
|
||||
finetune_mlp_modules=True, # Train MLPs
|
||||
r=16,
|
||||
lora_alpha=32,
|
||||
target_modules="all-linear",
|
||||
)
|
||||
|
||||
# Apply chat template (required for base models)
|
||||
processor = get_chat_template(processor, "gemma-3")
|
||||
|
||||
# Load VLM dataset (with images and messages)
|
||||
dataset = load_dataset("your-vlm-dataset", split="train", streaming=True)
|
||||
|
||||
# Enable training mode
|
||||
FastVisionModel.for_training(model)
|
||||
|
||||
# Train with VLM-specific collator
|
||||
trainer = SFTTrainer(
|
||||
model=model,
|
||||
train_dataset=dataset,
|
||||
processing_class=processor.tokenizer,
|
||||
data_collator=UnslothVisionDataCollator(model, processor),
|
||||
args=SFTConfig(
|
||||
output_dir="./vlm-output",
|
||||
per_device_train_batch_size=2,
|
||||
gradient_accumulation_steps=4,
|
||||
max_steps=500,
|
||||
learning_rate=2e-4,
|
||||
# VLM-specific settings
|
||||
remove_unused_columns=False,
|
||||
dataset_text_field="",
|
||||
dataset_kwargs={"skip_prepare_dataset": True},
|
||||
report_to="trackio",
|
||||
),
|
||||
)
|
||||
|
||||
trainer.train()
|
||||
```
|
||||
|
||||
## Key Differences from Standard TRL
|
||||
|
||||
| Aspect | Standard TRL | Unsloth |
|
||||
|--------|--------------|---------|
|
||||
| Model loading | `AutoModelForCausalLM.from_pretrained()` | `FastLanguageModel.from_pretrained()` |
|
||||
| LoRA setup | `PeftModel` / `LoraConfig` | `FastLanguageModel.get_peft_model()` |
|
||||
| VLM loading | Limited support | `FastVisionModel.from_pretrained()` |
|
||||
| VLM collator | Manual | `UnslothVisionDataCollator` |
|
||||
| Memory usage | Standard | ~60% less |
|
||||
| Training speed | Standard | ~2x faster |
|
||||
|
||||
## VLM Dataset Format
|
||||
|
||||
VLM datasets should have:
|
||||
- `images`: List of PIL images or image paths
|
||||
- `messages`: Conversation format with image references
|
||||
|
||||
```python
|
||||
{
|
||||
"images": [<PIL.Image>, ...],
|
||||
"messages": [
|
||||
{"role": "user", "content": [
|
||||
{"type": "image"},
|
||||
{"type": "text", "text": "Describe this image"}
|
||||
]},
|
||||
{"role": "assistant", "content": "This image shows..."}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Streaming Datasets
|
||||
|
||||
For large VLM datasets, use streaming to avoid disk space issues:
|
||||
|
||||
```python
|
||||
dataset = load_dataset(
|
||||
"your-vlm-dataset",
|
||||
split="train",
|
||||
streaming=True, # Stream from Hub
|
||||
)
|
||||
|
||||
# Must use max_steps with streaming (no epoch-based training)
|
||||
SFTConfig(max_steps=500, ...)
|
||||
```
|
||||
|
||||
## Saving Models
|
||||
|
||||
### Save LoRA Adapter
|
||||
|
||||
```python
|
||||
model.save_pretrained("./adapter")
|
||||
processor.save_pretrained("./adapter")
|
||||
|
||||
# Push to Hub
|
||||
model.push_to_hub("username/my-vlm-adapter")
|
||||
processor.push_to_hub("username/my-vlm-adapter")
|
||||
```
|
||||
|
||||
### Merge and Save Full Model
|
||||
|
||||
```python
|
||||
# Merge LoRA weights into base model
|
||||
model = model.merge_and_unload()
|
||||
|
||||
# Save merged model
|
||||
model.save_pretrained("./merged")
|
||||
tokenizer.save_pretrained("./merged")
|
||||
```
|
||||
|
||||
### Convert to GGUF
|
||||
|
||||
Unsloth models can be converted to GGUF for llama.cpp/Ollama:
|
||||
|
||||
```python
|
||||
# Save in 16-bit for GGUF conversion
|
||||
model.save_pretrained_gguf("./gguf", tokenizer, quantization_method="f16")
|
||||
|
||||
# Or directly quantize
|
||||
model.save_pretrained_gguf("./gguf", tokenizer, quantization_method="q4_k_m")
|
||||
```
|
||||
|
||||
## Qwen3-VL Specific Settings
|
||||
|
||||
For Qwen3-VL models, use these recommended settings:
|
||||
|
||||
**Instruct models:**
|
||||
```python
|
||||
temperature = 0.7
|
||||
top_p = 0.8
|
||||
presence_penalty = 1.5
|
||||
```
|
||||
|
||||
**Thinking models:**
|
||||
```python
|
||||
temperature = 1.0
|
||||
top_p = 0.95
|
||||
presence_penalty = 0.0
|
||||
```
|
||||
|
||||
## Hardware Requirements
|
||||
|
||||
| Model | Min VRAM (Unsloth 4-bit) | Recommended GPU |
|
||||
|-------|--------------------------|-----------------|
|
||||
| 2B-4B | 8GB | T4, L4 |
|
||||
| 7B-8B | 16GB | A10G, L4x4 |
|
||||
| 13B | 24GB | A10G-large |
|
||||
| 30B+ | 48GB+ | A100 |
|
||||
|
||||
## Example: Full VLM Training Script
|
||||
|
||||
See `scripts/unsloth_sft_example.py` for a complete production-ready example that includes:
|
||||
- Unsloth VLM setup
|
||||
- Streaming dataset support
|
||||
- Trackio monitoring
|
||||
- Hub push
|
||||
- CLI arguments
|
||||
|
||||
Run locally:
|
||||
```bash
|
||||
uv run scripts/unsloth_sft_example.py \
|
||||
--dataset trl-lib/Capybara \
|
||||
--max-steps 500 \
|
||||
--output-repo username/my-model
|
||||
```
|
||||
|
||||
Run on HF Jobs:
|
||||
```python
|
||||
hf_jobs("uv", {
|
||||
"script": "<script content>",
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "2h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"}
|
||||
})
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- `scripts/unsloth_sft_example.py` - Complete text LLM training example
|
||||
- [Unsloth Documentation](https://unsloth.ai/docs)
|
||||
- [LFM2.5 Guide](https://unsloth.ai/docs/models/tutorials/lfm2.5)
|
||||
- [Qwen3-VL Guide](https://unsloth.ai/docs/models/qwen3-vl-how-to-run-and-fine-tune)
|
||||
- [Unsloth GitHub](https://github.com/unslothai/unsloth)
|
||||
+424
@@ -0,0 +1,424 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "transformers>=4.36.0",
|
||||
# "peft>=0.7.0",
|
||||
# "torch>=2.0.0",
|
||||
# "accelerate>=0.24.0",
|
||||
# "huggingface_hub>=0.20.0",
|
||||
# "sentencepiece>=0.1.99",
|
||||
# "protobuf>=3.20.0",
|
||||
# "numpy",
|
||||
# "gguf",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
"""
|
||||
GGUF Conversion Script - Production Ready
|
||||
|
||||
This script converts a LoRA fine-tuned model to GGUF format for use with:
|
||||
- llama.cpp
|
||||
- Ollama
|
||||
- LM Studio
|
||||
- Other GGUF-compatible tools
|
||||
|
||||
PREREQUISITES (install these FIRST):
|
||||
- Ubuntu/Debian: sudo apt-get update && sudo apt-get install -y build-essential cmake
|
||||
- RHEL/CentOS: sudo yum groupinstall -y "Development Tools" && sudo yum install -y cmake
|
||||
- macOS: xcode-select --install && brew install cmake
|
||||
|
||||
Usage:
|
||||
Set environment variables:
|
||||
- ADAPTER_MODEL: Your fine-tuned model (e.g., "username/my-finetuned-model")
|
||||
- BASE_MODEL: Base model used for fine-tuning (e.g., "Qwen/Qwen2.5-0.5B")
|
||||
- OUTPUT_REPO: Where to upload GGUF files (e.g., "username/my-model-gguf")
|
||||
- HF_USERNAME: Your Hugging Face username (optional, for README)
|
||||
|
||||
Dependencies: All required packages are declared in PEP 723 header above.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from peft import PeftModel
|
||||
from huggingface_hub import HfApi
|
||||
import subprocess
|
||||
|
||||
|
||||
def check_system_dependencies():
|
||||
"""Check if required system packages are available."""
|
||||
print("🔍 Checking system dependencies...")
|
||||
|
||||
# Check for git
|
||||
if subprocess.run(["which", "git"], capture_output=True).returncode != 0:
|
||||
print(" ❌ git is not installed. Please install it:")
|
||||
print(" Ubuntu/Debian: sudo apt-get install git")
|
||||
print(" RHEL/CentOS: sudo yum install git")
|
||||
print(" macOS: brew install git")
|
||||
return False
|
||||
|
||||
# Check for make or cmake
|
||||
has_make = subprocess.run(["which", "make"], capture_output=True).returncode == 0
|
||||
has_cmake = subprocess.run(["which", "cmake"], capture_output=True).returncode == 0
|
||||
|
||||
if not has_make and not has_cmake:
|
||||
print(" ❌ Neither make nor cmake found. Please install build tools:")
|
||||
print(" Ubuntu/Debian: sudo apt-get install build-essential cmake")
|
||||
print(" RHEL/CentOS: sudo yum groupinstall 'Development Tools' && sudo yum install cmake")
|
||||
print(" macOS: xcode-select --install && brew install cmake")
|
||||
return False
|
||||
|
||||
print(" ✅ System dependencies found")
|
||||
return True
|
||||
|
||||
|
||||
def run_command(cmd, description):
|
||||
"""Run a command with error handling."""
|
||||
print(f" {description}...")
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
if result.stdout:
|
||||
print(f" {result.stdout[:200]}") # Show first 200 chars
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f" ❌ Command failed: {' '.join(cmd)}")
|
||||
if e.stdout:
|
||||
print(f" STDOUT: {e.stdout[:500]}")
|
||||
if e.stderr:
|
||||
print(f" STDERR: {e.stderr[:500]}")
|
||||
return False
|
||||
except FileNotFoundError:
|
||||
print(f" ❌ Command not found: {cmd[0]}")
|
||||
return False
|
||||
|
||||
|
||||
print("🔄 GGUF Conversion Script")
|
||||
print("=" * 60)
|
||||
|
||||
# Check system dependencies first
|
||||
if not check_system_dependencies():
|
||||
print("\n❌ Please install the missing system dependencies and try again.")
|
||||
sys.exit(1)
|
||||
|
||||
# Configuration from environment variables
|
||||
ADAPTER_MODEL = os.environ.get("ADAPTER_MODEL", "evalstate/qwen-capybara-medium")
|
||||
BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen2.5-0.5B")
|
||||
OUTPUT_REPO = os.environ.get("OUTPUT_REPO", "evalstate/qwen-capybara-medium-gguf")
|
||||
username = os.environ.get("HF_USERNAME", ADAPTER_MODEL.split('/')[0])
|
||||
|
||||
print(f"\n📦 Configuration:")
|
||||
print(f" Base model: {BASE_MODEL}")
|
||||
print(f" Adapter model: {ADAPTER_MODEL}")
|
||||
print(f" Output repo: {OUTPUT_REPO}")
|
||||
|
||||
# Step 1: Load base model and adapter
|
||||
print("\n🔧 Step 1: Loading base model and LoRA adapter...")
|
||||
print(" (This may take a few minutes)")
|
||||
|
||||
try:
|
||||
base_model = AutoModelForCausalLM.from_pretrained(
|
||||
BASE_MODEL,
|
||||
dtype=torch.float16,
|
||||
device_map="auto",
|
||||
trust_remote_code=True,
|
||||
)
|
||||
print(" ✅ Base model loaded")
|
||||
except Exception as e:
|
||||
print(f" ❌ Failed to load base model: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
# Load and merge adapter
|
||||
print(" Loading LoRA adapter...")
|
||||
model = PeftModel.from_pretrained(base_model, ADAPTER_MODEL)
|
||||
print(" ✅ Adapter loaded")
|
||||
|
||||
print(" Merging adapter with base model...")
|
||||
merged_model = model.merge_and_unload()
|
||||
print(" ✅ Models merged!")
|
||||
except Exception as e:
|
||||
print(f" ❌ Failed to merge models: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
# Load tokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained(ADAPTER_MODEL, trust_remote_code=True)
|
||||
print(" ✅ Tokenizer loaded")
|
||||
except Exception as e:
|
||||
print(f" ❌ Failed to load tokenizer: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
# Step 2: Save merged model temporarily
|
||||
print("\n💾 Step 2: Saving merged model...")
|
||||
merged_dir = "/tmp/merged_model"
|
||||
try:
|
||||
merged_model.save_pretrained(merged_dir, safe_serialization=True)
|
||||
tokenizer.save_pretrained(merged_dir)
|
||||
print(f" ✅ Merged model saved to {merged_dir}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Failed to save merged model: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
# Step 3: Install llama.cpp for conversion
|
||||
print("\n📥 Step 3: Setting up llama.cpp for GGUF conversion...")
|
||||
|
||||
# Clone llama.cpp repository
|
||||
if not run_command(
|
||||
["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"],
|
||||
"Cloning llama.cpp repository"
|
||||
):
|
||||
print(" Trying alternative clone method...")
|
||||
# Try shallow clone
|
||||
if not run_command(
|
||||
["git", "clone", "--depth", "1", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"],
|
||||
"Cloning llama.cpp (shallow)"
|
||||
):
|
||||
sys.exit(1)
|
||||
|
||||
# Install Python dependencies
|
||||
print(" Installing Python dependencies...")
|
||||
if not run_command(
|
||||
["pip", "install", "-r", "/tmp/llama.cpp/requirements.txt"],
|
||||
"Installing llama.cpp requirements"
|
||||
):
|
||||
print(" ⚠️ Some requirements may already be installed")
|
||||
|
||||
if not run_command(
|
||||
["pip", "install", "sentencepiece", "protobuf"],
|
||||
"Installing tokenizer dependencies"
|
||||
):
|
||||
print(" ⚠️ Tokenizer dependencies may already be installed")
|
||||
|
||||
# Step 4: Convert to GGUF (FP16)
|
||||
print("\n🔄 Step 4: Converting to GGUF format (FP16)...")
|
||||
gguf_output_dir = "/tmp/gguf_output"
|
||||
os.makedirs(gguf_output_dir, exist_ok=True)
|
||||
|
||||
convert_script = "/tmp/llama.cpp/convert_hf_to_gguf.py"
|
||||
model_name = ADAPTER_MODEL.split('/')[-1]
|
||||
gguf_file = f"{gguf_output_dir}/{model_name}-f16.gguf"
|
||||
|
||||
print(f" Running conversion...")
|
||||
if not run_command(
|
||||
[
|
||||
sys.executable, convert_script,
|
||||
merged_dir,
|
||||
"--outfile", gguf_file,
|
||||
"--outtype", "f16"
|
||||
],
|
||||
f"Converting to FP16"
|
||||
):
|
||||
print(" ❌ Conversion failed!")
|
||||
sys.exit(1)
|
||||
|
||||
print(f" ✅ FP16 GGUF created: {gguf_file}")
|
||||
|
||||
# Step 5: Quantize to different formats
|
||||
print("\n⚙️ Step 5: Creating quantized versions...")
|
||||
|
||||
# Build quantize tool using CMake (more reliable than make)
|
||||
print(" Building quantize tool with CMake...")
|
||||
os.makedirs("/tmp/llama.cpp/build", exist_ok=True)
|
||||
|
||||
# Configure with CMake
|
||||
if not run_command(
|
||||
["cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
|
||||
"-DGGML_CUDA=OFF"],
|
||||
"Configuring with CMake"
|
||||
):
|
||||
print(" ❌ CMake configuration failed")
|
||||
sys.exit(1)
|
||||
|
||||
# Build just the quantize tool
|
||||
if not run_command(
|
||||
["cmake", "--build", "/tmp/llama.cpp/build", "--target", "llama-quantize", "-j", "4"],
|
||||
"Building llama-quantize"
|
||||
):
|
||||
print(" ❌ Build failed!")
|
||||
sys.exit(1)
|
||||
|
||||
print(" ✅ Quantize tool built")
|
||||
|
||||
# Use the CMake build output path
|
||||
quantize_bin = "/tmp/llama.cpp/build/bin/llama-quantize"
|
||||
|
||||
# Common quantization formats
|
||||
quant_formats = [
|
||||
("Q4_K_M", "4-bit, medium quality (recommended)"),
|
||||
("Q5_K_M", "5-bit, higher quality"),
|
||||
("Q8_0", "8-bit, very high quality"),
|
||||
]
|
||||
|
||||
quantized_files = []
|
||||
for quant_type, description in quant_formats:
|
||||
print(f" Creating {quant_type} quantization ({description})...")
|
||||
quant_file = f"{gguf_output_dir}/{model_name}-{quant_type.lower()}.gguf"
|
||||
|
||||
if not run_command(
|
||||
[quantize_bin, gguf_file, quant_file, quant_type],
|
||||
f"Quantizing to {quant_type}"
|
||||
):
|
||||
print(f" ⚠️ Skipping {quant_type} due to error")
|
||||
continue
|
||||
|
||||
quantized_files.append((quant_file, quant_type))
|
||||
|
||||
# Get file size
|
||||
size_mb = os.path.getsize(quant_file) / (1024 * 1024)
|
||||
print(f" ✅ {quant_type}: {size_mb:.1f} MB")
|
||||
|
||||
if not quantized_files:
|
||||
print(" ❌ No quantized versions were created successfully")
|
||||
sys.exit(1)
|
||||
|
||||
# Step 6: Upload to Hub
|
||||
print("\n☁️ Step 6: Uploading to Hugging Face Hub...")
|
||||
api = HfApi()
|
||||
|
||||
# Create repo
|
||||
print(f" Creating repository: {OUTPUT_REPO}")
|
||||
try:
|
||||
api.create_repo(repo_id=OUTPUT_REPO, repo_type="model", exist_ok=True)
|
||||
print(" ✅ Repository ready")
|
||||
except Exception as e:
|
||||
print(f" ℹ️ Repository may already exist: {e}")
|
||||
|
||||
# Upload FP16 version
|
||||
print(" Uploading FP16 GGUF...")
|
||||
try:
|
||||
api.upload_file(
|
||||
path_or_fileobj=gguf_file,
|
||||
path_in_repo=f"{model_name}-f16.gguf",
|
||||
repo_id=OUTPUT_REPO,
|
||||
)
|
||||
print(" ✅ FP16 uploaded")
|
||||
except Exception as e:
|
||||
print(f" ❌ Upload failed: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
# Upload quantized versions
|
||||
for quant_file, quant_type in quantized_files:
|
||||
print(f" Uploading {quant_type}...")
|
||||
try:
|
||||
api.upload_file(
|
||||
path_or_fileobj=quant_file,
|
||||
path_in_repo=f"{model_name}-{quant_type.lower()}.gguf",
|
||||
repo_id=OUTPUT_REPO,
|
||||
)
|
||||
print(f" ✅ {quant_type} uploaded")
|
||||
except Exception as e:
|
||||
print(f" ❌ Upload failed for {quant_type}: {e}")
|
||||
continue
|
||||
|
||||
# Create README
|
||||
print("\n📝 Creating README...")
|
||||
readme_content = f"""---
|
||||
base_model: {BASE_MODEL}
|
||||
tags:
|
||||
- gguf
|
||||
- llama.cpp
|
||||
- quantized
|
||||
- trl
|
||||
- sft
|
||||
---
|
||||
|
||||
# {OUTPUT_REPO.split('/')[-1]}
|
||||
|
||||
This is a GGUF conversion of [{ADAPTER_MODEL}](https://huggingface.co/{ADAPTER_MODEL}), which is a LoRA fine-tuned version of [{BASE_MODEL}](https://huggingface.co/{BASE_MODEL}).
|
||||
|
||||
## Model Details
|
||||
|
||||
- **Base Model:** {BASE_MODEL}
|
||||
- **Fine-tuned Model:** {ADAPTER_MODEL}
|
||||
- **Training:** Supervised Fine-Tuning (SFT) with TRL
|
||||
- **Format:** GGUF (for llama.cpp, Ollama, LM Studio, etc.)
|
||||
|
||||
## Available Quantizations
|
||||
|
||||
| File | Quant | Size | Description | Use Case |
|
||||
|------|-------|------|-------------|----------|
|
||||
| {model_name}-f16.gguf | F16 | ~1GB | Full precision | Best quality, slower |
|
||||
| {model_name}-q8_0.gguf | Q8_0 | ~500MB | 8-bit | High quality |
|
||||
| {model_name}-q5_k_m.gguf | Q5_K_M | ~350MB | 5-bit medium | Good quality, smaller |
|
||||
| {model_name}-q4_k_m.gguf | Q4_K_M | ~300MB | 4-bit medium | Recommended - good balance |
|
||||
|
||||
## Usage
|
||||
|
||||
### With llama.cpp
|
||||
|
||||
```bash
|
||||
# Download model
|
||||
hf download {OUTPUT_REPO} {model_name}-q4_k_m.gguf
|
||||
|
||||
# Run with llama.cpp
|
||||
./llama-cli -m {model_name}-q4_k_m.gguf -p "Your prompt here"
|
||||
```
|
||||
|
||||
### With Ollama
|
||||
|
||||
1. Create a `Modelfile`:
|
||||
```
|
||||
FROM ./{model_name}-q4_k_m.gguf
|
||||
```
|
||||
|
||||
2. Create the model:
|
||||
```bash
|
||||
ollama create my-model -f Modelfile
|
||||
ollama run my-model
|
||||
```
|
||||
|
||||
### With LM Studio
|
||||
|
||||
1. Download the `.gguf` file
|
||||
2. Import into LM Studio
|
||||
3. Start chatting!
|
||||
|
||||
## License
|
||||
|
||||
Inherits the license from the base model: {BASE_MODEL}
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{{{OUTPUT_REPO.split('/')[-1].replace('-', '_')},
|
||||
author = {{{username}}},
|
||||
title = {{{OUTPUT_REPO.split('/')[-1]}}},
|
||||
year = {{2025}},
|
||||
publisher = {{Hugging Face}},
|
||||
url = {{https://huggingface.co/{OUTPUT_REPO}}}
|
||||
}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Converted to GGUF format using llama.cpp*
|
||||
"""
|
||||
|
||||
try:
|
||||
api.upload_file(
|
||||
path_or_fileobj=readme_content.encode(),
|
||||
path_in_repo="README.md",
|
||||
repo_id=OUTPUT_REPO,
|
||||
)
|
||||
print(" ✅ README uploaded")
|
||||
except Exception as e:
|
||||
print(f" ❌ README upload failed: {e}")
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("✅ GGUF Conversion Complete!")
|
||||
print(f"📦 Repository: https://huggingface.co/{OUTPUT_REPO}")
|
||||
print(f"\n📥 Download with:")
|
||||
print(f" hf download {OUTPUT_REPO} {model_name}-q4_k_m.gguf")
|
||||
print(f"\n🚀 Use with Ollama:")
|
||||
print(" 1. Download the GGUF file")
|
||||
print(f" 2. Create Modelfile: FROM ./{model_name}-q4_k_m.gguf")
|
||||
print(" 3. ollama create my-model -f Modelfile")
|
||||
print(" 4. ollama run my-model")
|
||||
print("=" * 60)
|
||||
@@ -0,0 +1,417 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
Dataset Format Inspector for TRL Training (LLM-Optimized Output)
|
||||
|
||||
Inspects Hugging Face datasets to determine TRL training compatibility.
|
||||
Uses Datasets Server API for instant results - no dataset download needed!
|
||||
|
||||
ULTRA-EFFICIENT: Uses HF Datasets Server API - completes in <2 seconds.
|
||||
|
||||
Usage with HF Jobs:
|
||||
hf_jobs("uv", {
|
||||
"script": "https://huggingface.co/datasets/evalstate/trl-helpers/raw/main/dataset_inspector.py",
|
||||
"script_args": ["--dataset", "your/dataset", "--split", "train"]
|
||||
})
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import json
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
from typing import List, Dict, Any
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Inspect dataset format for TRL training")
|
||||
parser.add_argument("--dataset", type=str, required=True, help="Dataset name")
|
||||
parser.add_argument("--split", type=str, default="train", help="Dataset split (default: train)")
|
||||
parser.add_argument("--config", type=str, default="default", help="Dataset config name (default: default)")
|
||||
parser.add_argument("--preview", type=int, default=150, help="Max chars per field preview")
|
||||
parser.add_argument("--samples", type=int, default=5, help="Number of samples to fetch (default: 5)")
|
||||
parser.add_argument("--json-output", action="store_true", help="Output as JSON")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def api_request(url: str) -> Dict:
|
||||
"""Make API request to Datasets Server"""
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=10) as response:
|
||||
return json.loads(response.read().decode())
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
return None
|
||||
raise Exception(f"API request failed: {e.code} {e.reason}")
|
||||
except Exception as e:
|
||||
raise Exception(f"API request failed: {str(e)}")
|
||||
|
||||
|
||||
def get_splits(dataset: str) -> Dict:
|
||||
"""Get available splits for dataset"""
|
||||
url = f"https://datasets-server.huggingface.co/splits?dataset={urllib.parse.quote(dataset)}"
|
||||
return api_request(url)
|
||||
|
||||
|
||||
def get_rows(dataset: str, config: str, split: str, offset: int = 0, length: int = 5) -> Dict:
|
||||
"""Get rows from dataset"""
|
||||
url = f"https://datasets-server.huggingface.co/rows?dataset={urllib.parse.quote(dataset)}&config={config}&split={split}&offset={offset}&length={length}"
|
||||
return api_request(url)
|
||||
|
||||
|
||||
def find_columns(columns: List[str], patterns: List[str]) -> List[str]:
|
||||
"""Find columns matching patterns"""
|
||||
return [c for c in columns if any(p in c.lower() for p in patterns)]
|
||||
|
||||
|
||||
def check_sft_compatibility(columns: List[str]) -> Dict[str, Any]:
|
||||
"""Check SFT compatibility"""
|
||||
has_messages = "messages" in columns
|
||||
has_text = "text" in columns
|
||||
has_prompt_completion = "prompt" in columns and "completion" in columns
|
||||
|
||||
ready = has_messages or has_text or has_prompt_completion
|
||||
|
||||
possible_prompt = find_columns(columns, ["prompt", "instruction", "question", "input"])
|
||||
possible_response = find_columns(columns, ["response", "completion", "output", "answer"])
|
||||
|
||||
return {
|
||||
"ready": ready,
|
||||
"reason": "messages" if has_messages else "text" if has_text else "prompt+completion" if has_prompt_completion else None,
|
||||
"possible_prompt": possible_prompt[0] if possible_prompt else None,
|
||||
"possible_response": possible_response[0] if possible_response else None,
|
||||
"has_context": "context" in columns,
|
||||
}
|
||||
|
||||
|
||||
def check_dpo_compatibility(columns: List[str]) -> Dict[str, Any]:
|
||||
"""Check DPO compatibility"""
|
||||
has_standard = "prompt" in columns and "chosen" in columns and "rejected" in columns
|
||||
|
||||
possible_prompt = find_columns(columns, ["prompt", "instruction", "question", "input"])
|
||||
possible_chosen = find_columns(columns, ["chosen", "preferred", "winner"])
|
||||
possible_rejected = find_columns(columns, ["rejected", "dispreferred", "loser"])
|
||||
|
||||
can_map = bool(possible_prompt and possible_chosen and possible_rejected)
|
||||
|
||||
return {
|
||||
"ready": has_standard,
|
||||
"can_map": can_map,
|
||||
"prompt_col": possible_prompt[0] if possible_prompt else None,
|
||||
"chosen_col": possible_chosen[0] if possible_chosen else None,
|
||||
"rejected_col": possible_rejected[0] if possible_rejected else None,
|
||||
}
|
||||
|
||||
|
||||
def check_grpo_compatibility(columns: List[str]) -> Dict[str, Any]:
|
||||
"""Check GRPO compatibility"""
|
||||
has_prompt = "prompt" in columns
|
||||
has_no_responses = "chosen" not in columns and "rejected" not in columns
|
||||
|
||||
possible_prompt = find_columns(columns, ["prompt", "instruction", "question", "input"])
|
||||
|
||||
return {
|
||||
"ready": has_prompt and has_no_responses,
|
||||
"can_map": bool(possible_prompt) and has_no_responses,
|
||||
"prompt_col": possible_prompt[0] if possible_prompt else None,
|
||||
}
|
||||
|
||||
|
||||
def check_kto_compatibility(columns: List[str]) -> Dict[str, Any]:
|
||||
"""Check KTO compatibility"""
|
||||
return {"ready": "prompt" in columns and "completion" in columns and "label" in columns}
|
||||
|
||||
|
||||
def generate_mapping_code(method: str, info: Dict[str, Any]) -> str:
|
||||
"""Generate mapping code for a training method"""
|
||||
if method == "SFT":
|
||||
if info["ready"]:
|
||||
return None
|
||||
|
||||
prompt_col = info.get("possible_prompt")
|
||||
response_col = info.get("possible_response")
|
||||
has_context = info.get("has_context", False)
|
||||
|
||||
if not prompt_col:
|
||||
return None
|
||||
|
||||
if has_context and response_col:
|
||||
return f"""def format_for_sft(example):
|
||||
text = f"Instruction: {{example['{prompt_col}']}}\n\n"
|
||||
if example.get('context'):
|
||||
text += f"Context: {{example['context']}}\n\n"
|
||||
text += f"Response: {{example['{response_col}']}}"
|
||||
return {{'text': text}}
|
||||
|
||||
dataset = dataset.map(format_for_sft, remove_columns=dataset.column_names)"""
|
||||
elif response_col:
|
||||
return f"""def format_for_sft(example):
|
||||
return {{'text': f"{{example['{prompt_col}']}}\n\n{{example['{response_col}']}}}}
|
||||
|
||||
dataset = dataset.map(format_for_sft, remove_columns=dataset.column_names)"""
|
||||
else:
|
||||
return f"""def format_for_sft(example):
|
||||
return {{'text': example['{prompt_col}']}}
|
||||
|
||||
dataset = dataset.map(format_for_sft, remove_columns=dataset.column_names)"""
|
||||
|
||||
elif method == "DPO":
|
||||
if info["ready"] or not info["can_map"]:
|
||||
return None
|
||||
|
||||
return f"""def format_for_dpo(example):
|
||||
return {{
|
||||
'prompt': example['{info['prompt_col']}'],
|
||||
'chosen': example['{info['chosen_col']}'],
|
||||
'rejected': example['{info['rejected_col']}'],
|
||||
}}
|
||||
|
||||
dataset = dataset.map(format_for_dpo, remove_columns=dataset.column_names)"""
|
||||
|
||||
elif method == "GRPO":
|
||||
if info["ready"] or not info["can_map"]:
|
||||
return None
|
||||
|
||||
return f"""def format_for_grpo(example):
|
||||
return {{'prompt': example['{info['prompt_col']}']}}
|
||||
|
||||
dataset = dataset.map(format_for_grpo, remove_columns=dataset.column_names)"""
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def format_value_preview(value: Any, max_chars: int) -> str:
|
||||
"""Format value for preview"""
|
||||
if value is None:
|
||||
return "None"
|
||||
elif isinstance(value, str):
|
||||
return value[:max_chars] + ("..." if len(value) > max_chars else "")
|
||||
elif isinstance(value, list):
|
||||
if len(value) > 0 and isinstance(value[0], dict):
|
||||
return f"[{len(value)} items] Keys: {list(value[0].keys())}"
|
||||
preview = str(value)
|
||||
return preview[:max_chars] + ("..." if len(preview) > max_chars else "")
|
||||
else:
|
||||
preview = str(value)
|
||||
return preview[:max_chars] + ("..." if len(preview) > max_chars else "")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
print(f"Fetching dataset info via Datasets Server API...")
|
||||
|
||||
try:
|
||||
# Get splits info
|
||||
splits_data = get_splits(args.dataset)
|
||||
if not splits_data or "splits" not in splits_data:
|
||||
print(f"ERROR: Could not fetch splits for dataset '{args.dataset}'")
|
||||
print(f" Dataset may not exist or is not accessible via Datasets Server API")
|
||||
sys.exit(1)
|
||||
|
||||
# Find the right config
|
||||
available_configs = set()
|
||||
split_found = False
|
||||
config_to_use = args.config
|
||||
|
||||
for split_info in splits_data["splits"]:
|
||||
available_configs.add(split_info["config"])
|
||||
if split_info["config"] == args.config and split_info["split"] == args.split:
|
||||
split_found = True
|
||||
|
||||
# If default config not found, try first available
|
||||
if not split_found and available_configs:
|
||||
config_to_use = list(available_configs)[0]
|
||||
print(f"Config '{args.config}' not found, trying '{config_to_use}'...")
|
||||
|
||||
# Get rows
|
||||
rows_data = get_rows(args.dataset, config_to_use, args.split, offset=0, length=args.samples)
|
||||
|
||||
if not rows_data or "rows" not in rows_data:
|
||||
print(f"ERROR: Could not fetch rows for dataset '{args.dataset}'")
|
||||
print(f" Split '{args.split}' may not exist")
|
||||
print(f" Available configs: {', '.join(sorted(available_configs))}")
|
||||
sys.exit(1)
|
||||
|
||||
rows = rows_data["rows"]
|
||||
if not rows:
|
||||
print(f"ERROR: No rows found in split '{args.split}'")
|
||||
sys.exit(1)
|
||||
|
||||
# Extract column info from first row
|
||||
first_row = rows[0]["row"]
|
||||
columns = list(first_row.keys())
|
||||
features = rows_data.get("features", [])
|
||||
|
||||
# Get total count if available
|
||||
total_examples = "Unknown"
|
||||
for split_info in splits_data["splits"]:
|
||||
if split_info["config"] == config_to_use and split_info["split"] == args.split:
|
||||
total_examples = f"{split_info.get('num_examples', 'Unknown'):,}" if isinstance(split_info.get('num_examples'), int) else "Unknown"
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
print(f"ERROR: {str(e)}")
|
||||
sys.exit(1)
|
||||
|
||||
# Run compatibility checks
|
||||
sft_info = check_sft_compatibility(columns)
|
||||
dpo_info = check_dpo_compatibility(columns)
|
||||
grpo_info = check_grpo_compatibility(columns)
|
||||
kto_info = check_kto_compatibility(columns)
|
||||
|
||||
# Determine recommended methods
|
||||
recommended = []
|
||||
if sft_info["ready"]:
|
||||
recommended.append("SFT")
|
||||
elif sft_info["possible_prompt"]:
|
||||
recommended.append("SFT (needs mapping)")
|
||||
|
||||
if dpo_info["ready"]:
|
||||
recommended.append("DPO")
|
||||
elif dpo_info["can_map"]:
|
||||
recommended.append("DPO (needs mapping)")
|
||||
|
||||
if grpo_info["ready"]:
|
||||
recommended.append("GRPO")
|
||||
elif grpo_info["can_map"]:
|
||||
recommended.append("GRPO (needs mapping)")
|
||||
|
||||
if kto_info["ready"]:
|
||||
recommended.append("KTO")
|
||||
|
||||
# JSON output mode
|
||||
if args.json_output:
|
||||
result = {
|
||||
"dataset": args.dataset,
|
||||
"config": config_to_use,
|
||||
"split": args.split,
|
||||
"total_examples": total_examples,
|
||||
"columns": columns,
|
||||
"features": [{"name": f["name"], "type": f["type"]} for f in features] if features else [],
|
||||
"compatibility": {
|
||||
"SFT": sft_info,
|
||||
"DPO": dpo_info,
|
||||
"GRPO": grpo_info,
|
||||
"KTO": kto_info,
|
||||
},
|
||||
"recommended_methods": recommended,
|
||||
}
|
||||
print(json.dumps(result, indent=2))
|
||||
sys.exit(0)
|
||||
|
||||
# Human-readable output optimized for LLM parsing
|
||||
print("=" * 80)
|
||||
print(f"DATASET INSPECTION RESULTS")
|
||||
print("=" * 80)
|
||||
|
||||
print(f"\nDataset: {args.dataset}")
|
||||
print(f"Config: {config_to_use}")
|
||||
print(f"Split: {args.split}")
|
||||
print(f"Total examples: {total_examples}")
|
||||
print(f"Samples fetched: {len(rows)}")
|
||||
|
||||
print(f"\n{'COLUMNS':-<80}")
|
||||
if features:
|
||||
for feature in features:
|
||||
print(f" {feature['name']}: {feature['type']}")
|
||||
else:
|
||||
for col in columns:
|
||||
print(f" {col}: (type info not available)")
|
||||
|
||||
print(f"\n{'EXAMPLE DATA':-<80}")
|
||||
example = first_row
|
||||
for col in columns:
|
||||
value = example.get(col)
|
||||
display = format_value_preview(value, args.preview)
|
||||
print(f"\n{col}:")
|
||||
print(f" {display}")
|
||||
|
||||
print(f"\n{'TRAINING METHOD COMPATIBILITY':-<80}")
|
||||
|
||||
# SFT
|
||||
print(f"\n[SFT] {'✓ READY' if sft_info['ready'] else '✗ NEEDS MAPPING'}")
|
||||
if sft_info["ready"]:
|
||||
print(f" Reason: Dataset has '{sft_info['reason']}' field")
|
||||
print(f" Action: Use directly with SFTTrainer")
|
||||
elif sft_info["possible_prompt"]:
|
||||
print(f" Detected: prompt='{sft_info['possible_prompt']}' response='{sft_info['possible_response']}'")
|
||||
print(f" Action: Apply mapping code (see below)")
|
||||
else:
|
||||
print(f" Status: Cannot determine mapping - manual inspection needed")
|
||||
|
||||
# DPO
|
||||
print(f"\n[DPO] {'✓ READY' if dpo_info['ready'] else '✗ NEEDS MAPPING' if dpo_info['can_map'] else '✗ INCOMPATIBLE'}")
|
||||
if dpo_info["ready"]:
|
||||
print(f" Reason: Dataset has 'prompt', 'chosen', 'rejected' fields")
|
||||
print(f" Action: Use directly with DPOTrainer")
|
||||
elif dpo_info["can_map"]:
|
||||
print(f" Detected: prompt='{dpo_info['prompt_col']}' chosen='{dpo_info['chosen_col']}' rejected='{dpo_info['rejected_col']}'")
|
||||
print(f" Action: Apply mapping code (see below)")
|
||||
else:
|
||||
print(f" Status: Missing required fields (prompt + chosen + rejected)")
|
||||
|
||||
# GRPO
|
||||
print(f"\n[GRPO] {'✓ READY' if grpo_info['ready'] else '✗ NEEDS MAPPING' if grpo_info['can_map'] else '✗ INCOMPATIBLE'}")
|
||||
if grpo_info["ready"]:
|
||||
print(f" Reason: Dataset has 'prompt' field")
|
||||
print(f" Action: Use directly with GRPOTrainer")
|
||||
elif grpo_info["can_map"]:
|
||||
print(f" Detected: prompt='{grpo_info['prompt_col']}'")
|
||||
print(f" Action: Apply mapping code (see below)")
|
||||
else:
|
||||
print(f" Status: Missing prompt field")
|
||||
|
||||
# KTO
|
||||
print(f"\n[KTO] {'✓ READY' if kto_info['ready'] else '✗ INCOMPATIBLE'}")
|
||||
if kto_info["ready"]:
|
||||
print(f" Reason: Dataset has 'prompt', 'completion', 'label' fields")
|
||||
print(f" Action: Use directly with KTOTrainer")
|
||||
else:
|
||||
print(f" Status: Missing required fields (prompt + completion + label)")
|
||||
|
||||
# Mapping code
|
||||
print(f"\n{'MAPPING CODE (if needed)':-<80}")
|
||||
|
||||
mapping_needed = False
|
||||
|
||||
sft_mapping = generate_mapping_code("SFT", sft_info)
|
||||
if sft_mapping:
|
||||
print(f"\n# For SFT Training:")
|
||||
print(sft_mapping)
|
||||
mapping_needed = True
|
||||
|
||||
dpo_mapping = generate_mapping_code("DPO", dpo_info)
|
||||
if dpo_mapping:
|
||||
print(f"\n# For DPO Training:")
|
||||
print(dpo_mapping)
|
||||
mapping_needed = True
|
||||
|
||||
grpo_mapping = generate_mapping_code("GRPO", grpo_info)
|
||||
if grpo_mapping:
|
||||
print(f"\n# For GRPO Training:")
|
||||
print(grpo_mapping)
|
||||
mapping_needed = True
|
||||
|
||||
if not mapping_needed:
|
||||
print("\nNo mapping needed - dataset is ready for training!")
|
||||
|
||||
print(f"\n{'SUMMARY':-<80}")
|
||||
print(f"Recommended training methods: {', '.join(recommended) if recommended else 'None (dataset needs formatting)'}")
|
||||
print(f"\nNote: Used Datasets Server API (instant, no download required)")
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(0)
|
||||
except Exception as e:
|
||||
print(f"ERROR: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
Estimate training time and cost for TRL jobs.
|
||||
|
||||
Usage with uv:
|
||||
uv run estimate_cost.py --model <model> --dataset <dataset> --hardware <flavor>
|
||||
|
||||
Example:
|
||||
uv run estimate_cost.py --model Qwen/Qwen2.5-0.5B --dataset trl-lib/Capybara --hardware a10g-large
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
||||
# Hardware costs per hour (approximate)
|
||||
HARDWARE_COSTS = {
|
||||
"t4-small": 0.75,
|
||||
"t4-medium": 1.50,
|
||||
"l4x1": 2.50,
|
||||
"a10g-small": 3.50,
|
||||
"a10g-large": 5.00,
|
||||
"a10g-largex2": 10.00,
|
||||
"a10g-largex4": 20.00,
|
||||
"a100-large": 10.00,
|
||||
}
|
||||
|
||||
# Model sizes in billions of parameters
|
||||
MODEL_SIZES = {
|
||||
"0.5B": 0.5,
|
||||
"1.5B": 1.5,
|
||||
"3B": 3,
|
||||
"7B": 7,
|
||||
"13B": 13,
|
||||
}
|
||||
|
||||
def estimate_training_time(model_params, dataset_size, epochs, hardware):
|
||||
"""Estimate training time in hours."""
|
||||
# Rough estimates based on empirical observations
|
||||
# These are approximations and actual times will vary
|
||||
|
||||
base_time_per_1k_examples = 0.1 # hours for 1B model on a10g-large
|
||||
|
||||
# Adjust for model size
|
||||
time = base_time_per_1k_examples * model_params * (dataset_size / 1000) * epochs
|
||||
|
||||
# Adjust for hardware (relative to a10g-large baseline)
|
||||
hardware_multipliers = {
|
||||
"t4-small": 2.0,
|
||||
"t4-medium": 1.5,
|
||||
"l4x1": 1.2,
|
||||
"a10g-small": 1.3,
|
||||
"a10g-large": 1.0,
|
||||
"a10g-largex2": 0.6,
|
||||
"a10g-largex4": 0.4,
|
||||
"a100-large": 0.7,
|
||||
}
|
||||
|
||||
multiplier = hardware_multipliers.get(hardware, 1.0)
|
||||
time *= multiplier
|
||||
|
||||
return time
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Estimate training cost for TRL jobs")
|
||||
parser.add_argument("--model", required=True, help="Model name or size (e.g., 'Qwen/Qwen2.5-0.5B' or '0.5B')")
|
||||
parser.add_argument("--dataset", required=True, help="Dataset name")
|
||||
parser.add_argument("--hardware", required=True, choices=HARDWARE_COSTS.keys(), help="Hardware flavor")
|
||||
parser.add_argument("--dataset-size", type=int, help="Override dataset size (number of examples)")
|
||||
parser.add_argument("--epochs", type=int, default=3, help="Number of training epochs")
|
||||
return parser.parse_args()
|
||||
|
||||
def extract_model_size(model_name):
|
||||
"""Extract model size from name or return parsed value."""
|
||||
for size_str, size_val in MODEL_SIZES.items():
|
||||
if size_str in model_name:
|
||||
return size_val
|
||||
|
||||
# Try to parse directly
|
||||
try:
|
||||
if "B" in model_name:
|
||||
return float(model_name.replace("B", ""))
|
||||
except:
|
||||
pass
|
||||
|
||||
return 1.0 # Default to 1B if can't determine
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
# Extract model parameters
|
||||
model_params = extract_model_size(args.model)
|
||||
print(f"📊 Model: {args.model} (~{model_params}B parameters)")
|
||||
|
||||
# Estimate dataset size (would need to load to get real size)
|
||||
if args.dataset_size:
|
||||
dataset_size = args.dataset_size
|
||||
else:
|
||||
# Common dataset sizes (approximations)
|
||||
dataset_sizes = {
|
||||
"trl-lib/Capybara": 16000,
|
||||
"Anthropic/hh-rlhf": 160000,
|
||||
}
|
||||
dataset_size = dataset_sizes.get(args.dataset, 10000)
|
||||
|
||||
print(f"📦 Dataset: {args.dataset} (~{dataset_size} examples)")
|
||||
print(f"🔄 Epochs: {args.epochs}")
|
||||
print(f"💻 Hardware: {args.hardware}")
|
||||
print()
|
||||
|
||||
# Estimate training time
|
||||
estimated_hours = estimate_training_time(model_params, dataset_size, args.epochs, args.hardware)
|
||||
estimated_cost = estimated_hours * HARDWARE_COSTS[args.hardware]
|
||||
|
||||
# Recommend timeout with buffer
|
||||
recommended_timeout_hours = estimated_hours * 1.3 # 30% buffer
|
||||
|
||||
print(f"⏱️ Estimated training time: {estimated_hours:.1f} hours")
|
||||
print(f"💰 Estimated cost: ${estimated_cost:.2f}")
|
||||
print(f"⏰ Recommended timeout: {recommended_timeout_hours:.1f}h (with 30% buffer)")
|
||||
print()
|
||||
|
||||
# Warnings and recommendations
|
||||
if estimated_hours > 4:
|
||||
print("⚠️ Long training time - consider:")
|
||||
print(" - Using faster hardware")
|
||||
print(" - Reducing epochs")
|
||||
print(" - Using a smaller dataset subset for testing")
|
||||
|
||||
if model_params >= 7 and args.hardware not in ["a10g-largex2", "a10g-largex4", "a100-large"]:
|
||||
print("⚠️ Large model - consider using:")
|
||||
print(" - Larger GPU (a100-large)")
|
||||
print(" - Multi-GPU setup (a10g-largex2 or a10g-largex4)")
|
||||
print(" - LoRA/PEFT for memory efficiency")
|
||||
|
||||
print()
|
||||
print("📋 Example job configuration:")
|
||||
print(f"""
|
||||
hf_jobs("uv", {{
|
||||
"script": "your_training_script.py",
|
||||
"flavor": "{args.hardware}",
|
||||
"timeout": "{recommended_timeout_hours:.0f}h",
|
||||
"secrets": {{"HF_TOKEN": "$HF_TOKEN"}}
|
||||
}})
|
||||
""")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+659
@@ -0,0 +1,659 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
Search Hugging Face benchmark datasets and fetch leaderboard results.
|
||||
|
||||
This script is designed to be pipeline-friendly:
|
||||
- search benchmark datasets by free text, alias, task, and modality
|
||||
- fetch dataset leaderboards in normalized JSON / NDJSON / table form
|
||||
- optionally read dataset ids from stdin for chaining
|
||||
|
||||
It uses HF_TOKEN automatically when present.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import textwrap
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
BASE_URL = "https://huggingface.co"
|
||||
DEFAULT_TIMEOUT = 30
|
||||
|
||||
ALIASES: dict[str, list[str]] = {
|
||||
"ocr": [
|
||||
"ocr",
|
||||
"olmocr",
|
||||
"pdf",
|
||||
"image-to-text",
|
||||
"screen",
|
||||
"screenspot",
|
||||
"markdown",
|
||||
"text recognition",
|
||||
],
|
||||
"coding": [
|
||||
"code",
|
||||
"coding",
|
||||
"software engineering",
|
||||
"programming",
|
||||
"swe",
|
||||
"terminal",
|
||||
"patch",
|
||||
"bug",
|
||||
"cuda",
|
||||
],
|
||||
"math": [
|
||||
"math",
|
||||
"reasoning",
|
||||
"gsm8k",
|
||||
"mmlu",
|
||||
"gpqa",
|
||||
"aime",
|
||||
"hmmt",
|
||||
],
|
||||
"retrieval": [
|
||||
"retrieval",
|
||||
"search",
|
||||
"mteb",
|
||||
"arguana",
|
||||
"bright",
|
||||
],
|
||||
"agents": [
|
||||
"agent",
|
||||
"agents",
|
||||
"terminal",
|
||||
"screen",
|
||||
"computer use",
|
||||
"tool use",
|
||||
],
|
||||
"asr": [
|
||||
"asr",
|
||||
"speech",
|
||||
"audio",
|
||||
"transcribe",
|
||||
"transcription",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class HfApiError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class FullHelpArgumentParser(argparse.ArgumentParser):
|
||||
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
self._search_parser: argparse.ArgumentParser | None = None
|
||||
self._leaderboard_parser: argparse.ArgumentParser | None = None
|
||||
|
||||
def format_help(self) -> str:
|
||||
text = super().format_help()
|
||||
extra_sections: list[str] = []
|
||||
|
||||
if self._search_parser is not None:
|
||||
extra_sections.append(
|
||||
"\nsearch command options:\n"
|
||||
+ textwrap.indent(self._search_parser.format_help().strip(), " ")
|
||||
)
|
||||
|
||||
if self._leaderboard_parser is not None:
|
||||
extra_sections.append(
|
||||
"\nleaderboard command options:\n"
|
||||
+ textwrap.indent(self._leaderboard_parser.format_help().strip(), " ")
|
||||
)
|
||||
|
||||
if extra_sections:
|
||||
text += "\n" + "\n".join(extra_sections) + "\n"
|
||||
return text
|
||||
|
||||
|
||||
def auth_headers() -> dict[str, str]:
|
||||
token = os.getenv("HF_TOKEN")
|
||||
return {"Authorization": f"Bearer {token}"} if token else {}
|
||||
|
||||
|
||||
def http_get_json(path: str, params: dict[str, Any] | None = None) -> Any:
|
||||
url = f"{BASE_URL}{path}"
|
||||
if params:
|
||||
pairs: list[tuple[str, str]] = []
|
||||
for key, value in params.items():
|
||||
if value is None:
|
||||
continue
|
||||
if isinstance(value, (list, tuple)):
|
||||
for item in value:
|
||||
pairs.append((key, str(item)))
|
||||
else:
|
||||
pairs.append((key, str(value)))
|
||||
if pairs:
|
||||
url = f"{url}?{urllib.parse.urlencode(pairs)}"
|
||||
|
||||
req = urllib.request.Request(url, headers=auth_headers())
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=DEFAULT_TIMEOUT) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
body = exc.read().decode("utf-8", errors="replace")
|
||||
raise HfApiError(f"{exc.code} {exc.reason} for {url}: {body[:500]}") from exc
|
||||
except urllib.error.URLError as exc:
|
||||
raise HfApiError(f"Request failed for {url}: {exc}") from exc
|
||||
|
||||
|
||||
def shorten(text: str, width: int) -> str:
|
||||
text = " ".join((text or "").split())
|
||||
if len(text) <= width:
|
||||
return text
|
||||
return text[: max(0, width - 1)] + "…"
|
||||
|
||||
|
||||
def first_text(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
if isinstance(value, list):
|
||||
return " ".join(first_text(v) for v in value)
|
||||
if isinstance(value, dict):
|
||||
return " ".join(first_text(v) for v in value.values())
|
||||
return str(value)
|
||||
|
||||
|
||||
def benchmark_catalog(limit: int = 500) -> list[dict[str, Any]]:
|
||||
data = http_get_json(
|
||||
"/api/datasets",
|
||||
params={"filter": "benchmark:official", "limit": limit, "full": "true"},
|
||||
)
|
||||
if not isinstance(data, list):
|
||||
raise HfApiError("Unexpected response while listing benchmark datasets")
|
||||
return data
|
||||
|
||||
|
||||
def dataset_search_blob(dataset: dict[str, Any]) -> str:
|
||||
card = dataset.get("cardData") or {}
|
||||
parts = [
|
||||
dataset.get("id", ""),
|
||||
dataset.get("description", ""),
|
||||
first_text(dataset.get("tags")),
|
||||
first_text(card.get("pretty_name")),
|
||||
first_text(card.get("tags")),
|
||||
first_text(card.get("task_categories")),
|
||||
first_text(card.get("task_ids")),
|
||||
]
|
||||
return " ".join(parts).lower()
|
||||
|
||||
|
||||
def dataset_search_fields(dataset: dict[str, Any]) -> dict[str, str]:
|
||||
card = dataset.get("cardData") or {}
|
||||
return {
|
||||
"id": first_text(dataset.get("id")).lower(),
|
||||
"pretty_name": first_text(card.get("pretty_name")).lower(),
|
||||
"tags": " ".join(
|
||||
[
|
||||
first_text(dataset.get("tags")),
|
||||
first_text(card.get("tags")),
|
||||
first_text(card.get("task_categories")),
|
||||
first_text(card.get("task_ids")),
|
||||
first_text(card.get("modality")),
|
||||
]
|
||||
).lower(),
|
||||
"description": first_text(dataset.get("description")).lower(),
|
||||
}
|
||||
|
||||
|
||||
def collect_prefixed_tags(dataset: dict[str, Any], prefixes: Iterable[str]) -> list[str]:
|
||||
prefixes = tuple(prefixes)
|
||||
tags = dataset.get("tags") or []
|
||||
card = dataset.get("cardData") or {}
|
||||
|
||||
out: list[str] = []
|
||||
for tag in tags:
|
||||
if isinstance(tag, str) and tag.startswith(prefixes):
|
||||
out.append(tag)
|
||||
|
||||
for key, prefix in (
|
||||
("task_categories", "task_categories:"),
|
||||
("task_ids", "task_ids:"),
|
||||
("modality", "modality:"),
|
||||
):
|
||||
values = card.get(key)
|
||||
if isinstance(values, list):
|
||||
for value in values:
|
||||
full_tag = f"{prefix}{value}"
|
||||
if full_tag.startswith(prefixes):
|
||||
out.append(full_tag)
|
||||
|
||||
deduped: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for tag in out:
|
||||
if tag not in seen:
|
||||
deduped.append(tag)
|
||||
seen.add(tag)
|
||||
return deduped
|
||||
|
||||
|
||||
def expand_aliases(aliases: list[str]) -> dict[str, list[str]]:
|
||||
expanded: dict[str, list[str]] = {}
|
||||
for alias in aliases:
|
||||
terms = ALIASES.get(alias.lower(), [alias])
|
||||
expanded[alias] = terms
|
||||
return expanded
|
||||
|
||||
|
||||
def matches_term(blob: str, term: str) -> bool:
|
||||
candidate = term.lower().strip()
|
||||
if not candidate:
|
||||
return False
|
||||
if re.fullmatch(r"[a-z0-9_]+", candidate):
|
||||
return re.search(rf"(?<![a-z0-9_]){re.escape(candidate)}(?![a-z0-9_])", blob) is not None
|
||||
return candidate in blob
|
||||
|
||||
|
||||
def score_dataset(
|
||||
dataset: dict[str, Any],
|
||||
queries: list[str],
|
||||
aliases: dict[str, list[str]],
|
||||
tasks: list[str],
|
||||
modalities: list[str],
|
||||
) -> dict[str, Any]:
|
||||
blob = dataset_search_blob(dataset)
|
||||
fields = dataset_search_fields(dataset)
|
||||
task_tags = collect_prefixed_tags(dataset, ["task_categories:", "task_ids:"])
|
||||
modality_tags = collect_prefixed_tags(dataset, ["modality:"])
|
||||
|
||||
score = 0
|
||||
reasons: list[str] = []
|
||||
|
||||
for query in queries:
|
||||
q = query.lower().strip()
|
||||
if q and any(matches_term(value, q) for value in fields.values()):
|
||||
score += 3
|
||||
reasons.append(f"query:{query}")
|
||||
|
||||
for alias_name, terms in aliases.items():
|
||||
matched_terms: list[str] = []
|
||||
alias_score = 0
|
||||
for term in terms:
|
||||
strong_match = any(
|
||||
matches_term(fields[field_name], term)
|
||||
for field_name in ("id", "pretty_name", "tags")
|
||||
)
|
||||
desc_match = matches_term(fields["description"], term)
|
||||
if strong_match:
|
||||
alias_score += 2
|
||||
matched_terms.append(term)
|
||||
elif desc_match:
|
||||
alias_score += 1
|
||||
matched_terms.append(term)
|
||||
if matched_terms:
|
||||
score += alias_score
|
||||
reasons.append(f"alias:{alias_name}=" + ",".join(matched_terms[:5]))
|
||||
|
||||
lower_task_tags = [t.lower() for t in task_tags]
|
||||
for task in tasks:
|
||||
task = task.lower().strip()
|
||||
if not task:
|
||||
continue
|
||||
exact = [
|
||||
tag
|
||||
for tag in lower_task_tags
|
||||
if tag == f"task_categories:{task}" or tag == f"task_ids:{task}"
|
||||
]
|
||||
fuzzy = matches_term(blob, task)
|
||||
if exact:
|
||||
score += 5
|
||||
reasons.append(f"task:{task}")
|
||||
elif fuzzy:
|
||||
score += 2
|
||||
reasons.append(f"task~:{task}")
|
||||
|
||||
lower_modality_tags = [m.lower() for m in modality_tags]
|
||||
for modality in modalities:
|
||||
modality = modality.lower().strip()
|
||||
if not modality:
|
||||
continue
|
||||
if f"modality:{modality}" in lower_modality_tags:
|
||||
score += 4
|
||||
reasons.append(f"modality:{modality}")
|
||||
|
||||
return {
|
||||
"dataset_id": dataset.get("id"),
|
||||
"score": score,
|
||||
"reasons": reasons,
|
||||
"task_tags": task_tags,
|
||||
"modality_tags": modality_tags,
|
||||
"benchmark_tags": collect_prefixed_tags(dataset, ["benchmark:"]),
|
||||
"pretty_name": (dataset.get("cardData") or {}).get("pretty_name"),
|
||||
"downloads": dataset.get("downloads"),
|
||||
"description": " ".join((dataset.get("description") or "").split()),
|
||||
}
|
||||
|
||||
|
||||
def search_benchmarks(
|
||||
queries: list[str],
|
||||
aliases: list[str],
|
||||
tasks: list[str],
|
||||
modalities: list[str],
|
||||
limit: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
datasets = benchmark_catalog(limit=500)
|
||||
alias_map = expand_aliases(aliases)
|
||||
|
||||
results = [score_dataset(ds, queries, alias_map, tasks, modalities) for ds in datasets]
|
||||
|
||||
active_filters = bool(queries or aliases or tasks or modalities)
|
||||
if active_filters:
|
||||
results = [row for row in results if row["score"] >= 2]
|
||||
|
||||
results.sort(
|
||||
key=lambda row: (
|
||||
-row["score"],
|
||||
-(row["downloads"] or 0),
|
||||
row["dataset_id"] or "",
|
||||
)
|
||||
)
|
||||
return results[:limit]
|
||||
|
||||
|
||||
def parse_repo_id(repo_id: str) -> tuple[str, str]:
|
||||
if "/" not in repo_id:
|
||||
raise ValueError(f"Expected <namespace>/<repo>, got: {repo_id}")
|
||||
namespace, repo = repo_id.split("/", 1)
|
||||
return namespace, repo
|
||||
|
||||
|
||||
def get_leaderboard(repo_id: str, task_id: str | None = None) -> list[dict[str, Any]]:
|
||||
namespace, repo = parse_repo_id(repo_id)
|
||||
params = {"task_id": task_id} if task_id else None
|
||||
data = http_get_json(f"/api/datasets/{namespace}/{repo}/leaderboard", params=params)
|
||||
if not isinstance(data, list):
|
||||
raise HfApiError(f"Unexpected leaderboard response for {repo_id}")
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for row in data:
|
||||
source = row.get("source") or {}
|
||||
normalized.append(
|
||||
{
|
||||
"dataset_id": repo_id,
|
||||
"task_id": task_id,
|
||||
"rank": row.get("rank"),
|
||||
"model_id": row.get("modelId"),
|
||||
"value": row.get("value"),
|
||||
"verified": row.get("verified"),
|
||||
"lower_is_better": row.get("lower_is_better"),
|
||||
"filename": row.get("filename"),
|
||||
"notes": row.get("notes"),
|
||||
"pull_request": row.get("pullRequest"),
|
||||
"source_name": source.get("name"),
|
||||
"source_url": source.get("url"),
|
||||
"source_is_external": source.get("isExternal"),
|
||||
}
|
||||
)
|
||||
return normalized
|
||||
|
||||
|
||||
def read_repo_ids_from_stdin() -> list[str]:
|
||||
if sys.stdin.isatty():
|
||||
return []
|
||||
|
||||
repo_ids: list[str] = []
|
||||
for raw_line in sys.stdin:
|
||||
line = raw_line.strip()
|
||||
if not line:
|
||||
continue
|
||||
if line.startswith("{"):
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
candidate = obj.get("dataset_id") or obj.get("id")
|
||||
if isinstance(candidate, str) and "/" in candidate:
|
||||
repo_ids.append(candidate)
|
||||
continue
|
||||
if "/" in line:
|
||||
repo_ids.append(line)
|
||||
return repo_ids
|
||||
|
||||
|
||||
def print_json(data: Any) -> None:
|
||||
json.dump(data, sys.stdout, indent=2, ensure_ascii=False)
|
||||
sys.stdout.write("\n")
|
||||
|
||||
|
||||
def print_ndjson(rows: list[dict[str, Any]]) -> None:
|
||||
for row in rows:
|
||||
sys.stdout.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def print_search_table(rows: list[dict[str, Any]]) -> None:
|
||||
if not rows:
|
||||
print("No benchmark datasets matched.")
|
||||
return
|
||||
|
||||
headers = ["dataset_id", "score", "modalities", "tasks", "reasons", "description"]
|
||||
widths = [34, 5, 18, 24, 30, 68]
|
||||
print(" ".join(h.ljust(w) for h, w in zip(headers, widths)))
|
||||
print(" ".join("-" * w for w in widths))
|
||||
for row in rows:
|
||||
values = [
|
||||
shorten(row.get("dataset_id") or "", widths[0]),
|
||||
str(row.get("score", "")),
|
||||
shorten(", ".join(row.get("modality_tags") or []), widths[2]),
|
||||
shorten(", ".join(row.get("task_tags") or []), widths[3]),
|
||||
shorten(", ".join(row.get("reasons") or []), widths[4]),
|
||||
shorten(row.get("description") or "", widths[5]),
|
||||
]
|
||||
print(" ".join(v.ljust(w) for v, w in zip(values, widths)))
|
||||
|
||||
|
||||
def print_leaderboard_table(rows: list[dict[str, Any]]) -> None:
|
||||
if not rows:
|
||||
print("No leaderboard rows returned.")
|
||||
return
|
||||
|
||||
headers = ["dataset_id", "rank", "model_id", "value", "verified", "source"]
|
||||
widths = [30, 5, 38, 10, 8, 28]
|
||||
print(" ".join(h.ljust(w) for h, w in zip(headers, widths)))
|
||||
print(" ".join("-" * w for w in widths))
|
||||
for row in rows:
|
||||
values = [
|
||||
shorten(str(row.get("dataset_id") or ""), widths[0]),
|
||||
str(row.get("rank") or ""),
|
||||
shorten(str(row.get("model_id") or ""), widths[2]),
|
||||
shorten(str(row.get("value") or ""), widths[3]),
|
||||
str(row.get("verified")),
|
||||
shorten(str(row.get("source_name") or ""), widths[5]),
|
||||
]
|
||||
print(" ".join(v.ljust(w) for v, w in zip(values, widths)))
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = FullHelpArgumentParser(
|
||||
prog="hf_benchmarks.py",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
description=textwrap.dedent(
|
||||
"""
|
||||
Search benchmark datasets and fetch leaderboard results from the Hugging Face Hub.
|
||||
|
||||
Workflow ideas:
|
||||
1) Discover candidate benchmarks:
|
||||
hf_benchmarks.py search --alias ocr
|
||||
hf_benchmarks.py search --alias coding
|
||||
hf_benchmarks.py search --task image-to-text --modality document
|
||||
|
||||
2) Inspect a leaderboard:
|
||||
hf_benchmarks.py leaderboard allenai/olmOCR-bench --top 10
|
||||
|
||||
3) Chain search -> leaderboard:
|
||||
hf_benchmarks.py search --alias coding --format ndjson \\
|
||||
| hf_benchmarks.py leaderboard --stdin --top 5 --format table
|
||||
"""
|
||||
),
|
||||
)
|
||||
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
search_parser = subparsers.add_parser(
|
||||
"search",
|
||||
help="Search benchmark datasets by query, alias, task, and modality",
|
||||
)
|
||||
search_parser.add_argument(
|
||||
"--query",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Free-text query to match against benchmark dataset metadata. Repeatable.",
|
||||
)
|
||||
search_parser.add_argument(
|
||||
"--alias",
|
||||
action="append",
|
||||
default=[],
|
||||
help=(
|
||||
"Convenience alias for common benchmark domains. Known aliases: "
|
||||
+ ", ".join(sorted(ALIASES))
|
||||
+ ". Repeatable."
|
||||
),
|
||||
)
|
||||
search_parser.add_argument(
|
||||
"--task",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Task to match, e.g. text-generation, image-to-text, question-answering. Repeatable.",
|
||||
)
|
||||
search_parser.add_argument(
|
||||
"--modality",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Modality to match, e.g. text, image, document, audio. Repeatable.",
|
||||
)
|
||||
search_parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=20,
|
||||
help="Maximum number of rows to print (default: 20).",
|
||||
)
|
||||
search_parser.add_argument(
|
||||
"--format",
|
||||
choices=["table", "json", "ndjson"],
|
||||
default="table",
|
||||
help="Output format (default: table).",
|
||||
)
|
||||
|
||||
leaderboard_parser = subparsers.add_parser(
|
||||
"leaderboard",
|
||||
help="Fetch normalized leaderboard rows for one or more benchmark datasets",
|
||||
)
|
||||
leaderboard_parser.add_argument(
|
||||
"datasets",
|
||||
nargs="*",
|
||||
help="Dataset repo ids (<namespace>/<repo>). Can also be supplied via stdin with --stdin.",
|
||||
)
|
||||
leaderboard_parser.add_argument(
|
||||
"--stdin",
|
||||
action="store_true",
|
||||
help="Read dataset ids from stdin. Accepts plain repo ids or NDJSON with dataset_id/id fields.",
|
||||
)
|
||||
leaderboard_parser.add_argument(
|
||||
"--task-id",
|
||||
default=None,
|
||||
help="Optional leaderboard task_id query parameter.",
|
||||
)
|
||||
leaderboard_parser.add_argument(
|
||||
"--top",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Only keep the top N results per leaderboard.",
|
||||
)
|
||||
leaderboard_parser.add_argument(
|
||||
"--format",
|
||||
choices=["table", "json", "ndjson"],
|
||||
default="table",
|
||||
help="Output format (default: table).",
|
||||
)
|
||||
|
||||
parser._search_parser = search_parser
|
||||
parser._leaderboard_parser = leaderboard_parser
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def run_search(args: argparse.Namespace) -> int:
|
||||
rows = search_benchmarks(
|
||||
queries=args.query,
|
||||
aliases=args.alias,
|
||||
tasks=args.task,
|
||||
modalities=args.modality,
|
||||
limit=args.limit,
|
||||
)
|
||||
|
||||
if args.format == "json":
|
||||
print_json(rows)
|
||||
elif args.format == "ndjson":
|
||||
print_ndjson(rows)
|
||||
else:
|
||||
print_search_table(rows)
|
||||
return 0
|
||||
|
||||
|
||||
def run_leaderboard(args: argparse.Namespace) -> int:
|
||||
repo_ids = list(args.datasets)
|
||||
if args.stdin:
|
||||
repo_ids.extend(read_repo_ids_from_stdin())
|
||||
|
||||
deduped: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for repo_id in repo_ids:
|
||||
if repo_id not in seen:
|
||||
deduped.append(repo_id)
|
||||
seen.add(repo_id)
|
||||
repo_ids = deduped
|
||||
|
||||
if not repo_ids:
|
||||
print("Error: provide dataset ids or use --stdin.", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
for repo_id in repo_ids:
|
||||
dataset_rows = get_leaderboard(repo_id, task_id=args.task_id)
|
||||
if args.top is not None:
|
||||
dataset_rows = dataset_rows[: args.top]
|
||||
rows.extend(dataset_rows)
|
||||
|
||||
if args.format == "json":
|
||||
print_json(rows)
|
||||
elif args.format == "ndjson":
|
||||
print_ndjson(rows)
|
||||
else:
|
||||
print_leaderboard_table(rows)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
if args.command == "search":
|
||||
return run_search(args)
|
||||
if args.command == "leaderboard":
|
||||
return run_leaderboard(args)
|
||||
parser.error(f"Unknown command: {args.command}")
|
||||
return 2
|
||||
except HfApiError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
except ValueError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "trl>=0.12.0",
|
||||
# "transformers>=4.36.0",
|
||||
# "accelerate>=0.24.0",
|
||||
# "trackio",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
"""
|
||||
Production-ready DPO training example for preference learning.
|
||||
|
||||
DPO (Direct Preference Optimization) trains models on preference pairs
|
||||
(chosen vs rejected responses) without requiring a reward model.
|
||||
|
||||
Usage with hf_jobs MCP tool:
|
||||
hf_jobs("uv", {
|
||||
"script": '''<paste this entire file>''',
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "3h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"},
|
||||
})
|
||||
|
||||
Or submit the script content directly inline without saving to a file.
|
||||
"""
|
||||
|
||||
import trackio
|
||||
from datasets import load_dataset
|
||||
from trl import DPOTrainer, DPOConfig
|
||||
|
||||
|
||||
# Load preference dataset
|
||||
print("📦 Loading dataset...")
|
||||
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
|
||||
print(f"✅ Dataset loaded: {len(dataset)} preference pairs")
|
||||
|
||||
# Create train/eval split
|
||||
print("🔀 Creating train/eval split...")
|
||||
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
|
||||
train_dataset = dataset_split["train"]
|
||||
eval_dataset = dataset_split["test"]
|
||||
print(f" Train: {len(train_dataset)} pairs")
|
||||
print(f" Eval: {len(eval_dataset)} pairs")
|
||||
|
||||
# Training configuration
|
||||
config = DPOConfig(
|
||||
# CRITICAL: Hub settings
|
||||
output_dir="qwen-dpo-aligned",
|
||||
push_to_hub=True,
|
||||
hub_model_id="username/qwen-dpo-aligned",
|
||||
hub_strategy="every_save",
|
||||
|
||||
# DPO-specific parameters
|
||||
beta=0.1, # KL penalty coefficient (higher = stay closer to reference)
|
||||
|
||||
# Training parameters
|
||||
num_train_epochs=1, # DPO typically needs fewer epochs than SFT
|
||||
per_device_train_batch_size=4,
|
||||
gradient_accumulation_steps=4,
|
||||
learning_rate=5e-7, # DPO uses much lower LR than SFT
|
||||
# max_length=1024, # Default - only set if you need different sequence length
|
||||
|
||||
# Logging & checkpointing
|
||||
logging_steps=10,
|
||||
save_strategy="steps",
|
||||
save_steps=100,
|
||||
save_total_limit=2,
|
||||
|
||||
# Evaluation - IMPORTANT: Only enable if eval_dataset provided
|
||||
eval_strategy="steps",
|
||||
eval_steps=100,
|
||||
|
||||
# Optimization
|
||||
warmup_ratio=0.1,
|
||||
lr_scheduler_type="cosine",
|
||||
|
||||
# Monitoring
|
||||
report_to="trackio", # Integrate with Trackio
|
||||
project="meaningful_project_name", # project name for the training name (trackio)
|
||||
run_name="baseline-run", #Descriptive name for this training run
|
||||
|
||||
)
|
||||
|
||||
# Initialize and train
|
||||
# Note: DPO requires an instruct-tuned model as the base
|
||||
print("🎯 Initializing trainer...")
|
||||
trainer = DPOTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B-Instruct", # Use instruct model, not base model
|
||||
train_dataset=train_dataset,
|
||||
eval_dataset=eval_dataset, # CRITICAL: Must provide eval_dataset when eval_strategy is enabled
|
||||
args=config,
|
||||
)
|
||||
|
||||
print("🚀 Starting DPO training...")
|
||||
trainer.train()
|
||||
|
||||
print("💾 Pushing to Hub...")
|
||||
trainer.push_to_hub()
|
||||
|
||||
# Finish Trackio tracking
|
||||
trackio.finish()
|
||||
|
||||
print("✅ Complete! Model at: https://huggingface.co/username/qwen-dpo-aligned")
|
||||
print("📊 View metrics at: https://huggingface.co/spaces/username/trackio")
|
||||
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "trl>=0.12.0",
|
||||
# "transformers>=4.36.0",
|
||||
# "accelerate>=0.24.0",
|
||||
# "trackio",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
"""
|
||||
Production-ready GRPO training example for online RL.
|
||||
|
||||
GRPO (Group Relative Policy Optimization) is an online RL method that
|
||||
optimizes relative to group performance. Best for tasks with automatic
|
||||
reward signals like code execution or math verification.
|
||||
|
||||
Usage with hf_jobs MCP tool:
|
||||
hf_jobs("uv", {
|
||||
"script": '''<paste this entire file>''',
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "4h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"},
|
||||
})
|
||||
|
||||
Or submit the script content directly inline without saving to a file.
|
||||
|
||||
Note: For most GRPO use cases, the TRL maintained script is recommended:
|
||||
https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py
|
||||
"""
|
||||
|
||||
import trackio
|
||||
from datasets import load_dataset
|
||||
from trl import GRPOTrainer, GRPOConfig
|
||||
|
||||
|
||||
# Load dataset (GRPO uses prompt-only format)
|
||||
dataset = load_dataset("trl-lib/math_shepherd", split="train")
|
||||
print(f"✅ Dataset loaded: {len(dataset)} prompts")
|
||||
|
||||
# Training configuration
|
||||
config = GRPOConfig(
|
||||
# CRITICAL: Hub settings
|
||||
output_dir="qwen-grpo-math",
|
||||
push_to_hub=True,
|
||||
hub_model_id="username/qwen-grpo-math",
|
||||
hub_strategy="every_save",
|
||||
|
||||
# Training parameters
|
||||
num_train_epochs=1,
|
||||
per_device_train_batch_size=4,
|
||||
gradient_accumulation_steps=4,
|
||||
learning_rate=1e-6,
|
||||
|
||||
# Logging & checkpointing
|
||||
logging_steps=10,
|
||||
save_strategy="steps",
|
||||
save_steps=100,
|
||||
save_total_limit=2,
|
||||
|
||||
# Optimization
|
||||
warmup_ratio=0.1,
|
||||
lr_scheduler_type="cosine",
|
||||
|
||||
# Monitoring
|
||||
report_to="trackio", # Integrate with Trackio
|
||||
project="meaningful_project_name", # project name for the training name (trackio)
|
||||
run_name="baseline-run", #Descriptive name for this training run
|
||||
|
||||
)
|
||||
|
||||
# Initialize and train
|
||||
# Note: GRPO requires an instruct-tuned model as the base
|
||||
trainer = GRPOTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B-Instruct",
|
||||
train_dataset=dataset,
|
||||
args=config,
|
||||
)
|
||||
|
||||
print("🚀 Starting GRPO training...")
|
||||
trainer.train()
|
||||
|
||||
print("💾 Pushing to Hub...")
|
||||
trainer.push_to_hub()
|
||||
|
||||
|
||||
print("✅ Complete! Model at: https://huggingface.co/username/qwen-grpo-math")
|
||||
print("📊 View metrics at: https://huggingface.co/spaces/username/trackio")
|
||||
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "trl>=0.12.0",
|
||||
# "peft>=0.7.0",
|
||||
# "transformers>=4.36.0",
|
||||
# "accelerate>=0.24.0",
|
||||
# "trackio",
|
||||
# ]
|
||||
# ///
|
||||
|
||||
"""
|
||||
Production-ready SFT training example with all best practices.
|
||||
|
||||
This script demonstrates:
|
||||
- Trackio integration for real-time monitoring
|
||||
- LoRA/PEFT for efficient training
|
||||
- Proper Hub saving configuration
|
||||
- Train/eval split for monitoring
|
||||
- Checkpoint management
|
||||
- Optimized training parameters
|
||||
|
||||
Usage with hf_jobs MCP tool:
|
||||
hf_jobs("uv", {
|
||||
"script": '''<paste this entire file>''',
|
||||
"flavor": "a10g-large",
|
||||
"timeout": "3h",
|
||||
"secrets": {"HF_TOKEN": "$HF_TOKEN"},
|
||||
})
|
||||
|
||||
Or submit the script content directly inline without saving to a file.
|
||||
"""
|
||||
|
||||
import trackio
|
||||
from datasets import load_dataset
|
||||
from peft import LoraConfig
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
|
||||
|
||||
# Load dataset
|
||||
print("📦 Loading dataset...")
|
||||
dataset = load_dataset("trl-lib/Capybara", split="train")
|
||||
print(f"✅ Dataset loaded: {len(dataset)} examples")
|
||||
|
||||
# Create train/eval split
|
||||
print("🔀 Creating train/eval split...")
|
||||
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
|
||||
train_dataset = dataset_split["train"]
|
||||
eval_dataset = dataset_split["test"]
|
||||
print(f" Train: {len(train_dataset)} examples")
|
||||
print(f" Eval: {len(eval_dataset)} examples")
|
||||
|
||||
# Note: For memory-constrained demos, skip eval by using full dataset as train_dataset
|
||||
# and removing eval_dataset, eval_strategy, and eval_steps from config below
|
||||
|
||||
# Training configuration
|
||||
config = SFTConfig(
|
||||
# CRITICAL: Hub settings
|
||||
output_dir="qwen-capybara-sft",
|
||||
push_to_hub=True,
|
||||
hub_model_id="username/qwen-capybara-sft",
|
||||
hub_strategy="every_save", # Push checkpoints
|
||||
|
||||
# Training parameters
|
||||
num_train_epochs=3,
|
||||
per_device_train_batch_size=4,
|
||||
gradient_accumulation_steps=4,
|
||||
learning_rate=2e-5,
|
||||
# max_length=1024, # Default - only set if you need different sequence length
|
||||
|
||||
# Logging & checkpointing
|
||||
logging_steps=10,
|
||||
save_strategy="steps",
|
||||
save_steps=100,
|
||||
save_total_limit=2,
|
||||
|
||||
# Evaluation - IMPORTANT: Only enable if eval_dataset provided
|
||||
eval_strategy="steps",
|
||||
eval_steps=100,
|
||||
|
||||
# Optimization
|
||||
warmup_ratio=0.1,
|
||||
lr_scheduler_type="cosine",
|
||||
|
||||
# Monitoring
|
||||
report_to="trackio", # Integrate with Trackio
|
||||
project="meaningful_project_name", # project name for the training name (trackio)
|
||||
run_name="baseline-run", #Descriptive name for this training run
|
||||
)
|
||||
|
||||
# LoRA configuration
|
||||
peft_config = LoraConfig(
|
||||
r=16,
|
||||
lora_alpha=32,
|
||||
lora_dropout=0.05,
|
||||
bias="none",
|
||||
task_type="CAUSAL_LM",
|
||||
target_modules=["q_proj", "v_proj"],
|
||||
)
|
||||
|
||||
# Initialize and train
|
||||
print("🎯 Initializing trainer...")
|
||||
trainer = SFTTrainer(
|
||||
model="Qwen/Qwen2.5-0.5B",
|
||||
train_dataset=train_dataset,
|
||||
eval_dataset=eval_dataset, # CRITICAL: Must provide eval_dataset when eval_strategy is enabled
|
||||
args=config,
|
||||
peft_config=peft_config,
|
||||
)
|
||||
|
||||
print("🚀 Starting training...")
|
||||
trainer.train()
|
||||
|
||||
print("💾 Pushing to Hub...")
|
||||
trainer.push_to_hub()
|
||||
|
||||
# Finish Trackio tracking
|
||||
trackio.finish()
|
||||
|
||||
print("✅ Complete! Model at: https://huggingface.co/username/qwen-capybara-sft")
|
||||
print("📊 View metrics at: https://huggingface.co/spaces/username/trackio")
|
||||
@@ -0,0 +1,512 @@
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "unsloth",
|
||||
# "datasets",
|
||||
# "trl==0.22.2",
|
||||
# "huggingface_hub[hf_transfer]",
|
||||
# "trackio",
|
||||
# "tensorboard",
|
||||
# "transformers==4.57.3",
|
||||
# ]
|
||||
# ///
|
||||
"""
|
||||
Fine-tune LLMs using Unsloth optimizations for ~60% less VRAM and 2x faster training.
|
||||
|
||||
Supports epoch-based or step-based training with optional eval split.
|
||||
Default model: LFM2.5-1.2B-Instruct (Liquid Foundation Model).
|
||||
|
||||
Epoch-based training (recommended for full datasets):
|
||||
uv run unsloth_sft_example.py \
|
||||
--dataset mlabonne/FineTome-100k \
|
||||
--num-epochs 1 \
|
||||
--eval-split 0.2 \
|
||||
--output-repo your-username/model-finetuned
|
||||
|
||||
Run on HF Jobs (1 epoch with eval):
|
||||
hf jobs uv run unsloth_sft_example.py \
|
||||
--flavor a10g-small --secrets HF_TOKEN --timeout 4h \
|
||||
-- --dataset mlabonne/FineTome-100k \
|
||||
--num-epochs 1 \
|
||||
--eval-split 0.2 \
|
||||
--output-repo your-username/model-finetuned
|
||||
|
||||
Step-based training (for quick tests):
|
||||
uv run unsloth_sft_example.py \
|
||||
--dataset mlabonne/FineTome-100k \
|
||||
--max-steps 500 \
|
||||
--output-repo your-username/model-finetuned
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
# Force unbuffered output for HF Jobs logs
|
||||
sys.stdout.reconfigure(line_buffering=True)
|
||||
sys.stderr.reconfigure(line_buffering=True)
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(levelname)s - %(message)s",
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def check_cuda():
|
||||
"""Check CUDA availability and exit if not available."""
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
logger.error("CUDA is not available. This script requires a GPU.")
|
||||
logger.error("Run on a machine with a CUDA-capable GPU or use HF Jobs:")
|
||||
logger.error(
|
||||
" hf jobs uv run unsloth_sft_example.py --flavor a10g-small ..."
|
||||
)
|
||||
sys.exit(1)
|
||||
logger.info(f"CUDA available: {torch.cuda.get_device_name(0)}")
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Fine-tune LLMs with Unsloth optimizations",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Quick test run
|
||||
uv run unsloth_sft_example.py \\
|
||||
--dataset mlabonne/FineTome-100k \\
|
||||
--max-steps 50 \\
|
||||
--output-repo username/model-test
|
||||
|
||||
# Full training with eval
|
||||
uv run unsloth_sft_example.py \\
|
||||
--dataset mlabonne/FineTome-100k \\
|
||||
--num-epochs 1 \\
|
||||
--eval-split 0.2 \\
|
||||
--output-repo username/model-finetuned
|
||||
|
||||
# With Trackio monitoring
|
||||
uv run unsloth_sft_example.py \\
|
||||
--dataset mlabonne/FineTome-100k \\
|
||||
--num-epochs 1 \\
|
||||
--output-repo username/model-finetuned \\
|
||||
--trackio-space username/trackio
|
||||
""",
|
||||
)
|
||||
|
||||
# Model and data
|
||||
parser.add_argument(
|
||||
"--base-model",
|
||||
default="LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
help="Base model (default: LiquidAI/LFM2.5-1.2B-Instruct)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dataset",
|
||||
required=True,
|
||||
help="Dataset in ShareGPT/conversation format (e.g., mlabonne/FineTome-100k)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-repo",
|
||||
required=True,
|
||||
help="HF Hub repo to push model to (e.g., 'username/model-finetuned')",
|
||||
)
|
||||
|
||||
# Training config
|
||||
parser.add_argument(
|
||||
"--num-epochs",
|
||||
type=float,
|
||||
default=None,
|
||||
help="Number of epochs (default: None). Use instead of --max-steps.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-steps",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Training steps (default: None). Use for quick tests or streaming.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch-size",
|
||||
type=int,
|
||||
default=2,
|
||||
help="Per-device batch size (default: 2)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--gradient-accumulation",
|
||||
type=int,
|
||||
default=4,
|
||||
help="Gradient accumulation steps (default: 4). Effective batch = batch-size * this",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--learning-rate",
|
||||
type=float,
|
||||
default=2e-4,
|
||||
help="Learning rate (default: 2e-4)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-seq-length",
|
||||
type=int,
|
||||
default=2048,
|
||||
help="Maximum sequence length (default: 2048)",
|
||||
)
|
||||
|
||||
# LoRA config
|
||||
parser.add_argument(
|
||||
"--lora-r",
|
||||
type=int,
|
||||
default=16,
|
||||
help="LoRA rank (default: 16). Higher = more capacity but more VRAM",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-alpha",
|
||||
type=int,
|
||||
default=16,
|
||||
help="LoRA alpha (default: 16). Same as r per Unsloth recommendation",
|
||||
)
|
||||
|
||||
# Logging
|
||||
parser.add_argument(
|
||||
"--trackio-space",
|
||||
default=None,
|
||||
help="HF Space for Trackio dashboard (e.g., 'username/trackio')",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--run-name",
|
||||
default=None,
|
||||
help="Custom run name for Trackio (default: auto-generated)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--save-local",
|
||||
default="unsloth-output",
|
||||
help="Local directory to save model (default: unsloth-output)",
|
||||
)
|
||||
|
||||
# Evaluation and data control
|
||||
parser.add_argument(
|
||||
"--eval-split",
|
||||
type=float,
|
||||
default=0.0,
|
||||
help="Fraction of data for evaluation (0.0-0.5). Default: 0.0 (no eval)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-samples",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Limit samples (default: None = use all)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--seed",
|
||||
type=int,
|
||||
default=3407,
|
||||
help="Random seed for reproducibility (default: 3407)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--merge-model",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Merge LoRA weights into base model before uploading (larger file, easier to use)",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
# Validate epochs/steps configuration
|
||||
if not args.num_epochs and not args.max_steps:
|
||||
args.num_epochs = 1
|
||||
logger.info("Using default --num-epochs=1")
|
||||
|
||||
# Determine training duration display
|
||||
if args.num_epochs:
|
||||
duration_str = f"{args.num_epochs} epoch(s)"
|
||||
else:
|
||||
duration_str = f"{args.max_steps} steps"
|
||||
|
||||
print("=" * 70)
|
||||
print("LLM Fine-tuning with Unsloth")
|
||||
print("=" * 70)
|
||||
print("\nConfiguration:")
|
||||
print(f" Base model: {args.base_model}")
|
||||
print(f" Dataset: {args.dataset}")
|
||||
print(f" Num samples: {args.num_samples or 'all'}")
|
||||
print(f" Eval split: {args.eval_split if args.eval_split > 0 else '(disabled)'}")
|
||||
print(f" Seed: {args.seed}")
|
||||
print(f" Training: {duration_str}")
|
||||
print(f" Batch size: {args.batch_size} x {args.gradient_accumulation} = {args.batch_size * args.gradient_accumulation}")
|
||||
print(f" Learning rate: {args.learning_rate}")
|
||||
print(f" LoRA rank: {args.lora_r}")
|
||||
print(f" Max seq length: {args.max_seq_length}")
|
||||
print(f" Output repo: {args.output_repo}")
|
||||
print(f" Trackio space: {args.trackio_space or '(not configured)'}")
|
||||
print()
|
||||
|
||||
# Check CUDA before heavy imports
|
||||
check_cuda()
|
||||
|
||||
# Enable fast transfers
|
||||
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
|
||||
|
||||
# Set Trackio space if provided
|
||||
if args.trackio_space:
|
||||
os.environ["TRACKIO_SPACE_ID"] = args.trackio_space
|
||||
logger.info(f"Trackio dashboard: https://huggingface.co/spaces/{args.trackio_space}")
|
||||
|
||||
# Import heavy dependencies
|
||||
from unsloth import FastLanguageModel
|
||||
from unsloth.chat_templates import standardize_data_formats, train_on_responses_only
|
||||
from datasets import load_dataset
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
from huggingface_hub import login
|
||||
|
||||
# Login to Hub
|
||||
token = os.environ.get("HF_TOKEN") or os.environ.get("hfjob")
|
||||
if token:
|
||||
login(token=token)
|
||||
logger.info("Logged in to Hugging Face Hub")
|
||||
else:
|
||||
logger.warning("HF_TOKEN not set - model upload may fail")
|
||||
|
||||
# 1. Load model
|
||||
print("\n[1/5] Loading model...")
|
||||
start = time.time()
|
||||
|
||||
model, tokenizer = FastLanguageModel.from_pretrained(
|
||||
model_name=args.base_model,
|
||||
max_seq_length=args.max_seq_length,
|
||||
load_in_4bit=False,
|
||||
load_in_8bit=False,
|
||||
load_in_16bit=True,
|
||||
full_finetuning=False,
|
||||
)
|
||||
|
||||
# Add LoRA adapters
|
||||
model = FastLanguageModel.get_peft_model(
|
||||
model,
|
||||
r=args.lora_r,
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "out_proj", "in_proj", "w1", "w2", "w3"],
|
||||
lora_alpha=args.lora_alpha,
|
||||
lora_dropout=0,
|
||||
bias="none",
|
||||
use_gradient_checkpointing="unsloth",
|
||||
random_state=args.seed,
|
||||
use_rslora=False,
|
||||
loftq_config=None,
|
||||
)
|
||||
print(f"Model loaded in {time.time() - start:.1f}s")
|
||||
|
||||
# 2. Load and prepare dataset
|
||||
print("\n[2/5] Loading dataset...")
|
||||
start = time.time()
|
||||
|
||||
dataset = load_dataset(args.dataset, split="train")
|
||||
print(f" Dataset has {len(dataset)} total samples")
|
||||
|
||||
if args.num_samples:
|
||||
dataset = dataset.select(range(min(args.num_samples, len(dataset))))
|
||||
print(f" Limited to {len(dataset)} samples")
|
||||
|
||||
# Auto-detect and normalize conversation column
|
||||
for col in ["messages", "conversations", "conversation"]:
|
||||
if col in dataset.column_names and isinstance(dataset[0][col], list):
|
||||
if col != "conversations":
|
||||
dataset = dataset.rename_column(col, "conversations")
|
||||
break
|
||||
dataset = standardize_data_formats(dataset)
|
||||
|
||||
# Apply chat template
|
||||
def formatting_prompts_func(examples):
|
||||
texts = tokenizer.apply_chat_template(
|
||||
examples["conversations"],
|
||||
tokenize=False,
|
||||
add_generation_prompt=False,
|
||||
)
|
||||
# Remove BOS token to avoid duplicates
|
||||
return {"text": [x.removeprefix(tokenizer.bos_token) for x in texts]}
|
||||
|
||||
dataset = dataset.map(formatting_prompts_func, batched=True)
|
||||
|
||||
# Split for evaluation if requested
|
||||
if args.eval_split > 0:
|
||||
split = dataset.train_test_split(test_size=args.eval_split, seed=args.seed)
|
||||
train_data = split["train"]
|
||||
eval_data = split["test"]
|
||||
print(f" Train: {len(train_data)} samples, Eval: {len(eval_data)} samples")
|
||||
else:
|
||||
train_data = dataset
|
||||
eval_data = None
|
||||
|
||||
print(f" Dataset ready in {time.time() - start:.1f}s")
|
||||
|
||||
# 3. Configure trainer
|
||||
print("\n[3/5] Configuring trainer...")
|
||||
|
||||
# Calculate steps per epoch for logging/eval intervals
|
||||
effective_batch = args.batch_size * args.gradient_accumulation
|
||||
num_samples = len(train_data)
|
||||
steps_per_epoch = num_samples // effective_batch
|
||||
|
||||
# Determine run name and logging steps
|
||||
if args.run_name:
|
||||
run_name = args.run_name
|
||||
elif args.num_epochs:
|
||||
run_name = f"unsloth-sft-{args.num_epochs}ep"
|
||||
else:
|
||||
run_name = f"unsloth-sft-{args.max_steps}steps"
|
||||
|
||||
if args.num_epochs:
|
||||
logging_steps = max(1, steps_per_epoch // 10)
|
||||
save_steps = max(1, steps_per_epoch // 4)
|
||||
else:
|
||||
logging_steps = max(1, args.max_steps // 20)
|
||||
save_steps = max(1, args.max_steps // 4)
|
||||
|
||||
# Determine reporting backend
|
||||
if args.trackio_space:
|
||||
report_to = ["tensorboard", "trackio"]
|
||||
else:
|
||||
report_to = ["tensorboard"]
|
||||
|
||||
training_config = SFTConfig(
|
||||
output_dir=args.save_local,
|
||||
dataset_text_field="text",
|
||||
per_device_train_batch_size=args.batch_size,
|
||||
gradient_accumulation_steps=args.gradient_accumulation,
|
||||
warmup_steps=5,
|
||||
num_train_epochs=args.num_epochs if args.num_epochs else 1,
|
||||
max_steps=args.max_steps if args.max_steps else -1,
|
||||
learning_rate=args.learning_rate,
|
||||
logging_steps=logging_steps,
|
||||
optim="adamw_8bit",
|
||||
weight_decay=0.01,
|
||||
lr_scheduler_type="linear",
|
||||
seed=args.seed,
|
||||
max_length=args.max_seq_length,
|
||||
report_to=report_to,
|
||||
run_name=run_name,
|
||||
push_to_hub=True,
|
||||
hub_model_id=args.output_repo,
|
||||
save_steps=save_steps,
|
||||
save_total_limit=3,
|
||||
)
|
||||
|
||||
# Add evaluation config if eval is enabled
|
||||
if eval_data:
|
||||
if args.num_epochs:
|
||||
training_config.eval_strategy = "epoch"
|
||||
print(" Evaluation enabled: every epoch")
|
||||
else:
|
||||
training_config.eval_strategy = "steps"
|
||||
training_config.eval_steps = max(1, args.max_steps // 5)
|
||||
print(f" Evaluation enabled: every {training_config.eval_steps} steps")
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model=model,
|
||||
tokenizer=tokenizer,
|
||||
train_dataset=train_data,
|
||||
eval_dataset=eval_data,
|
||||
args=training_config,
|
||||
)
|
||||
|
||||
# Train on responses only (mask user inputs)
|
||||
trainer = train_on_responses_only(
|
||||
trainer,
|
||||
instruction_part="<|im_start|>user\n",
|
||||
response_part="<|im_start|>assistant\n",
|
||||
)
|
||||
|
||||
# 4. Train
|
||||
print(f"\n[4/5] Training for {duration_str}...")
|
||||
if args.num_epochs:
|
||||
print(f" (~{steps_per_epoch} steps/epoch, {int(steps_per_epoch * args.num_epochs)} total steps)")
|
||||
start = time.time()
|
||||
|
||||
train_result = trainer.train()
|
||||
|
||||
train_time = time.time() - start
|
||||
total_steps = train_result.metrics.get("train_steps", args.max_steps or steps_per_epoch * args.num_epochs)
|
||||
print(f"\nTraining completed in {train_time / 60:.1f} minutes")
|
||||
print(f" Speed: {total_steps / train_time:.2f} steps/s")
|
||||
|
||||
# Print training metrics
|
||||
train_loss = train_result.metrics.get("train_loss")
|
||||
if train_loss:
|
||||
print(f" Final train loss: {train_loss:.4f}")
|
||||
|
||||
# Print eval results if eval was enabled
|
||||
if eval_data:
|
||||
print("\nRunning final evaluation...")
|
||||
try:
|
||||
eval_results = trainer.evaluate()
|
||||
eval_loss = eval_results.get("eval_loss")
|
||||
if eval_loss:
|
||||
print(f" Final eval loss: {eval_loss:.4f}")
|
||||
if train_loss:
|
||||
ratio = eval_loss / train_loss
|
||||
if ratio > 1.5:
|
||||
print(f" Warning: Eval loss is {ratio:.1f}x train loss - possible overfitting")
|
||||
else:
|
||||
print(f" Eval/train ratio: {ratio:.2f} - model generalizes well")
|
||||
except Exception as e:
|
||||
print(f" Warning: Final evaluation failed: {e}")
|
||||
print(" Continuing to save model...")
|
||||
|
||||
# 5. Save and push
|
||||
print("\n[5/5] Saving model...")
|
||||
|
||||
if args.merge_model:
|
||||
print("Merging LoRA weights into base model...")
|
||||
print(f"\nPushing merged model to {args.output_repo}...")
|
||||
model.push_to_hub_merged(
|
||||
args.output_repo,
|
||||
tokenizer=tokenizer,
|
||||
save_method="merged_16bit",
|
||||
)
|
||||
print(f"Merged model available at: https://huggingface.co/{args.output_repo}")
|
||||
else:
|
||||
model.save_pretrained(args.save_local)
|
||||
tokenizer.save_pretrained(args.save_local)
|
||||
print(f"Saved locally to {args.save_local}/")
|
||||
|
||||
print(f"\nPushing adapter to {args.output_repo}...")
|
||||
model.push_to_hub(args.output_repo, tokenizer=tokenizer)
|
||||
print(f"Adapter available at: https://huggingface.co/{args.output_repo}")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("Done!")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) == 1:
|
||||
print("=" * 70)
|
||||
print("LLM Fine-tuning with Unsloth")
|
||||
print("=" * 70)
|
||||
print("\nFine-tune language models with optional train/eval split.")
|
||||
print("\nFeatures:")
|
||||
print(" - ~60% less VRAM with Unsloth optimizations")
|
||||
print(" - 2x faster training vs standard methods")
|
||||
print(" - Epoch-based or step-based training")
|
||||
print(" - Optional evaluation to detect overfitting")
|
||||
print(" - Trains only on assistant responses (masked user inputs)")
|
||||
print("\nEpoch-based training:")
|
||||
print("\n uv run unsloth_sft_example.py \\")
|
||||
print(" --dataset mlabonne/FineTome-100k \\")
|
||||
print(" --num-epochs 1 \\")
|
||||
print(" --eval-split 0.2 \\")
|
||||
print(" --output-repo your-username/model-finetuned")
|
||||
print("\nHF Jobs example:")
|
||||
print("\n hf jobs uv run unsloth_sft_example.py \\")
|
||||
print(" --flavor a10g-small --secrets HF_TOKEN --timeout 4h \\")
|
||||
print(" -- --dataset mlabonne/FineTome-100k \\")
|
||||
print(" --num-epochs 1 \\")
|
||||
print(" --eval-split 0.2 \\")
|
||||
print(" --output-repo your-username/model-finetuned")
|
||||
print("\nFor full help: uv run unsloth_sft_example.py --help")
|
||||
print("=" * 70)
|
||||
sys.exit(0)
|
||||
|
||||
main()
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
name: huggingface-local-models
|
||||
description: "Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving."
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Hugging Face Local Models
|
||||
|
||||
Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with `llama-cli` or `llama-server`.
|
||||
|
||||
## Default Workflow
|
||||
|
||||
1. Search the Hub with `apps=llama.cpp`.
|
||||
2. Open `https://huggingface.co/<repo>?local-app=llama.cpp`.
|
||||
3. Prefer the exact HF local-app snippet and quant recommendation when it is visible.
|
||||
4. Confirm exact `.gguf` filenames with `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`.
|
||||
5. Launch with `llama-cli -hf <repo>:<QUANT>` or `llama-server -hf <repo>:<QUANT>`.
|
||||
6. Fall back to `--hf-repo` plus `--hf-file` when the repo uses custom file naming.
|
||||
7. Convert from Transformers weights only if the repo does not already expose GGUF files.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Install llama.cpp
|
||||
|
||||
```bash
|
||||
brew install llama.cpp
|
||||
winget install llama.cpp
|
||||
```
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp
|
||||
cd llama.cpp
|
||||
make
|
||||
```
|
||||
|
||||
### Authenticate for gated repos
|
||||
|
||||
```bash
|
||||
hf auth login
|
||||
```
|
||||
|
||||
### Search the Hub
|
||||
|
||||
```text
|
||||
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
```
|
||||
|
||||
### Run directly from the Hub
|
||||
|
||||
```bash
|
||||
llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
|
||||
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
|
||||
```
|
||||
|
||||
### Run an exact GGUF file
|
||||
|
||||
```bash
|
||||
llama-server \
|
||||
--hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
|
||||
--hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
|
||||
-c 4096
|
||||
```
|
||||
|
||||
### Convert only when no GGUF is available
|
||||
|
||||
```bash
|
||||
hf download <repo-without-gguf> --local-dir ./model-src
|
||||
python convert_hf_to_gguf.py ./model-src \
|
||||
--outfile model-f16.gguf \
|
||||
--outtype f16
|
||||
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
|
||||
```
|
||||
|
||||
### Smoke test a local server
|
||||
|
||||
With `llama-server` running (as started above), verify it responds:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer no-key" \
|
||||
-d '{
|
||||
"messages": [
|
||||
{"role": "user", "content": "Write a limerick about exception handling"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Quant Choice
|
||||
|
||||
- Prefer the exact quant that HF marks as compatible on the `?local-app=llama.cpp` page.
|
||||
- Keep repo-native labels such as `UD-Q4_K_M` instead of normalizing them.
|
||||
- Default to `Q4_K_M` unless the repo page or hardware profile suggests otherwise.
|
||||
- Prefer `Q5_K_M` or `Q6_K` for code or technical workloads when memory allows.
|
||||
- Consider `Q3_K_M`, `Q4_K_S`, or repo-specific `IQ` / `UD-*` variants for tighter RAM or VRAM budgets.
|
||||
- Treat `mmproj-*.gguf` files as projector weights, not the main checkpoint.
|
||||
|
||||
## Load References
|
||||
|
||||
- Read [hub-discovery.md](references/hub-discovery.md) for URL-first workflows, model search, tree API extraction, and command reconstruction.
|
||||
- Read [quantization.md](references/quantization.md) for format tables, model scaling, quality tradeoffs, and `imatrix`.
|
||||
- Read [hardware.md](references/hardware.md) for Metal, CUDA, ROCm, or CPU build and acceleration details.
|
||||
|
||||
## Resources
|
||||
|
||||
- llama.cpp: `https://github.com/ggml-org/llama.cpp`
|
||||
- Hugging Face GGUF + llama.cpp docs: `https://huggingface.co/docs/hub/gguf-llamacpp`
|
||||
- Hugging Face Local Apps docs: `https://huggingface.co/docs/hub/main/local-apps`
|
||||
- Hugging Face Local Agents docs: `https://huggingface.co/docs/hub/agents-local`
|
||||
- GGUF converter Space: `https://huggingface.co/spaces/ggml-org/gguf-my-repo`
|
||||
@@ -0,0 +1,38 @@
|
||||
## Hardware Acceleration
|
||||
|
||||
### Apple Silicon (Metal)
|
||||
|
||||
```bash
|
||||
make clean && make GGML_METAL=1
|
||||
llama-cli -m model.gguf -ngl 99 -p "Hello"
|
||||
```
|
||||
|
||||
### NVIDIA (CUDA)
|
||||
|
||||
```bash
|
||||
make clean && make GGML_CUDA=1
|
||||
llama-cli -m model.gguf -ngl 35 -p "Hello"
|
||||
|
||||
# Hybrid for large models
|
||||
llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20
|
||||
|
||||
# Multi-GPU split
|
||||
llama-cli -m large-model.gguf --tensor-split 0.5,0.5 -ngl 60
|
||||
```
|
||||
|
||||
### AMD (ROCm)
|
||||
|
||||
```bash
|
||||
make LLAMA_HIP=1
|
||||
llama-cli -m model.gguf -ngl 999
|
||||
```
|
||||
|
||||
### CPU
|
||||
|
||||
```bash
|
||||
# Match physical cores, not logical threads
|
||||
llama-cli -m model.gguf -t 8 -p "Hello"
|
||||
|
||||
# BLAS acceleration
|
||||
make LLAMA_OPENBLAS=1
|
||||
```
|
||||
@@ -0,0 +1,178 @@
|
||||
# Hugging Face URL Workflows for llama.cpp
|
||||
|
||||
Use URL-only workflows first. Do not require `hf` or API clients just to find GGUF files, choose a quant, or build a `llama-server` command.
|
||||
|
||||
## Contents
|
||||
|
||||
- Core URLs
|
||||
- Search for llama.cpp-compatible models
|
||||
- Use the local-app page for the recommended quant
|
||||
- Confirm exact files from the tree API
|
||||
- Build the command
|
||||
- Example: `unsloth/Qwen3.6-35B-A3B-GGUF`
|
||||
- Notes
|
||||
|
||||
## Core URLs
|
||||
|
||||
```text
|
||||
Search:
|
||||
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
||||
|
||||
Search with text:
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
|
||||
|
||||
Search with size bounds:
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
|
||||
Repo local-app view:
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
|
||||
Repo tree API:
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
|
||||
Repo file tree:
|
||||
https://huggingface.co/<repo>/tree/main
|
||||
```
|
||||
|
||||
## 1. Search for llama.cpp-compatible models
|
||||
|
||||
Start from the models page with `apps=llama.cpp`.
|
||||
|
||||
Use:
|
||||
|
||||
- `search=<term>` for model family names such as `Qwen`, `Gemma`, `Phi`, or `Mistral`
|
||||
- `num_parameters=min:0,max:24B` or similar if the user has hardware limits
|
||||
- `sort=trending` when the user wants popular repos right now
|
||||
|
||||
Do not start with random GGUF repos if the user has not chosen a model family yet. Search first, shortlist second.
|
||||
|
||||
Example: https://huggingface.co/models?search=Qwen&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
|
||||
## 2. Use the local-app page for the recommended quant
|
||||
|
||||
Open:
|
||||
|
||||
```text
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
```
|
||||
|
||||
Extract, in order:
|
||||
|
||||
1. The exact `Use this model` snippet, if it is visible as text
|
||||
2. The `Hardware compatibility` section from the fetched page text or HTML:
|
||||
- quant label
|
||||
- file size
|
||||
- bit-depth grouping
|
||||
3. Any extra launch flags shown in the snippet, such as `--jinja`
|
||||
|
||||
Treat the HF local-app snippet as the source of truth when it is visible.
|
||||
|
||||
Do this by reading the URL itself, not by assuming the UI rendered in a browser. If the fetched page source does not expose `Hardware compatibility`, say that the section was not text-visible and fall back to the tree API plus generic guidance from `quantization.md`.
|
||||
|
||||
## 3. Confirm exact files from the tree API
|
||||
|
||||
Open:
|
||||
|
||||
```text
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
```
|
||||
|
||||
Treat the JSON response as the source of truth for repo inventory.
|
||||
|
||||
Keep entries where:
|
||||
|
||||
- `type` is `file`
|
||||
- `path` ends with `.gguf`
|
||||
|
||||
Use these fields:
|
||||
|
||||
- `path` for the filename and subdirectory
|
||||
- `size` for the byte size
|
||||
- optionally `lfs.size` to confirm the LFS payload size
|
||||
|
||||
Separate files into:
|
||||
|
||||
- quantized single-file checkpoints, for example `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
|
||||
- projector weights, usually `mmproj-*.gguf`
|
||||
- BF16 shard files, usually under `BF16/`
|
||||
- everything else
|
||||
|
||||
Ignore unless the user asks:
|
||||
|
||||
- `README.md`
|
||||
- imatrix or calibration blobs
|
||||
|
||||
Use `https://huggingface.co/<repo>/tree/main` only as a human fallback if the API endpoint fails or the user wants the web view.
|
||||
|
||||
## 4. Build the command
|
||||
|
||||
Preferred order:
|
||||
|
||||
1. Copy the exact HF snippet from the local-app page
|
||||
2. If the page gives a clean quant label, use shorthand selection:
|
||||
|
||||
```bash
|
||||
llama-server -hf <repo>:<QUANT>
|
||||
```
|
||||
|
||||
3. If you need an exact file from the tree API, use the file-specific form:
|
||||
|
||||
```bash
|
||||
llama-server --hf-repo <repo> --hf-file <filename.gguf>
|
||||
```
|
||||
|
||||
4. For CLI usage instead of a server, use:
|
||||
|
||||
```bash
|
||||
llama-cli -hf <repo>:<QUANT>
|
||||
```
|
||||
|
||||
Use the exact-file form when the repo uses custom labels or nonstandard naming that could make `:<QUANT>` ambiguous.
|
||||
|
||||
## 5. Example: `unsloth/Qwen3.6-35B-A3B-GGUF`
|
||||
|
||||
Use these URLs:
|
||||
|
||||
```text
|
||||
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp
|
||||
https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true
|
||||
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main
|
||||
```
|
||||
|
||||
On the local-app page, the hardware compatibility section can expose entries such as:
|
||||
|
||||
- `UD-IQ4_XS` - 17.7 GB
|
||||
- `UD-Q4_K_S` - 20.9 GB
|
||||
- `UD-Q4_K_M` - 22.1 GB
|
||||
- `UD-Q5_K_M` - 26.5 GB
|
||||
- `UD-Q6_K` - 29.3 GB
|
||||
- `Q8_0` - 36.9 GB
|
||||
|
||||
On the tree API, you can confirm exact filenames such as:
|
||||
|
||||
- `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
|
||||
- `Qwen3.6-35B-A3B-UD-Q5_K_M.gguf`
|
||||
- `Qwen3.6-35B-A3B-UD-Q6_K.gguf`
|
||||
- `Qwen3.6-35B-A3B-Q8_0.gguf`
|
||||
- `mmproj-F16.gguf`
|
||||
|
||||
Good final output for this repo:
|
||||
|
||||
```text
|
||||
Repo: unsloth/Qwen3.6-35B-A3B-GGUF
|
||||
Recommended quant from HF: UD-Q4_K_M (22.1 GB)
|
||||
llama-server: llama-server --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
|
||||
Other GGUFs:
|
||||
- Qwen3.6-35B-A3B-UD-Q5_K_M.gguf - 26.5 GB
|
||||
- Qwen3.6-35B-A3B-UD-Q6_K.gguf - 29.3 GB
|
||||
- Qwen3.6-35B-A3B-Q8_0.gguf - 36.9 GB
|
||||
Projector:
|
||||
- mmproj-F16.gguf - 899 MB
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Repo-specific quant labels matter. Do not rewrite `UD-Q4_K_M` to `Q4_K_M` unless the page itself does.
|
||||
- `mmproj` files are projector weights for multimodal models, not the main language model checkpoint.
|
||||
- If the HF hardware compatibility panel is missing because the user has no hardware profile configured, or because the fetched page source did not expose it, still use the tree API plus generic quant guidance from `quantization.md`.
|
||||
- If the repo already has GGUFs, do not jump straight to conversion workflows.
|
||||
@@ -0,0 +1,256 @@
|
||||
# GGUF Quantization Guide
|
||||
|
||||
Complete guide to GGUF quantization formats and model conversion.
|
||||
|
||||
## Contents
|
||||
|
||||
- Hub-first quant selection
|
||||
- Quantization Formats
|
||||
- Converting Models
|
||||
- K-Quantization Methods
|
||||
- Quality Testing
|
||||
- Use Case Guide
|
||||
- Model Size Scaling
|
||||
- Finding Pre-Quantized Models
|
||||
- Importance Matrices (`imatrix`)
|
||||
- Troubleshooting
|
||||
|
||||
## Hub-first quant selection
|
||||
|
||||
Before using generic tables, open the model repo with:
|
||||
|
||||
```text
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
```
|
||||
|
||||
Prefer the exact quant labels and sizes shown in the `Hardware compatibility` section of the fetched `?local-app=llama.cpp` page text or HTML. Then confirm the matching filenames in:
|
||||
|
||||
```text
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
```
|
||||
|
||||
Use the Hub page first, and only fall back to the generic heuristics below when the repo page does not expose a clear recommendation.
|
||||
|
||||
## Quantization Formats
|
||||
|
||||
**GGUF** (GPT-Generated Unified Format) - Standard format for llama.cpp models.
|
||||
|
||||
### Format Comparison
|
||||
|
||||
| Format | Perplexity | Size (7B) | Tokens/sec | Notes |
|
||||
|--------|------------|-----------|------------|-------|
|
||||
| FP16 | 5.9565 (baseline) | 13.0 GB | 15 tok/s | Original quality |
|
||||
| Q8_0 | 5.9584 (+0.03%) | 7.0 GB | 25 tok/s | Nearly lossless |
|
||||
| **Q6_K** | 5.9642 (+0.13%) | 5.5 GB | 30 tok/s | Best quality/size |
|
||||
| **Q5_K_M** | 5.9796 (+0.39%) | 4.8 GB | 35 tok/s | Balanced |
|
||||
| **Q4_K_M** | 6.0565 (+1.68%) | 4.1 GB | 40 tok/s | **Recommended** |
|
||||
| Q4_K_S | 6.1125 (+2.62%) | 3.9 GB | 42 tok/s | Faster, lower quality |
|
||||
| Q3_K_M | 6.3184 (+6.07%) | 3.3 GB | 45 tok/s | Small models only |
|
||||
| Q2_K | 6.8673 (+15.3%) | 2.7 GB | 50 tok/s | Not recommended |
|
||||
|
||||
**Recommendation**: Use **Q4_K_M** for best balance of quality and speed.
|
||||
|
||||
## Converting Models
|
||||
|
||||
### Hugging Face to GGUF
|
||||
|
||||
```bash
|
||||
# 1. Download Hugging Face model
|
||||
hf download meta-llama/Llama-2-7b-chat-hf \
|
||||
--local-dir models/llama-2-7b-chat/
|
||||
|
||||
# 2. Convert to FP16 GGUF
|
||||
python convert_hf_to_gguf.py \
|
||||
models/llama-2-7b-chat/ \
|
||||
--outtype f16 \
|
||||
--outfile models/llama-2-7b-chat-f16.gguf
|
||||
|
||||
# 3. Quantize to Q4_K_M
|
||||
./llama-quantize \
|
||||
models/llama-2-7b-chat-f16.gguf \
|
||||
models/llama-2-7b-chat-Q4_K_M.gguf \
|
||||
Q4_K_M
|
||||
```
|
||||
|
||||
### Batch quantization
|
||||
|
||||
```bash
|
||||
# Quantize to multiple formats
|
||||
for quant in Q4_K_M Q5_K_M Q6_K Q8_0; do
|
||||
./llama-quantize \
|
||||
model-f16.gguf \
|
||||
model-${quant}.gguf \
|
||||
$quant
|
||||
done
|
||||
```
|
||||
|
||||
## K-Quantization Methods
|
||||
|
||||
**K-quants** use mixed precision for better quality:
|
||||
- Attention weights: Higher precision
|
||||
- Feed-forward weights: Lower precision
|
||||
|
||||
**Variants**:
|
||||
- `_S` (Small): Faster, lower quality
|
||||
- `_M` (Medium): Balanced (recommended)
|
||||
- `_L` (Large): Better quality, larger size
|
||||
|
||||
**Example**: `Q4_K_M`
|
||||
- `Q4`: 4-bit quantization
|
||||
- `K`: Mixed precision method
|
||||
- `M`: Medium quality
|
||||
|
||||
## Quality Testing
|
||||
|
||||
```bash
|
||||
# Calculate perplexity (quality metric)
|
||||
./llama-perplexity \
|
||||
-m model.gguf \
|
||||
-f wikitext-2-raw/wiki.test.raw \
|
||||
-c 512
|
||||
|
||||
# Lower perplexity = better quality
|
||||
# Baseline (FP16): ~5.96
|
||||
# Q4_K_M: ~6.06 (+1.7%)
|
||||
# Q2_K: ~6.87 (+15.3% - too much degradation)
|
||||
```
|
||||
|
||||
## Use Case Guide
|
||||
|
||||
### General purpose (chatbots, assistants)
|
||||
```
|
||||
Q4_K_M - Best balance
|
||||
Q5_K_M - If you have extra RAM
|
||||
```
|
||||
|
||||
### Code generation
|
||||
```
|
||||
Q5_K_M or Q6_K - Higher precision helps with code
|
||||
```
|
||||
|
||||
### Creative writing
|
||||
```
|
||||
Q4_K_M - Sufficient quality
|
||||
Q3_K_M - Acceptable for draft generation
|
||||
```
|
||||
|
||||
### Technical/medical
|
||||
```
|
||||
Q6_K or Q8_0 - Maximum accuracy
|
||||
```
|
||||
|
||||
### Edge devices (Raspberry Pi)
|
||||
```
|
||||
Q2_K or Q3_K_S - Fit in limited RAM
|
||||
```
|
||||
|
||||
## Model Size Scaling
|
||||
|
||||
### 7B parameter models
|
||||
|
||||
| Format | Size | RAM needed |
|
||||
|--------|------|------------|
|
||||
| Q2_K | 2.7 GB | 5 GB |
|
||||
| Q3_K_M | 3.3 GB | 6 GB |
|
||||
| Q4_K_M | 4.1 GB | 7 GB |
|
||||
| Q5_K_M | 4.8 GB | 8 GB |
|
||||
| Q6_K | 5.5 GB | 9 GB |
|
||||
| Q8_0 | 7.0 GB | 11 GB |
|
||||
|
||||
### 13B parameter models
|
||||
|
||||
| Format | Size | RAM needed |
|
||||
|--------|------|------------|
|
||||
| Q2_K | 5.1 GB | 8 GB |
|
||||
| Q3_K_M | 6.2 GB | 10 GB |
|
||||
| Q4_K_M | 7.9 GB | 12 GB |
|
||||
| Q5_K_M | 9.2 GB | 14 GB |
|
||||
| Q6_K | 10.7 GB | 16 GB |
|
||||
|
||||
### 70B parameter models
|
||||
|
||||
| Format | Size | RAM needed |
|
||||
|--------|------|------------|
|
||||
| Q2_K | 26 GB | 32 GB |
|
||||
| Q3_K_M | 32 GB | 40 GB |
|
||||
| Q4_K_M | 41 GB | 48 GB |
|
||||
| Q4_K_S | 39 GB | 46 GB |
|
||||
| Q5_K_M | 48 GB | 56 GB |
|
||||
|
||||
**Recommendation for 70B**: Use Q3_K_M or Q4_K_S to fit in consumer hardware.
|
||||
|
||||
## Finding Pre-Quantized Models
|
||||
|
||||
Use the Hub search with the llama.cpp app filter:
|
||||
|
||||
```text
|
||||
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
```
|
||||
|
||||
For a specific repo, open:
|
||||
|
||||
```text
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
```
|
||||
|
||||
Then launch directly from the Hub without extra Hub tooling:
|
||||
|
||||
```bash
|
||||
llama-cli -hf <repo>:Q4_K_M
|
||||
llama-server -hf <repo>:Q4_K_M
|
||||
```
|
||||
|
||||
If you need the exact file name from the tree API:
|
||||
|
||||
```bash
|
||||
llama-server --hf-repo <repo> --hf-file <filename.gguf>
|
||||
```
|
||||
|
||||
## Importance Matrices (imatrix)
|
||||
|
||||
**What**: Calibration data to improve quantization quality.
|
||||
|
||||
**Benefits**:
|
||||
- 10-20% perplexity improvement with Q4
|
||||
- Essential for Q3 and below
|
||||
|
||||
**Usage**:
|
||||
```bash
|
||||
# 1. Generate importance matrix
|
||||
./llama-imatrix \
|
||||
-m model-f16.gguf \
|
||||
-f calibration-data.txt \
|
||||
-o model.imatrix
|
||||
|
||||
# 2. Quantize with imatrix
|
||||
./llama-quantize \
|
||||
--imatrix model.imatrix \
|
||||
model-f16.gguf \
|
||||
model-Q4_K_M.gguf \
|
||||
Q4_K_M
|
||||
```
|
||||
|
||||
**Calibration data**:
|
||||
- Use domain-specific text (e.g., code for code models)
|
||||
- ~100MB of representative text
|
||||
- Higher quality data = better quantization
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Model outputs gibberish**:
|
||||
- Quantization too aggressive (Q2_K)
|
||||
- Try Q4_K_M or Q5_K_M
|
||||
- Verify model converted correctly
|
||||
|
||||
**Out of memory**:
|
||||
- Use lower quantization (Q4_K_S instead of Q5_K_M)
|
||||
- Offload fewer layers to GPU (`-ngl`)
|
||||
- Use smaller context (`-c 2048`)
|
||||
|
||||
**Slow inference**:
|
||||
- Higher quantization uses more compute
|
||||
- Q8_0 much slower than Q4_K_M
|
||||
- Consider speed vs quality trade-off
|
||||
@@ -0,0 +1,395 @@
|
||||
---
|
||||
name: huggingface-lora-space-builder
|
||||
description: Build and publish a Gradio demo on Hugging Face Spaces for a user-provided LoRA. Use when someone asks to create, generate, ship, or publish a Space, demo, Gradio app, or playground for a LoRA — including LoRAs for Qwen-Image, Qwen-Image-Edit, LTX-Video, Wan, FLUX, SDXL, or other diffusion base models. Also triggers when someone describes a LoRA they trained or hosts on the Hub and wants to share it. Covers picking the right base pipeline and `diffusers` inference recipe, designing a UI tailored to the LoRA's task and inputs (Union/multi-task control, edit, video, image, etc.), respecting model-card recommendations (trigger words, steps, guidance, LoRA scale, example inputs), and shipping to ZeroGPU hardware as a private Space by default.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Gradio LoRA Space Builder
|
||||
|
||||
Build and publish a Gradio demo on Hugging Face Spaces that runs inference with a user-provided LoRA. Use whenever someone asks to create, generate, ship, or publish "a Space", "a demo", "a Gradio app", or "a playground" for a LoRA — whether the base model is Qwen-Image, Qwen-Image-Edit, LTX, or another diffusion model. Also use when someone describes a LoRA they trained or hosts on the Hub and wants to share it. The default target is ZeroGPU hardware and the default inference library is `diffusers` when the base model supports it.
|
||||
|
||||
The output is a real, published Space (private by default) that the user can try in the browser, not a local script.
|
||||
|
||||
## What "good" looks like for these demos
|
||||
|
||||
The demo should feel handcrafted for this specific LoRA, not a generic template with the LoRA bolted on. Two LoRAs that share a task can still need different demos: a pose-control video LoRA and an outpainting video LoRA both take video in and produce video out, but the inputs the user provides, the preprocessing, and the controls are completely different. Recognizing that is the central job here.
|
||||
|
||||
Concretely, a good demo:
|
||||
|
||||
- Loads fast and runs fast — minimal model loading, sensible step count, no wasted computation per call.
|
||||
- Has a UI with exactly the controls this LoRA needs and nothing else. Excess sliders are a cost, not a feature.
|
||||
- Shows the user what's happening — progress, intermediate outputs where useful, the seed used, a clear error when input is missing.
|
||||
- Honors the LoRA's own recommendations from its model card: trigger words, recommended step count, recommended guidance scale, recommended LoRA scale, example inputs.
|
||||
- Is creative where creativity helps — interactive canvases, before/after sliders, side-by-side previews of intermediate processing — and plain where plainness is right.
|
||||
|
||||
## Workflow
|
||||
|
||||
Work through these phases in order. Information gathered in one phase decides the next.
|
||||
|
||||
1. Gather the LoRA info needed to pick a pipeline and design a UI.
|
||||
2. Pick the base pipeline and inference recipe.
|
||||
3. Design the UI for this specific LoRA's task and inputs.
|
||||
4. Write `app.py`, `requirements.txt`, and `README.md` together; show all three to the user for one batched approval.
|
||||
5. Publish the Space (private).
|
||||
|
||||
Don't drip-feed questions across multiple turns. Batch them.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Gather LoRA info
|
||||
|
||||
Required: a LoRA repo on the Hub (e.g. `username/my-lora`).
|
||||
|
||||
**First, try to read the repo without a token.** If it succeeds, the repo is public — proceed. If it fails with 401/403, the repo is private/gated and you need an authenticated session to read it. **Don't immediately ask for a token.** Check first whether the user is already authenticated.
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi, get_token
|
||||
|
||||
cached_token = get_token() # picks up HF_TOKEN env var or cached CLI login
|
||||
if cached_token:
|
||||
try:
|
||||
info = HfApi().whoami(token=cached_token)
|
||||
username = info["name"]
|
||||
# info also has fine-grained token scope info if applicable
|
||||
except Exception:
|
||||
cached_token = None # token exists but is invalid/expired
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
- If a valid cached token exists *and* it can read the repo, use it. No prompt needed.
|
||||
- If no cached token, or the cached token can't read this private repo, ask the user for a token — once, with the explanation below.
|
||||
|
||||
When asking for a token (and only when you actually need to ask):
|
||||
|
||||
> I need a Hugging Face access token with **write** scope (to read the LoRA if it's private/gated, and to publish the Space). Create one at https://huggingface.co/settings/tokens. Paste it here.
|
||||
|
||||
The same token will be reused for publishing in the final phase, so this is a one-time ask.
|
||||
|
||||
**Then read what's in the repo:**
|
||||
|
||||
- List the repo files (`huggingface_hub.HfApi().list_repo_files(repo_id)`). Look for `.safetensors`, `README.md`, example images/videos, multiple checkpoints.
|
||||
- Fetch the model card (`huggingface_hub.ModelCard.load(repo_id)`). The `data` dict has structured fields; the `text` has the README body.
|
||||
- If multiple `.safetensors` files exist, pick the right one — see "Picking the LoRA weights file" in `references/zerogpu-and-publishing.md`. Briefly: README-recommended file wins, then `pytorch_lora_weights.safetensors`, then latest training checkpoint, otherwise ask.
|
||||
|
||||
**From the model card, try to determine:**
|
||||
|
||||
- **Base model** — the `base_model` field, or text mentions in the README. Usually present. Use it to pick the pipeline reference file (see Phase 2).
|
||||
- **Task** — `pipeline_tag` if set, otherwise inferred from the base model and README text. The five tasks this skill handles: `text-to-image`, `image-to-image`, `text-to-video`, `image-to-video`, `video-to-video`.
|
||||
- **Trigger words** — often called "trigger word", "instance prompt", "activation word"; sometimes embedded in example prompts.
|
||||
- **Recommended inference recipe** — step count, guidance scale, true CFG scale, LoRA scale, resolution. Many LoRA cards include a Python snippet; trust its *parameters* (steps, guidance, CFG, LoRA scale, dtype). For *loading mechanics*, see `adapting-to-the-lora.md` — prefer `pipe.load_lora_weights(...)` over whatever loading approach the snippet uses.
|
||||
- **Example prompts and example media** — use these as Gradio examples in the UI.
|
||||
- **Sub-task / specific use case** — for image edits and video LoRAs, "what does this LoRA actually do" matters as much as the task category. A relighting LoRA, a face-swap LoRA, and a style LoRA all might be image-to-image, but the UI for each is different.
|
||||
|
||||
**When something can't be inferred, ask the user — once, in a single batched message.** Format the question to make answering trivial. For task category, list the five options as a numbered choice. For sub-task, give a one-line description ("what does this LoRA do? e.g. 'relight portraits', 'apply manga style', 'extend videos to wider aspect ratios'"). Don't ask if you can already infer it confidently from the base model or README.
|
||||
|
||||
If the model card has nothing helpful at all — no base model, no task, no example — surface that clearly: "The model card has no usable info. I'll need you to tell me: (1) base model, (2) what this LoRA does, (3) recommended step count and guidance scale if you know them."
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Pick the base pipeline
|
||||
|
||||
Two things to decide here: which reference file to load, and which pipeline class to use. They're not the same question — a base-model family file (e.g. `qwen-image.md`) covers multiple variants, and variants in the same family don't always share a pipeline class. Get this wrong and the Space loads but produces wrong output, or fails at startup.
|
||||
|
||||
**Step 1 — Load the reference file for this base model family.**
|
||||
|
||||
- `references/base-models/qwen-image.md` — covers Qwen-Image and Qwen-Image-Edit family (text-to-image and image-to-image).
|
||||
- `references/base-models/ltx.md` — covers LTX family (text-to-video, image-to-video, video-to-video, including IC-LoRAs).
|
||||
- `references/base-models/krea-2.md` — covers Krea 2 (K2), text-to-image (train on RAW, run inference/LoRAs on the Turbo distilled checkpoint).
|
||||
|
||||
If the base model isn't in one of these files, this skill doesn't have first-class support yet. Tell the user, and ask whether they want to proceed by analogy (use the closest model's recipe and adjust) or stop. Don't guess silently.
|
||||
|
||||
**Step 2 — Verify the pipeline class against the base model's own card. This step is mandatory, not optional.**
|
||||
|
||||
A new base model variant might use the same pipeline class with a different repo path, or a new pipeline class entirely. Don't trust the reference file's table alone — it's best-effort and can lag a recent release. Verify before committing:
|
||||
|
||||
```python
|
||||
from huggingface_hub import ModelCard
|
||||
base_card = ModelCard.load(base_model_id)
|
||||
# Read base_card.text — find the diffusers inference snippet, note the pipeline class it imports.
|
||||
```
|
||||
|
||||
The class imported in the base model card's diffusers snippet is the source of truth. Real examples where this matters:
|
||||
|
||||
- `Qwen-Image-Edit` uses `QwenImageEditPipeline`. `Qwen-Image-Edit-2509` and `Qwen-Image-Edit-2511` use `QwenImageEditPlusPipeline` — different class, different default parameters, takes a list of images instead of one. A LoRA targeting 2511 loaded onto `QwenImageEditPipeline` produces broken output.
|
||||
- LTX-Video uses `LTXPipeline`/`LTXImageToVideoPipeline`/`LTXConditionPipeline`. LTX-2 uses `LTX2Pipeline` from a different module path. LTX-2.3 sometimes needs a native pipeline outside diffusers.
|
||||
|
||||
If the base model card has no diffusers snippet at all, fall back to the reference file's table — and tell the user you're falling back, in case they know something the table doesn't.
|
||||
|
||||
The cost of this verification is one Hub fetch and a few seconds of reading. The cost of skipping it is the failure mode the previous bullet describes — a "working" Space that's quietly using the wrong class.
|
||||
|
||||
**Step 3 — Diffusers vs native pipeline.** Default to `diffusers` when the base model has a diffusers pipeline class. That's the case for Qwen-Image and Qwen-Image-Edit and most of LTX. Some LTX variants (notably LTX-2.3 with certain IC-LoRAs) need a native pipeline; the LTX reference says when. Diffusers gives standard `load_lora_weights` / `set_adapters` semantics; the native path needs LoRA-specific glue.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Design the UI for this LoRA
|
||||
|
||||
Don't reach for a template. Reason from the LoRA's task and inputs to a UI.
|
||||
|
||||
Read `references/tasks.md` for the per-task baseline UI patterns (what the standard inputs/outputs look like for T2I, I2I, T2V, I2V, V2V).
|
||||
|
||||
Then read `references/adapting-to-the-lora.md`, which is about *thinking through what this specific LoRA needs* — beyond the task category. That file is the most important one in this skill. The same task can need very different UIs: a pose-control LTX LoRA needs a video input and a pose-extraction preview; an outpaint LTX LoRA needs an aspect-ratio picker and a black-margin preview; a relighting Flux LoRA needs an image and a brush canvas for indicating where to add light. None of those reduce to "the V2V template" or "the I2I template".
|
||||
|
||||
**Self-check before writing the UI.** Write one sentence describing what a user does with this Space in 10 seconds. If that sentence doesn't distinguish this LoRA from any other LoRA of the same task, the UI isn't shaped enough yet.
|
||||
|
||||
Examples that pass the self-check:
|
||||
|
||||
- "Upload a video, pick a target aspect ratio, click Generate; the model fills the empty margins."
|
||||
- "Draw colored brush strokes where you want light, pick an illumination style, click Generate; the model relights the photo."
|
||||
- "Upload a video of someone moving and an image of a different character; the model produces a video of the character doing the motion."
|
||||
|
||||
Examples that fail:
|
||||
|
||||
- "Type a prompt and click generate." (Generic T2I — say more.)
|
||||
- "Upload an image and an instruction." (Generic edit — what kind of edit?)
|
||||
|
||||
**Gradio component freshness.** Gradio's component set evolves. Before defaulting to plain components, consider whether something newer fits better — for example `gr.ImageSlider` for before/after on edit LoRAs, `gr.BrowserState` for persistent prefs, `@gr.render` for UIs that change based on input. If you're unsure whether a component exists or what its signature is, web-fetch the current Gradio docs at https://www.gradio.app/docs rather than guessing.
|
||||
|
||||
**When stock and Hub custom components aren't enough — creative mode.** If the LoRA's natural input is a shape no Gradio component (built-in or on the Hub) expresses well — point sets, strokes, trajectories, multi-region annotations with metadata, 3D rotation gizmos, timeline scrubbers, anything where the user manipulates a thing on top of media — drop down to custom HTML/JS via `gr.HTML`. See `references/creative-mode.md` for the Gradio primitives (`gr.HTML`, `head=` injection, `elem_id` addressing, the two JS↔Python state-sync approaches), the discipline around defining a JSON wire format, and the pitfalls. Don't reach for creative mode just because it would be cool — reach for it when the LoRA's input shape demands it. And don't skip the Hub custom components rung above (e.g. `gradio_image_annotation`) before going fully bespoke.
|
||||
|
||||
**`gr.Examples` for media-input Spaces.** When no fitting example media is available from the model's own repo, pull from the shared input pools — split by modality so the HF dataset viewer can render proper thumbnails: images at [`linoyts/repo-to-space-example-inputs`](https://huggingface.co/datasets/linoyts/repo-to-space-example-inputs), videos at [`linoyts/repo-to-space-example-videos`](https://huggingface.co/datasets/linoyts/repo-to-space-example-videos). Both are CC0 with `categories` + natural-language `caption` metadata and the same filter/rank recipe in each dataset README. Pick 2–3 that fit the task, preprocess to the shapes the model expects, and bake the copies into the Space. Set `cache_examples=True, cache_mode="lazy"` so the first click caches without running examples at build time (see `references/zerogpu-and-publishing.md`).
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Write the Space files
|
||||
|
||||
Before writing, tell the user concretely what's about to happen — name the actual files. Not "I'll write the three files" but something like:
|
||||
|
||||
> "Now I'll write the three files needed to publish a Space: **`app.py`** (the Gradio demo and inference code), **`requirements.txt`** (Python dependencies), and **`README.md`** (Space configuration including ZeroGPU hardware setting). Then I'll show all three for your review before publishing."
|
||||
|
||||
This anchors the user in what's being produced. Don't say "three files" without naming them — it's vague and signals lack of commitment to the deliverable.
|
||||
|
||||
The three files are tightly coupled: `requirements.txt` is determined by what `app.py` imports, and the `README.md` YAML frontmatter sets the SDK version, hardware, and Space title that have to match. Write them together, then show all three to the user for approval in **one batched message** before publishing.
|
||||
|
||||
Read `references/zerogpu-and-publishing.md` for the ZeroGPU rules. The non-obvious ones:
|
||||
|
||||
- Models go on `cuda` at module level (not lazy-loaded inside the GPU function). ZeroGPU has a CUDA emulation that makes this work pre-allocation, and module-level placement is significantly faster than deferred placement.
|
||||
- The function that runs inference is decorated with `@spaces.GPU(duration=...)`. Pick a duration appropriate for the task — short for image generation, longer for video.
|
||||
- Don't use `torch.compile` — it's incompatible with ZeroGPU's process model.
|
||||
|
||||
### `app.py`
|
||||
|
||||
Compose from the pieces decided in Phases 1–3. Don't paste from a template. Each section should be there because it's needed:
|
||||
|
||||
- Imports — `gradio as gr`, `torch`, `spaces`, the pipeline class, anything the preprocessing needs.
|
||||
- Constants — `LORA_REPO`, `BASE_MODEL`, recommended step count, guidance, LoRA scale, trigger word.
|
||||
- Module-level model load — pipeline `from_pretrained`, `.to("cuda")`, `load_lora_weights`. If the LoRA repo is private, pass `token=os.environ["HF_TOKEN"]`.
|
||||
- Preprocessing functions (if any) — pose extraction, padding, mask building, etc. CPU code can run at module level; GPU code needs to be inside a `@spaces.GPU` function.
|
||||
- The inference function — decorated with `@spaces.GPU(duration=...)`. Validates inputs, applies trigger word, builds the pipeline kwargs, returns outputs.
|
||||
- The Gradio Blocks — the UI from Phase 3, wired to the inference function.
|
||||
|
||||
Common things to get right:
|
||||
|
||||
- Return the actually-used seed alongside the result so the user can reproduce.
|
||||
- `gr.Progress(track_tqdm=True)` on the inference function surfaces diffusers' internal progress bar.
|
||||
- Validate inputs — raise `gr.Error("Please upload an image first.")` when a required input is missing, rather than letting the pipeline fail with a cryptic error.
|
||||
- On `gr.Examples`, use `cache_examples=True, cache_mode="lazy"` — plain `cache_examples=True` runs examples at build time and fails on ZeroGPU; lazy mode defers caching to the first user click.
|
||||
- When `gr.Examples` has `fn=`, clicking a row calls `fn` with **only** the `inputs=` values, positionally — so every `fn` parameter not in `inputs` needs a default, or the click raises `TypeError: missing N required positional arguments`. The run event wires all components, so this passes manual and smoke tests and only breaks on the example click. Fix: default the extra params (keep `inputs=[prompt]`), or list every input with full example rows, or drop `fn`/`outputs` so a click just fills the fields.
|
||||
|
||||
### `requirements.txt`
|
||||
|
||||
Don't ship a fixed minimal list and hope for the best. The "minimal" list works for plain T2I LoRAs and breaks the moment the base model has a vision-language text encoder, video output, or any non-trivial preprocessing. **Derive `requirements.txt` from what the Space actually needs**, in this order:
|
||||
|
||||
1. **Every top-level non-stdlib import in `app.py`.** If `app.py` does `import cv2`, `requirements.txt` has `opencv-python`. If it does `from controlnet_aux import OpenposeDetector`, `requirements.txt` has `controlnet-aux`. Walk the imports mechanically. (Note the exclusions in the next paragraph — some imports are runtime built-ins and don't need to be listed.)
|
||||
2. **What the base-model reference's "Required dependencies" subsection says.** Each base-model file lists the non-obvious extras the pipeline pulls in — `torchvision` for Qwen-Image (Qwen 2.5-VL text encoder), `imageio[ffmpeg]` for LTX (video export), etc. Include all of them. These are the deps that aren't picked up from imports because the pipeline's components import them transitively at load time.
|
||||
3. **What the LoRA's own model card explicitly mentions installing.** If the LoRA README has its own `pip install` block, lift the deps from there.
|
||||
4. **The diffusers/ML stack:** `diffusers`, `transformers`, `accelerate`, `peft`, `safetensors`. Default to plain (unpinned). Switch `diffusers` to `git+https://github.com/huggingface/diffusers` if the base-model reference says the model needs it (recent releases often do — Qwen-Image-Edit-2511 is a current example).
|
||||
|
||||
**What *not* to list in `requirements.txt`:**
|
||||
|
||||
- **`gradio`** — controlled by the `sdk_version:` field in `README.md`'s YAML frontmatter, not by `requirements.txt`. Listing it in requirements is at best ignored, at worst causes a version conflict with the SDK. Set the version in the README only.
|
||||
- **`torch`** — provided by the Space runtime. Only add if you need a specific version pinned (rare, and usually a sign something else is wrong).
|
||||
- **`spaces`** — provided by the Space runtime. Only add if you need a specific version pinned.
|
||||
- **`huggingface_hub`** — provided by the Space runtime. Only add if you need a specific version pinned.
|
||||
|
||||
These four come pre-installed in the ZeroGPU container. Listing them anyway is the kind of "include rather than skip" instinct that's right for non-baseline deps but wrong for baseline ones, because pinning conflicts with the runtime's managed versions.
|
||||
|
||||
**Bias for everything else: include rather than skip when uncertain.** A package the Space doesn't actually use causes a slightly slower build. A missing required package causes a startup-time crash that's much harder for the user to diagnose. These costs aren't symmetric — the test failure that prompted this rule was exactly the second kind.
|
||||
|
||||
**But two specific deps are *not* safe to add reflexively** because they routinely cause more problems than they solve on ZeroGPU:
|
||||
|
||||
- `xformers` — pinned to specific torch versions, frequent source of conflicts. The ZeroGPU runtime ships torch 2.8+, so any pinned `xformers` version must support that. Additional gotcha on Blackwell: xformers' FA3 dispatch mis-gates the hardware (FA3 kernels are Hopper-only at `sm_90a`, but the dispatcher gates on `device_capability >= (9, 0)`, which also matches Blackwell) and crashes at kernel launch with `CUDA invalid argument`. If a Space using xformers attention hits this, disable FA3 dispatch at module load:
|
||||
|
||||
```python
|
||||
try:
|
||||
from xformers.ops.fmha import _set_use_fa3
|
||||
_set_use_fa3(False)
|
||||
except Exception:
|
||||
pass
|
||||
```
|
||||
|
||||
Only include `xformers` if `app.py` actually uses it.
|
||||
- `flash-attn` — needs a build step, often fails to install. Same torch 2.8+ alignment caveat as `xformers`. Only include if `app.py` actually uses it.
|
||||
|
||||
**Pin other versions only when you have a reason** (e.g. a known incompatibility, or matching a recipe from the model card).
|
||||
|
||||
### `README.md`
|
||||
|
||||
Spaces are configured by the YAML frontmatter at the top of `README.md`. This frontmatter is what selects ZeroGPU.
|
||||
|
||||
```
|
||||
---
|
||||
title: <human-readable title>
|
||||
emoji: 🎨
|
||||
colorFrom: pink
|
||||
colorTo: purple
|
||||
sdk: gradio
|
||||
sdk_version: <current Gradio version>
|
||||
app_file: app.py
|
||||
pinned: false
|
||||
hardware: zero-a10g
|
||||
short_description: <one short line for the Space tile, ~60 chars max>
|
||||
models:
|
||||
- <base model repo>
|
||||
- <lora repo>
|
||||
---
|
||||
|
||||
# <title>
|
||||
|
||||
A short description with links to the LoRA and base model.
|
||||
```
|
||||
|
||||
Key fields:
|
||||
|
||||
- `sdk: gradio` — required for ZeroGPU.
|
||||
- `sdk_version` — match the Gradio version you wrote against. Look up the current version (`pip index versions gradio`, or check https://www.gradio.app) rather than guessing.
|
||||
- `hardware: zero-a10g` — the legacy string for ZeroGPU. The actual hardware is NVIDIA RTX Pro 6000 Blackwell, but the identifier is `zero-a10g`. ZeroGPU is available to PRO, Team, and Enterprise accounts; if the user isn't subscribed, the Space will fall back to CPU. Mention this if you suspect they aren't on PRO.
|
||||
- `models:` — list base and LoRA repos. This enables Hub caching and discovery.
|
||||
- `short_description` — appears on the Space tile. **Keep it short (~60 characters or less).** The Hub's YAML validator rejects long values with a 400 from `https://huggingface.co/api/validate-yaml`, which surfaces as an `HfHubHTTPError` during `create_repo` or `upload_file`. The exact server-side limit isn't documented and may change, so target the visible-tile-length range rather than pushing right up to a cap. If you do hit the 400, the fix is almost always to shorten this field. One sentence describing what the Space does is plenty — the README body below the YAML is where you put longer prose.
|
||||
|
||||
### Single batched approval — order of operations matters
|
||||
|
||||
The discipline here is **write all three files first, then show them all together in one message**. Not "write app.py → talk about it → write requirements → talk about it → write README → talk about it." That rhythm produces three approval moments even if you don't explicitly ask for approval, because the user is being asked to react after each file.
|
||||
|
||||
Concretely:
|
||||
|
||||
1. **Write `app.py`, `requirements.txt`, and `README.md` in succession with no intervening prose.** No commentary between files. No "Now I'll write the next one." No description of what each file does as you produce it. Just the three files, back to back.
|
||||
2. **Then, in a single message, ask for approval covering all three at once.** Something like: "Here's the Space — `app.py` (N lines), `requirements.txt`, and `README.md`. Review and confirm to publish, or tell me what to change."
|
||||
3. The user responds once, covering whatever they want changed across any of the three files.
|
||||
|
||||
What to avoid:
|
||||
|
||||
- Walking through `app.py`'s structure or design choices after writing it but before writing the others. Save commentary for either the pre-writing announcement (Phase 4 opening) or the single approval message after all three exist.
|
||||
- Asking "ready for the next one?" or "want me to continue with requirements?" — those are implicit per-file approvals.
|
||||
- Showing one file inline and offering to "show the next when you're ready" — same trap.
|
||||
- Treating any of the three files as optional or as a follow-up. They are produced together as one deliverable.
|
||||
|
||||
If the user interrupts after seeing the first file with feedback or a question, that's fine — engage with it — but the rule still applies: the next time you produce code, produce all remaining files together, not one at a time.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Publish the Space
|
||||
|
||||
Use the authenticated session from Phase 1. Default to **private**, so the user can vet the Space before flipping it public. Confirm the target username with the user before creating: "I'll publish to `{username}/{space_name}` — confirm?"
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi, SpaceHardware
|
||||
|
||||
api = HfApi(token=hf_token)
|
||||
username = api.whoami()["name"]
|
||||
repo_id = f"{username}/{space_name}"
|
||||
|
||||
api.create_repo(
|
||||
repo_id=repo_id,
|
||||
repo_type="space",
|
||||
space_sdk="gradio",
|
||||
space_hardware=SpaceHardware.ZERO_A10G,
|
||||
private=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
|
||||
# Upload files
|
||||
for path in ["app.py", "requirements.txt", "README.md"]:
|
||||
api.upload_file(path_or_fileobj=path, path_in_repo=path,
|
||||
repo_id=repo_id, repo_type="space")
|
||||
```
|
||||
|
||||
If the LoRA repo itself is private/gated, the Space needs the token at runtime to download the LoRA. Set it as a Space secret:
|
||||
|
||||
```python
|
||||
api.add_space_secret(repo_id=repo_id, key="HF_TOKEN", value=HF_TOKEN)
|
||||
```
|
||||
|
||||
…and in `app.py`, load the LoRA with `token=os.environ["HF_TOKEN"]`.
|
||||
|
||||
**After upload**, run the smoke-test below before sharing — the build runs asynchronously and silent failures (wrong `weight_name`, missing dep, wrong pipeline class) only surface at first inference. **Once the smoke-test passes**, share the Space URL (`https://huggingface.co/spaces/{repo_id}`) and tell the user the Space is private — they'll need to be logged in to view it. Note that the build takes a few minutes; the logs are at `https://huggingface.co/spaces/{repo_id}/logs/container` if anything fails.
|
||||
|
||||
**Publish-time failures (before the build starts):**
|
||||
|
||||
- **`HfHubHTTPError: 400 Bad Request` from `https://huggingface.co/api/validate-yaml`** during `create_repo` or `upload_file`. The README YAML failed server-side validation. By far the most common cause is a `short_description` that's too long; sometimes a stray field or malformed value. Fix: shorten `short_description` to ~60 characters and retry. If shortening doesn't fix it, look for typos in field names or invalid values (e.g. unsupported colors in `colorFrom`/`colorTo`, an invalid `hardware` string).
|
||||
- **403 on `create_repo`** with `space_hardware="zero-a10g"`: user isn't on PRO/Team/Enterprise, so they can't request ZeroGPU at creation time. Fix: retry `create_repo` without `space_hardware`, leave `hardware: zero-a10g` in the README YAML — the Space gets created on CPU. The user can then either upgrade to PRO (auto-promotes to ZeroGPU) or apply for a [community GPU grant](https://huggingface.co/docs/hub/spaces-gpus#community-gpu-grants) (request via the Space's hardware settings).
|
||||
- **401/403 on `upload_file`**: token doesn't have write scope. Fix: ask the user for a write-scoped token.
|
||||
|
||||
**Common build failures (after the build starts):**
|
||||
|
||||
- LoRA `weight_name` mismatch in `load_lora_weights` → check the actual filename via `list_repo_files`.
|
||||
- Base model is gated and the token wasn't set as a Space secret.
|
||||
- ZeroGPU not allocated (user not on PRO) → Space falls back to CPU and is unusably slow.
|
||||
- Diffusers version doesn't recognize the pipeline class → pin to git diffusers in `requirements.txt`.
|
||||
- Missing dependency at module load → see `requirements.txt` derivation rules above; the most common case is a transitive dep like `torchvision` for Qwen-Image's text encoder.
|
||||
|
||||
If a build fails, offer to read the logs and propose a fix.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Smoke-test the Space
|
||||
|
||||
Before declaring the Space done and handing the URL to the user, exercise it once end-to-end. Several failure modes (wrong `weight_name`, wrong pipeline class, missing transitive dep, gated-base-model token issue) build cleanly and only surface at first inference. The `gradio` Python package ships a CLI that does exactly this — `gradio info` returns the endpoint signature, `gradio predict` runs an actual inference. Both ship with the `gradio` pip dependency the Space already needs, so they're available in any environment where this skill ran.
|
||||
|
||||
**Step 1 — Wait for the build.** `create_repo` returns immediately, but the container image is still building. Poll `HfApi().get_space_runtime(repo_id).stage` until it reaches `RUNNING`:
|
||||
|
||||
```python
|
||||
import time
|
||||
from huggingface_hub import HfApi
|
||||
api = HfApi(token=hf_token)
|
||||
while True:
|
||||
stage = api.get_space_runtime(repo_id).stage
|
||||
if stage == "RUNNING": break
|
||||
if stage in {"BUILD_ERROR", "RUNTIME_ERROR", "CONFIG_ERROR"}:
|
||||
raise RuntimeError(f"Build failed: {stage}. Logs: https://huggingface.co/spaces/{repo_id}/logs/container")
|
||||
time.sleep(15)
|
||||
```
|
||||
|
||||
If the build fails, fetch the container logs (`https://huggingface.co/spaces/{repo_id}/logs/container`), read the traceback, and propose a fix. Don't run `gradio info` against a Space that isn't running — it'll hang or 503.
|
||||
|
||||
**Step 2 — Verify the endpoint signature.** `gradio info {repo_id} --token {hf_token}` returns the exposed endpoints and their parameter types. Read the output and confirm: (a) the endpoint exists (default is `/predict`, but Blocks Spaces often have a custom name from the Python function name), (b) the parameters in order match what `app.py` declares, (c) file-typed params show `"type": "filepath"` as expected. If any of this is off, the user-facing UI may still appear correct but API calls will fail — fix and re-upload.
|
||||
|
||||
**Step 3 — Run one real inference.** Pick the lightest viable input — the simplest example from the LoRA card, or one of the `gr.Examples` entries. Pass `--token` for private Spaces. For file inputs, the payload uses `{"path": "...", "meta": {"_type": "gradio.FileData"}}`.
|
||||
|
||||
```bash
|
||||
# Text-to-image:
|
||||
gradio predict {repo_id} /predict '{"prompt": "...", "aspect_ratio": "1:1", ...}' --token $HF_TOKEN
|
||||
|
||||
# Image-to-image (file input):
|
||||
gradio predict {repo_id} /predict '{"input_image": {"path": "/tmp/sample.jpg", "meta": {"_type": "gradio.FileData"}}, "prompt": "..."}' --token $HF_TOKEN
|
||||
```
|
||||
|
||||
If you don't have a local sample image for I2I, lift one from the LoRA repo (`hf_hub_download(repo_id, filename="example.png")`) or the base model card.
|
||||
|
||||
**Caveat for creative-mode Spaces.** `gradio info` and `gradio predict` only exercise the Python endpoint — they tell you nothing about whether custom JS in a `gr.HTML` widget works. If the Space uses creative mode (see `references/creative-mode.md`), after the API smoke-test passes, **open the Space URL in a browser and verify the interaction once** before sharing. Server-side green plus broken JS is the most common failure mode for these.
|
||||
|
||||
**Step 4 — Interpret the result.**
|
||||
|
||||
- **Returns successfully and the output looks plausible** → done. Share the URL.
|
||||
- **HTTPError 503 / "Space is sleeping"** → the Space spun down between steps 1 and 3. Wake it (`api.restart_space(repo_id)`) and retry.
|
||||
- **Inference error mentioning `weight_name` / `safetensors`** → the LoRA filename in `app.py` doesn't match the actual file in the LoRA repo. Re-check `list_repo_files`, fix `weight_name=`, re-upload `app.py`.
|
||||
- **Inference error mentioning a missing pipeline class or attribute** → diffusers version too old. Switch `requirements.txt` to `git+https://github.com/huggingface/diffusers` and re-upload.
|
||||
- **`ImportError` at module load** → missing dep. Add it to `requirements.txt` and re-upload. The runtime logs (`/logs/run`) name the missing package.
|
||||
- **OOM** → reduce default resolution or step count, or pick a smaller base variant.
|
||||
- **Timeout / hangs** → bump `@spaces.GPU(duration=...)` and re-upload.
|
||||
|
||||
The smoke-test exists to convert these from "user discovers it and reports back" to "you discover it and fix it before sharing." Don't skip it because the build went green — green-build-broken-inference is the most common failure mode for Spaces with a non-trivial pipeline.
|
||||
|
||||
---
|
||||
|
||||
## What to avoid
|
||||
|
||||
- A generic "one demo for all LoRAs" template. The whole point of this skill is to tailor.
|
||||
- Lazy-loading the model inside the GPU function. Slow on ZeroGPU, and hides startup errors until first request.
|
||||
- `torch.compile`. Not supported on ZeroGPU.
|
||||
- `cache_examples=True` without `cache_mode="lazy"` on ZeroGPU.
|
||||
- `gr.Examples(fn=…)` whose `inputs` don't cover the inference function's required args — builds green, then crashes on the first example click with a missing-positional-argument error.
|
||||
- Uploading the LoRA weights into the Space repo. Pull from the LoRA's own Hub repo at runtime.
|
||||
- Asking for the HF token only at the end, then discovering the LoRA was private all along and you couldn't read the model card.
|
||||
- Exposing every diffusers knob. Pick the 1–3 controls that matter for this LoRA.
|
||||
- Long preambles in the chat reply once the Space is published. The Space URL is the deliverable; keep the wrap-up brief.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Adapting the demo to the specific LoRA
|
||||
|
||||
The task category (T2I, I2I, T2V, I2V, V2V) gets you a starting shape. It does not get you a UI. Two LoRAs in the same task category can need very different demos. This file is about the reasoning that gets you from "I know the task" to "I know what this demo should look like."
|
||||
|
||||
## The core question
|
||||
|
||||
For each LoRA, ask: **what does this LoRA actually need from the user, and what's the most natural way for the user to provide it?**
|
||||
|
||||
These are different questions. The model might need a pose video as conditioning. That doesn't mean the user has to provide a pose video — they can provide a regular video and the demo extracts pose from it. The model might need a black-bordered video for outpainting. That doesn't mean the user uploads a pre-bordered video — they upload a normal video and pick an aspect ratio, and the demo adds the borders.
|
||||
|
||||
The job of the demo is to translate between what the user has (a video, a photo, an idea) and what the model wants (pose conditioning, padded frames, masked latents, a prompt prefixed with a trigger word).
|
||||
|
||||
## Reading what the LoRA needs
|
||||
|
||||
Sources, in order of usefulness:
|
||||
|
||||
1. **The model card's example code snippet — for *parameters*.** If the README has a Python block showing how to call the LoRA, trust it for inference parameters: pipeline class, step count, guidance scale, true CFG, LoRA scale, dtype, resolution, negative prompt. Trust it for inputs: if it passes `image=...`, the demo takes an image; if it passes `image=...` and `mask_image=...`, the demo needs both.
|
||||
|
||||
**For *loading mechanics*, treat the snippet as a signal, not a directive.** Prefer the standard diffusers path: `pipe.load_lora_weights(repo_id, weight_name=...)`. This is the maintained, well-tested path that handles DoRA, rsLoRA, custom target modules, and most format variants when diffusers + PEFT are recent enough. If the model card uses something else — `PeftModel.from_pretrained(pipe.transformer, ...)`, `diffsynth_engine`, custom imports, manual state-dict surgery — that's a flag to investigate, not to copy.
|
||||
|
||||
Reasons model card authors reach for non-standard loading paths often don't transfer: training-time conventions, environment quirks (older diffusers/PEFT versions, CPU offload patterns), or even malformed configs that pass silently in their setup but crash elsewhere. (Real example: an `adapter_config.json` with `task_type: "DIFFUSION"` works locally on some PEFT versions but crashes on current PEFT, because PEFT's `TaskType` enum only contains NLP tasks; diffusers' loader bypasses this validation by reading the safetensors directly.)
|
||||
|
||||
Custom inference paths absolutely *can* work on ZeroGPU when needed (LTX-2.3 native pipeline is a real example). But default to diffusers because it's the standard, current, maintained path. Only adopt the model card's loading approach if `load_lora_weights` demonstrably can't handle this LoRA — and when you do, port it to the ZeroGPU constraints (module-level `.to('cuda')`, no `enable_model_cpu_offload`).
|
||||
|
||||
2. **Trigger words and prompt patterns.** "Use the trigger word X at the start of the prompt" → automatically prepend `X` to the user's prompt rather than asking them to type it. "Prompts should describe the scene as Y" → add prompt formatting examples or a placeholder. If the LoRA expects a structured input embedded in the prompt (like bounding-box coordinates or named regions), the UI should produce that structure for the user, not require them to type it.
|
||||
|
||||
3. **Example media in the repo.** Example outputs tell you what the LoRA does. Example inputs tell you what the user has to provide. Example *paired* inputs+outputs (input video → output video, input image → output image) tell you the transformation. Lift these into `gr.Examples` so the user can click through them.
|
||||
|
||||
4. **The model's task family.** A pose-conditioned model wants pose maps. A depth-conditioned model wants depth. An outpaint model wants padded frames with masked regions. Each implies preprocessing.
|
||||
|
||||
5. **Recommended hyperparameters.** Step count, guidance scale, true CFG, LoRA scale. Bake the recommended values in as defaults. Expose a slider only if the LoRA's behavior is sensitive to the value across a range (e.g. LoRA scale 0.7–1.3 produces meaningfully different results); otherwise it's just clutter.
|
||||
|
||||
When the model card has none of these, you have three options:
|
||||
|
||||
- **Infer from precedent.** If similar LoRAs exist for the same base model, study their demos.
|
||||
- **Ask the user.** Once, batched, with concrete questions: "Does this LoRA have a trigger word? What's a recommended step count? Got an example prompt?"
|
||||
- **Use sensible defaults from the base model.** Worse than the previous two; only fall back to this when nothing else is available.
|
||||
|
||||
## Verifying the pipeline class
|
||||
|
||||
The pipeline class is decided in Phase 2, not here. The procedure (read the base model's own card, trust its diffusers snippet over reference tables) is in `SKILL.md` under "Phase 2 — Pick the base pipeline." Mentioning it here as a pointer because skipping that verification is one of the most common ways to ship a broken Space — when in doubt, re-read Phase 2.
|
||||
|
||||
## From "what the LoRA needs" to UI shape
|
||||
|
||||
For each thing the LoRA expects as input, decide:
|
||||
|
||||
- **Where does the user get this from?** A pose video — extract from a regular video, or accept a pre-extracted pose video, or both? An aspect ratio — picker, sliders for width/height, or auto from input? A reference image — separate slot, or embedded in a "drag and drop here" area?
|
||||
|
||||
- **Can a more natural input be transformed into this?** Almost always yes. A user has a video, not a pose video. A user wants "wider", not "padded with black bars at these specific coordinates". The demo bridges the gap.
|
||||
|
||||
- **Should the user see the intermediate?** Often yes — a pose extraction preview, a letterbox preview, a generated mask preview. Showing the intermediate builds trust ("yes, the model is conditioning on what I expected") and helps the user iterate. But a preview that takes 10 seconds to generate every time the user changes a setting is a worse UX than no preview.
|
||||
|
||||
- **What's the smallest set of controls that lets the user actually drive this LoRA?** Anything beyond that is clutter. A LoRA that's only good at one specific transformation might need just an input slot and a Generate button.
|
||||
|
||||
## Examples of the reasoning, applied
|
||||
|
||||
**Pose-control video LoRA (V2V).**
|
||||
The model conditions on pose video. The user has a regular video. The demo takes a video, extracts pose, optionally takes an appearance reference image (for "the character looks like *this*, doing *that* motion"), runs inference, returns a video. The pose extraction is shown as a preview so the user knows what's being used. Aspect-ratio picker is irrelevant — the output matches the source.
|
||||
|
||||
**Outpaint video LoRA (V2V).**
|
||||
The model fills black-margined frames. The user has a video and wants it wider/taller. The demo takes a video, takes a target aspect ratio (dropdown: 16:9, 9:16, 1:1, etc.), pads frames with black bars to that aspect ratio, shows a preview of the padded first frame, runs inference. No appearance reference — the LoRA's job is to extend, not transform. If the LoRA's model card mentions that gamma correction helps for dark scenes, expose that as an Advanced toggle.
|
||||
|
||||
**Relight image LoRA (I2I).**
|
||||
The model relights based on prompt. The user has a photo and an idea of what lighting they want. The demo takes an image, takes a brush canvas where the user paints colored strokes indicating where light should come from, takes an illumination style dropdown ("golden hour", "neon", "studio"), takes an optional background change, builds the prompt, runs inference. Brush color and position become structured prompt content.
|
||||
|
||||
**Style image LoRA (T2I).**
|
||||
The model produces images in a specific style. The user has a prompt. The demo takes a prompt (with the trigger word auto-prepended), takes an aspect ratio, runs inference. That's the whole UI. No reference image, no brush canvas, no preprocessing — the LoRA does the work.
|
||||
|
||||
**Bounding-box drag-drop LoRA (I2I).**
|
||||
The model moves and resizes objects between two bounding boxes drawn on the image. The user has an image and an intent ("move that vase from here to there"). The demo takes an image, lets the user draw two boxes (red for source, green for target) directly on the image with a custom canvas component, runs inference. The boxes become the structured input the model expects.
|
||||
|
||||
**Identity-preserving I2V.**
|
||||
The model animates a character while preserving identity. The user has a still photo and a motion intent. The demo takes an image (the character), takes a prompt (the motion), runs inference. If the model also accepts a driving video for motion, expose that as an alternative input mode rather than a second mandatory input.
|
||||
|
||||
The pattern in all of these: start from what the user *has* and what they *want*, and build the UI that bridges to what the model needs.
|
||||
|
||||
## Things that change UI shape
|
||||
|
||||
Signals from the model card or the LoRA's behavior that should change the UI:
|
||||
|
||||
- **Few-step inference (≤ 8 steps).** Hide the steps slider — at this regime the model is recipe-locked. CFG is often 1.0 too. Lock these defaults rather than exposing them.
|
||||
- **Recommended LoRA scale ≠ 1.0** or scale-sensitive behavior. Expose a LoRA-scale slider centered on the recommended value.
|
||||
- **Multiple reference inputs.** Two image slots, clearly labeled (e.g. "appearance" and "pose source") with help text explaining the role of each.
|
||||
- **Optional inputs.** Make optional clearly optional — placeholder text, "(optional)" in the label, the demo runs without it.
|
||||
- **Multi-stage pipelines** (e.g. extract + generate, generate + refine). Show stage progress (`progress(0.3, desc="Extracting pose...")` then `progress(0.6, desc="Generating...")`). Otherwise the user stares at a blank progress bar for a long generation.
|
||||
- **Output is video > 5s.** Bump `@spaces.GPU(duration=...)` higher and warn the user in the UI that generation takes longer.
|
||||
- **LoRA expects structured prompt content** (coordinates, region tags, named entities). Build a small UI that produces that structure rather than asking the user to type it raw.
|
||||
|
||||
## Things that don't change UI shape
|
||||
|
||||
- The base model identity, beyond determining the pipeline class. A Qwen-Image style LoRA and a Flux style LoRA can have identical UIs.
|
||||
- Pure performance details (dtype, device map, attention impl). These belong in the model load code, not the UI.
|
||||
- The LoRA's training data composition. Interesting, not load-bearing.
|
||||
|
||||
## When LoRA loading fails
|
||||
|
||||
The right move when `pipe.load_lora_weights(...)` fails is to read the error and recognize the *category* of failure. Each category has a different fix path. Don't guess — different errors imply different things about whether the LoRA is salvageable on the diffusers path at all.
|
||||
|
||||
**Don't preemptively call conversion utilities.** `load_lora_weights` already calls the appropriate converters internally for the formats it knows about. Calling `convert_state_dict_to_diffusers` or similar before there's an error is redundant in the common case and risky if you guess wrong — you can mangle a state dict that would have loaded fine.
|
||||
|
||||
Failure categories:
|
||||
|
||||
- **Config-validation failure** (errors mentioning `task_type`, `peft_type`, "Invalid task type", `PeftConfig`, or anything from `peft/config.py`): the safetensors weights themselves may be fine; `adapter_config.json` is the problem. Fix paths: pass `weight_name=` explicitly so `load_lora_weights` reads the safetensors directly without going through PEFT's strict config parser; or download the safetensors and load via state dict. Falling back to `PeftModel.from_pretrained` will *not* help — that path crashes on the same config.
|
||||
|
||||
- **Missing keys / unexpected keys** ("Loading adapter weights from state_dict led to missing keys" or "led to unexpected keys"): the state dict's key naming doesn't match what diffusers' loader expects. This often means the LoRA was trained with a non-diffusers convention (kohya, ComfyUI, OneTrainer, custom training scripts), or with custom target modules. Fix paths: try diffusers' built-in conversion utilities (`convert_state_dict_to_diffusers`, base-model-specific converters in `diffusers.loaders`); if those don't help, the format may not be supported yet — surface this clearly to the user rather than silently using a partial load.
|
||||
|
||||
- **Shape mismatch errors**: the LoRA's tensor shapes don't match the base model's. Usually means the LoRA was trained against a different base model variant than the one being used (e.g. trained on Qwen-Image-Edit but loaded against Qwen-Image, or trained on FLUX.1-dev but loaded against FLUX.1-schnell). Fix: check the model card's `base_model` field carefully and switch to the correct base.
|
||||
|
||||
- **OOM during load or first inference**: not really a loading failure — the LoRA loaded but the combined base + LoRA + activations don't fit. Fix paths involve `pipe.enable_vae_tiling()`, smaller resolutions, FP8/quantized base variants. Not in scope for this section.
|
||||
|
||||
- **Missing keys for *some* weights only** (e.g. text encoder LoRA missing but transformer LoRA present): often a partial-coverage LoRA that only targets one component. May actually be intentional and may still work — generate a test image and see if the LoRA effect is present.
|
||||
|
||||
When none of these fit and `load_lora_weights` simply doesn't work, falling back to a non-diffusers path becomes a real option. At that point the model card's snippet becomes more useful — but port it to ZeroGPU constraints (no `enable_model_cpu_offload`, module-level `.to('cuda')`, models on `cuda` outside `@spaces.GPU`).
|
||||
@@ -0,0 +1,63 @@
|
||||
# Krea 2 reference
|
||||
|
||||
Krea 2 (K2) is a flow-matching **text-to-image** model: a 12B dense DiT (grouped-query attention) with a **Qwen3-VL** text encoder (multi-layer feature aggregation) and the **Qwen-Image VAE** (`AutoencoderKLQwenImage`). Pipeline class: `Krea2Pipeline`. It ships as two checkpoints designed to work together:
|
||||
|
||||
| Repo | Role | Use it for |
|
||||
|------|------|------------|
|
||||
| [`krea/Krea-2-Turbo`](https://huggingface.co/krea/Krea-2-Turbo) | 8-step **distilled** | **Inference / demos** |
|
||||
| [`krea/Krea-2-Raw`](https://huggingface.co/krea/Krea-2-Raw) | base, non-distilled | **LoRA training** — *not* for inference |
|
||||
|
||||
**For a LoRA Space, load Turbo.** Krea 2 LoRAs are *trained on RAW but run on Turbo* (they "express strongly on Turbo"), and RAW is explicitly not meant for inference — expect poor quality if you load it directly. So a demo should almost always use `krea/Krea-2-Turbo`. The official LoRA cards confirm this ("To be used on `krea/Krea-2-Turbo`").
|
||||
|
||||
> Needs a `diffusers` build that includes `Krea2Pipeline` (both repos are `library_name: diffusers`). If `from diffusers import Krea2Pipeline` fails, the installed `diffusers` predates the integration — update it.
|
||||
|
||||
## Required dependencies
|
||||
|
||||
- `diffusers` with `Krea2Pipeline`.
|
||||
- `transformers` recent enough for **Qwen3-VL** (the text encoder is a `Qwen3VLModel`, e.g. `Qwen/Qwen3-VL-4B-Instruct`).
|
||||
- **`torchvision`** — the Qwen3-VL processor pulls it in transitively; missing it is a startup `ImportError`. Always include it.
|
||||
- `sentencepiece` if you see tokenizer-related ImportErrors at startup.
|
||||
|
||||
## Default load + LoRA (Turbo)
|
||||
|
||||
```python
|
||||
import torch
|
||||
from diffusers import Krea2Pipeline
|
||||
|
||||
pipe = Krea2Pipeline.from_pretrained("krea/Krea-2-Turbo", torch_dtype=torch.bfloat16).to("cuda")
|
||||
pipe.transformer.load_lora_adapter("user/my-krea2-lora", weight_name="my_lora.safetensors")
|
||||
pipe.transformer.set_adapters("default", weights=1.0)
|
||||
|
||||
# include the LoRA's trigger word(s) from its card
|
||||
image = pipe(
|
||||
"a deer grazing in a forest, <trigger words>",
|
||||
num_inference_steps=8, guidance_scale=0.0,
|
||||
generator=torch.Generator("cuda").manual_seed(0),
|
||||
).images[0]
|
||||
```
|
||||
|
||||
Krea 2 LoRAs load through the **transformer's** adapter API (`pipe.transformer.load_lora_adapter(...)` + `pipe.transformer.set_adapters("default", weights=1.0)`), per the official LoRA cards — not the pipeline-level `pipe.load_lora_weights`. Honor the LoRA's **trigger word(s)** and recommended weight (1.0 by default).
|
||||
|
||||
## Inference recipe
|
||||
|
||||
- **Turbo (the demo default):** `num_inference_steps=8`, **`guidance_scale=0.0`** (guidance disabled), LoRA weight `1.0`.
|
||||
- **RAW:** training only — don't ship a RAW inference demo; its quality is intentionally low (it's the malleable base you fine-tune on, then run on Turbo).
|
||||
|
||||
## `guidance_scale` convention (gotcha)
|
||||
|
||||
Krea 2 enables guidance whenever **`guidance_scale > 0`** and computes velocity as `cond + guidance_scale * (cond − uncond)` (≡ the usual CFG formulation with scale `1 + guidance_scale`). So Turbo disables guidance with **`guidance_scale=0.0`** — not `1.0`.
|
||||
|
||||
## Resolution
|
||||
|
||||
`height`/`width` must be divisible by **16** (`vae_scale_factor * patch_size`); the pipeline rounds up to a multiple of 16 (with a warning) otherwise. Default 1024×1024.
|
||||
|
||||
## ZeroGPU duration
|
||||
|
||||
Standard T2I: place modules at module scope, `pipe.to("cuda")`, no `torch.compile`. Turbo 8-step 1024² is fast (≈ 20–40s) — set `@spaces.GPU(duration=...)` comfortably above that. The 12B DiT + Qwen3-VL encoder fits ZeroGPU; if you OOM on the default size, try `@spaces.GPU(size="xlarge")`.
|
||||
|
||||
## Things to watch
|
||||
|
||||
- **Load Turbo, not RAW, for demos.** RAW is the training base and not meant for inference.
|
||||
- **`guidance_scale=0` disables guidance** (Krea convention, unlike pipelines where 1.0 is "off"). Turbo = 8 steps, `guidance_scale=0.0`.
|
||||
- **LoRAs use `pipe.transformer.load_lora_adapter` + `set_adapters("default", …)`** (transformer-level), and have **trigger words** — read the card.
|
||||
- **New pipeline → update diffusers** if `from diffusers import Krea2Pipeline` fails.
|
||||
@@ -0,0 +1,205 @@
|
||||
# LTX reference
|
||||
|
||||
The LTX family covers `LTX-Video` (0.9.x series), `LTX-2`, and `LTX-2.3`. All target text-to-video, image-to-video, and video-to-video.
|
||||
|
||||
Diffusers support varies across versions:
|
||||
|
||||
| Base model series | Diffusers support | LoRA loading via diffusers? |
|
||||
|---------------------------|------------------------------------------|-----------------------------|
|
||||
| `Lightricks/LTX-Video` (0.9.x) | Yes — `LTXPipeline`, `LTXImageToVideoPipeline`, `LTXConditionPipeline` | Yes |
|
||||
| `Lightricks/LTX-2` | Yes — `LTX2Pipeline` from `diffusers.pipelines.ltx2` | Yes (recent diffusers) |
|
||||
| `Lightricks/LTX-2.3` | Partial — diffusers support is still limited overall, so the original `Lightricks/LTX-2.3` repo can still be used via the native path. Diffusers-converted variants exist at `dg845/LTX-2.3-Diffusers` and `dg845/LTX-2.3-Distilled-Diffusers`, supporting regular LoRAs via the standard pipelines, IC LoRAs via `LTX2InContextPipeline`, and HDR IC-LoRAs via `LTX2HDRPipeline` (PR #13572, merged 2026-05-15; needs `git+https://github.com/huggingface/diffusers`) | Yes on the diffusers variants — standard `load_lora_weights` + `set_adapters`. Native path still required for some configurations. |
|
||||
|
||||
When in doubt about LTX-2.3, check the LoRA's model card for an example snippet. If it imports from `ltx_video` or a native module rather than `diffusers`, use the native path.
|
||||
|
||||
## Pipelines (diffusers path)
|
||||
|
||||
> **Before using this table, verify against the base model's own card on the Hub.** This table is best-effort and can lag a recent release (LTX moves fast). The diffusers snippet on the base model's Hub page is source of truth. See `SKILL.md` Phase 2 for the procedure.
|
||||
|
||||
| Task | Pipeline | Pretrained ID |
|
||||
|-------------------------|-----------------------------------|------------------------------------------------------|
|
||||
| Text-to-video | `LTXPipeline` | `Lightricks/LTX-Video` |
|
||||
| Image-to-video | `LTXImageToVideoPipeline` | `Lightricks/LTX-Video` |
|
||||
| Video-to-video / keyframes | `LTXConditionPipeline` | `Lightricks/LTX-Video-0.9.5` or later |
|
||||
| Spatial upscale | `LTXLatentUpsamplePipeline` | `Lightricks/ltxv-spatial-upscaler-0.9.8` |
|
||||
| LTX-2 T2V/I2V | `LTX2Pipeline` | `Lightricks/LTX-2` |
|
||||
| LTX-2.3 in-context (IC LoRAs) | `LTX2InContextPipeline` | `dg845/LTX-2.3-Distilled-Diffusers` (preferred for demos; non-distilled `dg845/LTX-2.3-Diffusers` also works) |
|
||||
|
||||
`LTXConditionPipeline` is the workhorse for V2V — it takes a `LTXVideoCondition` (or list of them) plus optional first-frame image, and produces a video conditioned on the input.
|
||||
|
||||
## Required dependencies
|
||||
|
||||
LTX pipelines need extras beyond the standard diffusers/transformers/peft set, because video output requires file-format support:
|
||||
|
||||
- **`imageio`** and **`imageio-ffmpeg`** — required by `diffusers.utils.export_to_video`. Without them, video export fails at runtime even though model loading succeeds. Always include both.
|
||||
- **`sentencepiece`** — required by the T5 text encoder some LTX variants use. Include if you see tokenizer-related ImportErrors at startup.
|
||||
- **`av`** — `pyav`, useful when reading input videos in non-trivial formats. Include for V2V pipelines that take video input via `load_video`.
|
||||
|
||||
For LTX-2 and LTX-2.3, the latest diffusers from git is often required since pipeline classes (`LTX2Pipeline`, conditioning APIs) land before pip releases:
|
||||
|
||||
```
|
||||
git+https://github.com/huggingface/diffusers
|
||||
```
|
||||
|
||||
For LTX-2.3 native path:
|
||||
|
||||
```
|
||||
git+https://github.com/Lightricks/LTX-Video.git
|
||||
```
|
||||
|
||||
If `from_pretrained` fails with class-not-found errors for a recent LTX variant, switch to git diffusers.
|
||||
|
||||
## Default load (T2V)
|
||||
|
||||
```python
|
||||
import torch
|
||||
from diffusers import LTXPipeline
|
||||
from diffusers.utils import export_to_video
|
||||
|
||||
pipe = LTXPipeline.from_pretrained(
|
||||
"Lightricks/LTX-Video",
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
pipe.to("cuda")
|
||||
pipe.load_lora_weights("user/my-ltx-lora")
|
||||
```
|
||||
|
||||
## Default load (V2V via LTXConditionPipeline)
|
||||
|
||||
```python
|
||||
import torch
|
||||
from diffusers import LTXConditionPipeline
|
||||
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
|
||||
from diffusers.utils import load_video, export_to_video
|
||||
|
||||
pipe = LTXConditionPipeline.from_pretrained(
|
||||
"Lightricks/LTX-Video-0.9.5",
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
pipe.to("cuda")
|
||||
pipe.load_lora_weights("user/my-ltx-vlora")
|
||||
|
||||
video = load_video("input.mp4")
|
||||
condition = LTXVideoCondition(video=video, frame_index=0)
|
||||
|
||||
frames = pipe(
|
||||
conditions=[condition],
|
||||
prompt="...",
|
||||
negative_prompt="worst quality, jittery, blurry",
|
||||
width=768, height=512,
|
||||
num_frames=121,
|
||||
num_inference_steps=50,
|
||||
).frames[0]
|
||||
export_to_video(frames, "out.mp4", fps=24)
|
||||
```
|
||||
|
||||
## LTX-2 (diffusers path)
|
||||
|
||||
```python
|
||||
from diffusers.pipelines.ltx2 import LTX2Pipeline
|
||||
pipe = LTX2Pipeline.from_pretrained("Lightricks/LTX-2", torch_dtype=torch.bfloat16)
|
||||
```
|
||||
|
||||
LTX-2 supports a two-stage pipeline (base + latent upsample) for production quality. For a demo, the single-stage pipeline is usually sufficient and faster.
|
||||
|
||||
**Don't default two-stage on for IC-LoRAs — check the card.** The common LTX-2.3 two-stage recipe runs stage 2 as an x2 latent-upsample + refine **with the IC-LoRA disabled** (`disable_lora()`), re-rendering on the bare base — which re-degrades exactly the LoRA-conditioned detail (text, identity). Whether two-stage is appropriate is per-LoRA, per the card: e.g. the *In/Out-painting* card explicitly says "both tasks use a two-stage pipeline," whereas the *Ingredients* card recommends a single-pass 30-step recipe and never mentions two stages. Single-stage is the safer demo default unless the card calls for two-stage.
|
||||
|
||||
## Inference defaults
|
||||
|
||||
- **Resolution**: typically multiples of 32. Common sizes: 768×512, 1216×704, 704×480.
|
||||
- **Frame count**: typically `8k+1` (e.g. 121, 161, 257). Compute from `duration_seconds * fps` and round.
|
||||
- **fps**: 24 by default; some LoRAs are trained at different rates (16, 30) — check the model card.
|
||||
- **`num_inference_steps`**: 30–50 for non-distilled. Distilled checkpoints (look for "distilled" in the name) often run at 8–12.
|
||||
- **`negative_prompt`**: LTX is sensitive to negative prompts. A good default: `"worst quality, inconsistent motion, blurry, jittery, distorted"`.
|
||||
- **Distilled IC-LoRA — disable audio guidance too, not just `guidance_scale`.** `LTX2InContextPipeline` computes `do_classifier_free_guidance = guidance_scale > 1 OR audio_guidance_scale > 1`, and **`audio_guidance_scale` defaults to `7.0`**. So `guidance_scale=1.0` is not enough — CFG stays on via audio (and `stg_scale` defaults on), which doubles/mis-batches the forward against the in-context reference tokens (wrong recipe, sometimes a runtime error). For a distilled IC-LoRA pass all four: `pipe(..., guidance_scale=1.0, stg_scale=0.0, audio_guidance_scale=1.0, audio_stg_scale=0.0)`.
|
||||
|
||||
## Frame-count helper
|
||||
|
||||
```python
|
||||
def num_frames_for_duration(seconds, fps=24, base=8):
|
||||
raw = seconds * fps
|
||||
return ((int(raw) - 1) // base) * base + 1
|
||||
```
|
||||
|
||||
## Native pipeline path (LTX-2.3 fallback)
|
||||
|
||||
The native repo is a fallback when the diffusers path doesn't work, or for specific conditionings that diffusers may not support yet. Try `LTX2InContextPipeline` on the diffusers path first (see the IC-LoRAs section above).
|
||||
|
||||
**The native repo depends on the model generation** — pick by the LoRA's base model (and what its card's snippet imports):
|
||||
|
||||
| Base model | Native repo | Package(s) / import | Pipeline classes |
|
||||
|---|---|---|---|
|
||||
| `LTX-Video` (0.9.x) | `github.com/Lightricks/LTX-Video` | `ltx_video` | `from ltx_video.pipelines import LTXPipeline` |
|
||||
| `LTX-2`, `LTX-2.3` | `github.com/Lightricks/LTX-2` | `ltx-core` + `ltx-pipelines` (editable) | `ltx_pipelines.ic_lora.ICLoraPipeline`, `TI2VidOneStagePipeline`, … |
|
||||
|
||||
For the 0.9.x series the pattern looks like:
|
||||
|
||||
```bash
|
||||
# in requirements.txt
|
||||
git+https://github.com/Lightricks/LTX-Video.git
|
||||
```
|
||||
|
||||
```python
|
||||
from ltx_video.pipelines import LTXPipeline as NativeLTXPipeline
|
||||
# ... model loading per the native repo's README
|
||||
```
|
||||
|
||||
The native path doesn't use `load_lora_weights`. Instead, LoRAs are usually wired in at pipeline construction time, often via a list of LoRA configurations or by pointing at a fused checkpoint.
|
||||
|
||||
When the LoRA's model card has a Python snippet using the native repo, copy its construction pattern verbatim. The native API changes more often than diffusers' does, so don't paraphrase.
|
||||
|
||||
### LTX-2.x native path on ZeroGPU — gotchas
|
||||
|
||||
For `LTX-2` / `LTX-2.3` (the `ltx-core` + `ltx-pipelines` repo), clone + `pip install -e packages/ltx-core packages/ltx-pipelines` at runtime in `app.py` and pin a commit. Three things bite on ZeroGPU:
|
||||
|
||||
- **Native loader bypasses ZeroGPU virtualization → "No CUDA GPUs available" at startup.** The native safetensors loader does `safe_open(path, device="cuda")` and copies host→device inside safetensors' own C++ (`cudaMemcpy`), **bypassing `torch.Tensor.to`** — the call ZeroGPU patches to virtualize + pack module-scope weights. Nothing packs and module-scope placement raises *"No CUDA GPUs are available."* Fix: monkeypatch the loader to open on CPU then move with `.to`:
|
||||
```python
|
||||
with safetensors.safe_open(shard, framework="pt", device="cpu") as f:
|
||||
value = f.get_tensor(name).to(device=device) # torch path → ZeroGPU-virtualisable
|
||||
```
|
||||
Also: patch attention to **SDPA** (FA3 crashes on Blackwell ZeroGPU), and **never call `torch.cuda.*` / `get_device_capability()` at module scope** (it poisons virtualization).
|
||||
- **Native two-stage pins two 22B transformers → offload-disk overflow.** `ICLoraPipeline` builds two independent `ModelLedger`s (stage-1 with the LoRA, stage-2 without), each loading its **own** full transformer. The CLI loads them sequentially, but ZeroGPU pins all weights at module scope, so enabling stage 2 pins **both** (~143G) and overflows the offload disk (`OSError: [Errno 28] No space left on device`, ~96G cap). For a demo: pin stage 1, then have stage 2 **reuse stage 1's pinned modules** (`for n in [...]: setattr(s2, n, getattr(s1, n))`) so only one transformer is resident (~71G).
|
||||
- **`ICLoraPipeline` is distilled-only.** Per the LTX-2 `ltx-pipelines` guide, IC-LoRA inference runs in distilled-only mode (fixed 8-step sigmas, no `num_inference_steps`/`guidance_scale`/`negative_prompt`); the docs do not describe non-distilled IC-LoRA or IC-LoRA + CFG/STG (guidance lives only on full-model pipelines like `TI2VidOneStagePipeline`, and in two-stage is stage-1-only). So if a card recommends a *non-distilled* recipe (e.g. 30 steps + guidance + STG on a `…-dev` base), there is no stock IC-LoRA pipeline for it — use the supported distilled `ICLoraPipeline` and note the recipe difference, or fall back to the diffusers `LTX2InContextPipeline` (which exposes steps/guidance).
|
||||
|
||||
## IC-LoRAs (in-context conditioning)
|
||||
|
||||
LTX IC-LoRAs condition the model on a reference video alongside a first-frame image. The diffusers path uses `LTX2InContextPipeline` with standard `load_lora_weights` + `set_adapters`.
|
||||
|
||||
Common flavors:
|
||||
|
||||
- **Pose / depth / canny IC-LoRAs**: the reference video must be preprocessed into the control signal (pose skeletons, depth maps, edge maps) before being passed as conditioning. Passing a raw video with appearance information leaks color/style through.
|
||||
- **Outpaint IC-LoRAs**: the input video is padded with black margins to a target aspect ratio and passed as conditioning. The model fills the black regions.
|
||||
- **Frame interpolation / extension IC-LoRAs**: the input is a sparse set of keyframes; the model fills in between or extends.
|
||||
- **Audio-driven lip-sync IC-LoRAs**: take a reference video and a new audio track; the model re-syncs lip motion to match the new audio. UI needs both a video input and an audio input (e.g. for video translation, voiceover replacement, multilingual dubbing).
|
||||
|
||||
Each of these implies different preprocessing in the demo and different UI shape (an aspect-ratio picker for outpaint, a pose preview for pose-control, a frame-pair input for keyframe extension). Read `references/adapting-to-the-lora.md` and shape the UI to the specific IC-LoRA.
|
||||
|
||||
## Quantization gotcha (LTX-2.3, native path only)
|
||||
|
||||
On the native path, some LTX-2.3 IC-LoRAs ship with FP8 quantization policies that fuse LoRA deltas into transformer weights at model-load time, using a Triton CUDA kernel. ZeroGPU has no real CUDA at module-load time (only the emulation layer) — this can crash the Space at startup. The diffusers path via `LTX2InContextPipeline` does not hit this: LoRAs are loaded through PEFT as separate adapter weights and applied at runtime rather than fused into the base transformer at load.
|
||||
|
||||
Two workarounds when the user hits this on the native path:
|
||||
|
||||
- **Skip quantization for the LoRA-fusion stage.** LTX-2.3 has a two-stage pipeline; apply quantization only to the second (non-LoRA) stage. Set `stage_1_quantization=None`.
|
||||
- **Pre-fuse the LoRA into a standalone checkpoint.** Download base + LoRA, fuse on a dev GPU, push the fused checkpoint to the Hub under the user's namespace, and have the Space load the fused checkpoint with no LoRA.
|
||||
|
||||
The second option is preferable when the user plans to demo the same LoRA repeatedly — Space startup is much faster.
|
||||
|
||||
## ZeroGPU duration guidance for LTX
|
||||
|
||||
- Short T2V (3 seconds, 24fps, distilled): 60–90 seconds.
|
||||
- Standard T2V (5 seconds, 50 steps): 120–180 seconds.
|
||||
- I2V: similar to T2V at the same duration.
|
||||
- V2V with preprocessing (pose extraction, etc.): add 10–30 seconds for preprocessing overhead. Set duration to 180+.
|
||||
- LTX-2.3 two-stage: 240–360 seconds.
|
||||
|
||||
Set `@spaces.GPU(duration=...)` to comfortably exceed expected generation time.
|
||||
|
||||
## Things to watch
|
||||
|
||||
- **Latest pipelines often need git diffusers.** Pin to `git+https://github.com/huggingface/diffusers` in `requirements.txt` for any LTX variant released in the last few weeks.
|
||||
- **`git+diffusers` is a moving target — verify LTX2 output quality.** Bare `@main` inherits whatever's there. The LTX2 text connector carried a token-reversal regression (PR #13564) that scrambled prompt tokens/registers — degrading prompt adherence and fine detail (e.g. garbled on-screen text), worst on short prompts — until **PR #13931 (merged 2026-06-19)** fixed it. If LTX2 output looks weak or garbled, confirm the installed diffusers commit is **at or past #13931**, and prefer pinning to a known-good commit (`git+https://github.com/huggingface/diffusers@<sha>`) over bare `@main` for reproducibility.
|
||||
- **Don't `torch.compile`.** LTX is fast enough on ZeroGPU without it; compile is incompatible anyway.
|
||||
- **`enable_vae_tiling()` for higher resolutions.** LTX VAE memory grows with resolution; enable tiling for outputs above 768px on either axis.
|
||||
- **Negative prompts matter more than for image models.** Don't ship with an empty negative_prompt unless the LoRA's model card says so.
|
||||
- **Frame-rate mismatch can produce glitches.** If the LoRA was trained at 24 fps and the demo passes 30, motion looks wrong. Use the LoRA's recommended fps.
|
||||
@@ -0,0 +1,144 @@
|
||||
# Qwen-Image and Qwen-Image-Edit reference
|
||||
|
||||
The Qwen-Image family is fully supported in `diffusers`. Both base and edit variants accept LoRAs via the standard `load_lora_weights` interface.
|
||||
|
||||
## Pipelines
|
||||
|
||||
> **Before using this table, verify against the base model's own card on the Hub.** This table is best-effort and can lag a recent release. The diffusers snippet on the base model's Hub page is source of truth for which pipeline class to import. See `SKILL.md` Phase 2 for the procedure.
|
||||
|
||||
| Base model | Pipeline class | Task |
|
||||
|---------------------------------------|-------------------------------|-------------------------------------|
|
||||
| `Qwen/Qwen-Image` | `QwenImagePipeline` | Text-to-image |
|
||||
| `Qwen/Qwen-Image-Edit` | `QwenImageEditPipeline` | Image editing (instruction-driven) |
|
||||
| `Qwen/Qwen-Image-Edit-2509` | `QwenImageEditPlusPipeline` | Image editing, multi-image input |
|
||||
| `Qwen/Qwen-Image-Edit-2511` | `QwenImageEditPlusPipeline` | Image editing, latest variant |
|
||||
|
||||
The 2509 and 2511 variants use a *different* pipeline class than the original `QwenImageEditPipeline` — they take a list of input images and have different default parameters. Don't assume that variants in the same family share a pipeline class. Loading a 2511-trained LoRA onto `QwenImageEditPipeline` produces broken output; the failure is silent (no exception), so verifying against the base model card is the only way to catch it.
|
||||
|
||||
The 2511 variant integrates several popular community LoRAs into the base, which can mean a LoRA trained against earlier Qwen-Image-Edit may behave subtly differently when loaded against 2511; if the LoRA's model card specifies which Edit variant it was trained on, match it.
|
||||
|
||||
## Required dependencies
|
||||
|
||||
Qwen-Image and Qwen-Image-Edit pipelines need extras beyond the standard diffusers/transformers/peft set, because the text encoder is `Qwen2_5_VLForConditionalGeneration` (Qwen 2.5-VL):
|
||||
|
||||
- **`torchvision`** — required by `Qwen2VLVideoProcessor`, which the text encoder's processor pulls in transitively. Missing this is a startup-time `ImportError` ("Qwen2VLVideoProcessor requires the Torchvision library"). Always include in `requirements.txt` for any Qwen-Image Space.
|
||||
- **`sentencepiece`** — required by some Qwen tokenizer paths. Include if you see tokenizer-related ImportErrors at startup.
|
||||
|
||||
The 2511 variant in particular often requires the latest `diffusers` from git, since `QwenImageEditPlusPipeline` and 2511-specific fixes land before pip releases:
|
||||
|
||||
```
|
||||
git+https://github.com/huggingface/diffusers
|
||||
```
|
||||
|
||||
If `from_pretrained("Qwen/Qwen-Image-Edit-2511", ...)` fails with a class-not-found or attribute error, switch the requirement to git.
|
||||
|
||||
## Default load (T2I)
|
||||
|
||||
```python
|
||||
import torch
|
||||
from diffusers import QwenImagePipeline
|
||||
|
||||
pipe = QwenImagePipeline.from_pretrained(
|
||||
"Qwen/Qwen-Image",
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
pipe.to("cuda")
|
||||
pipe.load_lora_weights("user/my-qwen-lora")
|
||||
```
|
||||
|
||||
`pytorch_lora_weights.safetensors` is the conventional filename. If the repo has a different name, pass `weight_name="..."`.
|
||||
|
||||
For multiple adapters or when you want to control LoRA scale at inference time, use `set_adapters`:
|
||||
|
||||
```python
|
||||
pipe.load_lora_weights("user/my-qwen-lora", adapter_name="mylora")
|
||||
pipe.set_adapters(["mylora"], adapter_weights=[0.9])
|
||||
```
|
||||
|
||||
## Default load (Image Edit)
|
||||
|
||||
For original `Qwen-Image-Edit`:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from diffusers import QwenImageEditPipeline
|
||||
|
||||
pipe = QwenImageEditPipeline.from_pretrained(
|
||||
"Qwen/Qwen-Image-Edit",
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
pipe.to("cuda")
|
||||
pipe.load_lora_weights("user/my-qwen-edit-lora")
|
||||
```
|
||||
|
||||
For `Qwen-Image-Edit-2509` and `Qwen-Image-Edit-2511`:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from diffusers import QwenImageEditPlusPipeline
|
||||
|
||||
pipe = QwenImageEditPlusPipeline.from_pretrained(
|
||||
"Qwen/Qwen-Image-Edit-2511", # or 2509
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
pipe.to("cuda")
|
||||
pipe.load_lora_weights("user/my-qwen-edit-lora")
|
||||
```
|
||||
|
||||
`QwenImageEditPlusPipeline` accepts `image=<PIL>` or `image=[<PIL>, <PIL>, ...]` for multi-image edits. `QwenImageEditPipeline` accepts a single image. Default parameters differ slightly between the two — see "Inference defaults" below.
|
||||
|
||||
## Inference defaults
|
||||
|
||||
For non-distilled Qwen-Image:
|
||||
- `num_inference_steps`: 50 by default; the LoRA's model card may recommend lower.
|
||||
- `true_cfg_scale`: typical 4.0.
|
||||
- `width`/`height`: multiples of 16, ideally 1024 or 1328 along the long axis.
|
||||
|
||||
For Qwen-Image-Edit (original):
|
||||
- `num_inference_steps`: 30–50 typical, often less for distilled variants.
|
||||
- `true_cfg_scale`: 4.0 typical.
|
||||
- Input image gets internally resized; passing a reasonable resolution (1024px on the long side) is fine.
|
||||
|
||||
For Qwen-Image-Edit-2509 / 2511 (`QwenImageEditPlusPipeline`):
|
||||
- `num_inference_steps`: 40 typical for 2511; 50 for 2509.
|
||||
- `true_cfg_scale`: 4.0.
|
||||
- `guidance_scale`: 1.0 (the new pipeline uses `true_cfg_scale` as the active CFG; standard `guidance_scale` is kept at 1.0).
|
||||
- Input is a list of one or more PIL images.
|
||||
|
||||
For Lightning / few-step LoRAs (e.g. `lightx2v/Qwen-Image-Lightning-*`):
|
||||
- `num_inference_steps`: 4 or 8 (read the LoRA's model card — they ship 4-step and 8-step variants).
|
||||
- `true_cfg_scale`: usually 1.0 (CFG disabled).
|
||||
- Often comes with a custom scheduler config — see the lightx2v README for the exact `FlowMatchEulerDiscreteScheduler` config to use.
|
||||
|
||||
## Resolution buckets
|
||||
|
||||
Qwen-Image uses 16-pixel-aligned resolutions. When the user picks an aspect ratio, compute `width` and `height` as multiples of 16. A helper:
|
||||
|
||||
```python
|
||||
def round_to_bucket(w, h, multiple=16):
|
||||
return (w // multiple) * multiple, (h // multiple) * multiple
|
||||
```
|
||||
|
||||
For image-edit pipelines, resize the input image to the nearest bucket while preserving aspect; don't crop.
|
||||
|
||||
## ZeroGPU duration guidance
|
||||
|
||||
- Standard 50-step Qwen-Image T2I at 1024×1024: 60–90 seconds.
|
||||
- 4-step Lightning Qwen-Image: 15–25 seconds.
|
||||
- Qwen-Image-Edit 30 steps: 60–90 seconds.
|
||||
|
||||
Set `@spaces.GPU(duration=...)` accordingly.
|
||||
|
||||
## Common LoRA patterns on Qwen-Image
|
||||
|
||||
- **Style LoRAs (T2I).** Standard load. Trigger word usually present. UI: prompt + aspect ratio.
|
||||
- **Subject / character LoRAs (T2I).** Standard load. Trigger word almost always present. UI: prompt with the trigger pre-prepended in code, possibly an example prompt highlighting the trigger.
|
||||
- **Lighting / aesthetic LoRAs (T2I).** Often paired with a recommended LoRA scale ≠ 1.0 — check the model card.
|
||||
- **Edit LoRAs (image-to-image, on Qwen-Image-Edit).** Specific instructions baked in. The LoRA might require a specific instruction phrasing — match the pattern from the model card. UI: input image + instruction textbox.
|
||||
- **Lightning-distilled LoRAs.** Lock step count and CFG to recommended values; hide the sliders.
|
||||
|
||||
## Things to watch
|
||||
|
||||
- **VAE memory.** For 1328×1328 outputs, consider `pipe.enable_vae_tiling()` and `pipe.enable_vae_slicing()` after loading. Keep them off for smaller resolutions to avoid quality loss.
|
||||
- **Don't compile the transformer on ZeroGPU.** `torch.compile` won't work; the speedup options on ZeroGPU are limited to reducing steps or using FP8-distilled variants.
|
||||
- **Negative prompts work** but defaults are often empty string. Don't expose a negative prompt in the UI unless the LoRA's behavior actually benefits from it.
|
||||
@@ -0,0 +1,262 @@
|
||||
# Creative mode: custom HTML/JS UIs in Gradio
|
||||
|
||||
When a LoRA's natural input shape doesn't fit any standard Gradio component or Hub custom component, you can drop down to plain HTML/CSS/JS inside a Gradio app. This file is about *how* — the Gradio primitives that make it work, and the discipline that keeps it from turning into a tangled mess.
|
||||
|
||||
This is the third rung of the component ladder (see `tasks.md` → "Picking components"):
|
||||
|
||||
1. **Stock Gradio components.** First choice. Almost always enough for T2I and a lot of I2I.
|
||||
2. **Hub custom components.** `gradio_image_annotation`, `gradio_imageslider`, `gradio_modal`, `gradio_rangeslider`, etc. No JS, just a `pip install` and an import.
|
||||
3. **Creative mode (this file).** Custom HTML/JS, when the user's input shape is something none of the above expresses well — point sets, trajectories, brush strokes, region selections with metadata, timeline scrubbing, 3D rotation gizmos, color picking on regions, drag-resize handles on media, keyframed inputs, anything where the user manipulates *a thing on top of media*.
|
||||
|
||||
Skipping rung 2 is a common mistake. If a Hub custom component fits, use it — `gradio_image_annotation` already covers bbox drawing, label assignment, and basic editing without a single line of JS.
|
||||
|
||||
But rung 2 has its own discipline (see "Hub custom components are fragile" below). The most common failure mode for these Spaces is **the custom component silently fails to render**: the Python side imports fine, the page loads, the API smoke-test in Phase 6 of `SKILL.md` passes — and yet the widget just isn't on the page. A user opening the Space sees the surrounding layout (buttons, accordions, outputs) but no upload zone, no annotator, nothing. If you don't actually look at the rendered page in a browser, you'll ship a broken Space and not know it.
|
||||
|
||||
## Hub custom components are fragile
|
||||
|
||||
Treat any `gradio_*` package from the Hub as load-bearing-and-untested-against-your-Gradio-version until you've seen it render. The most common failure modes:
|
||||
|
||||
- **Version mismatch silent breakage.** A Hub component built against Gradio N may load (because its declared `gradio<N+2,>=N` range covers your `sdk_version`) but mount to an empty DOM node on a slightly newer Gradio. No traceback in the build logs. No Python error. The component simply doesn't appear, and the rest of the column flows up around the gap. This is what produces "Generate button at the top of an empty left column" Spaces.
|
||||
- **Stale releases.** Many Hub custom components were last published 1–2 years ago. The Gradio frontend has changed since. Check the package's release date against the Gradio version you're targeting; if there's a multi-major-version gap, expect breakage.
|
||||
- **Mismatched param shapes.** The component's Python signature accepts a parameter the Svelte side no longer reads. Your `disable_edit_boxes=True` does nothing, or worse, throws on the JS side and the whole component fails to mount.
|
||||
|
||||
Discipline before committing to a Hub component:
|
||||
|
||||
1. **Check the package's last release date** (PyPI page or `pip index versions <pkg>`). If it's older than the Gradio release in your `sdk_version`, treat it as suspect.
|
||||
2. **Smoke-test the component in isolation** — a five-line Gradio app with just that one component, locally, *before* integrating it into the full app. If it renders, fine. If it's missing or blank, you've caught the breakage cheaply.
|
||||
3. **When a Hub component fails to render, don't iterate on its kwargs.** Drop to either (a) split stock components (e.g. `gr.Image` for upload + a sibling widget for the box coords) or (b) rung 3 (custom HTML/JS via `gr.HTML`). Twiddling `disable_edit_boxes` / `use_default_label` / `sources` is not going to bring a Svelte component back from a JS-side mount failure.
|
||||
|
||||
The same discipline applies to less obviously "custom" components if they were added recently — `gr.ImageSlider`, for example, can render unexpectedly when paired with a custom component on the same page.
|
||||
|
||||
## When creative mode is the right call
|
||||
|
||||
Reach for it when the user's *natural* input is structurally outside what stock components express:
|
||||
|
||||
- **Spatial input on top of media.** Drawing arrows on a frame, painting strokes on an image, dropping points along a trajectory, selecting irregular regions, drawing curves.
|
||||
- **Multi-shape annotation.** Source-and-destination box pairs, multiple labeled regions, ordered sequences of points/boxes that are semantically distinct.
|
||||
- **Continuous-with-snapping controls.** 3D rotation gizmos, dial/wheel controls, timeline scrubbers — anything where a slider would technically work but feel wrong.
|
||||
- **Composite controls bound together.** A canvas plus a color picker plus a brush-size dial that all feed the same structured input, where binding three separate components and reasoning about their joint state is uglier than rolling one widget.
|
||||
- **Live preview that depends on multiple inputs.** Something the user wants to see *immediately* as they manipulate, where a server roundtrip per change is too slow.
|
||||
|
||||
If the input is "a number," "a string," or "an image," you don't need this. Don't build a custom canvas because it would look cool — build it because the LoRA's input shape demands it.
|
||||
|
||||
## The Gradio primitives
|
||||
|
||||
Before reading the patterns below, it's worth re-checking the current Gradio docs for anything that landed recently:
|
||||
|
||||
- `gr.HTML` — https://www.gradio.app/docs/gradio/html
|
||||
- Custom components — https://www.gradio.app/guides/custom-components-in-five-minutes
|
||||
- `Blocks.launch(head=, css=)` — https://www.gradio.app/docs/gradio/blocks#blocks-launch
|
||||
|
||||
WebFetch these if you're unsure about a signature. The custom-HTML surface area evolves and lagging on it produces Spaces that "work" in stale ways.
|
||||
|
||||
The primitives that creative mode is built from:
|
||||
|
||||
### `gr.HTML` for arbitrary markup
|
||||
|
||||
Drop any HTML into the page. The block becomes a regular Gradio component, but its content is whatever you write. You're responsible for everything inside it: layout, styling, interactivity.
|
||||
|
||||
```python
|
||||
gr.HTML("""
|
||||
<div id="my-widget" style="...">
|
||||
<canvas id="my-canvas" width="512" height="512"></canvas>
|
||||
<div id="my-status"></div>
|
||||
</div>
|
||||
""")
|
||||
```
|
||||
|
||||
### `demo.launch(head=..., css=...)`
|
||||
|
||||
Inject `<script>` and `<style>` tags into the page `<head>`. This is how you load external JS libraries (Three.js, p5, fabric, anime.js, …) or define page-wide CSS that needs to be in `<head>` rather than inline.
|
||||
|
||||
```python
|
||||
head = '<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>'
|
||||
css = '.fillable {max-width: 1200px !important}'
|
||||
demo.launch(head=head, css=css)
|
||||
```
|
||||
|
||||
Use the highest-quality CDN you can (cdnjs, jsdelivr, unpkg). Pin the version (`/three.js/r128/`, not `/three.js/latest/`). Loading from the LoRA author's personal server is a recipe for the Space breaking when their server moves.
|
||||
|
||||
### `elem_id` and `elem_classes` for addressing
|
||||
|
||||
Every Gradio component accepts `elem_id="..."` and `elem_classes=[...]`. JS uses these to find the rendered DOM nodes:
|
||||
|
||||
```python
|
||||
prompt_box = gr.Textbox(elem_id="my-prompt", elem_classes=["hidden-input"])
|
||||
```
|
||||
|
||||
```javascript
|
||||
const promptBox = document.getElementById("my-prompt");
|
||||
// Note: the Gradio component wraps an <input> or <textarea>;
|
||||
// you usually want the inner element:
|
||||
const inner = promptBox.querySelector("input, textarea");
|
||||
```
|
||||
|
||||
### Two ways JS pushes state into Python
|
||||
|
||||
This is the part that trips people up. Pick whichever fits your case and stick with it for the whole widget — mixing them in one app is confusing.
|
||||
|
||||
**Approach A — Hidden Gradio input + DOM event dispatch.** Define a hidden `gr.Textbox` (or `gr.JSON`, `gr.Number`, etc.). From JS, set its inner `<input>`/`<textarea>` value and dispatch synthetic `input` and `change` events so Gradio's reactivity fires.
|
||||
|
||||
```python
|
||||
state_json = gr.Textbox(value="{}", elem_id="state-json", visible=False)
|
||||
```
|
||||
|
||||
```javascript
|
||||
function setGradioValue(elemId, value) {
|
||||
const container = document.getElementById(elemId);
|
||||
if (!container) return;
|
||||
const el = container.querySelector("input, textarea");
|
||||
if (!el) return;
|
||||
const proto = el.tagName === "TEXTAREA"
|
||||
? HTMLTextAreaElement.prototype
|
||||
: HTMLInputElement.prototype;
|
||||
const setter = Object.getOwnPropertyDescriptor(proto, "value").set;
|
||||
setter.call(el, value);
|
||||
el.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
el.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
}
|
||||
|
||||
setGradioValue("state-json", JSON.stringify(myState));
|
||||
```
|
||||
|
||||
The native setter dance (`Object.getOwnPropertyDescriptor(...).set.call(el, value)`) is necessary because React intercepts plain `el.value = ...` assignments and they won't trigger a re-render. The DOM event dispatch is what makes Gradio see the change.
|
||||
|
||||
This approach works in any Gradio version, with any component type. Downsides: lots of glue, fragile to DOM structure changes, easy to write race conditions.
|
||||
|
||||
**Approach B — `gr.HTML` subclass with `html_template`/`js_on_load`.** Newer Gradio supports subclassing `gr.HTML` and providing custom props plus a `js_on_load` script. The script gets a `props` object with custom values and a `trigger()` function for emitting events back to Python. State binding is value-based (Gradio reads the `value` prop just like any other component).
|
||||
|
||||
```python
|
||||
class PointPicker(gr.HTML):
|
||||
def __init__(self, value=None, image_url=None, **kwargs):
|
||||
super().__init__(
|
||||
value=value or {"points": []},
|
||||
html_template="<canvas id='pp-canvas' width='512' height='512'></canvas>",
|
||||
js_on_load="""
|
||||
const canvas = document.getElementById('pp-canvas');
|
||||
canvas.addEventListener('click', (e) => {
|
||||
const rect = canvas.getBoundingClientRect();
|
||||
const x = (e.clientX - rect.left) / rect.width;
|
||||
const y = (e.clientY - rect.top) / rect.height;
|
||||
props.value = {points: [...(props.value?.points || []), {x, y}]};
|
||||
trigger('change', props.value);
|
||||
});
|
||||
""",
|
||||
image_url=image_url,
|
||||
**kwargs,
|
||||
)
|
||||
```
|
||||
|
||||
Cleaner, value-based, no hidden inputs. Downsides: requires a Gradio version recent enough to support the subclass shape, and the JS lives inside a Python string, which IDEs don't lint.
|
||||
|
||||
**Picking between A and B:** if you're starting fresh and on current Gradio, prefer B — the binding semantics are saner. Reach for A when you need to integrate with already-existing Gradio components (sliders that should mirror the canvas state, prompt boxes the canvas writes into) or when the UI shell is large enough that putting it inside a `gr.HTML` subclass is awkward.
|
||||
|
||||
### Two ways Python pushes state into JS
|
||||
|
||||
- **For B (subclass):** return `gr.update(prop=value)` from an event handler. The widget's polling loop sees the prop change and reacts.
|
||||
- **For A (hidden input):** write to a hidden Gradio Textbox from your event handler. JS polls the Textbox value (or hooks into Gradio's mutation observer) and reacts.
|
||||
|
||||
Either way, expect a small polling interval (50–200ms is typical). Don't poll faster — it will dominate CPU on weaker machines.
|
||||
|
||||
### `js=` on event handlers
|
||||
|
||||
For tiny JS-only transforms (e.g. "scroll to result on click"), event handlers accept a `js=` arg that runs in the browser without a server roundtrip:
|
||||
|
||||
```python
|
||||
btn.click(fn=infer, inputs=[...], outputs=[...], js="() => { window.scrollTo(0, 0); }")
|
||||
```
|
||||
|
||||
Useful for UI polish (scroll, focus, show/hide loaders), not for state management.
|
||||
|
||||
## The communication contract
|
||||
|
||||
The single most important discipline in creative mode: **define the JSON shape that flows between JS and Python, and treat it as the API.**
|
||||
|
||||
Write it down at the top of `app.py` as a comment, even one line. Pick names that survive translation between camelCase JS and snake_case Python (or commit to one and convert at the boundary):
|
||||
|
||||
```python
|
||||
# State shape on the JS↔Python wire:
|
||||
# {
|
||||
# "src": {"x1": float, "y1": float, "x2": float, "y2": float} | null,
|
||||
# "dst": {"x1": float, "y1": float, "x2": float, "y2": float} | null,
|
||||
# "label": string | null
|
||||
# }
|
||||
```
|
||||
|
||||
Once it's written down, both sides have a target to conform to and you'll catch shape drift early.
|
||||
|
||||
Common shape archetypes — none of these are required, just illustrative:
|
||||
|
||||
- **Point set:** `{"points": [{"x": 0.3, "y": 0.4}, ...]}`
|
||||
- **Stroke set:** `{"strokes": [{"color": "#ff0", "size": 12, "points": [...]}, ...]}`
|
||||
- **Regions:** `{"regions": [{"label": "subject", "bbox": {...}}, ...]}`
|
||||
- **Transform:** `{"azimuth": 45, "elevation": 0, "distance": 1.0}`
|
||||
- **Trajectory with time:** `{"keyframes": [{"t": 0.0, "x": ...}, {"t": 0.5, ...}]}`
|
||||
|
||||
All coordinates normalized to `[0, 1]` is almost always the right call — it survives image resizing without rescaling math.
|
||||
|
||||
## External libraries
|
||||
|
||||
Use what helps. Stay light when you can.
|
||||
|
||||
- **Pure canvas + DOM** is enough for: drawing rectangles/points/lines, brush strokes, drag-and-drop handles, image overlays.
|
||||
- **SVG** for: vector overlays, especially when the marks need to scale crisply or be styled with CSS.
|
||||
- **Three.js** for: 3D rotation gizmos, depth visualizations, anything where you need WebGL rendering of a small scene. Pulled in via `head=` script tag.
|
||||
- **fabric.js / paper.js / konva** for: complex shape-on-canvas interactions when raw canvas is getting ugly. Worth it once you've added more than ~200 lines of canvas glue.
|
||||
- **p5.js** for: rapid prototyping of generative-art-style canvases. Heavier than necessary for production.
|
||||
|
||||
When in doubt, raw canvas. Pulling in fabric for a single rectangle is overkill; pulling in three.js because you wanted "fancier sliders" is a smell.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
These bite repeatedly. Read before writing.
|
||||
|
||||
- **Silent mount failure of Hub custom components.** Covered in detail above. Worth restating because it's the failure mode that the API smoke-test cannot catch: the component imports, the page loads, and the widget is just absent from the DOM. Always verify in a browser; never trust "the build went green" as proof a Hub custom component renders.
|
||||
|
||||
- **Init race with Gradio mount.** The `js_on_load` (B) or a top-level `<script>` injected via `head` may run before Gradio has rendered the DOM nodes you're trying to hook into. Guard with a short `setTimeout` or wait for the element to exist:
|
||||
```javascript
|
||||
function init() {
|
||||
const el = document.getElementById("my-widget");
|
||||
if (!el) { setTimeout(init, 50); return; }
|
||||
// ... do work
|
||||
}
|
||||
init();
|
||||
```
|
||||
|
||||
- **Double init.** If the user navigates between tabs or Gradio re-renders, your init can fire twice. Stash a flag: `if (window.__myWidgetInited) return; window.__myWidgetInited = true;`.
|
||||
|
||||
- **Base64 image transfer cost.** Sending a full-resolution image as a base64 string through a hidden Textbox is fine for previews but punishingly slow for 4K images. Downscale on the JS side before stuffing into state, or pass the image through a real `gr.Image` component and only send the *interaction state* (boxes, points) through the hidden channel.
|
||||
|
||||
- **File inputs don't roundtrip through hidden Textboxes.** A hidden `gr.File` won't accept arbitrary JS-set values. If the user uploads via a custom `<input type="file">`, you have two options: (a) read the file in JS, base64-encode, and write to a hidden Textbox; (b) wire the custom file input to programmatically click a real `gr.File` (`document.querySelector("#real-file input").click()` is brittle but works).
|
||||
|
||||
- **Polling cadence.** 100ms is a reasonable default; 50ms feels snappy but burns CPU on weaker machines; 500ms feels laggy. Don't poll inside `requestAnimationFrame` for state sync — that's 60Hz and overkill.
|
||||
|
||||
- **Mobile / touch.** If the widget lives in a Space people will open on their phones, handle `touchstart`/`touchmove`/`touchend` alongside mouse events. The two-event-system thing is annoying but unavoidable.
|
||||
|
||||
- **ZeroGPU duration when payloads grow.** If the custom UI lets users submit large state (many points, big strokes, multiple boxes), the server-side processing time can grow with input size. Re-check `@spaces.GPU(duration=...)` against worst-case payloads.
|
||||
|
||||
- **Custom run buttons.** If you build your own "Run" button in HTML and have it trigger a real Gradio button via `.click()`, make sure state-sync (write to hidden inputs) happens *before* the click, with enough delay for events to propagate (`setTimeout(() => realBtn.click(), 50)` is the usual fix).
|
||||
|
||||
- **CSS isolation.** Spaces inject their own CSS. If your widget styles get clobbered, increase specificity (`#my-widget .foo` instead of `.foo`) or use `!important` sparingly. Don't fight it with inline styles for everything — that gets unreadable fast.
|
||||
|
||||
## Smoke-test caveat — applies to rung 2 AND rung 3
|
||||
|
||||
`gradio info` / `gradio predict` (Phase 6 of `SKILL.md`) only exercise the Python endpoint. They tell you nothing about what actually renders. For any Space that uses a Hub custom component (rung 2) or custom HTML/JS (rung 3), the Python side can be perfectly correct *and* the user can see a broken UI.
|
||||
|
||||
The two distinct failure modes:
|
||||
|
||||
- **Rung 2 silent mount failure.** Hub custom component imports cleanly, gets a slot in the Gradio config, and just doesn't appear in the rendered DOM. You see the components around it but the widget itself is gone — leaving e.g. a Generate button at the top of an otherwise empty column. No error anywhere. See "Hub custom components are fragile" above.
|
||||
- **Rung 3 broken JS.** Server-side green, but `js_on_load` errored, or a CDN script failed to load, or your event handlers never bound. The widget mounts but doesn't do anything when clicked.
|
||||
|
||||
After the API smoke-test in Phase 6 passes, **open the Space URL in a browser and verify both that every component is visible and that one full interaction (upload → click Generate → see result) works** before sharing. This is not optional for these Spaces — it's the only check that catches the failure modes above.
|
||||
|
||||
If the user is in an environment where you can't drive a browser yourself, ask them to open the Space URL and confirm the upload zone is visible and accepts an image before declaring the Space done. The cost of asking is one extra message; the cost of skipping is shipping a Space that looks empty.
|
||||
|
||||
## Real-world examples
|
||||
|
||||
These two Spaces use the patterns above for very different LoRAs. They're useful as concrete proof the patterns work, not as templates to copy:
|
||||
|
||||
- **3D camera control via Three.js** (Approach B, `gr.HTML` subclass): https://huggingface.co/spaces/multimodalart/qwen-image-multiple-angles-3d-camera
|
||||
- **Bounding-box drag/resize via canvas** (Approach A, hidden inputs + custom Run button): https://huggingface.co/spaces/linoyts/FLUX.2-klein-Move
|
||||
|
||||
A short look at each is worth it before designing a new widget — they show what "production polish" feels like in this register (snap-to-nearest animations, status overlays, cursor changes on hover, mobile-touch handling). But your widget will look different, because your LoRA wants different inputs.
|
||||
@@ -0,0 +1,124 @@
|
||||
# Tasks: per-task baseline UI patterns
|
||||
|
||||
This file describes the *baseline* UI shape for each task category. Use it after the LoRA's task is identified, as a starting skeleton. Then read `adapting-to-the-lora.md` to shape the actual UI to the specific LoRA — the baseline is rarely the right final answer.
|
||||
|
||||
The five tasks: text-to-image, image-to-image, text-to-video, image-to-video, video-to-video.
|
||||
|
||||
## Common to all tasks
|
||||
|
||||
- Layout: two-column `gr.Row` with `equal_height=True`. Inputs on the left, outputs on the right.
|
||||
- One primary `gr.Button("Generate", variant="primary", size="lg")`. No secondary buttons unless they do meaningfully different things.
|
||||
- An `Advanced` accordion (`gr.Accordion("Advanced", open=False)`) for power-user controls that most users will never touch (seed, randomize seed, advanced sampler params).
|
||||
- Always include a seed control with a "Randomize seed" checkbox, and return the actually-used seed alongside the result so users can reproduce.
|
||||
- Wire up `prompt.submit` as well as button click for text inputs, so Enter works.
|
||||
- Use `gr.Progress(track_tqdm=True)` so diffusers' internal progress bar surfaces.
|
||||
|
||||
## Text-to-image (T2I)
|
||||
|
||||
**Inputs:** prompt (`gr.Textbox`, `lines=2`, with example placeholder). Aspect ratio or resolution control. Optional: negative prompt for models that support it.
|
||||
|
||||
**Outputs:** `gr.Image` (or `gr.Gallery` if returning more than one).
|
||||
|
||||
**Standard advanced controls:** seed, randomize seed, num inference steps, guidance scale. Hide steps and guidance entirely for few-step LoRAs (Lightning, Turbo, schnell).
|
||||
|
||||
**Aspect ratio handling:** offer a dropdown of common ratios with width/height auto-derived. Snap dimensions to the model's native bucket size (16 for most diffusion transformers, 8 for older UNet-based). Show width/height as read-only display.
|
||||
|
||||
**Examples:** lift example prompts from the LoRA model card into a `gr.Examples` block. Use `cache_examples=True, cache_mode="lazy"` so caching defers to first click instead of failing at build time on ZeroGPU.
|
||||
|
||||
## Image-to-image (I2I)
|
||||
|
||||
**Inputs:** input image (`gr.Image(type="pil")` or `numpy` depending on what your processing wants), instruction or prompt (`gr.Textbox`).
|
||||
|
||||
**Outputs:** `gr.Image`. For edit LoRAs, consider `gr.ImageSlider` (built-in) for before/after comparison instead of a separate image.
|
||||
|
||||
**Resolution handling:** for instruction-edit pipelines (Qwen-Image-Edit, Flux Kontext, Flux.2 Klein), resize the input to the nearest multiple of the model's bucket size, preserving aspect. Don't crop unless the LoRA expects a specific aspect.
|
||||
|
||||
**Validation:** raise `gr.Error("Please upload an image first.")` when the input is empty. Consider disabling Generate until an image is loaded (`run_button.interactive = False`, flip on `input_image.change`).
|
||||
|
||||
**Sub-task variants change a lot here.** Read `adapting-to-the-lora.md` — relighting, face swap, object move, style transfer, instruction edits, inpainting all live under "I2I" but call for different UIs.
|
||||
|
||||
## Text-to-video (T2V)
|
||||
|
||||
**Inputs:** prompt. Duration slider (1–10 seconds typical, depending on the model's max). Resolution/aspect picker.
|
||||
|
||||
**Outputs:** `gr.Video(autoplay=True)`. Set `format="mp4"` and pick fps explicitly (24 is a safe default; some models prefer 16 or 30).
|
||||
|
||||
**Standard advanced controls:** seed, randomize seed, fps. Steps usually locked for distilled video models.
|
||||
|
||||
**Duration awareness:** set `@spaces.GPU(duration=...)` to comfortably exceed expected generation time. For 5-second 720p video, 180+ seconds of GPU time is realistic. Tell the user in the UI that generation takes a while ("Generating a 5s video takes about 2 minutes").
|
||||
|
||||
**Frame count math:** most video diffusion models want frame counts that are `8k+1` or similar. Compute `num_frames` from `duration * fps` and round to the nearest valid value rather than passing arbitrary frames. The base-model reference file says what's valid for each model.
|
||||
|
||||
## Image-to-video (I2V)
|
||||
|
||||
**Inputs:** input image (the first frame, or a stylistic reference depending on the LoRA), prompt describing the motion. Duration.
|
||||
|
||||
**Outputs:** `gr.Video`.
|
||||
|
||||
**Aspect ratio:** auto-detect from the input image and snap to the model's nearest bucket. Show the chosen resolution as info text.
|
||||
|
||||
**Variants:** some I2V LoRAs use the input image as the literal first frame; others use it as a stylistic reference and generate a new first frame from the prompt. The model card usually says which. The UI for both is similar; the difference is in how `image=` is passed to the pipeline and whether a "use as first frame" toggle makes sense.
|
||||
|
||||
## Video-to-video (V2V)
|
||||
|
||||
**Inputs:** at minimum, a source video. Almost always also a prompt. Often additional inputs depending on what the LoRA does (reference image for appearance, mask, control video, etc.).
|
||||
|
||||
**Outputs:** `gr.Video`. For LoRAs that do preprocessing on the input (pose extraction, depth estimation, padding), show the preprocessed intermediate as a second smaller video alongside the result, so the user sees what the model actually saw.
|
||||
|
||||
**This is where adaptation matters most.** "V2V" alone tells you almost nothing about the UI. Pose-control, depth-control, canny-control, outpainting, inpainting, style transfer, motion transfer, frame interpolation, and upscaling are all V2V and all need different UIs. Always read `adapting-to-the-lora.md` and the per-base-model file before designing.
|
||||
|
||||
**Common patterns:**
|
||||
|
||||
- Preprocessing preview: a small `gr.Video(height=240)` showing the extracted pose / depth / canny / padded video. Update it on input change so the user sees the preprocessed result before clicking Generate.
|
||||
- Two-input layout for motion-transfer LoRAs: source video + appearance image, clearly labeled.
|
||||
- Aspect-ratio picker only when the LoRA actually changes aspect (outpainting). For pose/depth/canny control, output aspect matches input.
|
||||
|
||||
## Picking components
|
||||
|
||||
Walk this ladder in order. Stop at the first rung that fits the LoRA's input shape.
|
||||
|
||||
**1. Stock Gradio components.** First choice almost always:
|
||||
|
||||
- `gr.ImageSlider` — built-in before/after comparison for edit LoRAs.
|
||||
- `gr.ImageEditor` — upload + paint on top of an image. The right pick for any LoRA whose "input shape" is "a region of the image" expressed by painting — object removal trained on red-highlighted regions, relight trained on colored brush strokes, scribble-conditioned edits. Constrain the brush with `gr.Brush(default_color="#ff0000", colors=["#ff0000"])` so the user can only paint in the color the LoRA was trained on; the editor returns `{"background", "layers", "composite"}` and the `composite` is what you feed the pipeline. Used in production by `linoyts/QIE-2509-Object-Remover-Bbox-v3` (qie-2509-object-remover) — don't reach for `gradio_image_annotation` for these tasks just because the LoRA was trained "with bboxes"; the user-facing shape is a painted region, not a literal box.
|
||||
- `@gr.render` — UI that changes shape based on input (e.g. show extra controls only when an input is uploaded).
|
||||
- `gr.Examples` — clickable example inputs. Almost always worth including. Lift from the LoRA's model card.
|
||||
- `gr.BrowserState` — persist user preferences (preferred aspect ratio, last seed, etc.) across sessions.
|
||||
- `gr.DeepLinkButton` — share a specific generation as a URL.
|
||||
|
||||
**2. Hub custom components.** A `pip install` and an import, no JS to maintain:
|
||||
|
||||
- `gradio_image_annotation` — bbox/point annotation on top of an image. Right when the LoRA literally needs box *coordinates* as structured input (e.g. drag-and-drop "move from box A to box B" LoRAs, region-tagged edits). Wrong when the LoRA wants a painted region — use `gr.ImageEditor` instead.
|
||||
- `gradio_imageslider` — alternative before/after slider with extra controls.
|
||||
- `gradio_modal` — modal dialogs.
|
||||
- `gradio_rangeslider` — dual-handle range slider.
|
||||
|
||||
Browse the rest at https://www.gradio.app/custom-components/gallery before going further down the ladder.
|
||||
|
||||
**3. Creative mode (custom HTML/JS).** When stock and Hub custom components both come up short — point sets, strokes, trajectories, region selections with metadata, 3D rotation gizmos, timeline scrubbers, anything where the user manipulates a thing on top of media. See `creative-mode.md` for the Gradio primitives, the JS↔Python communication contract, and the pitfalls. Don't skip rung 2 to get here — `gradio_image_annotation` already covers a lot of what looks like it needs custom HTML.
|
||||
|
||||
Themes: default to `gr.themes.Citrus()`.
|
||||
|
||||
Before defaulting to a plain component or guessing at a custom one, web-fetch the current Gradio docs at https://www.gradio.app/docs.
|
||||
|
||||
## Gradio 6.x gotchas
|
||||
|
||||
The current `sdk_version` (6.x at time of writing — verify with `pip index versions gradio`) changed a few things that older recipes get wrong. The failures are easy to miss because they happen at the Space's first import, not when you write the file locally.
|
||||
|
||||
- **`theme=` and `css=` moved from `gr.Blocks(...)` to `demo.launch(...)`.** Passing them to `Blocks` now emits a deprecation warning and the styling silently doesn't apply. Always:
|
||||
|
||||
```python
|
||||
with gr.Blocks(title="...") as demo:
|
||||
...
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch(theme=gr.themes.Citrus(), css=CSS)
|
||||
```
|
||||
|
||||
Spaces runs `app.py` as `__main__`, so the `launch()` call executes.
|
||||
|
||||
- **Some component kwargs were removed.** `gr.Image` no longer accepts `show_download_button` (the same change affects a handful of other components). The Space fails at import with `TypeError: __init__() got an unexpected keyword argument 'show_download_button'` — not surfaced until the container actually starts. When in doubt, web-fetch the current docs for the specific component before passing non-obvious kwargs.
|
||||
|
||||
- **The Gradio version the Space runs at is set by `sdk_version:` in the README YAML, *not* by `requirements.txt`.** Pinning `gradio` in `requirements.txt` is at best ignored and at worst causes a runtime mismatch; set the version once, in the README, and write `app.py` against that version.
|
||||
|
||||
If the first build fails on a `TypeError` or signature mismatch in a Gradio component, this is the most common cause — read `/logs/container` (build) or `/logs/run` (runtime), look at the line in `app.py`, and check the current component signature.
|
||||
@@ -0,0 +1,196 @@
|
||||
# ZeroGPU and publishing
|
||||
|
||||
ZeroGPU is the default hardware target. It's a shared serverless GPU pool: GPU is allocated on each request, held for the duration of a `@spaces.GPU` function call, and released. The key implications for the demo's code shape:
|
||||
|
||||
## ZeroGPU rules
|
||||
|
||||
**Models go on `cuda` at module level, not lazy-loaded inside the GPU function.**
|
||||
|
||||
```python
|
||||
import torch
|
||||
import spaces
|
||||
from diffusers import QwenImagePipeline
|
||||
|
||||
pipe = QwenImagePipeline.from_pretrained("Qwen/Qwen-Image", torch_dtype=torch.bfloat16)
|
||||
pipe.to("cuda")
|
||||
pipe.load_lora_weights("user/my-lora")
|
||||
|
||||
@spaces.GPU(duration=60)
|
||||
def generate(prompt):
|
||||
return pipe(prompt).images[0]
|
||||
```
|
||||
|
||||
ZeroGPU uses a CUDA emulation mode that lets `.to("cuda")` work outside `@spaces.GPU` functions during startup. Module-level placement is significantly faster than deferred placement because CUDA transfers are optimized for startup-time placement. Lazy-loading inside `@spaces.GPU` is discouraged.
|
||||
|
||||
**The `@spaces.GPU` decorator wraps the function that needs the GPU.**
|
||||
|
||||
The default duration is 60 seconds. Set it higher for longer tasks: `@spaces.GPU(duration=120)` or `@spaces.GPU(duration=300)`. Set it lower if the task reliably finishes faster — lower duration means higher queue priority. For tasks where duration varies with input, pass a function: `@spaces.GPU(duration=lambda *args: ...)`.
|
||||
|
||||
**GPU size: `large` (default, 48GB VRAM) or `xlarge` (96GB, full Blackwell).** Specify with `@spaces.GPU(size="xlarge")` when a single inference needs more than 48GB — large base video models, high-resolution generation, or heavy multi-stage pipelines. `xlarge` consumes 2× the daily quota per second and queues longer, so only reach for it when `large` actually OOMs.
|
||||
|
||||
Typical durations:
|
||||
|
||||
- Few-step T2I (4-8 steps): 30-60 seconds.
|
||||
- Standard T2I (20-50 steps): 60-90 seconds.
|
||||
- I2I / instruction edits: 60-90 seconds.
|
||||
- Short video (3-5 seconds): 120-180 seconds.
|
||||
- Long video / multi-stage: 180-300 seconds.
|
||||
|
||||
**Don't use `torch.compile`.** It's incompatible with ZeroGPU's process model (the GPU process forks per call). The decorator is a no-op outside ZeroGPU, so `pipe(...)` runs uncompiled in both environments.
|
||||
|
||||
**Validate inputs at the top of the GPU function.** Raising `gr.Error(...)` inside a `@spaces.GPU` function still consumes some GPU quota for the allocation. Validate before doing real work, or move validation into a non-decorated function called by the UI.
|
||||
|
||||
**Use `cache_examples=True` with `cache_mode="lazy"` on `gr.Examples`.** Plain `cache_examples=True` runs the function at build time, before a GPU is allocated, and will fail. `cache_mode="lazy"` defers caching to the first time a user clicks each example — the GPU is available, and subsequent clicks return the cached result instantly.
|
||||
|
||||
**Match `gr.Examples(fn=…)` to the function signature.** A click calls `fn` with only the `inputs=` components, positionally; extra parameters without defaults raise `TypeError: missing N required positional arguments`. The run event wires every component, so this only breaks on the example click — not manual or smoke tests. Fix: default the extra params, or pass the full `inputs=` list with full example rows.
|
||||
|
||||
**Don't initialize CUDA from outside the controlled paths.** `pipe.to("cuda")` is fine (CUDA emulation handles it). Calling `torch.cuda.something()` directly at module level can break the process model — when in doubt, do it inside the GPU function or skip it.
|
||||
|
||||
**ZeroGPU requires PRO/Team/Enterprise.** A free-tier user can create a Space with `hardware: zero-a10g` in the README, but it'll fall back to CPU. If the user isn't on a supporting plan, mention this and point them at two paths: upgrade to PRO (unlocks ZeroGPU directly), or apply for a [community GPU grant](https://huggingface.co/docs/hub/spaces-gpus#community-gpu-grants) (request free paid GPU hardware via the Space's hardware settings, subject to approval).
|
||||
|
||||
## HF Hub patterns
|
||||
|
||||
### Authentication — check first, ask only if needed
|
||||
|
||||
Don't ask for a token reflexively. Check whether the user is already authenticated, and only prompt if there's no usable session.
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi, get_token
|
||||
|
||||
def resolve_auth():
|
||||
"""Returns (token, username) or (None, None) if no usable auth."""
|
||||
cached = get_token() # picks up HF_TOKEN env var or cached CLI login
|
||||
if not cached:
|
||||
return None, None
|
||||
try:
|
||||
info = HfApi().whoami(token=cached)
|
||||
return cached, info["name"]
|
||||
except Exception:
|
||||
return None, None # token exists but is invalid/expired
|
||||
```
|
||||
|
||||
Decision tree:
|
||||
|
||||
- **User already authenticated and the LoRA repo is public**: use the existing token. Confirm the username with the user before publishing ("I'll publish to `{username}` — confirm?").
|
||||
- **User already authenticated and the LoRA repo is private**: try `api.repo_info(repo_id, token=cached)`. If it succeeds, the existing token has the right scope — proceed. If it fails (token doesn't have access to that repo), ask for a token with broader access.
|
||||
- **No cached token**: ask the user. One ask, with the explanation: "I need a Hugging Face access token with **write** scope. Create one at https://huggingface.co/settings/tokens. Paste it here." The same token will be reused for publishing.
|
||||
|
||||
The default flow on a Hugging Face Space, in a logged-in user's local environment with `huggingface-cli login`, or in any environment with `HF_TOKEN` set, will *not* require asking the user for a token. Asking is the fallback, not the default.
|
||||
|
||||
### Reading the LoRA repo
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi, ModelCard
|
||||
|
||||
api = HfApi(token=hf_token) # token may be None for public repos
|
||||
|
||||
try:
|
||||
info = api.repo_info(repo_id) # 401/403 → private/gated; need token
|
||||
except Exception as e:
|
||||
# Handle private/gated repo case
|
||||
...
|
||||
|
||||
files = api.list_repo_files(repo_id)
|
||||
card = ModelCard.load(repo_id, token=hf_token)
|
||||
base_model = card.data.get("base_model")
|
||||
pipeline_tag = card.data.get("pipeline_tag")
|
||||
readme_text = card.text
|
||||
```
|
||||
|
||||
### Picking the LoRA weights file
|
||||
|
||||
Many LoRA repos contain a single `.safetensors` file and the choice is trivial. But some contain several — variants (4-step / 8-step distillations, FP16 vs BF16, different ranks), training-history checkpoints (`epoch-10.safetensors`, `epoch-20.safetensors`), or genuinely different methods (`lora.safetensors` + `lora_dora.safetensors`). Pick in this order, stopping at the first match:
|
||||
|
||||
1. **The README recommends a specific file.** This is the strongest signal — if the author bothered to name a file, that's the choice. Look for filenames inside inference snippets (especially `weight_name="..."` arguments), in "recommended" or "best" callouts, in comparison tables ranking variants, or in any prose like "use X for Y." If the README clearly points at one file, use it without asking.
|
||||
|
||||
2. **No README recommendation, and `pytorch_lora_weights.safetensors` exists at the repo root.** Use it. This is the diffusers convention and a safe default.
|
||||
|
||||
3. **Neither, but the multiple files look like training checkpoints** (filenames with patterns like `epoch-N`, `step-N`, `checkpoint-N`, or a numeric progression like `lora-1.safetensors`, `lora-2.safetensors`, `lora-3.safetensors`). Default to the highest-numbered / latest one, but mention the choice in the response so the user can override: "Repo has epoch-10, epoch-20, epoch-30; using epoch-30 — let me know if you want a different one."
|
||||
|
||||
4. **Otherwise** — files look like alternative variants (`*-4steps` vs `*-8steps`, `*-fp16` vs `*-bf16`, `lora` vs `lora_dora`), or names are opaque (`v2.safetensors`, `final.safetensors`, `output.safetensors`), or there's no clear "latest." Ask, with a one-line description of each option based on what the filenames suggest. Don't pick blindly — the wrong choice produces a working Space that's silently using the wrong weights.
|
||||
|
||||
This reasoning happens once, in Phase 1. The chosen filename is then passed to `load_lora_weights` via `weight_name="..."` in `app.py`.
|
||||
|
||||
### Loading a private LoRA in `app.py`
|
||||
|
||||
```python
|
||||
import os
|
||||
pipe.load_lora_weights("user/private-lora", token=os.environ["HF_TOKEN"])
|
||||
```
|
||||
|
||||
### Creating and publishing the Space
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi, SpaceHardware
|
||||
|
||||
api = HfApi(token=hf_token)
|
||||
username = api.whoami()["name"]
|
||||
repo_id = f"{username}/{space_name}"
|
||||
|
||||
api.create_repo(
|
||||
repo_id=repo_id,
|
||||
repo_type="space",
|
||||
space_sdk="gradio",
|
||||
space_hardware=SpaceHardware.ZERO_A10G,
|
||||
private=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
|
||||
# Set HF token as a Space secret if the LoRA or base model is private/gated
|
||||
api.add_space_secret(repo_id=repo_id, key="HF_TOKEN", value=hf_token)
|
||||
|
||||
# Upload files
|
||||
for path in ["app.py", "requirements.txt", "README.md"]:
|
||||
api.upload_file(
|
||||
path_or_fileobj=path,
|
||||
path_in_repo=path,
|
||||
repo_id=repo_id,
|
||||
repo_type="space",
|
||||
)
|
||||
```
|
||||
|
||||
The Space starts building automatically once files are pushed.
|
||||
|
||||
### `SpaceHardware.ZERO_A10G`
|
||||
|
||||
The string value is `"zero-a10g"`. This is a legacy name from when ZeroGPU ran on A10Gs; the actual hardware is NVIDIA RTX Pro 6000 Blackwell, but the identifier stuck. Both `SpaceHardware.ZERO_A10G` and the literal `"zero-a10g"` work. Prefer the enum for clarity.
|
||||
|
||||
If `create_repo` rejects the hardware (typically because the user isn't on PRO), retry without `space_hardware=`, set the README's `hardware: zero-a10g` anyway, and tell the user the Space will run on CPU until they either upgrade to PRO or apply for a [community GPU grant](https://huggingface.co/docs/hub/spaces-gpus#community-gpu-grants) (request form lives in the Space's hardware settings).
|
||||
|
||||
### Updating an existing Space
|
||||
|
||||
If the user already has a Space they want to update (rather than creating fresh), `create_repo` with `exist_ok=True` is a no-op on the existing repo. `upload_file` overwrites. Existing secrets and hardware settings are preserved. Don't delete and recreate the Space — they'll lose stars, comments, and any custom config.
|
||||
|
||||
## After publishing
|
||||
|
||||
The Space URL is `https://huggingface.co/spaces/{repo_id}`. Build logs are at `https://huggingface.co/spaces/{repo_id}/logs/container`. Runtime logs at `https://huggingface.co/spaces/{repo_id}/logs/run`.
|
||||
|
||||
When sharing the URL with the user:
|
||||
|
||||
- Note that the Space is private — they need to be logged in to view it.
|
||||
- Note that the build takes a few minutes the first time.
|
||||
- Offer to look at the logs if anything fails.
|
||||
- Don't add a long postamble — they want to click the link, not read more text.
|
||||
|
||||
**Confirm a redeploy is actually live before testing it.** An `app.py`-only push does **not** change the Space's reported `runtime.stage` — the old replica keeps serving "RUNNING" while the new build swaps in, so a `gradio_client` test can silently hit **stale code**. To be sure: push → `api.restart_space(repo)` → poll until the stage leaves and returns to RUNNING → grep the boot logs (`/logs/run`) for a unique `[VERSION] …` marker you printed at module scope → then test. Also set `demo.launch(show_error=True)` so `gradio_client` surfaces the real traceback instead of a generic `AppError`.
|
||||
|
||||
## Publish-time failures (before the build starts)
|
||||
|
||||
These happen during `create_repo` or `upload_file`, *before* the Space build pipeline runs. Diagnose by reading the exception, not the container logs (the container hasn't started yet).
|
||||
|
||||
- **`HfHubHTTPError: 400 Bad Request` from `https://huggingface.co/api/validate-yaml`.** The README's YAML frontmatter failed server-side validation. By far the most common cause is `short_description` exceeding the server's length cap (the cap isn't documented and may change; targeting ~60 characters keeps you well clear). Other causes include typos in field names (`hardware` vs `hardwre`), invalid color values in `colorFrom`/`colorTo`, an unrecognized `hardware` string, or a malformed `models:` list. Fix: open `README.md`, shorten `short_description`, double-check the other YAML fields, retry. If the user gave you a long description for the Space, put the long version in the README body below the YAML — that's the right home for prose.
|
||||
|
||||
- **`HfHubHTTPError: 403 Forbidden` on `create_repo` with `space_hardware="zero-a10g"`.** The user's account can't request ZeroGPU at creation time (typically because they're not on PRO/Team/Enterprise). Fix: retry `create_repo` without the `space_hardware` argument; keep `hardware: zero-a10g` in the README YAML. The Space gets created on CPU. Point the user at two paths to get off CPU: upgrade to PRO (auto-promotes the Space to ZeroGPU), or apply for a [community GPU grant](https://huggingface.co/docs/hub/spaces-gpus#community-gpu-grants) (request via the Space's hardware settings).
|
||||
|
||||
- **`HfHubHTTPError: 401/403` on `upload_file`.** Token lacks write scope. Fix: ask the user for a write-scoped token (or use a fine-grained token with write permission on this specific Space).
|
||||
|
||||
- **`RepositoryNotFoundError` on `upload_file` immediately after `create_repo`.** Race condition; very rare. Fix: small `time.sleep(1)` between create and upload, or retry the upload.
|
||||
|
||||
## Common build failures
|
||||
|
||||
- **`weight_name` mismatch in `load_lora_weights`.** The actual file in the repo is named differently. Fix: `api.list_repo_files(repo_id)` to find the real filename; pass `weight_name=` explicitly.
|
||||
- **Gated base model, no token.** The base model (e.g. `black-forest-labs/FLUX.1-dev`) requires accepting a license. Fix: ensure the user has accepted the license on the Hub, and the token is set as a Space secret.
|
||||
- **Diffusers version too old for the pipeline class.** The base model was released after the latest pinned diffusers. Fix: change `requirements.txt` from `diffusers` to `git+https://github.com/huggingface/diffusers`.
|
||||
- **CUDA OOM on first request.** The model is too big for the 48GB VRAM available on the default `large` size. Solutions, in order of preference: pick a smaller or quantized variant (FP8, smaller checkpoint); request `@spaces.GPU(size="xlarge")` to get the full 96GB (costs 2× quota and queues longer); enable model offloading (`pipe.enable_model_cpu_offload()` — conflicts with ZeroGPU's process model, last resort only).
|
||||
- **`cache_examples=True` failure.** Build-time GPU isn't available on ZeroGPU. Fix: add `cache_mode="lazy"` so caching happens on first user click instead of at build.
|
||||
- **Free-tier user, hardware not allocated.** Space falls back to CPU. The build succeeds but inference is unusably slow. Fix: user upgrades to PRO, or removes `hardware: zero-a10g` and lives with CPU.
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
name: huggingface-paper-publisher
|
||||
description: Publish and manage research papers on Hugging Face Hub. Supports creating paper pages, linking papers to models/datasets, claiming authorship, and generating professional markdown-based research articles.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Overview
|
||||
|
||||
Publish, manage, and link research papers on the Hugging Face Hub — arXiv indexing, model/dataset linking, authorship management, and professional markdown research-article generation.
|
||||
|
||||
## Integration with the HF Ecosystem
|
||||
|
||||
Paper Pages (index/discover), arXiv integration (auto-indexing from arXiv IDs), model/dataset linking via metadata, authorship verification, and a research-article template.
|
||||
|
||||
## Dependencies
|
||||
|
||||
The included `scripts/paper_manager.py` uses PEP 723 inline dependencies — prefer `uv run` over manual environment setup. Requires `HF_TOKEN` (write access) in the environment. **All paths in the commands below are relative to this skill's directory** — `cd` there first, or use the full path.
|
||||
|
||||
## Commands Reference
|
||||
|
||||
**Index a paper from arXiv** (or visit `https://huggingface.co/papers/{arxiv-id}` directly):
|
||||
```bash
|
||||
uv run scripts/paper_manager.py index --arxiv-id "2301.12345"
|
||||
uv run scripts/paper_manager.py check --arxiv-id "2301.12345" # check if already indexed
|
||||
```
|
||||
|
||||
**Link a paper to a model/dataset/space** — the Hub extracts the arXiv ID and adds an `arxiv:<PAPER_ID>` tag, making the repo discoverable from the Paper Page:
|
||||
```bash
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/repo-name" --repo-type "model|dataset|space" \
|
||||
--arxiv-id "2301.12345" [--arxiv-ids "id1,id2,id3"] \
|
||||
[--citation "Full citation text"] [--create-pr]
|
||||
```
|
||||
|
||||
**Claim authorship** (or manually: visit the paper page, click your name, "Claim authorship", await admin verification):
|
||||
```bash
|
||||
uv run scripts/paper_manager.py claim --arxiv-id "2301.12345" --email "your.email@institution.edu"
|
||||
uv run scripts/paper_manager.py check-authorship --arxiv-id "2301.12345"
|
||||
```
|
||||
|
||||
**Manage profile visibility** (or via account settings → Papers):
|
||||
```bash
|
||||
uv run scripts/paper_manager.py list-my-papers
|
||||
uv run scripts/paper_manager.py toggle-visibility --arxiv-id "2301.12345" --show true|false
|
||||
```
|
||||
|
||||
**Create a research article** from a template (`standard` — traditional structure; `modern` — Distill-inspired, dynamic TOC, responsive; `arxiv` — arXiv-style; `ml-report` — ML experiment report):
|
||||
```bash
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "standard|modern|arxiv|ml-report" --title "Paper Title" \
|
||||
[--authors "Author1, Author2"] [--abstract "Abstract text"] [--output "paper.md"]
|
||||
uv run scripts/paper_manager.py convert --input "paper.md" --output "paper.html" [--style "modern|classic"]
|
||||
```
|
||||
|
||||
**Search, inspect, validate:**
|
||||
```bash
|
||||
uv run scripts/paper_manager.py search --query "transformer attention"
|
||||
uv run scripts/paper_manager.py info --arxiv-id "2301.12345" --format "json"
|
||||
uv run scripts/paper_manager.py citation --arxiv-id "2301.12345" --format "bibtex"
|
||||
uv run scripts/paper_manager.py validate --repo-id "username/model-name" --repo-type "model"
|
||||
```
|
||||
|
||||
## Standard Paper Template Structure
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Your Paper Title
|
||||
authors: Jane Doe, John Smith
|
||||
affiliations: University X, Lab Y
|
||||
date: 2025-01-15
|
||||
arxiv: 2301.12345
|
||||
tags: [machine-learning, nlp, fine-tuning]
|
||||
---
|
||||
|
||||
# Abstract
|
||||
# 1. Introduction
|
||||
# 2. Related Work
|
||||
# 3. Methodology
|
||||
# 4. Experiments
|
||||
# 5. Results
|
||||
# 6. Discussion
|
||||
# 7. Conclusion
|
||||
# References
|
||||
```
|
||||
|
||||
The `modern` template adds a dynamic table of contents, responsive layout, code highlighting, interactive figures, LaTeX math, and citation/author-affiliation linking.
|
||||
|
||||
## YAML Metadata for Linked Cards
|
||||
|
||||
Model and dataset cards need proper frontmatter for the Hub to extract the arXiv tag correctly:
|
||||
|
||||
```yaml
|
||||
---
|
||||
language: [en]
|
||||
license: apache-2.0
|
||||
tags: [text-generation, transformers, llm]
|
||||
library_name: transformers
|
||||
---
|
||||
```
|
||||
|
||||
Followed by a `# Model Name` section referencing `[Our Paper](https://arxiv.org/abs/2301.12345)` and a `## Citation` section with a fenced `bibtex` block containing the standard `@article{...}` entry (author, title, journal, year).
|
||||
|
||||
Dataset cards follow the same pattern with `task_categories`/`size_categories` instead of `library_name`.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
**Publish new research:** create the article -> submit to arXiv externally -> `index` -> `link` to the model -> `claim` authorship.
|
||||
|
||||
**Link an existing paper to multiple repos:** `check` -> `index` if needed -> `link` once per model/dataset/space repo-id.
|
||||
|
||||
**Update a model with a paper reference:** `hf download user/model README.md` -> `link` with `--citation` — the script adds YAML metadata if missing, inserts the arXiv link, adds the formatted citation, and preserves existing content.
|
||||
|
||||
**Batch-link multiple papers to one repo:**
|
||||
```bash
|
||||
for arxiv_id in "2301.12345" "2302.67890"; do
|
||||
uv run scripts/paper_manager.py link --repo-id "username/model-name" --repo-type "model" --arxiv-id "$arxiv_id"
|
||||
done
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
Index papers as soon as published; keep citations consistent across related repos. Add YAML frontmatter with correct license and task tags to every card. Claim authorship with an institutional email and keep visibility settings current. Link papers to every relevant model/dataset/Space, with BibTeX citations. Use one template consistently within a project, and include code/data links.
|
||||
|
||||
## Error Handling & Troubleshooting
|
||||
|
||||
- **Paper not found**: arXiv ID not indexed yet — visit `hf.co/papers/{arxiv-id}` to trigger indexing.
|
||||
- **Permission denied**: `HF_TOKEN` lacks write access to the repository.
|
||||
- **Invalid YAML**: malformed frontmatter in the README.
|
||||
- **Authorship claim failed / not verified**: email doesn't match author records, or awaiting admin review.
|
||||
- **Already claimed**: another user has claimed authorship — contact HF support with proof if this is wrong.
|
||||
- **arXiv tag not appearing**: ensure the README includes a properly formatted arXiv URL.
|
||||
- **Rate limiting**: too many API requests in a short time — space out batch operations.
|
||||
|
||||
## Resources
|
||||
|
||||
- [Hugging Face Paper Pages](https://huggingface.co/papers)
|
||||
- [Model Cards Guide](https://huggingface.co/docs/hub/en/model-cards)
|
||||
- [Dataset Cards Guide](https://huggingface.co/docs/hub/en/datasets-cards)
|
||||
- [tfrere's research article template](https://huggingface.co/spaces/tfrere/research-article-template) — complements this skill for writing; use this skill to publish/link/manage authorship afterward.
|
||||
- [arXiv submission guide](https://arxiv.org/help/submit)
|
||||
@@ -0,0 +1,326 @@
|
||||
# Example Usage: HF Paper Publisher Skill
|
||||
|
||||
This document demonstrates common workflows for publishing research papers on Hugging Face Hub.
|
||||
|
||||
## Example 1: Index an Existing arXiv Paper
|
||||
|
||||
If you've already published a paper on arXiv and want to make it discoverable on Hugging Face:
|
||||
|
||||
```bash
|
||||
# Check if paper exists
|
||||
uv run scripts/paper_manager.py check --arxiv-id "2301.12345"
|
||||
|
||||
# Index the paper
|
||||
uv run scripts/paper_manager.py index --arxiv-id "2301.12345"
|
||||
|
||||
# Get paper information
|
||||
uv run scripts/paper_manager.py info --arxiv-id "2301.12345"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```json
|
||||
{
|
||||
"exists": true,
|
||||
"url": "https://huggingface.co/papers/2301.12345",
|
||||
"arxiv_id": "2301.12345",
|
||||
"arxiv_url": "https://arxiv.org/abs/2301.12345"
|
||||
}
|
||||
```
|
||||
|
||||
## Example 2: Link Paper to Your Model
|
||||
|
||||
After indexing a paper, link it to your model repository:
|
||||
|
||||
```bash
|
||||
# Link single paper
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/my-awesome-model" \
|
||||
--repo-type "model" \
|
||||
--arxiv-id "2301.12345"
|
||||
|
||||
# Link multiple papers
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/my-awesome-model" \
|
||||
--repo-type "model" \
|
||||
--arxiv-ids "2301.12345,2302.67890"
|
||||
```
|
||||
|
||||
This will:
|
||||
1. Download the model's README.md
|
||||
2. Add or update YAML frontmatter
|
||||
3. Insert paper references with links
|
||||
4. Upload the updated README
|
||||
5. Hub automatically creates `arxiv:2301.12345` tags
|
||||
|
||||
## Example 3: Link Paper to Dataset
|
||||
|
||||
Same process for datasets:
|
||||
|
||||
```bash
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/my-dataset" \
|
||||
--repo-type "dataset" \
|
||||
--arxiv-id "2301.12345" \
|
||||
--citation "$(cat citation.bib)"
|
||||
```
|
||||
|
||||
## Example 4: Create a New Research Article
|
||||
|
||||
Generate a research paper from template:
|
||||
|
||||
```bash
|
||||
# Create with standard template
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "standard" \
|
||||
--title "Efficient Fine-Tuning of Large Language Models" \
|
||||
--authors "Jane Doe, John Smith" \
|
||||
--abstract "We propose a novel approach to fine-tuning..." \
|
||||
--output "paper.md"
|
||||
|
||||
# Create with modern template
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "modern" \
|
||||
--title "Vision Transformers for Medical Imaging" \
|
||||
--output "medical_vit_paper.md"
|
||||
|
||||
# Create ML experiment report
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "ml-report" \
|
||||
--title "BERT Fine-tuning Experiment Results" \
|
||||
--output "bert_experiment_report.md"
|
||||
```
|
||||
|
||||
## Example 5: Generate Citations
|
||||
|
||||
Get formatted citations for papers:
|
||||
|
||||
```bash
|
||||
# BibTeX format
|
||||
uv run scripts/paper_manager.py citation \
|
||||
--arxiv-id "2301.12345" \
|
||||
--format "bibtex"
|
||||
```
|
||||
|
||||
Output:
|
||||
```bibtex
|
||||
@article{arxiv2301_12345,
|
||||
title={Efficient Fine-Tuning of Large Language Models},
|
||||
author={Doe, Jane and Smith, John},
|
||||
journal={arXiv preprint arXiv:2301.12345},
|
||||
year={2023}
|
||||
}
|
||||
```
|
||||
|
||||
## Example 6: Complete Workflow - New Paper
|
||||
|
||||
Full workflow from paper creation to publication:
|
||||
|
||||
```bash
|
||||
# Step 1: Create research article
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "modern" \
|
||||
--title "Novel Architecture for Multimodal Learning" \
|
||||
--authors "Alice Chen, Bob Kumar" \
|
||||
--output "multimodal_paper.md"
|
||||
|
||||
# Step 2: Edit the paper (use your favorite editor)
|
||||
# vim multimodal_paper.md
|
||||
|
||||
# Step 3: Submit to arXiv (external process)
|
||||
# Upload to arxiv.org, receive arXiv ID: 2312.99999
|
||||
|
||||
# Step 4: Index on Hugging Face
|
||||
uv run scripts/paper_manager.py index --arxiv-id "2312.99999"
|
||||
|
||||
# Step 5: Link to your models/datasets
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "alice/multimodal-model-v1" \
|
||||
--repo-type "model" \
|
||||
--arxiv-id "2312.99999"
|
||||
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "alice/multimodal-dataset" \
|
||||
--repo-type "dataset" \
|
||||
--arxiv-id "2312.99999"
|
||||
|
||||
# Step 6: Generate citation for README
|
||||
uv run scripts/paper_manager.py citation \
|
||||
--arxiv-id "2312.99999" \
|
||||
--format "bibtex" > citation.bib
|
||||
```
|
||||
|
||||
## Example 7: Batch Link Papers
|
||||
|
||||
Link multiple papers to multiple repositories:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# List of papers
|
||||
PAPERS=("2301.12345" "2302.67890" "2303.11111")
|
||||
|
||||
# List of models
|
||||
MODELS=("username/model-a" "username/model-b" "username/model-c")
|
||||
|
||||
# Link each paper to each model
|
||||
for paper in "${PAPERS[@]}"; do
|
||||
for model in "${MODELS[@]}"; do
|
||||
echo "Linking $paper to $model..."
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "$model" \
|
||||
--repo-type "model" \
|
||||
--arxiv-id "$paper"
|
||||
done
|
||||
done
|
||||
```
|
||||
|
||||
## Example 8: Update Model Card with Paper Info
|
||||
|
||||
Get paper info and manually update model card:
|
||||
|
||||
```bash
|
||||
# Get paper information
|
||||
uv run scripts/paper_manager.py info \
|
||||
--arxiv-id "2301.12345" \
|
||||
--format "text" > paper_info.txt
|
||||
|
||||
# View the information
|
||||
cat paper_info.txt
|
||||
|
||||
# Manually incorporate into your model card or use the link command
|
||||
```
|
||||
|
||||
## Example 9: Search and Discover Papers
|
||||
|
||||
```bash
|
||||
# Search for papers (opens browser)
|
||||
uv run scripts/paper_manager.py search \
|
||||
--query "transformer attention mechanism"
|
||||
```
|
||||
|
||||
## Example 10: Working with tfrere's Template
|
||||
|
||||
This skill complements [tfrere's research article template](https://huggingface.co/spaces/tfrere/research-article-template):
|
||||
|
||||
```bash
|
||||
# 1. Use tfrere's Space to create a beautiful web-based paper
|
||||
# Visit: https://huggingface.co/spaces/tfrere/research-article-template
|
||||
|
||||
# 2. Export your paper content to markdown
|
||||
|
||||
# 3. Submit to arXiv
|
||||
|
||||
# 4. Use this skill to index and link
|
||||
uv run scripts/paper_manager.py index --arxiv-id "YOUR_ARXIV_ID"
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "your-username/your-model" \
|
||||
--arxiv-id "YOUR_ARXIV_ID"
|
||||
```
|
||||
|
||||
## Example 11: Error Handling
|
||||
|
||||
```bash
|
||||
# Check if paper exists before linking
|
||||
if uv run scripts/paper_manager.py check --arxiv-id "2301.12345" | grep -q '"exists": true'; then
|
||||
echo "Paper exists, proceeding with link..."
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/model" \
|
||||
--arxiv-id "2301.12345"
|
||||
else
|
||||
echo "Paper doesn't exist, indexing first..."
|
||||
uv run scripts/paper_manager.py index --arxiv-id "2301.12345"
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/model" \
|
||||
--arxiv-id "2301.12345"
|
||||
fi
|
||||
```
|
||||
|
||||
## Example 12: CI/CD Integration
|
||||
|
||||
Add to your `.github/workflows/update-paper.yml`:
|
||||
|
||||
```yaml
|
||||
name: Update Paper Links
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
update:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Set up uv
|
||||
uses: astral-sh/setup-uv@v5
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Link paper to model
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "${{ github.repository_owner }}/model-name" \
|
||||
--repo-type "model" \
|
||||
--arxiv-id "2301.12345"
|
||||
```
|
||||
|
||||
## Tips and Best Practices
|
||||
|
||||
1. **Always check if paper exists** before indexing to avoid unnecessary operations
|
||||
2. **Use meaningful commit messages** when linking papers to repositories
|
||||
3. **Include full citations** in model cards for proper attribution
|
||||
4. **Link papers to all relevant artifacts** (models, datasets, spaces)
|
||||
5. **Generate BibTeX citations** for easy reference by others
|
||||
6. **Keep paper visibility updated** in your HF profile settings
|
||||
7. **Use templates consistently** within your research group
|
||||
8. **Version control your papers** alongside code
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Paper not found after indexing
|
||||
|
||||
```bash
|
||||
# Visit the URL directly to trigger indexing
|
||||
open "https://huggingface.co/papers/2301.12345"
|
||||
|
||||
# Wait a few seconds, then check again
|
||||
uv run scripts/paper_manager.py check --arxiv-id "2301.12345"
|
||||
```
|
||||
|
||||
### Permission denied when linking
|
||||
|
||||
```bash
|
||||
# Verify your token has write access
|
||||
echo $HF_TOKEN
|
||||
|
||||
# Set token if missing
|
||||
export HF_TOKEN="your_token_here"
|
||||
|
||||
# Or use .env file
|
||||
echo "HF_TOKEN=your_token_here" > .env
|
||||
```
|
||||
|
||||
### arXiv ID format issues
|
||||
|
||||
```bash
|
||||
# The script handles various formats:
|
||||
uv run scripts/paper_manager.py check --arxiv-id "2301.12345"
|
||||
uv run scripts/paper_manager.py check --arxiv-id "arxiv:2301.12345"
|
||||
uv run scripts/paper_manager.py check --arxiv-id "https://arxiv.org/abs/2301.12345"
|
||||
|
||||
# All are equivalent and will be normalized
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Explore the [Paper Pages documentation](https://huggingface.co/docs/hub/en/paper-pages)
|
||||
- Check out [tfrere's research template](https://huggingface.co/spaces/tfrere/research-article-template)
|
||||
- Browse [papers on HF](https://huggingface.co/papers)
|
||||
- Learn about [model cards](https://huggingface.co/docs/hub/en/model-cards)
|
||||
@@ -0,0 +1,216 @@
|
||||
# Quick Reference Guide
|
||||
|
||||
## Essential Commands
|
||||
|
||||
### Paper Indexing
|
||||
```bash
|
||||
# Index from arXiv
|
||||
uv run scripts/paper_manager.py index --arxiv-id "2301.12345"
|
||||
|
||||
# Check if exists
|
||||
uv run scripts/paper_manager.py check --arxiv-id "2301.12345"
|
||||
```
|
||||
|
||||
### Linking Papers
|
||||
```bash
|
||||
# Link to model
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/model" \
|
||||
--repo-type "model" \
|
||||
--arxiv-id "2301.12345"
|
||||
|
||||
# Link to dataset
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/dataset" \
|
||||
--repo-type "dataset" \
|
||||
--arxiv-id "2301.12345"
|
||||
|
||||
# Link multiple papers
|
||||
uv run scripts/paper_manager.py link \
|
||||
--repo-id "username/model" \
|
||||
--repo-type "model" \
|
||||
--arxiv-ids "2301.12345,2302.67890"
|
||||
```
|
||||
|
||||
### Creating Papers
|
||||
```bash
|
||||
# Standard template
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "standard" \
|
||||
--title "Paper Title" \
|
||||
--output "paper.md"
|
||||
|
||||
# Modern template
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "modern" \
|
||||
--title "Paper Title" \
|
||||
--authors "Author1, Author2" \
|
||||
--abstract "Abstract text" \
|
||||
--output "paper.md"
|
||||
|
||||
# ML Report
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "ml-report" \
|
||||
--title "Experiment Report" \
|
||||
--output "report.md"
|
||||
|
||||
# arXiv style
|
||||
uv run scripts/paper_manager.py create \
|
||||
--template "arxiv" \
|
||||
--title "Paper Title" \
|
||||
--output "paper.md"
|
||||
```
|
||||
|
||||
### Citations
|
||||
```bash
|
||||
# Generate BibTeX
|
||||
uv run scripts/paper_manager.py citation \
|
||||
--arxiv-id "2301.12345" \
|
||||
--format "bibtex"
|
||||
```
|
||||
|
||||
### Paper Info
|
||||
```bash
|
||||
# JSON format
|
||||
uv run scripts/paper_manager.py info \
|
||||
--arxiv-id "2301.12345" \
|
||||
--format "json"
|
||||
|
||||
# Text format
|
||||
uv run scripts/paper_manager.py info \
|
||||
--arxiv-id "2301.12345" \
|
||||
--format "text"
|
||||
```
|
||||
|
||||
## URL Formats
|
||||
|
||||
### Hugging Face Paper Pages
|
||||
- View paper: `https://huggingface.co/papers/{arxiv-id}`
|
||||
- Example: `https://huggingface.co/papers/2301.12345`
|
||||
|
||||
### arXiv
|
||||
- Abstract: `https://arxiv.org/abs/{arxiv-id}`
|
||||
- PDF: `https://arxiv.org/pdf/{arxiv-id}.pdf`
|
||||
- Example: `https://arxiv.org/abs/2301.12345`
|
||||
|
||||
## YAML Metadata Format
|
||||
|
||||
### Model Card
|
||||
```yaml
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- text-generation
|
||||
- transformers
|
||||
library_name: transformers
|
||||
---
|
||||
```
|
||||
|
||||
### Dataset Card
|
||||
```yaml
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: cc-by-4.0
|
||||
task_categories:
|
||||
- text-generation
|
||||
size_categories:
|
||||
- 10K<n<100K
|
||||
---
|
||||
```
|
||||
|
||||
## arXiv ID Formats
|
||||
|
||||
All these formats work:
|
||||
- `2301.12345`
|
||||
- `arxiv:2301.12345`
|
||||
- `https://arxiv.org/abs/2301.12345`
|
||||
- `https://arxiv.org/pdf/2301.12345.pdf`
|
||||
|
||||
## Environment Setup
|
||||
|
||||
### Set Token
|
||||
```bash
|
||||
export HF_TOKEN="your_token"
|
||||
```
|
||||
|
||||
### Or use .env file
|
||||
```bash
|
||||
echo "HF_TOKEN=your_token" > .env
|
||||
```
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. Index & Link
|
||||
```bash
|
||||
uv run scripts/paper_manager.py index --arxiv-id "2301.12345"
|
||||
uv run scripts/paper_manager.py link --repo-id "user/model" --arxiv-id "2301.12345"
|
||||
```
|
||||
|
||||
### 2. Create & Publish
|
||||
```bash
|
||||
uv run scripts/paper_manager.py create --template "modern" --title "Title" --output "paper.md"
|
||||
# Edit paper.md
|
||||
# Submit to arXiv → get ID
|
||||
uv run scripts/paper_manager.py index --arxiv-id "NEW_ID"
|
||||
uv run scripts/paper_manager.py link --repo-id "user/model" --arxiv-id "NEW_ID"
|
||||
```
|
||||
|
||||
### 3. Batch Link
|
||||
```bash
|
||||
for id in "2301.12345" "2302.67890"; do
|
||||
uv run scripts/paper_manager.py link --repo-id "user/model" --arxiv-id "$id"
|
||||
done
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Paper not found
|
||||
Visit `https://huggingface.co/papers/{arxiv-id}` to trigger indexing
|
||||
|
||||
### Permission denied
|
||||
Check `HF_TOKEN` is set and has write access
|
||||
|
||||
### arXiv API errors
|
||||
Wait a moment and retry - arXiv has rate limits
|
||||
|
||||
## Tips
|
||||
|
||||
1. Always check paper exists before linking
|
||||
2. Use templates for consistency
|
||||
3. Include full citations in model cards
|
||||
4. Link papers to all relevant artifacts
|
||||
5. Keep citations up to date
|
||||
|
||||
## Templates Available
|
||||
|
||||
- `standard` - Traditional academic paper
|
||||
- `modern` - Web-friendly format (Distill-style)
|
||||
- `arxiv` - arXiv journal format
|
||||
- `ml-report` - ML experiment documentation
|
||||
|
||||
## File Locations
|
||||
|
||||
- Scripts: `scripts/paper_manager.py`
|
||||
- Templates: `templates/*.md`
|
||||
- Examples: `examples/example_usage.md`
|
||||
- This guide: `references/quick_reference.md`
|
||||
|
||||
## Getting Help
|
||||
|
||||
```bash
|
||||
# Command help
|
||||
uv run scripts/paper_manager.py --help
|
||||
|
||||
# Subcommand help
|
||||
uv run scripts/paper_manager.py link --help
|
||||
```
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [Full documentation](../SKILL.md)
|
||||
- [Usage examples](../examples/example_usage.md)
|
||||
- [HF Paper Pages](https://huggingface.co/papers)
|
||||
- [tfrere's template](https://huggingface.co/spaces/tfrere/research-article-template)
|
||||
+606
@@ -0,0 +1,606 @@
|
||||
#!/usr/bin/env -S uv run
|
||||
# /// script
|
||||
# requires-python = ">=3.10"
|
||||
# dependencies = [
|
||||
# "huggingface_hub",
|
||||
# "pyyaml",
|
||||
# "requests",
|
||||
# "python-dotenv",
|
||||
# ]
|
||||
# ///
|
||||
"""
|
||||
Paper Manager for Hugging Face Hub
|
||||
Manages paper indexing, linking, authorship, and article creation.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import re
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Dict, Any
|
||||
from datetime import datetime
|
||||
|
||||
try:
|
||||
from huggingface_hub import HfApi, hf_hub_download, get_token
|
||||
import yaml
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
except ImportError as e:
|
||||
print(f"Error: Missing required dependency: {e}")
|
||||
print("Tip: run this script with `uv run scripts/paper_manager.py ...`.")
|
||||
sys.exit(1)
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class PaperManager:
|
||||
"""Manages paper publishing operations on Hugging Face Hub."""
|
||||
|
||||
def __init__(self, hf_token: Optional[str] = None):
|
||||
"""Initialize Paper Manager with HF token."""
|
||||
self.token = hf_token or os.getenv("HF_TOKEN") or get_token()
|
||||
if not self.token:
|
||||
print("Warning: No HF_TOKEN found. Some operations will fail.")
|
||||
self.api = HfApi(token=self.token)
|
||||
|
||||
def index_paper(self, arxiv_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Index a paper on Hugging Face from arXiv.
|
||||
|
||||
Args:
|
||||
arxiv_id: arXiv identifier (e.g., "2301.12345")
|
||||
|
||||
Returns:
|
||||
dict: Status information
|
||||
"""
|
||||
# Clean and validate arXiv ID
|
||||
try:
|
||||
arxiv_id = self._clean_arxiv_id(arxiv_id)
|
||||
except ValueError as e:
|
||||
print(f"Error: {e}")
|
||||
return {"status": "error", "message": str(e)}
|
||||
|
||||
print(f"Indexing paper {arxiv_id} on Hugging Face...")
|
||||
|
||||
# Check if paper exists
|
||||
paper_url = f"https://huggingface.co/papers/{arxiv_id}"
|
||||
|
||||
try:
|
||||
response = requests.get(paper_url, timeout=10)
|
||||
if response.status_code == 200:
|
||||
print(f"✓ Paper already indexed at {paper_url}")
|
||||
return {"status": "exists", "url": paper_url}
|
||||
else:
|
||||
print(f"Paper not indexed. Visit {paper_url} to trigger indexing.")
|
||||
print("The paper will be automatically indexed when you first visit the URL.")
|
||||
return {"status": "not_indexed", "url": paper_url, "action": "visit_url"}
|
||||
except requests.RequestException as e:
|
||||
print(f"Error checking paper status: {e}")
|
||||
return {"status": "error", "message": str(e)}
|
||||
|
||||
def check_paper(self, arxiv_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if a paper exists on Hugging Face.
|
||||
|
||||
Args:
|
||||
arxiv_id: arXiv identifier
|
||||
|
||||
Returns:
|
||||
dict: Paper status and metadata
|
||||
"""
|
||||
try:
|
||||
arxiv_id = self._clean_arxiv_id(arxiv_id)
|
||||
except ValueError as e:
|
||||
return {"exists": False, "error": str(e)}
|
||||
paper_url = f"https://huggingface.co/papers/{arxiv_id}"
|
||||
|
||||
try:
|
||||
response = requests.get(paper_url, timeout=10)
|
||||
if response.status_code == 200:
|
||||
return {
|
||||
"exists": True,
|
||||
"url": paper_url,
|
||||
"arxiv_id": arxiv_id,
|
||||
"arxiv_url": f"https://arxiv.org/abs/{arxiv_id}"
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"exists": False,
|
||||
"arxiv_id": arxiv_id,
|
||||
"index_url": paper_url,
|
||||
"message": f"Visit {paper_url} to index this paper"
|
||||
}
|
||||
except requests.RequestException as e:
|
||||
return {"exists": False, "error": str(e)}
|
||||
|
||||
def link_paper_to_repo(
|
||||
self,
|
||||
repo_id: str,
|
||||
arxiv_id: str,
|
||||
repo_type: str = "model",
|
||||
citation: Optional[str] = None,
|
||||
create_pr: bool = False
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Link a paper to a model/dataset/space repository.
|
||||
|
||||
Args:
|
||||
repo_id: Repository identifier (e.g., "username/repo-name")
|
||||
arxiv_id: arXiv identifier
|
||||
repo_type: Type of repository ("model", "dataset", or "space")
|
||||
citation: Optional full citation text
|
||||
create_pr: Create a PR instead of direct commit
|
||||
|
||||
Returns:
|
||||
dict: Operation status
|
||||
"""
|
||||
try:
|
||||
arxiv_id = self._clean_arxiv_id(arxiv_id)
|
||||
except ValueError as e:
|
||||
print(f"Error: {e}")
|
||||
return {"status": "error", "message": str(e)}
|
||||
|
||||
print(f"Linking paper {arxiv_id} to {repo_type} {repo_id}...")
|
||||
|
||||
try:
|
||||
# Download current README
|
||||
readme_path = hf_hub_download(
|
||||
repo_id=repo_id,
|
||||
filename="README.md",
|
||||
repo_type=repo_type,
|
||||
token=self.token
|
||||
)
|
||||
|
||||
with open(readme_path, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
|
||||
# Parse or create YAML frontmatter
|
||||
updated_content = self._add_paper_to_readme(content, arxiv_id, citation)
|
||||
|
||||
# Upload updated README
|
||||
commit_message = f"Add paper reference: arXiv:{arxiv_id}"
|
||||
|
||||
if create_pr:
|
||||
# Create PR (not implemented in basic version)
|
||||
print("PR creation not yet implemented. Committing directly.")
|
||||
|
||||
self.api.upload_file(
|
||||
path_or_fileobj=updated_content.encode('utf-8'),
|
||||
path_in_repo="README.md",
|
||||
repo_id=repo_id,
|
||||
repo_type=repo_type,
|
||||
commit_message=commit_message,
|
||||
token=self.token
|
||||
)
|
||||
|
||||
paper_url = f"https://huggingface.co/papers/{arxiv_id}"
|
||||
repo_url = f"https://huggingface.co/{repo_id}"
|
||||
|
||||
print(f"✓ Successfully linked paper to repository")
|
||||
print(f" Paper: {paper_url}")
|
||||
print(f" Repo: {repo_url}")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"paper_url": paper_url,
|
||||
"repo_url": repo_url,
|
||||
"arxiv_id": arxiv_id
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error linking paper: {e}")
|
||||
return {"status": "error", "message": str(e)}
|
||||
|
||||
def _add_paper_to_readme(
|
||||
self,
|
||||
content: str,
|
||||
arxiv_id: str,
|
||||
citation: Optional[str] = None
|
||||
) -> str:
|
||||
"""
|
||||
Add paper reference to README content.
|
||||
|
||||
Args:
|
||||
content: Current README content
|
||||
arxiv_id: arXiv identifier
|
||||
citation: Optional citation text
|
||||
|
||||
Returns:
|
||||
str: Updated README content
|
||||
"""
|
||||
arxiv_url = f"https://arxiv.org/abs/{arxiv_id}"
|
||||
hf_paper_url = f"https://huggingface.co/papers/{arxiv_id}"
|
||||
|
||||
# Check if YAML frontmatter exists
|
||||
yaml_pattern = r'^---\s*\n(.*?)\n---\s*\n'
|
||||
match = re.match(yaml_pattern, content, re.DOTALL)
|
||||
|
||||
if match:
|
||||
# YAML exists, check if paper already referenced
|
||||
if arxiv_id in content:
|
||||
print(f"Paper {arxiv_id} already referenced in README")
|
||||
return content
|
||||
|
||||
# Add to existing content (after YAML)
|
||||
yaml_end = match.end()
|
||||
before = content[:yaml_end]
|
||||
after = content[yaml_end:]
|
||||
else:
|
||||
# No YAML, add minimal frontmatter
|
||||
yaml_content = "---\n---\n\n"
|
||||
before = yaml_content
|
||||
after = content
|
||||
|
||||
# Add paper reference section with boundary markers
|
||||
paper_section = "\n<!-- paper-manager:start -->\n"
|
||||
paper_section += f"## Paper\n\n"
|
||||
paper_section += f"This {'model' if 'model' in content.lower() else 'work'} is based on research presented in:\n\n"
|
||||
paper_section += f"**[View on arXiv]({arxiv_url})** | "
|
||||
paper_section += f"**[View on Hugging Face]({hf_paper_url})**\n\n"
|
||||
|
||||
if citation:
|
||||
safe_citation = self._sanitize_text(citation)
|
||||
paper_section += f"### Citation\n\n```bibtex\n{safe_citation}\n```\n\n"
|
||||
|
||||
paper_section += "<!-- paper-manager:end -->\n"
|
||||
|
||||
# Insert after YAML, before main content
|
||||
updated_content = before + paper_section + after
|
||||
|
||||
return updated_content
|
||||
|
||||
def create_research_article(
|
||||
self,
|
||||
template: str,
|
||||
title: str,
|
||||
output: str,
|
||||
authors: Optional[str] = None,
|
||||
abstract: Optional[str] = None
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a research article from template.
|
||||
|
||||
Args:
|
||||
template: Template name ("standard", "modern", "arxiv", "ml-report")
|
||||
title: Paper title
|
||||
output: Output filename
|
||||
authors: Comma-separated author names
|
||||
abstract: Abstract text
|
||||
|
||||
Returns:
|
||||
dict: Creation status
|
||||
"""
|
||||
print(f"Creating research article with '{template}' template...")
|
||||
|
||||
# Load template
|
||||
template_dir = Path(__file__).parent.parent / "templates"
|
||||
template_file = template_dir / f"{template}.md"
|
||||
|
||||
if not template_file.exists():
|
||||
return {
|
||||
"status": "error",
|
||||
"message": f"Template '{template}' not found at {template_file}"
|
||||
}
|
||||
|
||||
with open(template_file, 'r', encoding='utf-8') as f:
|
||||
template_content = f.read()
|
||||
|
||||
# Prepare safe values for different contexts
|
||||
date_str = datetime.now().strftime("%Y-%m-%d")
|
||||
safe_title_body = self._sanitize_text(title)
|
||||
authors_val = authors if authors else "Your Name"
|
||||
safe_authors_body = self._sanitize_text(authors_val)
|
||||
abstract_val = abstract if abstract else "Abstract to be written..."
|
||||
safe_abstract_body = self._sanitize_text(abstract_val)
|
||||
|
||||
# Split frontmatter from body for context-aware escaping
|
||||
fm_pattern = r'^(---\s*\n)(.*?\n)(---\s*\n)'
|
||||
fm_match = re.match(fm_pattern, template_content, re.DOTALL)
|
||||
|
||||
if fm_match:
|
||||
fm_open, fm_body, fm_close = fm_match.group(1), fm_match.group(2), fm_match.group(3)
|
||||
body = template_content[fm_match.end():]
|
||||
|
||||
# YAML-escape values in frontmatter
|
||||
fm_body = fm_body.replace("{{TITLE}}", self._escape_yaml_value(title))
|
||||
fm_body = fm_body.replace("{{AUTHORS}}", self._escape_yaml_value(authors_val))
|
||||
fm_body = fm_body.replace("{{DATE}}", date_str)
|
||||
|
||||
# Sanitize values in body
|
||||
body = body.replace("{{TITLE}}", safe_title_body)
|
||||
body = body.replace("{{AUTHORS}}", safe_authors_body)
|
||||
body = body.replace("{{ABSTRACT}}", safe_abstract_body)
|
||||
body = body.replace("{{DATE}}", date_str)
|
||||
|
||||
content = fm_open + fm_body + fm_close + body
|
||||
else:
|
||||
# No frontmatter — sanitize everything
|
||||
content = template_content.replace("{{TITLE}}", safe_title_body)
|
||||
content = content.replace("{{DATE}}", date_str)
|
||||
content = content.replace("{{AUTHORS}}", safe_authors_body)
|
||||
content = content.replace("{{ABSTRACT}}", safe_abstract_body)
|
||||
|
||||
# Write output
|
||||
with open(output, 'w', encoding='utf-8') as f:
|
||||
f.write(content)
|
||||
|
||||
print(f"✓ Research article created at {output}")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"output": output,
|
||||
"template": template
|
||||
}
|
||||
|
||||
def get_arxiv_info(self, arxiv_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch paper information from arXiv API.
|
||||
|
||||
Args:
|
||||
arxiv_id: arXiv identifier
|
||||
|
||||
Returns:
|
||||
dict: Paper metadata
|
||||
"""
|
||||
try:
|
||||
arxiv_id = self._clean_arxiv_id(arxiv_id)
|
||||
except ValueError as e:
|
||||
return {"error": str(e)}
|
||||
api_url = f"https://export.arxiv.org/api/query?id_list={arxiv_id}"
|
||||
|
||||
try:
|
||||
response = requests.get(api_url, timeout=10)
|
||||
response.raise_for_status()
|
||||
|
||||
# Parse XML response (simplified)
|
||||
content = response.text
|
||||
|
||||
# Extract basic info with regex (proper XML parsing would be better)
|
||||
title_match = re.search(r'<title>(.*?)</title>', content, re.DOTALL)
|
||||
authors_matches = re.findall(r'<name>(.*?)</name>', content)
|
||||
summary_match = re.search(r'<summary>(.*?)</summary>', content, re.DOTALL)
|
||||
|
||||
# Sanitize all text extracted from the external API
|
||||
raw_title = title_match.group(1).strip() if title_match else None
|
||||
raw_authors = authors_matches[1:] if len(authors_matches) > 1 else []
|
||||
raw_abstract = summary_match.group(1).strip() if summary_match else None
|
||||
|
||||
return {
|
||||
"arxiv_id": arxiv_id,
|
||||
"title": self._sanitize_text(raw_title) if raw_title else None,
|
||||
"authors": [self._sanitize_text(a) for a in raw_authors],
|
||||
"abstract": self._sanitize_text(raw_abstract) if raw_abstract else None,
|
||||
"arxiv_url": f"https://arxiv.org/abs/{arxiv_id}",
|
||||
"pdf_url": f"https://arxiv.org/pdf/{arxiv_id}.pdf"
|
||||
}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
def generate_citation(
|
||||
self,
|
||||
arxiv_id: str,
|
||||
format: str = "bibtex"
|
||||
) -> str:
|
||||
"""
|
||||
Generate citation for a paper.
|
||||
|
||||
Args:
|
||||
arxiv_id: arXiv identifier
|
||||
format: Citation format ("bibtex", "apa", "mla")
|
||||
|
||||
Returns:
|
||||
str: Formatted citation
|
||||
"""
|
||||
try:
|
||||
arxiv_id = self._clean_arxiv_id(arxiv_id)
|
||||
except ValueError as e:
|
||||
return f"Error: {e}"
|
||||
|
||||
info = self.get_arxiv_info(arxiv_id)
|
||||
|
||||
if "error" in info:
|
||||
return f"Error fetching paper info: {info['error']}"
|
||||
|
||||
if format == "bibtex":
|
||||
# Generate BibTeX citation
|
||||
key = f"arxiv{arxiv_id.replace('.', '_')}"
|
||||
raw_authors = " and ".join(info.get("authors", ["Unknown"]))
|
||||
raw_title = info.get("title", "Untitled")
|
||||
year = arxiv_id.split(".")[0][:2] # Extract year from ID (simplified)
|
||||
year = f"20{year}" if int(year) < 50 else f"19{year}"
|
||||
|
||||
# Escape BibTeX structural characters in untrusted values
|
||||
safe_title = raw_title.replace('{', r'\{').replace('}', r'\}')
|
||||
safe_authors = raw_authors.replace('{', r'\{').replace('}', r'\}')
|
||||
|
||||
citation = f"""@article{{{key},
|
||||
title={{{safe_title}}},
|
||||
author={{{safe_authors}}},
|
||||
journal={{arXiv preprint arXiv:{arxiv_id}}},
|
||||
year={{{year}}}
|
||||
}}"""
|
||||
return citation
|
||||
|
||||
return f"Format '{format}' not yet implemented"
|
||||
|
||||
# Patterns for valid arXiv IDs
|
||||
_ARXIV_ID_MODERN = re.compile(r'^\d{4}\.\d{4,5}(v\d+)?$')
|
||||
_ARXIV_ID_LEGACY = re.compile(r'^[a-zA-Z\-]+/\d{7}(v\d+)?$')
|
||||
|
||||
@staticmethod
|
||||
def _clean_arxiv_id(arxiv_id: str) -> str:
|
||||
"""Clean, normalize, and validate arXiv ID.
|
||||
|
||||
Raises:
|
||||
ValueError: If the cleaned ID does not match a valid arXiv format.
|
||||
"""
|
||||
# Remove common prefixes and whitespace
|
||||
arxiv_id = arxiv_id.strip()
|
||||
arxiv_id = re.sub(r'^(arxiv:|arXiv:)', '', arxiv_id, flags=re.IGNORECASE)
|
||||
arxiv_id = re.sub(r'https?://arxiv\.org/(abs|pdf)/', '', arxiv_id)
|
||||
arxiv_id = arxiv_id.replace('.pdf', '')
|
||||
|
||||
# Validate format
|
||||
if not (PaperManager._ARXIV_ID_MODERN.match(arxiv_id)
|
||||
or PaperManager._ARXIV_ID_LEGACY.match(arxiv_id)):
|
||||
raise ValueError(
|
||||
f"Invalid arXiv ID: {arxiv_id!r}. "
|
||||
"Expected format: YYMM.NNNNN[vN] or category/YYMMNNN[vN]"
|
||||
)
|
||||
|
||||
return arxiv_id
|
||||
|
||||
@staticmethod
|
||||
def _escape_yaml_value(value: str) -> str:
|
||||
"""Escape a string for safe use as a YAML scalar value.
|
||||
|
||||
Wraps in double quotes and escapes internal quotes and backslashes
|
||||
to prevent YAML injection via crafted titles/authors.
|
||||
"""
|
||||
value = value.replace('\\', '\\\\').replace('"', '\\"')
|
||||
return f'"{value}"'
|
||||
|
||||
@staticmethod
|
||||
def _sanitize_text(text: str) -> str:
|
||||
"""Sanitize untrusted text for safe inclusion in Markdown/YAML output.
|
||||
|
||||
Normalizes whitespace, strips control characters, and neutralizes
|
||||
markdown code-fence breakout and YAML document delimiters.
|
||||
"""
|
||||
# Remove control characters (keep newlines and tabs)
|
||||
text = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', text)
|
||||
# Normalize whitespace runs (collapse multiple spaces/tabs, preserve single newlines)
|
||||
text = re.sub(r'[^\S\n]+', ' ', text)
|
||||
text = re.sub(r'\n{3,}', '\n\n', text)
|
||||
# Neutralize markdown code fence breakout
|
||||
text = text.replace('```', r'\`\`\`')
|
||||
# Neutralize YAML document delimiters at line start
|
||||
text = re.sub(r'^---', r'\\---', text, flags=re.MULTILINE)
|
||||
return text.strip()
|
||||
|
||||
|
||||
def main():
|
||||
"""Main CLI entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Paper Manager for Hugging Face Hub",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
|
||||
subparsers = parser.add_subparsers(dest="command", help="Command to execute")
|
||||
|
||||
# Index command
|
||||
index_parser = subparsers.add_parser("index", help="Index a paper from arXiv")
|
||||
index_parser.add_argument("--arxiv-id", required=True, help="arXiv paper ID")
|
||||
|
||||
# Check command
|
||||
check_parser = subparsers.add_parser("check", help="Check if paper exists")
|
||||
check_parser.add_argument("--arxiv-id", required=True, help="arXiv paper ID")
|
||||
|
||||
# Link command
|
||||
link_parser = subparsers.add_parser("link", help="Link paper to repository")
|
||||
link_parser.add_argument("--repo-id", required=True, help="Repository ID")
|
||||
link_parser.add_argument("--repo-type", default="model", choices=["model", "dataset", "space"])
|
||||
link_parser.add_argument("--arxiv-id", help="Single arXiv ID")
|
||||
link_parser.add_argument("--arxiv-ids", help="Comma-separated arXiv IDs")
|
||||
link_parser.add_argument("--citation", help="Full citation text")
|
||||
link_parser.add_argument("--create-pr", action="store_true", help="Create PR instead of direct commit")
|
||||
|
||||
# Create command
|
||||
create_parser = subparsers.add_parser("create", help="Create research article")
|
||||
create_parser.add_argument("--template", required=True, help="Template name")
|
||||
create_parser.add_argument("--title", required=True, help="Paper title")
|
||||
create_parser.add_argument("--output", required=True, help="Output filename")
|
||||
create_parser.add_argument("--authors", help="Comma-separated authors")
|
||||
create_parser.add_argument("--abstract", help="Abstract text")
|
||||
|
||||
# Info command
|
||||
info_parser = subparsers.add_parser("info", help="Get paper information")
|
||||
info_parser.add_argument("--arxiv-id", required=True, help="arXiv paper ID")
|
||||
info_parser.add_argument("--format", default="json", choices=["json", "text"])
|
||||
|
||||
# Citation command
|
||||
citation_parser = subparsers.add_parser("citation", help="Generate citation")
|
||||
citation_parser.add_argument("--arxiv-id", required=True, help="arXiv paper ID")
|
||||
citation_parser.add_argument("--format", default="bibtex", choices=["bibtex", "apa", "mla"])
|
||||
|
||||
# Search command
|
||||
search_parser = subparsers.add_parser("search", help="Search papers")
|
||||
search_parser.add_argument("--query", required=True, help="Search query")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.command:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
# Initialize manager
|
||||
manager = PaperManager()
|
||||
|
||||
# Execute command
|
||||
if args.command == "index":
|
||||
result = manager.index_paper(args.arxiv_id)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.command == "check":
|
||||
result = manager.check_paper(args.arxiv_id)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.command == "link":
|
||||
arxiv_ids = []
|
||||
if args.arxiv_id:
|
||||
arxiv_ids.append(args.arxiv_id)
|
||||
if args.arxiv_ids:
|
||||
arxiv_ids.extend([id.strip() for id in args.arxiv_ids.split(",")])
|
||||
|
||||
if not arxiv_ids:
|
||||
print("Error: Must provide --arxiv-id or --arxiv-ids")
|
||||
sys.exit(1)
|
||||
|
||||
for arxiv_id in arxiv_ids:
|
||||
result = manager.link_paper_to_repo(
|
||||
repo_id=args.repo_id,
|
||||
arxiv_id=arxiv_id,
|
||||
repo_type=args.repo_type,
|
||||
citation=args.citation,
|
||||
create_pr=args.create_pr
|
||||
)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.command == "create":
|
||||
result = manager.create_research_article(
|
||||
template=args.template,
|
||||
title=args.title,
|
||||
output=args.output,
|
||||
authors=args.authors,
|
||||
abstract=args.abstract
|
||||
)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.command == "info":
|
||||
result = manager.get_arxiv_info(args.arxiv_id)
|
||||
if args.format == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
if "error" in result:
|
||||
print(f"Error: {result['error']}")
|
||||
else:
|
||||
print(f"Title: {result.get('title')}")
|
||||
print(f"Authors: {', '.join(result.get('authors', []))}")
|
||||
print(f"arXiv URL: {result.get('arxiv_url')}")
|
||||
print(f"\nAbstract:\n{result.get('abstract')}")
|
||||
|
||||
elif args.command == "citation":
|
||||
citation = manager.generate_citation(args.arxiv_id, args.format)
|
||||
print(citation)
|
||||
|
||||
elif args.command == "search":
|
||||
print(f"Searching for: {args.query}")
|
||||
print("Search functionality coming soon!")
|
||||
print(f"Visit: https://huggingface.co/papers?search={args.query}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
title: {{TITLE}}
|
||||
authors: {{AUTHORS}}
|
||||
date: {{DATE}}
|
||||
arxiv:
|
||||
layout: arxiv
|
||||
---
|
||||
|
||||
# {{TITLE}}
|
||||
|
||||
<div class="arxiv-header">
|
||||
|
||||
**{{AUTHORS}}**
|
||||
|
||||
*Submitted to arXiv: {{DATE}}*
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
**Abstract**—{{ABSTRACT}}
|
||||
|
||||
**Index Terms**—Machine Learning, Deep Learning, Neural Networks
|
||||
|
||||
---
|
||||
|
||||
## I. INTRODUCTION
|
||||
|
||||
**T**HIS paper presents [brief overview of the contribution]. The main contributions of this work are:
|
||||
|
||||
- Contribution 1: Description
|
||||
- Contribution 2: Description
|
||||
- Contribution 3: Description
|
||||
|
||||
The rest of this paper is organized as follows: Section II reviews related work, Section III describes the proposed methodology, Section IV presents experimental results, and Section V concludes the paper.
|
||||
|
||||
## II. RELATED WORK
|
||||
|
||||
### A. Subarea 1
|
||||
|
||||
Discussion of relevant prior work in subarea 1.
|
||||
|
||||
### B. Subarea 2
|
||||
|
||||
Discussion of relevant prior work in subarea 2.
|
||||
|
||||
### C. Comparison with Prior Art
|
||||
|
||||
Table comparing existing methods:
|
||||
|
||||
| Method | Year | Approach | Limitation |
|
||||
|--------|------|----------|------------|
|
||||
| Method A [1] | 2020 | Description | Issue |
|
||||
| Method B [2] | 2021 | Description | Issue |
|
||||
| Method C [3] | 2023 | Description | Issue |
|
||||
|
||||
## III. METHODOLOGY
|
||||
|
||||
### A. Problem Formulation
|
||||
|
||||
Let $X = \{x_1, x_2, ..., x_n\}$ be the input space and $Y = \{y_1, y_2, ..., y_m\}$ be the output space. We aim to learn a function $f: X \rightarrow Y$ that minimizes:
|
||||
|
||||
$$
|
||||
\mathcal{L}(\theta) = \sum_{i=1}^{N} \ell(f(x_i; \theta), y_i) + \lambda R(\theta)
|
||||
$$
|
||||
|
||||
where $\theta$ represents model parameters, $\ell$ is the loss function, and $R(\theta)$ is a regularization term.
|
||||
|
||||
### B. Model Architecture
|
||||
|
||||
Describe the model architecture in detail.
|
||||
|
||||
**Input Layer**: Description
|
||||
|
||||
**Hidden Layers**: Let $h^{(l)}$ denote the activation of layer $l$:
|
||||
|
||||
$$
|
||||
h^{(l)} = \sigma(W^{(l)}h^{(l-1)} + b^{(l)})
|
||||
$$
|
||||
|
||||
where $\sigma$ is the activation function, $W^{(l)}$ is the weight matrix, and $b^{(l)}$ is the bias vector.
|
||||
|
||||
**Output Layer**: Description
|
||||
|
||||
### C. Training Algorithm
|
||||
|
||||
**Algorithm 1**: Training Procedure
|
||||
|
||||
```
|
||||
1: Input: Training data D = {(xi, yi)}
|
||||
2: Initialize parameters θ
|
||||
3: for epoch = 1 to max_epochs do
|
||||
4: for each mini-batch B ⊂ D do
|
||||
5: Compute loss: L(θ) = 1/|B| Σ ℓ(f(xi; θ), yi)
|
||||
6: Update: θ ← θ - η∇θL(θ)
|
||||
7: end for
|
||||
8: end for
|
||||
9: Return: Trained parameters θ*
|
||||
```
|
||||
|
||||
### D. Complexity Analysis
|
||||
|
||||
**Time Complexity**: The training algorithm has time complexity $O(NTE)$ where $N$ is the dataset size, $T$ is the number of epochs, and $E$ is the per-example computation cost.
|
||||
|
||||
**Space Complexity**: The model requires $O(P)$ space where $P$ is the number of parameters.
|
||||
|
||||
## IV. EXPERIMENTS
|
||||
|
||||
### A. Experimental Setup
|
||||
|
||||
**Datasets**: We evaluate on the following benchmarks:
|
||||
|
||||
1. **Dataset A**: Description (size, splits, characteristics)
|
||||
2. **Dataset B**: Description
|
||||
3. **Dataset C**: Description
|
||||
|
||||
**Baselines**: We compare against:
|
||||
|
||||
- Baseline 1 [4]: Description
|
||||
- Baseline 2 [5]: Description
|
||||
- Baseline 3 [6]: Description
|
||||
|
||||
**Evaluation Metrics**: Performance is measured using:
|
||||
|
||||
- Metric 1: Definition
|
||||
- Metric 2: Definition
|
||||
- Metric 3: Definition
|
||||
|
||||
**Implementation Details**: All experiments are conducted using:
|
||||
|
||||
- Framework: PyTorch 2.0
|
||||
- Hardware: NVIDIA A100 GPUs
|
||||
- Hyperparameters: Learning rate $\eta = 10^{-4}$, batch size $B = 32$, epochs $T = 100$
|
||||
|
||||
### B. Quantitative Results
|
||||
|
||||
**TABLE I: MAIN RESULTS**
|
||||
|
||||
| Method | Dataset A | Dataset B | Dataset C | Average |
|
||||
|--------|-----------|-----------|-----------|---------|
|
||||
| Baseline 1 [4] | 82.3 | 78.5 | 80.1 | 80.3 |
|
||||
| Baseline 2 [5] | 85.7 | 82.1 | 83.9 | 83.9 |
|
||||
| Baseline 3 [6] | 88.1 | 85.3 | 86.7 | 86.7 |
|
||||
| **Ours** | **91.2** | **88.9** | **90.1** | **90.1** |
|
||||
|
||||
Our method achieves state-of-the-art performance across all three benchmarks, with an average improvement of 3.4 percentage points over the previous best method.
|
||||
|
||||
### C. Ablation Study
|
||||
|
||||
**TABLE II: ABLATION STUDY RESULTS**
|
||||
|
||||
| Configuration | Dataset A | Δ |
|
||||
|---------------|-----------|---|
|
||||
| Full Model | 91.2 | - |
|
||||
| w/o Component A | 88.7 | -2.5 |
|
||||
| w/o Component B | 89.4 | -1.8 |
|
||||
| w/o Component C | 90.5 | -0.7 |
|
||||
|
||||
The ablation study demonstrates that all components contribute to the final performance, with Component A having the largest impact.
|
||||
|
||||
### D. Qualitative Analysis
|
||||
|
||||
**Fig. 1**: Visualization of learned representations using t-SNE projection.
|
||||
|
||||
**Fig. 2**: Example predictions showing correct classifications and failure cases.
|
||||
|
||||
### E. Computational Efficiency
|
||||
|
||||
**TABLE III: COMPUTATIONAL REQUIREMENTS**
|
||||
|
||||
| Method | Parameters | FLOPs | Inference (ms) |
|
||||
|--------|------------|-------|----------------|
|
||||
| Baseline 1 [4] | 50M | 10G | 8.2 |
|
||||
| Baseline 2 [5] | 100M | 25G | 15.7 |
|
||||
| Baseline 3 [6] | 200M | 50G | 28.3 |
|
||||
| **Ours** | **80M** | **18G** | **12.1** |
|
||||
|
||||
Our method achieves superior performance while maintaining reasonable computational costs.
|
||||
|
||||
## V. DISCUSSION
|
||||
|
||||
### A. Analysis of Results
|
||||
|
||||
The experimental results demonstrate that [analysis].
|
||||
|
||||
### B. Limitations
|
||||
|
||||
Current limitations include:
|
||||
|
||||
1. Limitation 1: Description
|
||||
2. Limitation 2: Description
|
||||
3. Limitation 3: Description
|
||||
|
||||
### C. Broader Impact
|
||||
|
||||
Potential applications include:
|
||||
|
||||
- Application 1: Description
|
||||
- Application 2: Description
|
||||
- Application 3: Description
|
||||
|
||||
**Ethical Considerations**: [Discussion of potential risks and mitigation strategies]
|
||||
|
||||
## VI. CONCLUSION
|
||||
|
||||
This paper presented {{TITLE}}, which achieves [main achievement]. The key contributions are:
|
||||
|
||||
1. Contribution 1: Summary
|
||||
2. Contribution 2: Summary
|
||||
3. Contribution 3: Summary
|
||||
|
||||
Future work will focus on [future directions].
|
||||
|
||||
## ACKNOWLEDGMENTS
|
||||
|
||||
The authors thank [acknowledgments]. This work was supported by [funding sources].
|
||||
|
||||
## REFERENCES
|
||||
|
||||
[1] Author A et al., "Paper Title," *Conference Name*, 2020.
|
||||
|
||||
[2] Author B et al., "Paper Title," *Journal Name*, vol. X, no. Y, pp. Z-W, 2021.
|
||||
|
||||
[3] Author C et al., "Paper Title," *arXiv preprint arXiv:XXXX.XXXXX*, 2023.
|
||||
|
||||
[4] Author D et al., "Baseline 1 Paper," *Conference*, 2019.
|
||||
|
||||
[5] Author E et al., "Baseline 2 Paper," *Conference*, 2021.
|
||||
|
||||
[6] Author F et al., "Baseline 3 Paper," *Conference*, 2023.
|
||||
|
||||
---
|
||||
|
||||
## APPENDIX A: ADDITIONAL EXPERIMENTS
|
||||
|
||||
Supplementary experimental results.
|
||||
|
||||
## APPENDIX B: PROOF OF THEOREM
|
||||
|
||||
**Theorem 1**: Statement of theorem.
|
||||
|
||||
**Proof**: Detailed proof.
|
||||
|
||||
## APPENDIX C: HYPERPARAMETERS
|
||||
|
||||
Complete list of hyperparameters used in all experiments:
|
||||
|
||||
| Hyperparameter | Value | Description |
|
||||
|----------------|-------|-------------|
|
||||
| Learning rate | $10^{-4}$ | Initial learning rate |
|
||||
| Batch size | 32 | Training batch size |
|
||||
| Epochs | 100 | Number of training epochs |
|
||||
| Optimizer | AdamW | Optimization algorithm |
|
||||
| Weight decay | 0.01 | L2 regularization coefficient |
|
||||
| Warmup steps | 1000 | LR warmup duration |
|
||||
| Dropout | 0.1 | Dropout probability |
|
||||
|
||||
---
|
||||
|
||||
<style>
|
||||
.arxiv-header {
|
||||
text-align: center;
|
||||
margin-bottom: 2em;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: 'Computer Modern', serif;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
h1 {
|
||||
text-align: center;
|
||||
font-size: 1.8em;
|
||||
margin-top: 1em;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 1.3em;
|
||||
margin-top: 1.5em;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
h3 {
|
||||
font-size: 1.1em;
|
||||
font-style: italic;
|
||||
margin-top: 1em;
|
||||
}
|
||||
|
||||
table {
|
||||
margin: 1em auto;
|
||||
border-collapse: collapse;
|
||||
}
|
||||
|
||||
th, td {
|
||||
border: 1px solid #000;
|
||||
padding: 0.5em;
|
||||
text-align: center;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,358 @@
|
||||
---
|
||||
title: {{TITLE}}
|
||||
authors: {{AUTHORS}}
|
||||
date: {{DATE}}
|
||||
type: ml-experiment-report
|
||||
tags: [machine-learning, experiment-report]
|
||||
---
|
||||
|
||||
# {{TITLE}}
|
||||
|
||||
**Machine Learning Experiment Report**
|
||||
|
||||
**Researchers**: {{AUTHORS}}
|
||||
**Date**: {{DATE}}
|
||||
**Status**: Draft / Final / In Review
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
{{ABSTRACT}}
|
||||
|
||||
### Key Findings
|
||||
- Finding 1
|
||||
- Finding 2
|
||||
- Finding 3
|
||||
|
||||
### Recommendations
|
||||
- Recommendation 1
|
||||
- Recommendation 2
|
||||
|
||||
---
|
||||
|
||||
## 1. Objective
|
||||
|
||||
### 1.1 Research Question
|
||||
|
||||
What specific question are we trying to answer?
|
||||
|
||||
### 1.2 Success Criteria
|
||||
|
||||
How will we measure success?
|
||||
|
||||
- **Metric 1**: Target value
|
||||
- **Metric 2**: Target value
|
||||
- **Metric 3**: Target value
|
||||
|
||||
### 1.3 Constraints
|
||||
|
||||
- Computational budget
|
||||
- Time constraints
|
||||
- Data availability
|
||||
|
||||
---
|
||||
|
||||
## 2. Dataset
|
||||
|
||||
### 2.1 Data Description
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Name** | Dataset name |
|
||||
| **Source** | Origin of data |
|
||||
| **Size** | Number of examples |
|
||||
| **Features** | Feature count and types |
|
||||
| **Target** | What we're predicting |
|
||||
| **License** | Usage rights |
|
||||
|
||||
### 2.2 Data Splits
|
||||
|
||||
| Split | Size | Percentage |
|
||||
|-------|------|------------|
|
||||
| Train | X examples | Y% |
|
||||
| Validation | X examples | Y% |
|
||||
| Test | X examples | Y% |
|
||||
|
||||
### 2.3 Data Quality
|
||||
|
||||
- **Missing Values**: Analysis and handling
|
||||
- **Outliers**: Detection and treatment
|
||||
- **Imbalance**: Class distribution
|
||||
- **Preprocessing**: Transformations applied
|
||||
|
||||
### 2.4 Exploratory Analysis
|
||||
|
||||
Key insights from data exploration:
|
||||
|
||||
1. Pattern 1
|
||||
2. Pattern 2
|
||||
3. Pattern 3
|
||||
|
||||
---
|
||||
|
||||
## 3. Model
|
||||
|
||||
### 3.1 Architecture
|
||||
|
||||
Describe the model architecture:
|
||||
|
||||
```
|
||||
Input → Layer 1 → Layer 2 → ... → Output
|
||||
```
|
||||
|
||||
### 3.2 Model Specifications
|
||||
|
||||
| Component | Configuration |
|
||||
|-----------|--------------|
|
||||
| **Type** | Model family |
|
||||
| **Parameters** | Total count |
|
||||
| **Layers** | Number and types |
|
||||
| **Activation** | Functions used |
|
||||
| **Dropout** | Regularization rate |
|
||||
|
||||
### 3.3 Baseline Models
|
||||
|
||||
What are we comparing against?
|
||||
|
||||
1. **Baseline 1**: Simple baseline (e.g., majority class)
|
||||
2. **Baseline 2**: Standard approach (e.g., logistic regression)
|
||||
3. **Baseline 3**: Previous best method
|
||||
|
||||
---
|
||||
|
||||
## 4. Training
|
||||
|
||||
### 4.1 Hyperparameters
|
||||
|
||||
| Hyperparameter | Value | Rationale |
|
||||
|----------------|-------|-----------|
|
||||
| Learning Rate | 1e-4 | Tuned via grid search |
|
||||
| Batch Size | 32 | GPU memory constraint |
|
||||
| Epochs | 100 | Based on validation |
|
||||
| Optimizer | AdamW | Standard for transformers |
|
||||
| Weight Decay | 0.01 | Regularization |
|
||||
| LR Schedule | Cosine | Smooth convergence |
|
||||
|
||||
### 4.2 Training Process
|
||||
|
||||
```python
|
||||
# Training pseudocode
|
||||
for epoch in range(num_epochs):
|
||||
train_loss = train_one_epoch(model, train_loader)
|
||||
val_loss = validate(model, val_loader)
|
||||
if val_loss < best_loss:
|
||||
save_checkpoint(model)
|
||||
```
|
||||
|
||||
### 4.3 Computational Resources
|
||||
|
||||
| Resource | Specification |
|
||||
|----------|--------------|
|
||||
| **Hardware** | GPU model and count |
|
||||
| **Memory** | RAM and VRAM |
|
||||
| **Training Time** | Hours/days |
|
||||
| **Cost** | Estimated compute cost |
|
||||
|
||||
### 4.4 Training Curves
|
||||
|
||||
Include plots of:
|
||||
- Training loss over time
|
||||
- Validation loss over time
|
||||
- Learning rate schedule
|
||||
- Other relevant metrics
|
||||
|
||||
---
|
||||
|
||||
## 5. Results
|
||||
|
||||
### 5.1 Quantitative Results
|
||||
|
||||
| Model | Accuracy | Precision | Recall | F1 | AUC |
|
||||
|-------|----------|-----------|--------|-------|-----|
|
||||
| Baseline 1 | 0.65 | 0.64 | 0.66 | 0.65 | 0.70 |
|
||||
| Baseline 2 | 0.78 | 0.77 | 0.79 | 0.78 | 0.82 |
|
||||
| **Ours** | **0.89** | **0.88** | **0.90** | **0.89** | **0.93** |
|
||||
|
||||
### 5.2 Statistical Significance
|
||||
|
||||
- **P-value**: Statistical test results
|
||||
- **Confidence Intervals**: 95% CI for key metrics
|
||||
- **Multiple Runs**: Mean ± std over N runs
|
||||
|
||||
### 5.3 Per-Class Performance
|
||||
|
||||
| Class | Precision | Recall | F1 | Support |
|
||||
|-------|-----------|--------|-----|---------|
|
||||
| Class 1 | 0.90 | 0.88 | 0.89 | 500 |
|
||||
| Class 2 | 0.87 | 0.91 | 0.89 | 450 |
|
||||
| Class 3 | 0.88 | 0.89 | 0.88 | 550 |
|
||||
|
||||
### 5.4 Qualitative Results
|
||||
|
||||
#### Success Cases
|
||||
|
||||
Examples where the model performs well.
|
||||
|
||||
#### Failure Cases
|
||||
|
||||
Examples where the model fails and why.
|
||||
|
||||
---
|
||||
|
||||
## 6. Analysis
|
||||
|
||||
### 6.1 Ablation Study
|
||||
|
||||
| Configuration | Score | Change |
|
||||
|---------------|-------|--------|
|
||||
| Full Model | 0.89 | - |
|
||||
| - Feature Set A | 0.85 | -0.04 |
|
||||
| - Feature Set B | 0.87 | -0.02 |
|
||||
| - Augmentation | 0.86 | -0.03 |
|
||||
|
||||
### 6.2 Error Analysis
|
||||
|
||||
What types of errors is the model making?
|
||||
|
||||
1. **Error Type 1**: Frequency and cause
|
||||
2. **Error Type 2**: Frequency and cause
|
||||
3. **Error Type 3**: Frequency and cause
|
||||
|
||||
### 6.3 Feature Importance
|
||||
|
||||
Which features matter most?
|
||||
|
||||
| Feature | Importance | Notes |
|
||||
|---------|------------|-------|
|
||||
| Feature 1 | 0.35 | Most predictive |
|
||||
| Feature 2 | 0.28 | Secondary signal |
|
||||
| Feature 3 | 0.15 | Marginal impact |
|
||||
|
||||
---
|
||||
|
||||
## 7. Robustness
|
||||
|
||||
### 7.1 Cross-Dataset Evaluation
|
||||
|
||||
How does the model generalize to other datasets?
|
||||
|
||||
| Dataset | Score | Notes |
|
||||
|---------|-------|-------|
|
||||
| Original | 0.89 | Training distribution |
|
||||
| Dataset A | 0.82 | Similar domain |
|
||||
| Dataset B | 0.71 | Different domain |
|
||||
|
||||
### 7.2 Adversarial Robustness
|
||||
|
||||
Performance under adversarial conditions.
|
||||
|
||||
### 7.3 Fairness Analysis
|
||||
|
||||
Performance across demographic groups or sensitive attributes.
|
||||
|
||||
---
|
||||
|
||||
## 8. Deployment Considerations
|
||||
|
||||
### 8.1 Model Size
|
||||
|
||||
- **Parameters**: Total count
|
||||
- **Disk Size**: MB/GB on disk
|
||||
- **Memory**: Runtime memory usage
|
||||
|
||||
### 8.2 Inference Speed
|
||||
|
||||
| Batch Size | Latency | Throughput |
|
||||
|------------|---------|------------|
|
||||
| 1 | 10ms | 100 QPS |
|
||||
| 8 | 45ms | 178 QPS |
|
||||
| 32 | 150ms | 213 QPS |
|
||||
|
||||
### 8.3 Production Requirements
|
||||
|
||||
- **Dependencies**: Software requirements
|
||||
- **Infrastructure**: Hardware needs
|
||||
- **Monitoring**: What to track in production
|
||||
- **Fallback**: Backup strategy
|
||||
|
||||
---
|
||||
|
||||
## 9. Conclusions
|
||||
|
||||
### 9.1 Summary
|
||||
|
||||
Key takeaways from the experiment.
|
||||
|
||||
### 9.2 Did We Meet Objectives?
|
||||
|
||||
| Objective | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Objective 1 | ✅ Met | Achieved target |
|
||||
| Objective 2 | ⚠️ Partial | Close to target |
|
||||
| Objective 3 | ❌ Not Met | Needs more work |
|
||||
|
||||
### 9.3 Lessons Learned
|
||||
|
||||
What did we learn from this experiment?
|
||||
|
||||
1. Lesson 1
|
||||
2. Lesson 2
|
||||
3. Lesson 3
|
||||
|
||||
---
|
||||
|
||||
## 10. Next Steps
|
||||
|
||||
### 10.1 Short-term (1-2 weeks)
|
||||
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
- [ ] Task 3
|
||||
|
||||
### 10.2 Medium-term (1-2 months)
|
||||
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
- [ ] Task 3
|
||||
|
||||
### 10.3 Long-term (3+ months)
|
||||
|
||||
- [ ] Task 1
|
||||
- [ ] Task 2
|
||||
- [ ] Task 3
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
1. Reference 1
|
||||
2. Reference 2
|
||||
3. Reference 3
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### A. Hyperparameter Search
|
||||
|
||||
Results from hyperparameter tuning.
|
||||
|
||||
### B. Additional Experiments
|
||||
|
||||
Supplementary experiments not included in main text.
|
||||
|
||||
### C. Code
|
||||
|
||||
Links to code repositories:
|
||||
- Training code: [link]
|
||||
- Evaluation code: [link]
|
||||
- Model checkpoint: [link]
|
||||
|
||||
### D. Data Card
|
||||
|
||||
Detailed data documentation following standard practices.
|
||||
|
||||
### E. Model Card
|
||||
|
||||
Model documentation following responsible AI practices.
|
||||
@@ -0,0 +1,319 @@
|
||||
---
|
||||
title: {{TITLE}}
|
||||
authors: {{AUTHORS}}
|
||||
date: {{DATE}}
|
||||
arxiv:
|
||||
tags: [machine-learning, ai]
|
||||
layout: modern
|
||||
---
|
||||
|
||||
<div class="header">
|
||||
|
||||
# {{TITLE}}
|
||||
|
||||
<div class="authors">
|
||||
{{AUTHORS}}
|
||||
</div>
|
||||
|
||||
<div class="date">
|
||||
{{DATE}}
|
||||
</div>
|
||||
|
||||
<div class="links">
|
||||
[arXiv](#) · [PDF](#) · [Code](#) · [Demo](#)
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
<div class="abstract">
|
||||
|
||||
{{ABSTRACT}}
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
Modern research requires clear, accessible communication. This template provides a clean, web-friendly format inspired by Distill and modern scientific publications.
|
||||
|
||||
<div class="key-insight">
|
||||
💡 **Key Insight**: Present your main contribution upfront to engage readers immediately.
|
||||
</div>
|
||||
|
||||
### Why This Matters
|
||||
|
||||
Explain the significance of your work in plain language. What real-world problems does it solve?
|
||||
|
||||
### Our Approach
|
||||
|
||||
Summarize your methodology at a high level before diving into details.
|
||||
|
||||
---
|
||||
|
||||
## Background
|
||||
|
||||
<div class="definition">
|
||||
**Definition**: Clearly define key terms and concepts early in the paper.
|
||||
</div>
|
||||
|
||||
Provide context necessary to understand your contribution without overwhelming readers with details.
|
||||
|
||||
### Problem Statement
|
||||
|
||||
Formally state the problem you're addressing.
|
||||
|
||||
### Challenges
|
||||
|
||||
What makes this problem difficult?
|
||||
|
||||
1. **Challenge 1**: Description
|
||||
2. **Challenge 2**: Description
|
||||
3. **Challenge 3**: Description
|
||||
|
||||
---
|
||||
|
||||
## Method
|
||||
|
||||
Present your approach with clear visual aids and intuitive explanations.
|
||||
|
||||
<div class="figure">
|
||||
|
||||
```
|
||||
[Diagram of your architecture goes here]
|
||||
```
|
||||
|
||||
**Figure 1**: Overview of the proposed method. Caption explains the key components.
|
||||
|
||||
</div>
|
||||
|
||||
### Model Architecture
|
||||
|
||||
Describe your model systematically:
|
||||
|
||||
```python
|
||||
# Pseudocode example
|
||||
class YourModel:
|
||||
def __init__(self):
|
||||
self.encoder = Encoder()
|
||||
self.decoder = Decoder()
|
||||
|
||||
def forward(self, x):
|
||||
z = self.encoder(x)
|
||||
output = self.decoder(z)
|
||||
return output
|
||||
```
|
||||
|
||||
### Training Strategy
|
||||
|
||||
Explain how you train the model, including:
|
||||
|
||||
- **Objective Function**: Mathematical formulation
|
||||
- **Optimization**: Algorithm and hyperparameters
|
||||
- **Regularization**: Techniques to prevent overfitting
|
||||
|
||||
---
|
||||
|
||||
## Experiments
|
||||
|
||||
### Setup
|
||||
|
||||
<div class="experiment-details">
|
||||
|
||||
| Component | Configuration |
|
||||
|-----------|--------------|
|
||||
| **Dataset** | Name, Size, Split |
|
||||
| **Hardware** | GPU Type, RAM |
|
||||
| **Framework** | PyTorch 2.0, Transformers |
|
||||
| **Training Time** | Hours/Days |
|
||||
|
||||
</div>
|
||||
|
||||
### Results
|
||||
|
||||
Present results clearly with tables and visualizations.
|
||||
|
||||
<div class="results-table">
|
||||
|
||||
| Model | Accuracy | F1 Score | Params | Speed |
|
||||
|-------|----------|----------|--------|-------|
|
||||
| Baseline | 85.2% | 0.84 | 100M | 100 tok/s |
|
||||
| **Ours** | **92.1%** | **0.91** | 120M | 95 tok/s |
|
||||
| SOTA | 90.5% | 0.89 | 300M | 60 tok/s |
|
||||
|
||||
</div>
|
||||
|
||||
<div class="insight">
|
||||
🔍 **Observation**: Our method achieves state-of-the-art performance with fewer parameters.
|
||||
</div>
|
||||
|
||||
### Analysis
|
||||
|
||||
Deep dive into what the results reveal:
|
||||
|
||||
1. **Performance**: How does your method compare?
|
||||
2. **Efficiency**: What are the computational costs?
|
||||
3. **Robustness**: How does it perform across different scenarios?
|
||||
|
||||
---
|
||||
|
||||
## Ablation Study
|
||||
|
||||
Systematically evaluate each component's contribution.
|
||||
|
||||
<div class="ablation-results">
|
||||
|
||||
| Configuration | Score | Δ |
|
||||
|---------------|-------|---|
|
||||
| Full Model | 92.1% | - |
|
||||
| - Component A | 89.3% | -2.8% |
|
||||
| - Component B | 90.1% | -2.0% |
|
||||
| - Component C | 91.5% | -0.6% |
|
||||
|
||||
</div>
|
||||
|
||||
**Conclusion**: All components contribute meaningfully, with Component A being most critical.
|
||||
|
||||
---
|
||||
|
||||
## Discussion
|
||||
|
||||
### What We Learned
|
||||
|
||||
Synthesize insights from your experiments.
|
||||
|
||||
### Limitations
|
||||
|
||||
<div class="limitations">
|
||||
|
||||
⚠️ **Current Limitations**:
|
||||
|
||||
1. Performance on domain X is limited
|
||||
2. Computational requirements are high
|
||||
3. Requires large training datasets
|
||||
|
||||
</div>
|
||||
|
||||
### Future Directions
|
||||
|
||||
Where should the community go next?
|
||||
|
||||
- **Direction 1**: Description
|
||||
- **Direction 2**: Description
|
||||
- **Direction 3**: Description
|
||||
|
||||
---
|
||||
|
||||
## Related Work
|
||||
|
||||
Compare and contrast with existing methods.
|
||||
|
||||
### Prior Approaches
|
||||
|
||||
| Method | Year | Key Idea | Limitation |
|
||||
|--------|------|----------|------------|
|
||||
| Method A | 2020 | Approach 1 | Issue X |
|
||||
| Method B | 2021 | Approach 2 | Issue Y |
|
||||
| Method C | 2023 | Approach 3 | Issue Z |
|
||||
|
||||
### How We Differ
|
||||
|
||||
Clearly articulate what's novel about your work.
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
<div class="conclusion">
|
||||
|
||||
We presented **{{TITLE}}**, which achieves:
|
||||
|
||||
1. ✅ **Main contribution 1**
|
||||
2. ✅ **Main contribution 2**
|
||||
3. ✅ **Main contribution 3**
|
||||
|
||||
Our results demonstrate [key finding], opening new directions for [future work].
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Reproducibility
|
||||
|
||||
<div class="reproducibility">
|
||||
|
||||
### Code & Data
|
||||
|
||||
- **Code**: [github.com/username/repo](#)
|
||||
- **Models**: [huggingface.co/username/model](#)
|
||||
- **Datasets**: [huggingface.co/datasets/username/dataset](#)
|
||||
- **Demo**: [huggingface.co/spaces/username/demo](#)
|
||||
|
||||
### Citation
|
||||
|
||||
```bibtex
|
||||
@article{yourpaper2025,
|
||||
title={{{{TITLE}}}},
|
||||
author={{{{AUTHORS}}}},
|
||||
year={2025},
|
||||
journal={arXiv preprint}
|
||||
}
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thank funding agencies, collaborators, and computing resources that made this work possible.
|
||||
|
||||
---
|
||||
|
||||
<div class="appendix">
|
||||
|
||||
## Appendix
|
||||
|
||||
### A. Additional Results
|
||||
|
||||
Supplementary experiments and extended results.
|
||||
|
||||
### B. Hyperparameters
|
||||
|
||||
Complete training configuration:
|
||||
|
||||
```yaml
|
||||
learning_rate: 1e-4
|
||||
batch_size: 32
|
||||
epochs: 100
|
||||
optimizer: AdamW
|
||||
scheduler: cosine
|
||||
warmup_steps: 1000
|
||||
```
|
||||
|
||||
### C. Dataset Details
|
||||
|
||||
Detailed information about datasets used.
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
<style>
|
||||
.header { text-align: center; margin-bottom: 2em; }
|
||||
.authors { font-size: 1.2em; margin: 0.5em 0; }
|
||||
.date { color: #666; margin: 0.5em 0; }
|
||||
.links { margin-top: 1em; }
|
||||
.abstract { background: #f5f5f5; padding: 1.5em; border-radius: 8px; margin: 1em 0; }
|
||||
.key-insight, .insight { background: #e8f4f8; border-left: 4px solid #2196F3; padding: 1em; margin: 1em 0; }
|
||||
.definition { background: #fff3e0; border-left: 4px solid #ff9800; padding: 1em; margin: 1em 0; }
|
||||
.limitations { background: #ffebee; border-left: 4px solid #f44336; padding: 1em; margin: 1em 0; }
|
||||
.conclusion { background: #e8f5e9; border-left: 4px solid #4caf50; padding: 1.5em; margin: 1em 0; }
|
||||
.figure { text-align: center; margin: 2em 0; }
|
||||
.experiment-details, .results-table, .ablation-results { margin: 1em 0; }
|
||||
.reproducibility { background: #f5f5f5; padding: 1.5em; border-radius: 8px; margin: 2em 0; }
|
||||
</style>
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
title: {{TITLE}}
|
||||
authors: {{AUTHORS}}
|
||||
date: {{DATE}}
|
||||
arxiv:
|
||||
tags: [machine-learning, deep-learning]
|
||||
---
|
||||
|
||||
# {{TITLE}}
|
||||
|
||||
**{{AUTHORS}}**
|
||||
|
||||
*{{DATE}}*
|
||||
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
{{ABSTRACT}}
|
||||
|
||||
---
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
Provide background and motivation for your research. Explain:
|
||||
- What problem are you addressing?
|
||||
- Why is it important?
|
||||
- What is novel about your approach?
|
||||
|
||||
### 1.1 Motivation
|
||||
|
||||
Describe the real-world context and importance of the problem.
|
||||
|
||||
### 1.2 Contributions
|
||||
|
||||
List the main contributions of your work:
|
||||
1. First contribution
|
||||
2. Second contribution
|
||||
3. Third contribution
|
||||
|
||||
---
|
||||
|
||||
## 2. Related Work
|
||||
|
||||
Survey previous research relevant to your work. Organize by:
|
||||
- Different approaches to the problem
|
||||
- Complementary methods
|
||||
- Alternative solutions
|
||||
|
||||
### 2.1 Previous Approaches
|
||||
|
||||
Discuss earlier methods and their limitations.
|
||||
|
||||
### 2.2 Recent Advances
|
||||
|
||||
Highlight recent developments in the field.
|
||||
|
||||
---
|
||||
|
||||
## 3. Background
|
||||
|
||||
Provide necessary technical background for understanding your work.
|
||||
|
||||
### 3.1 Problem Formulation
|
||||
|
||||
Formally define the problem you're solving.
|
||||
|
||||
### 3.2 Preliminaries
|
||||
|
||||
Introduce key concepts, notation, and terminology.
|
||||
|
||||
---
|
||||
|
||||
## 4. Methodology
|
||||
|
||||
Describe your approach in detail.
|
||||
|
||||
### 4.1 Overview
|
||||
|
||||
Provide a high-level description of your method.
|
||||
|
||||
### 4.2 Model Architecture
|
||||
|
||||
Detail the technical components of your system.
|
||||
|
||||
### 4.3 Training Procedure
|
||||
|
||||
Explain how the model is trained.
|
||||
|
||||
### 4.4 Implementation Details
|
||||
|
||||
Provide reproducibility information:
|
||||
- Hyperparameters
|
||||
- Hardware requirements
|
||||
- Software dependencies
|
||||
|
||||
---
|
||||
|
||||
## 5. Experiments
|
||||
|
||||
Present your experimental setup and results.
|
||||
|
||||
### 5.1 Datasets
|
||||
|
||||
Describe the datasets used for evaluation.
|
||||
|
||||
### 5.2 Evaluation Metrics
|
||||
|
||||
Define the metrics used to assess performance.
|
||||
|
||||
### 5.3 Baselines
|
||||
|
||||
List comparison methods.
|
||||
|
||||
### 5.4 Experimental Setup
|
||||
|
||||
Detail the experimental configuration.
|
||||
|
||||
---
|
||||
|
||||
## 6. Results
|
||||
|
||||
Present and analyze your findings.
|
||||
|
||||
### 6.1 Main Results
|
||||
|
||||
Report primary experimental results.
|
||||
|
||||
| Model | Dataset | Metric | Score |
|
||||
|-------|---------|--------|-------|
|
||||
| Baseline | Dataset A | Accuracy | 0.85 |
|
||||
| Ours | Dataset A | Accuracy | 0.92 |
|
||||
|
||||
### 6.2 Ablation Studies
|
||||
|
||||
Analyze the contribution of different components.
|
||||
|
||||
### 6.3 Qualitative Analysis
|
||||
|
||||
Provide examples and case studies.
|
||||
|
||||
---
|
||||
|
||||
## 7. Discussion
|
||||
|
||||
Interpret your results and discuss implications.
|
||||
|
||||
### 7.1 Analysis
|
||||
|
||||
What do the results tell us?
|
||||
|
||||
### 7.2 Limitations
|
||||
|
||||
Acknowledge limitations of your approach.
|
||||
|
||||
### 7.3 Broader Impact
|
||||
|
||||
Discuss societal implications and potential applications.
|
||||
|
||||
---
|
||||
|
||||
## 8. Conclusion
|
||||
|
||||
Summarize your work and contributions.
|
||||
|
||||
### 8.1 Summary
|
||||
|
||||
Recap the main findings.
|
||||
|
||||
### 8.2 Future Work
|
||||
|
||||
Suggest directions for future research.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thank collaborators, funding sources, and computational resources.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
1. Author A, et al. "Paper Title." Conference/Journal, Year.
|
||||
2. Author B, et al. "Another Paper." Conference/Journal, Year.
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### A. Additional Experiments
|
||||
|
||||
Supplementary experimental results.
|
||||
|
||||
### B. Implementation Details
|
||||
|
||||
Code snippets and configuration details.
|
||||
|
||||
### C. Hyperparameters
|
||||
|
||||
Complete list of hyperparameters used.
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
name: huggingface-papers
|
||||
description: Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata such as authors, linked models/datasets/spaces, Github repo and project page. Use when the user shares a Hugging Face paper page URL, an arXiv URL or ID, or asks to summarize, explain, or analyze an AI research paper.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Hugging Face Paper Pages
|
||||
|
||||
Hugging Face Paper pages (hf.co/papers) is a platform built on top of arXiv (arxiv.org), specifically for research papers in the field of artificial intelligence (AI) and computer science. Hugging Face users can submit their paper at hf.co/papers/submit, which features it on the Daily Papers feed (hf.co/papers). Each day, users can upvote papers and comment on papers. Each paper page allows authors to:
|
||||
- claim their paper (by clicking their name on the `authors` field). This makes the paper page appear on their Hugging Face profile.
|
||||
- link the associated model checkpoints, datasets and Spaces by including the HF paper or arXiv URL in the model card, dataset card or README of the Space
|
||||
- link the Github repository and/or project page URLs
|
||||
- link the HF organization. This also makes the paper page appear on the Hugging Face organization page.
|
||||
|
||||
Whenever someone mentions a HF paper or arXiv abstract/PDF URL in a model card, dataset card or README of a Space repository, the paper will be automatically indexed. Note that not all papers indexed on Hugging Face are also submitted to daily papers. The latter is more a manner of promoting a research paper. Papers can only be submitted to daily papers up until 14 days after their publication date on arXiv.
|
||||
|
||||
The Hugging Face team has built an easy-to-use API to interact with paper pages. Content of the papers can be fetched as markdown, or structured metadata can be returned such as author names, linked models/datasets/spaces, linked Github repo and project page.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User shares a Hugging Face paper page URL (e.g. `https://huggingface.co/papers/2602.08025`)
|
||||
- User shares a Hugging Face markdown paper page URL (e.g. `https://huggingface.co/papers/2602.08025.md`)
|
||||
- User shares an arXiv URL (e.g. `https://arxiv.org/abs/2602.08025` or `https://arxiv.org/pdf/2602.08025`)
|
||||
- User mentions a arXiv ID (e.g. `2602.08025`)
|
||||
- User asks you to summarize, explain, or analyze an AI research paper
|
||||
|
||||
## Parsing the paper ID
|
||||
|
||||
It's recommended to parse the paper ID (arXiv ID) from whatever the user provides:
|
||||
|
||||
| Input | Paper ID |
|
||||
| --- | --- |
|
||||
| `https://huggingface.co/papers/2602.08025` | `2602.08025` |
|
||||
| `https://huggingface.co/papers/2602.08025.md` | `2602.08025` |
|
||||
| `https://arxiv.org/abs/2602.08025` | `2602.08025` |
|
||||
| `https://arxiv.org/pdf/2602.08025` | `2602.08025` |
|
||||
| `2602.08025v1` | `2602.08025v1` |
|
||||
| `2602.08025` | `2602.08025` |
|
||||
|
||||
This allows you to provide the paper ID into any of the hub API endpoints mentioned below.
|
||||
|
||||
### Fetch the paper page as markdown
|
||||
|
||||
The content of a paper can be fetched as markdown like so:
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/papers/{PAPER_ID}.md"
|
||||
```
|
||||
|
||||
This should return the Hugging Face paper page as markdown. This relies on the HTML version of the paper at https://arxiv.org/html/{PAPER_ID}.
|
||||
|
||||
There are 2 exceptions:
|
||||
- Not all arXiv papers have an HTML version. If the HTML version of the paper does not exist, then the content falls back to the HTML of the Hugging Face paper page.
|
||||
- If it results in a 404, it means the paper is not yet indexed on hf.co/papers. See [Error handling](#error-handling) for info.
|
||||
|
||||
Alternatively, you can request markdown from the normal paper page URL, like so:
|
||||
|
||||
```bash
|
||||
curl -s -H "Accept: text/markdown" "https://huggingface.co/papers/{PAPER_ID}"
|
||||
```
|
||||
|
||||
### Paper Pages API Endpoints
|
||||
|
||||
All endpoints use the base URL `https://huggingface.co`.
|
||||
|
||||
#### Get structured metadata
|
||||
|
||||
Fetch the paper metadata as JSON using the Hugging Face REST API:
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/papers/{PAPER_ID}"
|
||||
```
|
||||
|
||||
This returns structured metadata that can include:
|
||||
|
||||
- authors (names and Hugging Face usernames, in case they have claimed the paper)
|
||||
- media URLs (uploaded when submitting the paper to Daily Papers)
|
||||
- summary (abstract) and AI-generated summary
|
||||
- project page and GitHub repository
|
||||
- organization and engagement metadata (number of upvotes)
|
||||
|
||||
To find models linked to the paper, use:
|
||||
|
||||
```bash
|
||||
curl https://huggingface.co/api/models?filter=arxiv:{PAPER_ID}
|
||||
```
|
||||
|
||||
To find datasets linked to the paper, use:
|
||||
|
||||
```bash
|
||||
curl https://huggingface.co/api/datasets?filter=arxiv:{PAPER_ID}
|
||||
```
|
||||
|
||||
To find spaces linked to the paper, use:
|
||||
|
||||
```bash
|
||||
curl https://huggingface.co/api/spaces?filter=arxiv:{PAPER_ID}
|
||||
```
|
||||
|
||||
#### Claim paper authorship
|
||||
|
||||
Claim authorship of a paper for a Hugging Face user:
|
||||
|
||||
```bash
|
||||
curl "https://huggingface.co/api/settings/papers/claim" \
|
||||
--request POST \
|
||||
--header "Content-Type: application/json" \
|
||||
--header "Authorization: Bearer $HF_TOKEN" \
|
||||
--data '{
|
||||
"paperId": "{PAPER_ID}",
|
||||
"claimAuthorId": "{AUTHOR_ENTRY_ID}",
|
||||
"targetUserId": "{USER_ID}"
|
||||
}'
|
||||
```
|
||||
|
||||
- Endpoint: `POST /api/settings/papers/claim`
|
||||
- Body:
|
||||
- `paperId` (string, required): arXiv paper identifier being claimed
|
||||
- `claimAuthorId` (string): author entry on the paper being claimed, 24-char hex ID
|
||||
- `targetUserId` (string): HF user who should receive the claim, 24-char hex ID
|
||||
- Response: paper authorship claim result, including the claimed paper ID
|
||||
|
||||
#### Get daily papers
|
||||
|
||||
Fetch the Daily Papers feed:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $HF_TOKEN" \
|
||||
"https://huggingface.co/api/daily_papers?p=0&limit=20&date=2017-07-21&sort=publishedAt"
|
||||
```
|
||||
|
||||
- Endpoint: `GET /api/daily_papers`
|
||||
- Query parameters:
|
||||
- `p` (integer): page number
|
||||
- `limit` (integer): number of results, between 1 and 100
|
||||
- `date` (string): RFC 3339 full-date, for example `2017-07-21`
|
||||
- `week` (string): ISO week, for example `2024-W03`
|
||||
- `month` (string): month value, for example `2024-01`
|
||||
- `submitter` (string): filter by submitter
|
||||
- `sort` (enum): `publishedAt` or `trending`
|
||||
- Response: list of daily papers
|
||||
|
||||
#### List papers
|
||||
|
||||
List arXiv papers sorted by published date:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $HF_TOKEN" \
|
||||
"https://huggingface.co/api/papers?cursor={CURSOR}&limit=20"
|
||||
```
|
||||
|
||||
- Endpoint: `GET /api/papers`
|
||||
- Query parameters:
|
||||
- `cursor` (string): pagination cursor
|
||||
- `limit` (integer): page size, 1-100 (same bounds as the daily-papers endpoint above)
|
||||
- Response: list of papers
|
||||
|
||||
#### Search papers
|
||||
|
||||
Perform hybrid semantic and full-text search on papers:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $HF_TOKEN" \
|
||||
"https://huggingface.co/api/papers/search?q=vision+language&limit=20"
|
||||
```
|
||||
|
||||
This searches over the paper title, authors, and content.
|
||||
|
||||
- Endpoint: `GET /api/papers/search`
|
||||
- Query parameters:
|
||||
- `q` (string): search query, max length 250
|
||||
- `limit` (integer): number of results, between 1 and 120
|
||||
- Response: matching papers
|
||||
|
||||
#### Index a paper
|
||||
|
||||
Insert a paper from arXiv by ID. If the paper is already indexed, only its authors can re-index it:
|
||||
|
||||
```bash
|
||||
curl "https://huggingface.co/api/papers/index" \
|
||||
--request POST \
|
||||
--header "Content-Type: application/json" \
|
||||
--header "Authorization: Bearer $HF_TOKEN" \
|
||||
--data '{
|
||||
"arxivId": "{ARXIV_ID}"
|
||||
}'
|
||||
```
|
||||
|
||||
- Endpoint: `POST /api/papers/index`
|
||||
- Body:
|
||||
- `arxivId` (string, required): arXiv ID to index, for example `2301.00001`
|
||||
- Pattern: `^\d{4}\.\d{4,5}$`
|
||||
- Response: empty JSON object on success
|
||||
|
||||
#### Update paper links
|
||||
|
||||
Update the project page, GitHub repository, or submitting organization for a paper. The requester must be the paper author, the Daily Papers submitter, or a papers admin:
|
||||
|
||||
```bash
|
||||
curl "https://huggingface.co/api/papers/{PAPER_OBJECT_ID}/links" \
|
||||
--request POST \
|
||||
--header "Content-Type: application/json" \
|
||||
--header "Authorization: Bearer $HF_TOKEN" \
|
||||
--data '{
|
||||
"projectPage": "https://example.com",
|
||||
"githubRepo": "https://github.com/org/repo",
|
||||
"organizationId": "{ORGANIZATION_ID}"
|
||||
}'
|
||||
```
|
||||
|
||||
- Endpoint: `POST /api/papers/{paperId}/links`
|
||||
- Path parameters:
|
||||
- `paperId` (string, required): Hugging Face paper object ID
|
||||
- Body:
|
||||
- `githubRepo` (string, nullable): GitHub repository URL
|
||||
- `organizationId` (string, nullable): organization ID, 24-char hex ID
|
||||
- `projectPage` (string, nullable): project page URL
|
||||
- Response: empty JSON object on success
|
||||
|
||||
## Error Handling
|
||||
|
||||
- **404 on `https://huggingface.co/papers/{PAPER_ID}` or `md` endpoint**: the paper is not indexed on Hugging Face paper pages yet.
|
||||
- **404 on `/api/papers/{PAPER_ID}`**: the paper may not be indexed on Hugging Face paper pages yet.
|
||||
- **Paper ID not found**: verify the extracted arXiv ID, including any version suffix
|
||||
|
||||
### Fallbacks
|
||||
|
||||
If the Hugging Face paper page does not contain enough detail for the user's question:
|
||||
|
||||
- Check the regular paper page at `https://huggingface.co/papers/{PAPER_ID}`
|
||||
- Fall back to the arXiv page or PDF for the original source:
|
||||
- `https://arxiv.org/abs/{PAPER_ID}`
|
||||
- `https://arxiv.org/pdf/{PAPER_ID}`
|
||||
|
||||
## Notes
|
||||
|
||||
- No authentication is required for public paper pages.
|
||||
- Write endpoints such as claim authorship, index paper, and update paper links require `Authorization: Bearer $HF_TOKEN`.
|
||||
- Prefer the `.md` endpoint for reliable machine-readable output.
|
||||
- Prefer `/api/papers/{PAPER_ID}` when you need structured JSON fields instead of page markdown.
|
||||
@@ -0,0 +1,241 @@
|
||||
---
|
||||
name: huggingface-spaces
|
||||
description: Build, deploy, and maintain applications on Hugging Face Spaces — Gradio / Docker / Static SDKs, ZeroGPU and dedicated hardware, model loading, debugging, buckets, inference providers, community grants. Use whenever the user asks to create or host an app on Hugging Face, port code onto ZeroGPU, fix a Space that won't build or run, or otherwise work with `hf spaces …`, `@spaces.GPU`, Space README frontmatter, or the `spaces` Python package.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Hugging Face Spaces
|
||||
|
||||
Hugging Face Spaces host machine-learning applications. There are 1M+ today; each Space is a git repo. This skill covers creating, building, debugging, and maintaining them.
|
||||
|
||||
## 0. Getting ready
|
||||
|
||||
Before anything else:
|
||||
|
||||
1. Check the `hf` CLI is installed: `which hf`. If not, `pip install -U huggingface_hub`.
|
||||
2. Check the user is logged in: `hf auth whoami`. If not, run `hf auth login` — it prints a URL and a one-time code; ask the user to open the URL and enter the code, then login completes automatically (OAuth, no token needed). Alternatively, pass a write-scoped token from https://huggingface.co/settings/tokens with `--token`.
|
||||
3. Note `whoami`'s `canPay` and `isPro` flags — they gate hardware choices below.
|
||||
|
||||
The `hf-cli` skill teaches an agent every `hf` command and is the recommended companion to this one. Install it with `hf skills add hf-cli` (add `--claude --global` to install for Claude Code as well, user-level).
|
||||
|
||||
## 1. What a Space is
|
||||
|
||||
A Space is a git repo with three possible SDKs:
|
||||
|
||||
- **Gradio** — most Spaces. Python, fast iteration, supports ZeroGPU.
|
||||
- **Docker** — arbitrary container. Use when you need a non-Python stack or a pre-built template (Streamlit, Argilla, Shiny, etc. — full list at https://huggingface.co/docs/hub/spaces-sdks-docker). Does **not** support ZeroGPU.
|
||||
- **Static** — plain HTML, or a React/Svelte/Vue project built at deploy time. Use for in-browser ML (transformers.js / WebGPU / WebAssembly / onnxruntime-web), project pages, interactive reports, or Spaces that orchestrate other Spaces. No hardware needed.
|
||||
|
||||
### Hardware tiers
|
||||
|
||||
Free, no creator cost: **`cpu-basic`** and **`zero-a10g`** (ZeroGPU). Static Spaces are also free and don't need hardware.
|
||||
|
||||
**`cpu-basic`** — 2 vCPU / 16 GB. For data viz, API-proxy Spaces, small CPU-bound models.
|
||||
|
||||
**ZeroGPU (`zero-a10g`)** — dynamic, per-request GPU allocation on NVIDIA RTX PRO 6000 Blackwell (sm_120). Two sizes: `large` (half MIG, 48 GB, 1× quota) and `xlarge` (full, 96 GB, 2× quota). Free for the Space creator; Space visitors consume their own daily quota (~5 min free / 40 min Pro / 60 min Enterprise). **Gradio-only**, **PyTorch-first**. Requires the creator to be on a PRO / Team / Enterprise plan.
|
||||
|
||||
**Dedicated GPU** (T4, L4, A10G, L40S, A100, H200) — billed to the Space creator by the hour. List + pricing: `hf spaces hardware`. Only the creator can attach these, and only if `canPay=True`. Use when ZeroGPU genuinely doesn't fit — non-PyTorch main model with heavy init, very-large-model long-context inference, etc.
|
||||
|
||||
If a non-PRO user has a use case that wants ZeroGPU, you can still build it: create a `cpu-basic` Space, code the app for ZeroGPU, push, then request a community grant. See [`references/grants.md`](references/grants.md).
|
||||
|
||||
For the authoritative reference: https://huggingface.co/docs/hub/spaces-overview
|
||||
|
||||
## 2. Look for an existing demo first
|
||||
|
||||
Before deciding how to build anything, search for prior art:
|
||||
|
||||
```bash
|
||||
hf spaces search "<model name or task>" --sdk gradio --limit 10
|
||||
```
|
||||
|
||||
If someone has built a similar Space, read its `app.py` and `requirements.txt` — that gives you the working pattern. Saves a lot of blind iteration. Mention to the user what you found before committing to an approach.
|
||||
|
||||
## 3. Decide SDK and hardware
|
||||
|
||||
Follow the user's explicit request first. If they were vague:
|
||||
|
||||
- **Default for a public ML demo**: Gradio + ZeroGPU. Use this unless something below applies.
|
||||
- **The model's only inference path is non-PyTorch** (ONNX / TF / JAX / vLLM as the MAIN model, with heavy init): dedicated GPU.
|
||||
- But: marginal non-torch tools (a small ONNX preprocessor, a TF utility) inside a torch-main pipeline are fine on ZeroGPU. The hijack only patches torch; init the non-torch lib inside `@spaces.GPU` and pay the short per-call init cost.
|
||||
- **Tiny / CPU-bound model, or API-proxy Space**: `cpu-basic` (`hardware`-free isn't applicable to Gradio).
|
||||
- **Browser-side ML or project page**: Static.
|
||||
- **Container with non-Python stack**: Docker.
|
||||
|
||||
### Sourcing the model
|
||||
|
||||
- **GitHub repo** — clone locally to read structure. If it already has a Gradio demo, the minimal viable path is to adapt it onto ZeroGPU (see [`references/zerogpu.md`](references/zerogpu.md)). Otherwise: read the README + inference code, prefer the PyTorch path, estimate VRAM (bf16 ≈ `params_B × 2` GB; 48 GB fits ≤24B params at bf16, or much larger with quantization — see [`references/zerogpu.md`](references/zerogpu.md) for quantization on ZeroGPU).
|
||||
- **HF model repo** — read its README, follow any linked GitHub.
|
||||
- **Paper / blog post** — look for an official or unofficial implementation. Don't reimplement unless trivial or the user explicitly asks.
|
||||
- **Vague request** — search Spaces first; surface results.
|
||||
|
||||
If the model genuinely won't fit, check **Inference Providers** as an alternative: see [`references/inference-providers.md`](references/inference-providers.md). This avoids hosting the model at all.
|
||||
|
||||
## 4. Create the Space
|
||||
|
||||
```bash
|
||||
hf repos create <namespace>/<name> --type space --space-sdk <gradio|docker|static> \
|
||||
[--flavor zero-a10g|cpu-basic|<paid-flavor>] \
|
||||
[--secrets KEY=val] [--env KEY=val] \
|
||||
--public|--private|--protected \
|
||||
--exist-ok
|
||||
```
|
||||
|
||||
- `--space-sdk` is required.
|
||||
- `--flavor` selects hardware. `zero-a10g` is the (legacy) identifier for ZeroGPU. Omit for `cpu-basic`. Run `hf spaces hardware` for the full paid list and pricing.
|
||||
- Visibility: `--public` (anyone can view), `--private` (only you), `--protected` (app is reachable but git repo / Files tab is private).
|
||||
- `--secrets KEY=val` becomes an environment variable inside the Space and is **not** visible to visitors. Use for API keys, gated-repo tokens (`HF_TOKEN=hf_…`), etc. Can also be set later via `hf spaces secrets set <id> KEY=val`.
|
||||
- `--env KEY=val` is **visible to visitors** — use only for non-sensitive config (`GRADIO_SSR_MODE=false`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, etc.).
|
||||
|
||||
> Note: `hardware:` in the README YAML is silently ignored — hardware is only set via `--flavor` at creation, or later via `hf spaces settings <id> --hardware <name>`.
|
||||
|
||||
## 5. Build the app
|
||||
|
||||
The Space now exists at `https://huggingface.co/spaces/<namespace>/<name>` but is empty.
|
||||
|
||||
### README.md frontmatter
|
||||
|
||||
Always required:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: ...
|
||||
emoji: 🚀 # pick something representative
|
||||
colorFrom: blue # red|yellow|green|blue|indigo|purple|pink|gray (only these)
|
||||
colorTo: indigo
|
||||
sdk: gradio # gradio | docker | static
|
||||
sdk_version: 6.15.1 # latest stable unless you have a reason*
|
||||
app_file: app.py # gradio only (docker / static use Dockerfile / index.html)
|
||||
short_description: ... # ≤ 60 chars (server rejects longer)
|
||||
python_version: "3.12" # ZeroGPU officially supports 3.10.13 and 3.12.12
|
||||
startup_duration_timeout: 30m # default; bump to 1h for big LLMs / heavy downloads
|
||||
---
|
||||
```
|
||||
|
||||
\* Default to the current latest stable, and **look up what that is** (`pip index versions gradio`, or the version a freshly-created Space defaults to) — the number above is a placeholder that goes stale, don't reuse it. Only pin older when the latest genuinely doesn't work for this Space: a custom component pins it, or you're adapting an existing demo and don't want to rewrite for 5.x→6.x breaking changes. If you need a 5.x, pick `5.50.0` (latest of the series; still supports custom components).
|
||||
|
||||
All frontmatter options: https://huggingface.co/docs/hub/spaces-config-reference
|
||||
|
||||
### Minimal ZeroGPU Gradio app
|
||||
|
||||
```python
|
||||
import spaces # MUST come before torch / diffusers / transformers
|
||||
import torch
|
||||
import gradio as gr
|
||||
from diffusers import DiffusionPipeline
|
||||
|
||||
pipe = DiffusionPipeline.from_pretrained("<repo>", torch_dtype=torch.bfloat16).to("cuda")
|
||||
|
||||
@spaces.GPU(duration=60)
|
||||
def generate(prompt: str):
|
||||
"""Generate an image from a text prompt.""" # docstring → API / MCP tool description
|
||||
return pipe(prompt).images[0]
|
||||
|
||||
gr.Interface(fn=generate, inputs=gr.Text(), outputs=gr.Image()).launch(mcp_server=True)
|
||||
```
|
||||
|
||||
Three rules — full treatment in [`references/zerogpu.md`](references/zerogpu.md):
|
||||
|
||||
1. **`import spaces` before torch / any CUDA-touching import.** It monkey-patches `torch.cuda.*`; once CUDA is initialized in the main process, it's too late.
|
||||
2. **Load the model at module scope, `.to("cuda")` eagerly.** ZeroGPU intercepts the call, packs weights to disk, and streams them into VRAM on the first `@spaces.GPU` entry. Lazy loading inside the decorator costs every user.
|
||||
3. **Decorate the function Gradio binds.** Estimate `duration` to the realistic worst case (smaller = higher queue priority and tighter quota check). For input-dependent runtime, pass a callable.
|
||||
|
||||
### Examples, docstrings, and MCP
|
||||
|
||||
- **Add `gr.Examples` whenever it makes sense** (the app takes input and representative inputs exist) — prefer the model/repo's own official examples. Keep example rows to the few inputs a user actually varies (prompt, image) and give the handler defaults for the rest (steps, seed, guidance) so a row is `["a prompt"]`, not a wall of knobs. Use `cache_examples=True, cache_mode="lazy"`. See [`references/gradio.md`](references/gradio.md).
|
||||
- **Give every API-triggered function a docstring and type hints.** Each Gradio event handler is exposed over the API; the docstring + signature are what a caller — and the MCP tool schema — sees.
|
||||
- **Launch with `demo.launch(mcp_server=True)`** (Gradio 5+) so the Space doubles as an MCP server: each API function becomes an MCP tool described by its docstring and hints.
|
||||
|
||||
### requirements.txt
|
||||
|
||||
Short version:
|
||||
|
||||
- **Do NOT list**: `gradio`, `spaces`, `huggingface_hub` (preinstalled and platform-managed; pinning them causes resolution failures or silently breaks the ZeroGPU runtime).
|
||||
- **Do list if you use them**: `torchvision`, `torchaudio` (not preinstalled), plus everything else (`diffusers`, `transformers`, `accelerate`, `sentencepiece`, …).
|
||||
- ZeroGPU only accepts torch `2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`. Default to leaving torch unpinned (the runtime preinstalls the latest). Only pin when a dep forces it.
|
||||
- For prebuilt CUDA-extension wheels (`flash_attn`, `xformers`, `pytorch3d`, `nvdiffrast`, `diff_gaussian_rasterization`, `torchmcubes`): use the prebuilt Blackwell wheels at `https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/tree/main/wheels`. Full mapping + caveats in [`references/requirements.md`](references/requirements.md).
|
||||
|
||||
### Per-SDK depth
|
||||
|
||||
- **Gradio patterns** (themes, `gr.Examples`, streaming, custom HTML components, `gr.Server`): [`references/gradio.md`](references/gradio.md).
|
||||
- **Docker**: https://huggingface.co/docs/hub/spaces-sdks-docker. Examples: `hf spaces list --filter docker`.
|
||||
- **Static**: https://huggingface.co/docs/hub/spaces-sdks-static. For built SPAs, set `app_build_command: npm run build` and `app_file: dist/index.html` in frontmatter.
|
||||
- **ZeroGPU specifics** (decorator semantics, sizing, AoTI, generators, concurrency, pickle / `gr.State` across the worker boundary): [`references/zerogpu.md`](references/zerogpu.md) — read this whenever the Space targets ZeroGPU.
|
||||
|
||||
|
||||
## 6. Iterate on the Space, not locally
|
||||
|
||||
Try to build a release candidate from the user quest locally and push it — then use the live URL as your test loop. The Space environment is the only one that matters; do not try to test locally. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing.
|
||||
|
||||
Push files with `hf upload <namespace>/<name> . --repo-type space`. **`--repo-type space` is required** — `hf upload` defaults to a *model* repo and will otherwise upload to (and silently create) a model repo of the same name. Add `--exclude "**/__pycache__/**"` so local bytecode caches aren't committed into the Space.
|
||||
|
||||
Once pushed, pick the cheapest update mechanism for each change — hot-reload for pure Python edits, `hf upload` for code-only files hot-reload can't touch, full rebuild only when `requirements.txt` / `Dockerfile` / README frontmatter actually changed. Full ladder + footguns (hot-reload poisoning factory reboot, runtime.sha lag, etc.) in [`references/debugging.md`](references/debugging.md).
|
||||
|
||||
## 7. Verify
|
||||
|
||||
Don't trust `RUNNING` alone — the app can be running but broken. Four steps, in order:
|
||||
|
||||
**A. Alive?** Stage + hardware:
|
||||
```bash
|
||||
hf spaces info <ns>/<name> --expand runtime
|
||||
```
|
||||
|
||||
**B. Logs clean post-boot?** Read the run log to confirm startup finished without warnings or silent fallbacks:
|
||||
```bash
|
||||
hf spaces logs <ns>/<name> --tail 200
|
||||
```
|
||||
Look for model-load completion, no import warnings, no "falling back to CPU" / dtype downgrade messages, no `RUNNING` masking a half-broken app.
|
||||
|
||||
**C. API actually responds.** With logs still tailing in another terminal (`hf spaces logs <ns>/<name> --follow`), call the endpoint:
|
||||
```python
|
||||
from gradio_client import Client, handle_file
|
||||
import os
|
||||
c = Client("<ns>/<name>", token=os.environ["HF_TOKEN"], httpx_kwargs={"timeout": 600})
|
||||
print(c.view_api()) # discover endpoints — don't guess
|
||||
result = c.predict(..., api_name="/generate")
|
||||
```
|
||||
|
||||
**D. Sniff output AND logs.** HTTP 200 ≠ correct output. Check both:
|
||||
```python
|
||||
head = open(result, "rb").read(16)
|
||||
# glTF / \x89PNG / RIFF…WEBP / RIFF…WAVE / [4:8]==b"ftyp" → png/jpg/webp/wav/mp4
|
||||
```
|
||||
And look at the run log emitted during the call — silent fallbacks (model snapping to a different size, missing optional dep, dtype downgrade) only show up there.
|
||||
|
||||
Full smoke-test patterns (streaming endpoints, OAuth-gated Spaces, `gr.Server` custom routes): [`references/debugging.md`](references/debugging.md).
|
||||
|
||||
## 8. Permanent storage (buckets)
|
||||
|
||||
Spaces are stateless — `/data` is wiped on restart. If the Space needs to persist user uploads, generations, logs, or interact with a long-lived store, mount a **bucket**:
|
||||
|
||||
```bash
|
||||
hf buckets create <ns>/<bucket-name> # --private optional
|
||||
hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data # read-write at /data
|
||||
```
|
||||
|
||||
Buckets are paid storage; check `canPay` and confirm with the user. Full patterns (read-fast / write-durable, public bucket URLs, model-cache anti-pattern): [`references/buckets.md`](references/buckets.md).
|
||||
|
||||
## 9. When things break
|
||||
|
||||
Order of operations:
|
||||
|
||||
1. Read the logs: `hf spaces logs <id> --build --follow` (build error) or `hf spaces logs <id> --follow` (runtime error). Find the **first** error, not the last.
|
||||
2. Grep [`references/known-errors.md`](references/known-errors.md) for the error string. Check if this is a known issue before trying your own fix — most common ZeroGPU / Gradio / dependency errors have a 1–2 line fix there.
|
||||
3. Iterate using the cheapest rung from [`references/debugging.md`](references/debugging.md). The vast majority of issues resolve with log-reading + smoke-test loops; interactive dev mode + SSH is a heavy-hammer last resort.
|
||||
|
||||
If you solve an error that wasn't in the known-errors list, suggest the user PR it back to this skill so future runs benefit.
|
||||
|
||||
---
|
||||
|
||||
## Reference index
|
||||
|
||||
| When to read | File |
|
||||
|---|---|
|
||||
| **How ZeroGPU works** + correct patterns (decorator, sizing, pickle, generators, real-time, AoTI) | [`references/zerogpu.md`](references/zerogpu.md) |
|
||||
| **Iterate + debug**: logs, rung ladder, smoke testing (and dev mode + SSH as a last resort) | [`references/debugging.md`](references/debugging.md) |
|
||||
| **Error-string lookup** — the single place for all error symptoms (Spaces, ZeroGPU, Gradio, deps) | [`references/known-errors.md`](references/known-errors.md) |
|
||||
| Pinning deps, picking wheels, torch-family alignment | [`references/requirements.md`](references/requirements.md) |
|
||||
| `gr.Examples` (add when it makes sense), themes, custom HTML components, `gr.Server`, MCP server (`mcp_server=True`) | [`references/gradio.md`](references/gradio.md) |
|
||||
| Persistent storage, public bucket URLs | [`references/buckets.md`](references/buckets.md) |
|
||||
| Community grant requests (non-PRO needing ZeroGPU) | [`references/grants.md`](references/grants.md) |
|
||||
| Provider proxy (zero-VRAM big LLM via Cerebras / Fireworks / Together / etc.) | [`references/inference-providers.md`](references/inference-providers.md) |
|
||||
| **3D Spaces: generation, CUDA extensions, output formats, and model recipes (incl. gaussian splatting)** | [`references/3d-generation.md`](references/3d-generation.md) |
|
||||
@@ -0,0 +1,109 @@
|
||||
# CUDA extensions for 3D Spaces on ZeroGPU
|
||||
|
||||
General ZeroGPU rules live in [`zerogpu.md`](zerogpu.md); wheel-tag anatomy and the prebuilt Blackwell wheel dataset live in [`requirements.md`](requirements.md). This file covers what makes 3D generation models harder than diffusion image models: native CUDA/C++ extensions and heavyweight multi-stage pipelines.
|
||||
|
||||
## The core constraint: no nvcc at build time
|
||||
|
||||
The ZeroGPU build container has **no CUDA toolkit and no GPU**. Any package that compiles CUDA at pip-install time (nvdiffrast, diff-gaussian-rasterization, torchmcubes, spconv-from-source, custom rasterizers, flash-attn sdist) **cannot be installed from `requirements.txt`** the normal way. At *runtime*, a CUDA toolkit is mounted at `/cuda-image/usr/local/cuda-13.0` (nvcc available once the app process is up), and the GPUs are RTX PRO 6000 Blackwell (**sm_120**).
|
||||
|
||||
Every working 3D Space uses one of three strategies, in order of preference:
|
||||
|
||||
### Strategy 1 — prebuilt wheels in requirements.txt (best)
|
||||
|
||||
Direct wheel URLs whose tags match the runtime exactly. Check sources in this order:
|
||||
|
||||
1. **The `multimodalart/zerogpu-blackwell-wheels` dataset** — covers `nvdiffrast`, `diff_gaussian_rasterization` (upstream Inria API), `torchmcubes`, `flash_attn`, `xformers`, `pytorch3d`. Cell-picking rules and per-package caveats in [`requirements.md`](requirements.md).
|
||||
2. **The official Space for your model family** — its requirements.txt often links wheels for the model's bespoke extensions (e.g. `microsoft/TRELLIS.2` links Blackwell wheels for `flex_gemm`, `nvdiffrast`, `nvdiffrec_render`, `cumesh`, `o_voxel` on GitHub Releases).
|
||||
3. **Build one yourself** on any CUDA machine (`TORCH_CUDA_ARCH_LIST="12.0+PTX" pip wheel .`) and host it in a Hub repo or commit it to the Space. No CUDA machine at hand? **Build it on HF Jobs** — pick a devel image matching the target runtime and let the job upload the wheel to a Hub repo:
|
||||
|
||||
```bash
|
||||
hf jobs run --flavor rtx-pro-6000 --timeout 30m --secrets HF_TOKEN \
|
||||
pytorch/pytorch:2.11.0-cuda13.0-cudnn9-devel \
|
||||
bash -c 'export TORCH_CUDA_ARCH_LIST="12.0+PTX" && \
|
||||
pip wheel --no-build-isolation --no-deps \
|
||||
git+https://github.com/<org>/<extension>.git -w /tmp/wheels && \
|
||||
pip install -U huggingface_hub && \
|
||||
hf upload <user>/zerogpu-wheels /tmp/wheels wheels --repo-type dataset'
|
||||
```
|
||||
|
||||
Then reference `https://huggingface.co/datasets/<user>/zerogpu-wheels/resolve/main/wheels/<file>.whl` in requirements.txt, or download and commit the wheel to the Space (Strategy 2). Three things to line up: the image's **torch + CUDA** must match the Space's pins (the tag above matches today's torch 2.11/cu130 ZeroGPU runtime — adjust as it moves), and the image's **Python** sets the wheel's cp tag, so check it matches the Space's `python_version:`. The `rtx-pro-6000` flavor is the same Blackwell sm_120 silicon as ZeroGPU, so the same job can `pip install` the fresh wheel and smoke-test the kernel before uploading; a cheaper flavor also works for compile-only (the arch comes from `TORCH_CUDA_ARCH_LIST`, not the attached GPU). Jobs bill by the minute and require a paid plan — a typical extension builds in well under 30 minutes.
|
||||
|
||||
When requirements contain a wheel URL, **pin `torch==` and `python_version:` to match its tags** (see `requirements.md`) — otherwise a base-image bump silently breaks the wheel. A tag mismatch is an `ImportError` (or a kernel-launch crash for a missing arch) at first import.
|
||||
|
||||
### Strategy 2 — wheel file committed to the Space repo
|
||||
|
||||
Same as Strategy 1 but the `.whl` lives in the Space repo and is installed at module-import time in `app.py` (the Hunyuan3D pattern, for their bespoke `custom_rasterizer`):
|
||||
|
||||
```python
|
||||
import subprocess, shlex
|
||||
subprocess.run(shlex.split("pip install custom_rasterizer-0.1-cp310-cp310-linux_x86_64.whl"), check=True)
|
||||
```
|
||||
|
||||
Use when you built the wheel yourself and don't want an external URL dependency. The cp tag dictates `python_version:`.
|
||||
|
||||
### Strategy 3 — JIT compile at startup (fallback, e.g. for forks with no wheel)
|
||||
|
||||
**CPU-only extensions** (pybind11, plain C++): compile at module import with g++ — seconds, no GPU needed. Hunyuan3D's mesh painter:
|
||||
|
||||
```python
|
||||
os.system("cd /home/user/app/hy3dpaint/DifferentiableRenderer && bash compile_mesh_painter.sh")
|
||||
# the script is one line: c++ -O3 -shared -std=c++11 -fPIC $(python -m pybind11 --includes) x.cpp -o x$(python3-config --extension-suffix)
|
||||
```
|
||||
|
||||
**CUDA extensions**, two variants seen in production:
|
||||
|
||||
*At module import, no GPU attached* (TripoSR's torchmcubes before its wheel existed, SF3D's texture_baker). Compiling needs only nvcc/headers, not a GPU — but you **must force the arch list** since nvcc can't probe a device:
|
||||
|
||||
```python
|
||||
subprocess.run(
|
||||
shlex.split("pip install --no-build-isolation ./texture_baker"),
|
||||
env={**os.environ, "TORCH_CUDA_ARCH_LIST": "12.0+PTX"},
|
||||
check=True,
|
||||
)
|
||||
```
|
||||
|
||||
*Inside a one-shot `@spaces.GPU(duration=600)` setup function called at module import* (`trellis-community/TRELLIS`, `dylanebert/LGM-mini` — the latter for the ashawkey diff-gaussian-rasterization fork that has no wheel). The full pattern: point `CUDA_HOME` at `/cuda-image/usr/local/cuda-13.0`, silence torch's CUDA-version check (torch cu128 vs nvcc 13.0) via a `sitecustomize.py` that no-ops `torch.utils.cpp_extension._check_cuda_version`, `pip install --no-build-isolation --no-deps git+<repo>`, then preload `ctypes.CDLL(".../libcudart.so.13", mode=ctypes.RTLD_GLOBAL)`. Copy it verbatim from one of those Spaces; don't reinvent it.
|
||||
|
||||
JIT costs: slower cold starts, a burned GPU allocation (second variant), and breakage when the runtime image moves. Prefer wheels; when you do JIT-build, log clearly so failures are diagnosable from the run logs — **a failed `os.system("pip install ...")` does not kill the app**; it resurfaces later as an ImportError or a silently degraded feature.
|
||||
|
||||
## Attention backends on Blackwell
|
||||
|
||||
Covered in [`requirements.md`](requirements.md) (flash_attn / xformers wheels, FA3-on-sm_120 crashes) and [`zerogpu.md`](zerogpu.md) → Attention backends. 3D-specific note: when a model exposes a backend env var (`ATTN_BACKEND` in TRELLIS: `xformers|flash_attn|sdpa|naive`), `sdpa` is the zero-dependency safe choice; set it before importing the model library.
|
||||
|
||||
## Durations for 3D workloads
|
||||
|
||||
Declared duration is billed against quota as requested (not actual runtime) and drives queue priority — declare honestly:
|
||||
|
||||
| Task | duration |
|
||||
|---|---|
|
||||
| Fast-tier single forward (TripoSR, SF3D), splat decode (TripoSplat, LGM) | default 60 (or lower for queue priority) |
|
||||
| Hunyuan3D shape-only | 40–60 |
|
||||
| TRELLIS / TRELLIS.2 generation; GLB extraction | 120 each |
|
||||
| Hunyuan3D shape + RGB texture | 90 |
|
||||
| Hunyuan3D-2.1 shape + PBR texture | 180 |
|
||||
|
||||
Multi-stage pipelines: split stages into separate `@spaces.GPU` functions **only when the user can act between them** (TRELLIS's generate → re-extract-at-different-quality split). Otherwise one decorated function per user action — each GPU entry costs a queue pass and a pickle round-trip.
|
||||
|
||||
## Passing 3D data across the GPU boundary
|
||||
|
||||
`@spaces.GPU` functions run in a forked process; args/returns cross via pickle (full rules in [`zerogpu.md`](zerogpu.md)). For 3D specifically:
|
||||
|
||||
- **Meshes/splats**: write files to a per-session temp path inside the GPU function, return the path string. Gaussian objects in these codebases are custom classes (CUDA tensors, sometimes locks) — they can never cross the boundary; do preprocess → sample → decode → file-write in one decorated function and return only paths.
|
||||
- **Intermediate latents between GPU calls**: CPU numpy dicts in `gr.State` (the TRELLIS pattern), rebuilt on cuda in the next call.
|
||||
- **`torch.cuda.empty_cache()` at the end of each GPU function** — 3D pipelines fragment VRAM fast on warm workers.
|
||||
- Never decorate bound methods (ml-sharp's `ModelWrapper` holds a non-picklable `threading.RLock`; wrap module-level functions instead).
|
||||
|
||||
## Memory
|
||||
|
||||
ZeroGPU `large` (48 GB) fits everything covered here (heaviest: Hunyuan3D-2.1 full PBR ~29 GB, TRELLIS.2 ≥24 GB). Don't request `size="xlarge"` unless a real OOM shows in the run logs. `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` (before importing torch) is the first fix for fragmentation OOMs on multi-stage pipelines.
|
||||
|
||||
## Failure checklist for 3D Spaces
|
||||
|
||||
(Also grep [`known-errors.md`](known-errors.md) — general errors are catalogued there.)
|
||||
|
||||
1. `ImportError: ...so: undefined symbol` / `cannot open shared object` → wheel-tag mismatch. Re-check torch pin, `python_version:`, CUDA suffix.
|
||||
2. `RuntimeError: CUDA error: invalid argument` at first attention call → FA3-on-Blackwell dispatch; see `requirements.md`.
|
||||
3. `nvcc not found` in **build** logs → a source CUDA dep leaked into requirements.txt; move it to a wheel or startup compile.
|
||||
4. `No space left on device` → offload-disk overflow; too many model variants/stages pinned at module scope.
|
||||
5. Hang at first request right after generation completes → a CUDA tensor (often nested in a dict/dataclass) crossed the pickle boundary.
|
||||
6. App RUNNING but a feature is missing (e.g. texture button gone) → a startup `pip install`/compile failed silently; read the run log from the top.
|
||||
@@ -0,0 +1,98 @@
|
||||
# 3D generation Spaces
|
||||
|
||||
Building a Space whose output is a 3D asset — image (occasionally text) in, downloadable and in-browser-viewable **mesh** (GLB/OBJ/PLY/STL) or **gaussian splat** (`.ply`/`.splat`) out. Read this whenever the user wants a Space/demo/playground for a 3D generation model, whether a stock checkpoint or their own finetuned variant. NeRF pipelines and world/scene models (HunyuanWorld, HY-World) are not covered.
|
||||
|
||||
The standard workflow in `SKILL.md` applies (create → build → iterate → verify). This file is the 3D entry point; the sub-references below cover the 3D-specific deltas.
|
||||
|
||||
## Sub-references
|
||||
|
||||
| When to read | File |
|
||||
|---|---|
|
||||
| CUDA/C++ extension handling — prebuilt wheels, startup-compile tricks, the dominant failure mode | `[3d-cuda-extensions.md](3d-cuda-extensions.md)` |
|
||||
| Output formats (GLB/OBJ/PLY/STL, splats), viewers, orientation, preprocessing, temp-file patterns | `[3d-outputs.md](3d-outputs.md)` |
|
||||
| Model selection details and per-family recipes (TRELLIS, Hunyuan3D, TripoSR, SF3D, …) | `[3d-models.md](3d-models.md)` |
|
||||
| Gaussian-splat deliverables (TripoSplat, LGM) — pure-PyTorch splat decode | `[3d-gsplat.md](3d-gsplat.md)` |
|
||||
|
||||
## What makes 3D Spaces different
|
||||
|
||||
1. **Native extensions.** These models need CUDA/C++ extensions (nvdiffrast, rasterizers, marching cubes, texture bakers) that cannot compile at Space build time. Working Spaces use prebuilt wheels or startup-compile tricks — [`3d-cuda-extensions.md`](3d-cuda-extensions.md). Getting this wrong is the dominant failure mode.
|
||||
2. **Vendored model code.** None of the major 3D models are cleanly pip-installable; every official Space carries the model library as a source tree in the Space repo. There is no `diffusers`-style one-liner. (Don't trust PyPI lookalikes: `trellis-3d` ships the Python tree without the CUDA extensions or the real dependency set.)
|
||||
|
||||
Both points make **"duplicate the official Space, then edit"** beat "assemble from scratch" here more than for any other model class.
|
||||
|
||||
## Picking a model
|
||||
|
||||
| Priority | Pick | Why | Reference |
|
||||
|---|---|---|---|
|
||||
| Best quality, PBR textures | **TRELLIS.2** | current quality bar; MIT; ≥24 GB (fits ZeroGPU large) | [`3d-models.md`](3d-models.md) |
|
||||
| Textured output, finetuning story | **Hunyuan3D-2.1** (2.0 for lighter/faster) | official training code → common finetune target; PBR paint stage; non-commercial license | [`3d-models.md`](3d-models.md) |
|
||||
| Speed / high-traffic / simplicity | **TripoSR** (untextured, MIT) or **SF3D** (textured, gated) | single forward pass, ~60s duration, tiny codebase | [`3d-models.md`](3d-models.md) |
|
||||
| Gaussians alongside mesh, multi-image input | **TRELLIS 1** | dual gaussian+mesh decoder; lighter than TRELLIS.2 | [`3d-models.md`](3d-models.md) |
|
||||
| Gaussian splats as the deliverable | **TripoSplat** (or LGM) | pure-PyTorch splat decode, no rasterizer needed; `gr.Model3D` renders splat `.ply` natively | [`3d-gsplat.md`](3d-gsplat.md) |
|
||||
|
||||
**Check gating and license in Phase 0, not at first build.** `stabilityai/stable-fast-3d` and `stable-point-aware-3d` are gated (`gated: auto`) — the user must have accepted the license and the Space needs `HF_TOKEN` as a secret; verify access with `hf repos info` / `HfApi().model_info()` up front. `tencent/Hunyuan3D-*` is ungated but **non-commercial** — flag before the Space goes public. TRELLIS/TripoSR/TripoSplat/LGM are MIT. Apple SHARP is research-only (`apple-amlr`).
|
||||
|
||||
If the model isn't covered by a reference file (Direct3D, TripoSG, Step1X-3D, a new release…), proceed by analogy with the user's agreement: find its official/most-liked *working* Space, fetch its actual files, and apply the same analysis — never guess a 3D dependency stack from memory.
|
||||
|
||||
## Deployment path: duplicate vs build
|
||||
|
||||
**Duplicate the official Space** (default for stock checkpoints, and for finetunes that are a `from_pretrained` swap — each reference file documents the swap):
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi
|
||||
api = HfApi()
|
||||
api.duplicate_repo("tencent/Hunyuan3D-2", to_id=f"{username}/my-hunyuan3d",
|
||||
repo_type="space", private=True, space_hardware="zero-a10g",
|
||||
space_secrets=[{"key": "HF_TOKEN", "value": hf_token}], # only if a gated/private repo is involved
|
||||
exist_ok=True)
|
||||
```
|
||||
|
||||
(`duplicate_space` is the deprecated older name.) Confirm the source Space is currently RUNNING first (`hf spaces info <id> --expand runtime`) — a paused/broken source means bitrot. Duplication copies *files only*; pass `space_hardware=`/`space_secrets=` explicitly. Then `hf download <repo> --repo-type space --local-dir .`, make the minimal edits (checkpoint repo-id, title/README, UI trims), and push back with `hf upload ... --repo-type space`. **Resist rewriting working extension-handling code you don't fully understand** — sitecustomize patches, `ctypes.CDLL` preloads, autotune caches, and `zero.startup()` calls all look redundant until removed.
|
||||
|
||||
**Build from source** (custom UI, shape-only trims, no live official Space): start from the TripoSR skeleton (the ~200-line app documented in [`3d-models.md`](3d-models.md) — the cleanest template), vendor the model library tree from the official Space or GitHub, and follow [`3d-cuda-extensions.md`](3d-cuda-extensions.md). Budget more debugging iterations than an image-model Space.
|
||||
|
||||
Either way, **fetch the official Space's live files first** (`https://huggingface.co/spaces/{id}/raw/main/{path}`) and treat them — not the reference files' pins — as source of truth. The reference files record what shipped as of mid-2026; the ZeroGPU runtime moves and official Spaces track it.
|
||||
|
||||
## UI design
|
||||
|
||||
Read [`3d-outputs.md`](3d-outputs.md) for formats, viewers, preprocessing, and temp-file patterns. Baseline image-to-3D UI:
|
||||
|
||||
- **Input**: image upload → visible preprocessing preview (background removal + crop — show what the model actually receives) → generate.
|
||||
- **Controls**: only the knobs this model responds to. Seed + randomize always; then per family: sampler steps/guidance (TRELLIS, Hunyuan3D), marching-cubes resolution (TripoSR), remesh/vertex-count/texture-size (SF3D), decimation + texture size at extraction (TRELLIS.2). Don't surface every config field of a research codebase.
|
||||
- **Output**: `gr.Model3D` (untextured meshes, splats) or `LitModel3D` with HDR lighting (textured/PBR), plus `gr.DownloadButton`s for GLB and secondary formats. Two-stage models (TRELLIS) show a fast turntable preview before the slower GLB extraction.
|
||||
- **Examples**: 3–6 known-good images lifted from the official Space's assets, `cache_examples=True, cache_mode="lazy"`.
|
||||
|
||||
## Verify: 3D-specific additions
|
||||
|
||||
On top of the standard smoke test in `SKILL.md` §7:
|
||||
|
||||
1. These apps expose **chained endpoints, not one `/predict`** — e.g. TripoSR: `/preprocess` (image → segmented image) then `/generate` (→ mesh files). `Client(...).view_api()` first, then call in sequence, feeding the first result into the second via `handle_file(...)`.
|
||||
2. **Validate the returned file, not its existence.** Meshes:
|
||||
|
||||
```python
|
||||
import trimesh
|
||||
m = trimesh.load("out.glb", force="scene")
|
||||
geoms = list(m.geometry.values()) if hasattr(m, "geometry") else [m]
|
||||
assert geoms and sum(g.faces.shape[0] for g in geoms) > 0, "empty mesh"
|
||||
```
|
||||
|
||||
Gaussian splat `.ply` (no faces — check point count and splat attributes):
|
||||
|
||||
```python
|
||||
from plyfile import PlyData
|
||||
v = PlyData.read("out.ply")["vertex"]
|
||||
assert v.count > 1000, "suspiciously few gaussians"
|
||||
assert {"f_dc_0", "opacity", "scale_0", "rot_0"} <= set(v.data.dtype.names), "not a gaussian ply"
|
||||
```
|
||||
|
||||
3. **Look at it in the browser once.** Orientation (upright? facing forward?), texture presence, viewer lighting — the failure modes a programmatic check can't catch ([`3d-outputs.md`](3d-outputs.md) → Orientation).
|
||||
4. Startup-time extension compiles fail *silently* (app still boots, feature degrades) — grep the run log from the top even when everything looks green. Failure checklist: bottom of [`3d-cuda-extensions.md`](3d-cuda-extensions.md).
|
||||
|
||||
## What to avoid
|
||||
|
||||
- Assembling a TRELLIS/Hunyuan-class Space from scratch when a running official Space exists to duplicate.
|
||||
- Trusting reference-file version pins over the live official Space's files.
|
||||
- Compiling CUDA extensions via `requirements.txt` (no nvcc at build time), or `pip install`-ing PyPI lookalike packages for the model libraries.
|
||||
- Returning meshes/gaussians as in-memory objects from `@spaces.GPU` functions — write files, return paths.
|
||||
- Fixed output filenames (`output.glb`) — concurrent users clobber each other.
|
||||
- Declaring green after `RUNNING` + a returned file. Load the mesh, count faces/points, and look at it once.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Gaussian splatting reference
|
||||
|
||||
Image-to-3D models whose output is a **gaussian splat** (a point cloud of oriented 3D gaussians) rather than a mesh. Deliverable file: gaussian `.ply` (the INRIA convention — the interchange format everything reads) and optionally `.splat` (antimatter15 32-byte records; smaller, no view-dependent color).
|
||||
|
||||
| Model | Repo | License / gating | Official Space (working reference) |
|
||||
|---|---|---|---|
|
||||
| TripoSplat | `VAST-AI/TripoSplat` | **MIT, ungated** | `VAST-AI/TripoSplat` (ZeroGPU, running, Gradio 6) |
|
||||
| LGM | `ashawkey/LGM` (+ `dylanebert/LGM` diffusers port) | **MIT, ungated** | `dylanebert/LGM-mini` and `ashawkey/LGM` (ZeroGPU, running) |
|
||||
| Apple SHARP | `apple/Sharp` | **apple-amlr — research-only**, ungated | `notaneimu/ml-sharp-3d-viewer` (currently CPU; code is ZeroGPU-ready) |
|
||||
| Splatter Image | `szymanowiczs/splatter-image-v1` | MIT | `szymanowiczs/splatter_image` (ZeroGPU, running; old Gradio, minimal skeleton) |
|
||||
| Splatt3R (stereo pair → scene) | `brandonsmart/splatt3r_v1.0` | research code | `brandonsmart/splatt3r` (ZeroGPU, running) |
|
||||
|
||||
TRELLIS 1 also exports gaussian `.ply` alongside its mesh — covered in `3d-models.md`.
|
||||
|
||||
> **Verify against the live official Space before writing files.** Facts below are as of 2026-07.
|
||||
|
||||
## The one decision that shapes everything: rasterize server-side or not?
|
||||
|
||||
Rendering gaussians on the server needs a CUDA rasterizer (`diff-gaussian-rasterization` or `gsplat`). Prebuilt Blackwell (sm_120) wheel availability is narrow: the `multimodalart/zerogpu-blackwell-wheels` dataset (see [`requirements.md`](requirements.md)) ships `diff_gaussian_rasterization` with the **upstream Inria API only** (2-tuple return) — the ashawkey fork most LGM-family code imports (4-tuple with alpha+depth) has no wheel, and gsplat's own wheel index tops out at torch 2.4/cu124. Two viable shapes:
|
||||
|
||||
**Shape A — no server-side rendering (default, strongly preferred).** The model decodes gaussians as plain PyTorch tensors; you write a `.ply` and let the *browser* render it. No CUDA extension at all — `requirements.txt` can be as small as `gradio, torch, torchvision, numpy, safetensors, pillow, tqdm` (TripoSplat's, in full). This is how every modern splat Space works (TripoSplat, SHARP, Splatt3R, Splatter Image; Splatt3R's requirements literally comment out the rasterizer dep).
|
||||
|
||||
**Shape B — a real rasterizer (only for server-rendered orbit videos).** If the code uses the upstream Inria API, take the prebuilt wheel from `requirements.md`'s dataset and be done. `ashawkey/LGM` needs its fork, so its Space JIT-builds it — the GPU-worker bootstrap documented in `3d-cuda-extensions.md` Strategy 3: `@spaces.GPU(duration=600)` setup function at module import, `CUDA_HOME=/cuda-image/usr/local/cuda-13.0`, `TORCH_CUDA_ARCH_LIST="12.0"`, sitecustomize no-op of `_check_cuda_version`, `pip install --no-build-isolation --no-deps git+https://github.com/graphdeco-inria/diff-gaussian-rasterization.git`, then `ctypes.CDLL(".../libcudart.so.13", RTLD_GLOBAL)`. Copy it from `dylanebert/LGM-mini` verbatim. Don't take this on unless the user explicitly wants server-rendered video output.
|
||||
|
||||
## Viewing splats
|
||||
|
||||
- **`gr.Model3D` renders gaussian `.ply` and `.splat` natively** (Babylon.js; supported at least since Gradio 4.25, current in 6.x). Return the file path like any other output. `display_mode` is ignored for splats; `clear_color`, `camera_position`, `height` still apply. This is the zero-effort default — use it.
|
||||
- **Custom JS viewers** give better splat sorting and controls when the demo is the product: TripoSplat serves a Spark.js (`@sparkjsdev/spark`) viewer page in an iframe; ml-sharp vendors a static PlayCanvas SuperSplat build and serves it via `gr.set_static_paths(...)` + `gr.HTML` iframe with `/gradio_api/file=...` URLs. Only go here if `gr.Model3D`'s rendering visibly undersells the model.
|
||||
- Always add a `gr.DownloadButton` for the `.ply` — splat users take the file to their own viewer/engine.
|
||||
|
||||
## Output writing
|
||||
|
||||
Gaussian `.ply` (INRIA convention): binary little-endian, vertex props `x,y,z,nx,ny,nz,f_dc_0..2,(f_rest_*),opacity(logit),scale_0..2(log),rot_0..3(quat)`. Model codebases ship their own writer (`save_ply` in LGM/SHARP/TripoSplat) — use it; don't hand-roll unless porting.
|
||||
|
||||
Optional `.splat` conversion for lighter downloads (~32 bytes/gaussian, drops view-dependent SH): TripoSplat's `triposplat.py::to_splat_bytes` is a complete pure-numpy reference — position f32×3, scale f32×3 (linear), RGBA u8×4 (`rgb = (f_dc·0.2820948 + 0.5)·255`, alpha = sigmoid(opacity)), quaternion u8×4 (`q·128+128`), records sorted by opacity×volume.
|
||||
|
||||
Watch orientation here too: splat conventions differ from viewers' (TripoSplat applies `[[1,0,0],[0,0,-1],[0,1,0]]` on export). If the splat renders sideways in `gr.Model3D`, fix it at export, same as meshes.
|
||||
|
||||
## ZeroGPU pickle discipline (bites harder here)
|
||||
|
||||
Gaussian objects in these codebases are custom classes (often holding CUDA tensors or locks) — **they cannot cross the `@spaces.GPU` boundary**. Two proven patterns:
|
||||
|
||||
- Do *everything* — preprocess, sample, decode, `.ply` write — inside one decorated function and return only path strings (TripoSplat, with an explicit comment to that effect).
|
||||
- Never decorate bound methods; wrap module-level functions (`spaces.GPU(duration=180)(predict_to_ply)` — ml-sharp, whose `ModelWrapper` holds a non-picklable `threading.RLock`).
|
||||
|
||||
Durations: TripoSplat and LGM-mini run on the bare `@spaces.GPU` default 60s; SHARP uses 180. Splat decoding is fast — the multiview-diffusion stage (LGM) is what eats time.
|
||||
|
||||
## Model-specific notes
|
||||
|
||||
- **TripoSplat** — the current default pick: MIT, running official ZeroGPU Space, fp16 weights well within 48 GB, ~262k gaussians per generation, BiRefNet background removal built in. Caveat when duplicating: the official Space uses a fully custom Gradio 6 `gradio.Server` + `index.html` frontend, not `gr.Blocks` — its HTTP API is bespoke, so either keep it wholesale or rebuild a plain `gr.Blocks` + `gr.Model3D` UI around `TripoSplatPipeline` (the pipeline code is self-contained in `triposplat.py`/`model.py`). Weights download at startup via `hf download VAST-AI/TripoSplat --local-dir ckpts`.
|
||||
- **LGM** — image → 4 multiview images (ImageDream) → gaussians. `dylanebert/LGM-mini` is the clean diffusers-style duplicate target (both pipelines via `from_pretrained(..., trust_remote_code=True)`, output straight into `gr.Model3D`); `ashawkey/LGM` is the video-rendering variant (Shape B). Both carry an xformers→SDPA monkeypatch for Blackwell — keep it when duplicating (the `xformers 0.0.34` Blackwell wheel in `requirements.md`'s dataset makes the patch unnecessary if you modernize the requirements instead, but the patch is harmless).
|
||||
- **SHARP** — pip-installable (`sharp @ git+https://github.com/apple/ml-sharp.git@<sha>`), predicts a camera-space *scene* splat from one photo. Research-only license (`apple-amlr`) — tell the user before they build anything commercial. `preload_from_hub` in the frontmatter pre-bakes the checkpoint into the Space image.
|
||||
- **Splatter Image** — the minimal skeleton (~CVPR 2024, per-pixel gaussians, MIT): good template bones, but its Space pins Gradio 4.27 and a cu113 torch via `pre-requirements.txt` — modernize rather than copy pins.
|
||||
@@ -0,0 +1,227 @@
|
||||
# 3D models: TRELLIS / TRELLIS.2, TripoSR, Stable Fast 3D, SPAR3D, Hunyuan3D
|
||||
|
||||
Per-family deployment recipes for mesh-generation models, distilled from the live files of their official Spaces. Gaussian splatting models live in [`3d-gsplat.md`](3d-gsplat.md); the overall workflow in [`3d-generation.md`](3d-generation.md).
|
||||
|
||||
## TRELLIS family
|
||||
|
||||
The TRELLIS family covers `microsoft/TRELLIS-image-large` (TRELLIS 1, ~1.2B) and `microsoft/TRELLIS.2-4B` (TRELLIS.2). Both are image-to-3D. Both are MIT licensed and ungated — no token needed to download weights.
|
||||
|
||||
| Variant | Model repo | Output | VRAM | Official Space (working reference) |
|
||||
|---|---|---|---|---|
|
||||
| TRELLIS 1 | `microsoft/TRELLIS-image-large` (mirror: `JeffreyXiang/TRELLIS-image-large`) | Gaussians + mesh → textured GLB, gaussian `.ply` | ~16 GB | `trellis-community/TRELLIS` (ZeroGPU, running) |
|
||||
| TRELLIS.2 | `microsoft/TRELLIS.2-4B` | High-fidelity PBR-textured GLB | ≥24 GB | `microsoft/TRELLIS.2` (ZeroGPU, running) |
|
||||
|
||||
There is no usable pip package (`trellis-3d` on PyPI ships only the pure-Python tree without the CUDA extensions or real dependency set; nothing exists for TRELLIS.2). **Both official Spaces vendor the library source (`trellis/` or `trellis2/`) directly in the Space repo.** Deploying means duplicating the official Space or copying its tree — not `pip install`.
|
||||
|
||||
> **Verify against the live official Space before writing files.** Every pin below (torch/CUDA/Python versions, wheel URLs) reflects what the official Space shipped as of 2026-07 and tracks the ZeroGPU runtime, which changes. Fetch the current files first — `hf download <space-id> --repo-type space --local-dir .` or read `https://huggingface.co/spaces/<space-id>/raw/main/requirements.txt` — and treat those as source of truth.
|
||||
|
||||
### TRELLIS.2 recipe (prebuilt-wheels strategy)
|
||||
|
||||
The `microsoft/TRELLIS.2` Space compiles nothing: every CUDA extension is a prebuilt wheel whose tags match the ZeroGPU runtime exactly (`+torch2.11.0.cu130`, `cp312`, built for Blackwell sm_120). Its frontmatter pins `python_version: 3.12` and `sdk_version: 6.1.0`, and requirements.txt (as of 2026-07) is:
|
||||
|
||||
```
|
||||
--extra-index-url https://download.pytorch.org/whl/cu130
|
||||
|
||||
torch==2.11.0
|
||||
torchvision==0.26.0
|
||||
triton==3.6.0
|
||||
pillow==12.0.0
|
||||
imageio==2.37.2
|
||||
imageio-ffmpeg==0.6.0
|
||||
tqdm==4.67.1
|
||||
easydict==1.13
|
||||
opencv-python-headless==4.12.0.88
|
||||
trimesh==4.10.1
|
||||
transformers==4.57.3
|
||||
zstandard==0.25.0
|
||||
kornia==0.8.2
|
||||
timm==1.0.22
|
||||
git+https://github.com/EasternJournalist/utils3d.git@9a4eb15e4021b67b12c460c7057d642626897ec8
|
||||
https://github.com/adithyaxx/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu13torch2.11cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
|
||||
https://github.com/LDYang694/Storages/releases/download/rtxpro6000/flex_gemm-1.0.0%2Btorch2.11.0.cu130-cp312-cp312-linux_x86_64.whl
|
||||
https://github.com/LDYang694/Storages/releases/download/rtxpro6000/nvdiffrast-0.4.0%2Btorch2.11.0.cu130-cp312-cp312-linux_x86_64.whl
|
||||
https://github.com/LDYang694/Storages/releases/download/rtxpro6000/nvdiffrec_render-0.0.0%2Btorch2.11.0.cu130-cp312-cp312-linux_x86_64.whl
|
||||
https://github.com/LDYang694/Storages/releases/download/rtxpro6000/cumesh-0.0.1%2Btorch2.11.0.cu130-cp312-cp312-linux_x86_64.whl
|
||||
https://github.com/LDYang694/Storages/releases/download/rtxpro6000/o_voxel-0.0.1%2Btorch2.11.0.cu130-cp312-cp312-linux_x86_64.whl
|
||||
```
|
||||
|
||||
Wheel tags, torch pin, and `python_version:` must agree — see `3d-cuda-extensions.md` for the tag anatomy. TRELLIS.2 uses `flex_gemm` for sparse conv (no spconv).
|
||||
|
||||
Module-level setup (order matters — env vars before any torch/trellis import):
|
||||
|
||||
```python
|
||||
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
|
||||
os.environ["ATTN_BACKEND"] = "flash_attn"
|
||||
os.environ["FLEX_GEMM_AUTOTUNE_CACHE_PATH"] = os.path.join(..., "autotune_cache.json")
|
||||
|
||||
pipeline = Trellis2ImageTo3DPipeline.from_pretrained("microsoft/TRELLIS.2-4B")
|
||||
pipeline.rembg_model = None # official Space outsources rembg to briaai/BRIA-RMBG-2.0 via gradio_client
|
||||
pipeline.low_vram = False
|
||||
pipeline.cuda()
|
||||
```
|
||||
|
||||
**Keep `autotune_cache.json` when duplicating.** The Space commits a pre-computed flex_gemm autotune cache so kernel autotuning doesn't re-run on every fresh ZeroGPU worker. Deleting it makes first-request latency much worse.
|
||||
|
||||
GPU functions in the official Space: `@spaces.GPU(duration=120)` on generation (`pipeline.run(...)` with per-stage sampler params, `pipeline_type` of `512` / `1024_cascade` / `1536_cascade`) and `@spaces.GPU(duration=120)` on GLB extraction (`pipeline.decode_latent(...)` + `o_voxel.postprocess.to_glb(...)`). Generation and extraction are split into two GPU calls with latents passed between them as CPU numpy in `gr.State` — keep that split; it lets users re-extract at different decimation/texture sizes without regenerating.
|
||||
|
||||
### TRELLIS 1 recipe (JIT-compile strategy)
|
||||
|
||||
The `trellis-community/TRELLIS` Space demonstrates the other viable strategy: compile nvdiffrast + diff-gaussian-rasterization **at first boot on a GPU worker**, inside `@spaces.GPU(duration=600)`, using the nvcc that ZeroGPU mounts at runtime (`/cuda-image/usr/local/cuda-13.0`). Its requirements.txt (torch 2.8.0, `spconv-cu120==2.3.6`, `xformers`, `rembg`, no python_version pin → 3.10) installs everything that *can* come from PyPI; the JIT step handles the rest with:
|
||||
|
||||
- `TORCH_CUDA_ARCH_LIST="12.0"` (Blackwell — nvcc can't probe a GPU that isn't attached yet)
|
||||
- a `sitecustomize.py` that no-ops `torch.utils.cpp_extension._check_cuda_version` (torch built for cu128, nvcc is 13.0)
|
||||
- `pip install --no-build-isolation ./extensions/nvdiffrast` (vendored source), then the mip-splatting `diff-gaussian-rasterization` from a shallow git clone
|
||||
- afterwards, `ctypes.CDLL("<cuda-libdir>/libcudart.so.13", mode=ctypes.RTLD_GLOBAL)` so the freshly built extensions find the CUDA 13 runtime
|
||||
|
||||
Backend env vars, set at the very top of `app.py`: `SPCONV_ALGO=native`, `ATTN_BACKEND=xformers`. Because ZeroGPU is Blackwell, xformers must be forced onto Cutlass kernels (its Flash-Attn-3 dispatch crashes on sm_120) — the Space monkeypatches `xformers.ops.memory_efficient_attention` to default `op=(fmha.cutlass.FwOp, fmha.cutlass.BwOp)`. Don't drop that patch when duplicating.
|
||||
|
||||
GPU functions: `@spaces.GPU(duration=120)` for generate-and-extract-GLB (single- and multi-image modes, 120-frame turntable preview video via `render_utils.render_video`, GLB via `postprocessing_utils.to_glb(gs, mesh, simplify=0.95, texture_size=1024)`); bare `@spaces.GPU` for gaussian `.ply` extraction. Viewer: `LitModel3D` from `gradio_litmodel3d==0.0.1` (installed at runtime with `--no-deps` if missing). The Space also sets `demo.launch(mcp_server=True)` — TRELLIS demos double as MCP servers; keep it.
|
||||
|
||||
### Deploying a finetuned TRELLIS variant
|
||||
|
||||
Both pipelines resolve everything from a `pipeline.json` at the model repo root, so a finetuned checkpoint that keeps the repo layout is a one-line swap:
|
||||
|
||||
```python
|
||||
pipeline = TrellisImageTo3DPipeline.from_pretrained("user/my-finetuned-trellis") # or Trellis2ImageTo3DPipeline
|
||||
```
|
||||
|
||||
If the finetuned repo is private, pass the token via env (`HF_TOKEN` Space secret) — `from_pretrained` in the vendored trellis code goes through `huggingface_hub` and picks it up. If the user only has raw checkpoint files (no `pipeline.json`), copy `pipeline.json` + config layout from the base repo into their model repo first.
|
||||
|
||||
### Picking between the two
|
||||
|
||||
- **TRELLIS.2** — best quality, PBR materials, 3 resolution tiers; heavier (≥24 GB, fine on ZeroGPU large's 48 GB), Gradio 6, Python 3.12. Default choice for a new Space.
|
||||
- **TRELLIS 1** — lighter (~16 GB), dual gaussian+mesh output (the gaussian `.ply` path matters if the user wants splats), multi-image conditioning modes, and the community Space is a cleaner codebase to modify. Choose when the user wants gaussians, multi-image input, or minimal VRAM.
|
||||
|
||||
### Durations and misc
|
||||
|
||||
- Generation: `duration=120` is what both official Spaces use; TRELLIS.2 at 1536³ pushes ~60s of pure compute on H100-class hardware, so don't go below 120.
|
||||
- Call `torch.cuda.empty_cache()` at the end of each GPU function (both Spaces do) — sequential calls on a warm worker otherwise accumulate.
|
||||
- TRELLIS.2's mesh simplify cap: `mesh.simplify(16777216)` — nvdiffrast's face-count limit.
|
||||
- Preprocessing (alpha-bbox crop, resize to ≤1024, background removal) runs on CPU outside the GPU function.
|
||||
|
||||
## Mesh workhorses: TripoSR, Stable Fast 3D, SPAR3D, Hunyuan3D
|
||||
|
||||
Two groups in this file. The **fast tier** (TripoSR, SF3D, SPAR3D): single-forward-pass image-to-mesh, seconds per generation, small VRAM, simple codebases — pick when the user wants a snappy demo, a high-traffic public Space (short `@spaces.GPU` durations = less quota per click and better queue priority), or can accept lower fidelity than TRELLIS.2. And **Hunyuan3D-2/2.1** (Tencent): two-stage — a flow-matching DiT generates the shape, then an optional multiview-diffusion "Paint" stage textures it — pick when the user wants textured output and TRELLIS.2 doesn't fit, or when they finetuned a Hunyuan3D DiT (the 2.1 release includes official training code, so finetuned shape checkpoints are common).
|
||||
|
||||
| Model | Repo | Output | License / gating | Official Space |
|
||||
|---|---|---|---|---|
|
||||
| TripoSR | `stabilityai/TripoSR` | untextured mesh (OBJ + GLB) | **MIT, ungated** | `stabilityai/TripoSR` (ZeroGPU, running) |
|
||||
| Stable Fast 3D (SF3D) | `stabilityai/stable-fast-3d` | UV-unwrapped **textured** GLB | stabilityai-ai-community, **gated (auto)** | `stabilityai/stable-fast-3d` (ZeroGPU, running) |
|
||||
| SPAR3D | `stabilityai/stable-point-aware-3d` | textured GLB + editable point cloud | stabilityai-ai-community, **gated (auto)** | `stabilityai/stable-point-aware-3d` (dedicated L4, **currently BUILD_ERROR** — reference code only) |
|
||||
| Hunyuan3D-2.0 | `tencent/Hunyuan3D-2` (subfolder `hunyuan3d-dit-v2-0`; also `-mini`, `-mv`) | GLB, RGB-textured (Paint 2.0) | tencent-hunyuan-community, **ungated, non-commercial** | `tencent/Hunyuan3D-2` (ZeroGPU, running) |
|
||||
| Hunyuan3D-2.1 | `tencent/Hunyuan3D-2.1` (subfolder `hunyuan3d-dit-v2-1`) | GLB, **PBR**-textured (Paint 2.1) | tencent-hunyuan-community, **ungated, non-commercial** | `tencent/Hunyuan3D-2.1` (ZeroGPU, **paused** — code is still the reference) |
|
||||
|
||||
**Licenses and gating matter here.** The two Stability models are `gated: auto` — the deploying user must accept the license on the model page, and the Space needs their `HF_TOKEN` as a secret to download weights at startup; if the gate isn't accepted, `from_pretrained` fails with 401 at build — check before publishing. The Hunyuan models are ungated (no token needed) but the community license restricts commercial use — surface that before the user flips a Space public. TripoSR is MIT and needs nothing.
|
||||
|
||||
> **Verify against the live official Space before writing files.** Pins below are as of 2026-07. Fetch current files via `https://huggingface.co/spaces/<id>/raw/main/<path>` first.
|
||||
|
||||
### TripoSR — the minimal recipe (~200-line app)
|
||||
|
||||
The simplest working 3D Space on the Hub, and the best starting skeleton for a from-scratch mesh demo. `tsr/` package vendored in the Space; frontmatter pins `python_version: 3.10.13`, `sdk_version: 4.20.1`.
|
||||
|
||||
One CUDA extension, JIT-built at module import (marching cubes):
|
||||
|
||||
```python
|
||||
subprocess.run(
|
||||
shlex.split("pip install --no-build-isolation git+https://github.com/tatsy/torchmcubes.git"),
|
||||
env={**os.environ, "TORCH_CUDA_ARCH_LIST": "12.0+PTX"}, # no GPU visible at startup — force Blackwell arch
|
||||
check=True,
|
||||
)
|
||||
```
|
||||
|
||||
requirements.txt carries the build toolchain for it (`scikit-build-core>=0.10`, `pybind11>=2.10`, `cmake`, `ninja`) plus `rembg`, `onnxruntime`, `trimesh`, `omegaconf`, `einops`, `transformers`, `Pillow`, `huggingface-hub`. No torch pin (base image). (A prebuilt `torchmcubes` Blackwell wheel now exists in the dataset described in [`requirements.md`](requirements.md) — cleaner than the JIT build when building fresh.)
|
||||
|
||||
```python
|
||||
model = TSR.from_pretrained("stabilityai/TripoSR", config_name="config.yaml", weight_name="model.ckpt")
|
||||
model.renderer.set_chunk_size(131072)
|
||||
model.to(device)
|
||||
|
||||
@spaces.GPU # default 60s is plenty — the forward pass is ~1s
|
||||
def generate(image, mc_resolution):
|
||||
scene_codes = model(image, device)
|
||||
mesh = model.extract_mesh(scene_codes, resolution=mc_resolution)[0]
|
||||
mesh = to_gradio_3d_orientation(mesh) # axis fix — see 3d-outputs.md
|
||||
... # export OBJ + GLB to NamedTemporaryFiles, return both paths
|
||||
```
|
||||
|
||||
Preprocessing on CPU, outside the GPU function: `rembg` background removal → `resize_foreground(image, 0.85)` → composite onto gray. UI: input image + "marching cubes resolution" slider (32–320) + two `gr.Model3D` tabs (OBJ and GLB).
|
||||
|
||||
### Stable Fast 3D — textured output, source-built extensions
|
||||
|
||||
`sf3d/` vendored; frontmatter `python_version: 3.10.13`, `sdk_version: 4.41.0`. Two vendored extension source dirs built at module import (they're commented out in requirements.txt with a note — the "HF hack"):
|
||||
|
||||
```python
|
||||
os.system(
|
||||
'CPPFLAGS="-include utility" TORCH_CUDA_ARCH_LIST="12.0+PTX" USE_CUDA=1 '
|
||||
"pip install -vv --no-build-isolation ./texture_baker ./uv_unwrapper"
|
||||
)
|
||||
```
|
||||
|
||||
`texture_baker` is a torch CUDAExtension (the `CPPFLAGS="-include utility"` is required for its build), `uv_unwrapper` a plain C++ extension. Other deps of note: `open_clip_torch`, `rembg[gpu]`, `pynanoinstantmeshes` + `gpytoolbox` (remeshing), `gradio-litmodel3d==0.0.1`.
|
||||
|
||||
```python
|
||||
model = SF3D.from_pretrained("stabilityai/stable-fast-3d",
|
||||
config_name="config.yaml", weight_name="model.safetensors")
|
||||
model.eval().to(device) # gated repo — needs HF_TOKEN secret with accepted license
|
||||
|
||||
@spaces.GPU # default 60s
|
||||
def run(input_image, remesh_option, vertex_count, texture_size):
|
||||
with torch.autocast(device_type=device, dtype=torch.bfloat16):
|
||||
model_batch = create_batch(input_image)
|
||||
...
|
||||
trimesh_mesh.export(tmp.name, file_type="glb", include_normals=True)
|
||||
```
|
||||
|
||||
UI controls that earn their place: foreground ratio, remesh (None/Triangle/Quad), target vertex count, texture size (512–2048). Viewer: `LitModel3D` with selectable HDR environment maps — textured output deserves image-based lighting (see [`3d-outputs.md`](3d-outputs.md)).
|
||||
|
||||
### SPAR3D — reference code only, don't duplicate blindly
|
||||
|
||||
Two-stage: point-cloud diffusion → mesh, with a `gradio_pointcloudeditor` step so users can edit the intermediate point cloud before meshing. Distinctive UI idea worth stealing. But the official Space runs on **dedicated L4, not ZeroGPU** (no `import spaces`, torch pinned to 2.5.1, real nvcc assumed) and is currently in BUILD_ERROR. To deploy SPAR3D on ZeroGPU you'd port it: add `spaces`, drop the torch pin, apply the SF3D extension-build pattern (same vendored `texture_baker`/`uv_unwrapper` + a prebuilt `pynim` wheel), and decorate the two stages. Budget real debugging time; propose SF3D instead unless the user specifically needs the point-cloud editing stage. Its background removal uses `transparent-background` (InSPyReNet) rather than rembg.
|
||||
|
||||
### Hunyuan3D-2 / 2.1 — textured two-stage generation
|
||||
|
||||
Both official Spaces vendor the model code in the Space repo (`hy3dgen/` for 2.0; `hy3dshape/` + `hy3dpaint/` for 2.1) — there's a `setup.py` but nothing is pip-installed; imports resolve from cwd. Two native extensions are handled at module-import time in `gradio_app.py`:
|
||||
|
||||
```python
|
||||
# custom_rasterizer: prebuilt CUDA wheel committed to the Space repo (no nvcc at build time)
|
||||
subprocess.run(shlex.split("pip install custom_rasterizer-0.1-cp310-cp310-linux_x86_64.whl"), check=True)
|
||||
# mesh painter: CPU-only pybind11 extension, compiled with plain g++ in seconds
|
||||
os.system("cd /home/user/app/hy3dpaint/DifferentiableRenderer && bash compile_mesh_painter.sh")
|
||||
```
|
||||
|
||||
The cp310 wheel means Python 3.10 — pin `python_version: "3.10"` explicitly when duplicating so a runtime default bump can't break it.
|
||||
|
||||
**Requirements notes (2.1)** — fully pinned; the non-obvious entries: `--extra-index-url https://download.blender.org/pypi/` + `bpy==4.0` (Blender-as-pip; installs headless with no display setup — see [`3d-outputs.md`](3d-outputs.md) → Headless Blender — though the Space's OBJ→GLB conversion actually runs on trimesh+pygltflib); `realesrgan==0.3.0` + `basicsr==1.4.2` (texture upscaling — needs the Space's `torchvision_fix.py` shim, applied at the top of `gradio_app.py` *and* again before the texgen import, because basicsr imports the removed `torchvision.transforms.functional_tensor`); `cupy-cuda12x`, `pymeshlab`, `xatlas`, `open3d`, `trimesh`, `pygltflib` (mesh stack); **no torch pin**; `pydantic==2.10.6` (gradio compat).
|
||||
|
||||
**Model loading** (module level):
|
||||
|
||||
```python
|
||||
from hy3dshape.pipelines import Hunyuan3DDiTFlowMatchingPipeline # hy3dgen.shapegen on 2.0
|
||||
shape_pipe = Hunyuan3DDiTFlowMatchingPipeline.from_pretrained(
|
||||
"tencent/Hunyuan3D-2.1", subfolder="hunyuan3d-dit-v2-1", use_safetensors=False,
|
||||
) # 2.0: repo "tencent/Hunyuan3D-2", subfolder "hunyuan3d-dit-v2-0", use_safetensors=True, + .enable_flashvdm()
|
||||
|
||||
# Texture stage — wrap in try/except and degrade to shape-only if it fails (official HAS_TEXTUREGEN pattern)
|
||||
from hy3dpaint.textureGenPipeline import Hunyuan3DPaintPipeline, Hunyuan3DPaintConfig
|
||||
paint_pipe = Hunyuan3DPaintPipeline(Hunyuan3DPaintConfig(max_num_view=8, resolution=768))
|
||||
```
|
||||
|
||||
**GPU decorators**: `@spaces.GPU(duration=40–60)` shape-only; `@spaces.GPU(duration=90)` (2.0) / `@spaces.GPU(duration=180)` (2.1 PBR) shape+texture. VRAM (2.1, per its README): ~10 GB shape, ~21 GB texture, ~29 GB both — fits ZeroGPU large, but the full PBR path is the heaviest recipe in these references; don't shave the duration.
|
||||
|
||||
**Output path gotcha (2.1)**: textured meshes export as **OBJ first, then convert to GLB** via the Space's `convert_utils` (obj2gltf with PBR materials) — direct GLB export from the paint pipeline core-dumps. Keep the conversion step when duplicating.
|
||||
|
||||
**Serving quirk**: both official Spaces use a custom `<model-viewer>` HTML iframe + FastAPI `StaticFiles` + `gr.mount_gradio_app` + manual `uvicorn.run` instead of `demo.launch()` — which requires calling `from spaces import zero; zero.startup()` manually before uvicorn. **When building fresh rather than duplicating, skip all of this** — export a GLB, show it in `gr.Model3D`/`LitModel3D`, and let `demo.launch()` do its job. Only keep the iframe machinery when duplicating wholesale.
|
||||
|
||||
**Local-run pattern worth copying**: 2.1's `gradio_app.py` defines a no-op `spaces.GPU` decorator when not running on Spaces (`ENV != "Huggingface"`), so the same file runs on a local GPU box unchanged.
|
||||
|
||||
**Deploying a finetuned Hunyuan3D variant** — the official 2.1 release ships training code (`hy3dshape/` includes trainers and configs), so "I finetuned Hunyuan3D, make me a demo" is a real request. The shape pipeline resolves `subfolder` inside the model repo:
|
||||
|
||||
```python
|
||||
shape_pipe = Hunyuan3DDiTFlowMatchingPipeline.from_pretrained(
|
||||
"user/my-finetuned-hunyuan3d", subfolder="hunyuan3d-dit-v2-1",
|
||||
)
|
||||
```
|
||||
|
||||
If the user's repo keeps the official layout (config + checkpoint under the dit subfolder), it's a repo-id swap; if they saved a bare checkpoint, mirror the base repo's subfolder structure into their repo first. Finetunes are almost always **shape-only** — keep the stock Paint stage from the base repo for texturing (it conditions on the input image, not the shape checkpoint). Private repo → `HF_TOKEN` Space secret.
|
||||
|
||||
**Which Hunyuan variant**: **2.1** — PBR texturing, the one with official training code, the default for finetunes, heaviest. **2.0** — lighter, faster (flashvdm), RGB textures, official Space actually running today (known-good duplicate target); `-mini` and `-mv` (multiview-conditioned) subfolder variants live in the same repo. **Shape-only demo** (skip Paint entirely) — halves the dependency surface (no bpy/realesrgan/basicsr/cupy) and cuts duration to 40–60s; offer it when the user just wants geometry.
|
||||
@@ -0,0 +1,114 @@
|
||||
# Mesh outputs: formats, viewers, preprocessing
|
||||
|
||||
## Formats
|
||||
|
||||
**GLB is the default output format.** It's self-contained (geometry + materials + textures in one binary file), renders in every in-browser viewer, and is what all the major official 3D Spaces produce. Offer OBJ/PLY/STL as secondary downloads, not as the primary view.
|
||||
|
||||
| Format | Carries | When to offer |
|
||||
|---|---|---|
|
||||
| GLB | geometry + normals + UVs + PBR materials + embedded textures | always — primary output |
|
||||
| OBJ (+MTL) | geometry, UVs; textures via sidecar files | DCC-tool users; note multi-file awkwardness in a web download |
|
||||
| PLY | geometry, vertex colors; also the container for gaussian splats (see `3d-gsplat.md`) | point clouds, vertex-colored meshes, splats |
|
||||
| STL | bare geometry | 3D-printing crowd |
|
||||
|
||||
Conversion is trimesh one-liners — the Hunyuan3D Spaces expose exactly this as an "export" accordion:
|
||||
|
||||
```python
|
||||
mesh = trimesh.load("out.glb")
|
||||
mesh.export("out.obj") # or .ply / .stl — inferred from extension
|
||||
```
|
||||
|
||||
Two exceptions to "just use trimesh":
|
||||
|
||||
- **PBR materials**: trimesh's GLB export handles baseColor fine but degrades full PBR (metallic/roughness/normal maps). Hunyuan3D-2.1 exports OBJ then converts with obj2gltf specifically because its direct GLB export core-dumps — when a pipeline ships its own exporter (`to_glb`, `convert_utils`, `o_voxel.postprocess`), use it instead of round-tripping through trimesh.
|
||||
- **Normals**: pass `include_normals=True` on trimesh GLB exports (SF3D does) — missing normals render matte-black in some viewers.
|
||||
|
||||
## Orientation
|
||||
|
||||
The #1 "it works but looks wrong" bug. Model output conventions (often Z-up, or −Y-forward from the training renders) don't match the viewers' glTF convention (Y-up, +Z toward camera), so meshes come out lying face-down or backwards. TripoSR's fix:
|
||||
|
||||
```python
|
||||
def to_gradio_3d_orientation(mesh):
|
||||
mesh.apply_transform(trimesh.transformations.rotation_matrix(-np.pi/2, [1, 0, 0])) # Z-up → Y-up
|
||||
mesh.apply_transform(trimesh.transformations.rotation_matrix( np.pi/2, [0, 1, 0]))
|
||||
return mesh
|
||||
```
|
||||
|
||||
The exact rotation is per-model — copy it from the official Space's app (search for `rotation_matrix` or `apply_transform`). If output is mirrored, the model bakes a flipped X; TripoSR's OBJ export flips `mesh.vertices[:, 0]` back. Verify orientation visually during the smoke-test; it can't be caught from exit codes.
|
||||
|
||||
## Viewers
|
||||
|
||||
- **`gr.Model3D`** — built-in, zero extra deps, fine for untextured or baseColor meshes; also renders gaussian splat `.ply`/`.splat` natively (see `3d-gsplat.md`). Useful kwargs: `display_mode="solid"` (default point cloud rendering ruins meshes in some versions; ignored for splats), `clear_color=(0.25, 0.25, 0.25, 1.0)`, `height=...`. Default choice.
|
||||
- **`LitModel3D`** (`gradio_litmodel3d==0.0.1`, a Hub custom component) — adds image-based lighting with HDR environment maps. Worth the extra dep for **textured/PBR** output, where flat lighting hides the texture quality (SF3D and trellis-community both use it, SF3D with a selectable `.hdr` set). Textured model + plain `gr.Model3D` undersells the result.
|
||||
- **Custom `<model-viewer>` iframe + FastAPI static mount** — what the Hunyuan Spaces do. Maximum control, but requires abandoning `demo.launch()` for manual uvicorn plus a manual `spaces.zero.startup()` call. Don't build this fresh; only keep it when duplicating a Space that already has it.
|
||||
- **Turntable preview** — TRELLIS renders a 120-frame orbit video (`imageio.mimsave`, fps=15) shown in `gr.Video` *before* the user commits to GLB extraction; TRELLIS.2 uses a base64-JPEG JS orbiter. Good pattern when extraction is a separate, slower step; skip it for fast-tier models where the mesh is ready immediately.
|
||||
|
||||
Always pair the viewer with `gr.DownloadButton` for the raw file — the viewer is a preview, the file is the deliverable.
|
||||
|
||||
## Input preprocessing (image-to-3D)
|
||||
|
||||
Every image-to-3D model expects a **segmented foreground object**, roughly centered, on neutral/transparent background. The shared recipe, run on CPU *outside* the GPU function:
|
||||
|
||||
1. **Respect an existing alpha channel.** If the upload is RGBA with a real alpha, skip segmentation (TRELLIS checks this first).
|
||||
2. **Otherwise remove the background** — `rembg` (u2net, onnxruntime; the common choice), `transparent-background` (InSPyReNet; SPAR3D), or BiRefNet. TRELLIS.2 outsources to the `briaai/BRIA-RMBG-2.0` Space via `gradio_client` — fine for an official Space, but a fragile external dependency for a user Space; prefer local rembg.
|
||||
3. **Crop to the alpha bbox, resize** (≤1024 for TRELLIS-class, 512 for fast tier), **rescale the foreground** to ~85% of frame (`foreground_ratio` slider in the Stability Spaces), composite onto neutral gray or keep alpha.
|
||||
|
||||
Show the preprocessed image in the UI before generation — users need to see what the model actually gets, and a bad segmentation explains a bad mesh. Loading the rembg session at startup (TRELLIS runs one dummy `preprocess_image` at boot) avoids a first-click latency spike.
|
||||
|
||||
## Temp files and concurrency
|
||||
|
||||
Handlers run concurrently; never write to fixed paths. The pattern used by the official Spaces:
|
||||
|
||||
```python
|
||||
demo = gr.Blocks(delete_cache=(600, 600)) # sweep files older than 600s every 600s
|
||||
|
||||
@demo.load(outputs=...)
|
||||
def start_session(req: gr.Request):
|
||||
session_dir = os.path.join(TMP_ROOT, str(req.session_hash))
|
||||
os.makedirs(session_dir, exist_ok=True)
|
||||
```
|
||||
|
||||
or simply `tempfile.NamedTemporaryFile(suffix=".glb", delete=False)` per call (TripoSR/SF3D). Either works; never a bare `"output.glb"`.
|
||||
|
||||
## Headless Blender (bpy / renders)
|
||||
|
||||
Blender shows up in 3D Spaces two ways: as the **`bpy` pip module** (`--extra-index-url https://download.blender.org/pypi/` + `bpy==<pin>`; Hunyuan3D-2.1 pins `bpy==4.0` — which installs cleanly on ZeroGPU with no display setup, though that Space's actual GLB conversion runs on trimesh+pygltflib), or as a **standalone binary** driven by a render script (TRELLIS-dataset-toolkit-style turntables, rigging/preprocessing tools). Spaces containers have no display; the working patterns, all verified from running Spaces:
|
||||
|
||||
- **pip `bpy` needs no xvfb.** Mesh ops, Cycles, and Workbench renders work headless — bpy creates its GL context via EGL/Mesa, not X11. The `packages.txt` seen in every working bpy demo Space (e.g. `radames/gradio-blender-bpy`) is just:
|
||||
|
||||
```
|
||||
libegl1-mesa-dev
|
||||
libgl1-mesa-dev
|
||||
```
|
||||
|
||||
ZeroGPU gotcha: **`import bpy` takes 30–60 s** — never import it inside a `@spaces.GPU` function (it burns the duration budget and aborts the task). Import at module scope, or run a persistent bpy worker in the main process (`VAST-AI/SkinTokens` does file-based IPC to one).
|
||||
- **A modern Blender binary (≥3.4) needs no xvfb either**: download the tarball at startup, run `blender -b -noaudio --python script.py -- <args>` as a subprocess (`MajorDaniel/UniRig` on ZeroGPU). `packages.txt` then carries the binary's shared-lib deps: `libx11-6 libxi6 libxrender1 libxxf86vm1 libfontconfig1 libsm6 libxkbcommon0 libxkbcommon-x11-0 libgl1-mesa-glx`. Blender's manual is explicit that CLI/background rendering needs no display; a GPU Cycles subprocess must be launched from **inside** `@spaces.GPU` (it inherits the worker's CUDA), while display/env setup is process-global and belongs at module scope.
|
||||
- **Reach for Xvfb only when forced**: a pinned pre-3.4 Blender binary (no EGL headless support yet), EEVEE on legacy setups, or non-Blender GL stacks (Open3D/VTK offscreen, pyglet). On the **gradio SDK**, don't launch raw `Xvfb :99 &` — the container lacks the `/tmp/.X11-unix` permissions for it (UniRig documents this) — use `pyvirtualdisplay` at module scope, the pattern `gradient-spaces/GuideFlow3D` uses to render Cycles through Blender 3.0.1 headlessly. `packages.txt`:
|
||||
|
||||
```
|
||||
xvfb
|
||||
libx11-6
|
||||
libgl1
|
||||
libxrender1
|
||||
libxi6
|
||||
libxkbcommon-x11-0
|
||||
libsm6
|
||||
```
|
||||
|
||||
and at the very top of `app.py`, before anything touches Blender/GL (plus `pyvirtualdisplay` in requirements.txt):
|
||||
|
||||
```python
|
||||
if os.environ.get("DISPLAY") is None:
|
||||
from pyvirtualdisplay import Display # drives Xvfb with user-owned sockets
|
||||
display = Display(visible=0, size=(1920, 1080))
|
||||
display.start()
|
||||
os.environ.setdefault("DISPLAY", f":{display.display}")
|
||||
```
|
||||
|
||||
On the **Docker SDK** you control the image: `xvfb-run -a -s '-screen 0 1024x768x24' <cmd>` (or `Xvfb :99 ... & export DISPLAY=:99`), adding Mesa software-GL env on CPU hardware (`LIBGL_ALWAYS_SOFTWARE=1`, `GALLIUM_DRIVER=llvm`).
|
||||
- **Keep Space renders on Cycles.** EEVEE is GPU-rasterization by design (no CPU path — headless EEVEE exists on Linux ≥3.4 via EGL, but under Xvfb/software-GL it crawls); Cycles renders fine on CPU or on CUDA inside `@spaces.GPU`.
|
||||
- **pyrender** (not Blender, but the same "needs GL" problem): use EGL, not Xvfb — `os.environ["PYOPENGL_PLATFORM"] = "egl"` with `libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev mesa-common-dev` in packages.txt (`H-Liu1997/EMAGE` on ZeroGPU).
|
||||
|
||||
## Examples
|
||||
|
||||
`gr.Examples` with a handful of good input images makes or breaks first impressions of a 3D demo. Source them from the official Space's `assets/`/`examples/` dir (they're chosen to work well) or the model repo. On ZeroGPU use `cache_examples=True, cache_mode="lazy"`. Objects that demo well: single centered objects with clear silhouettes — figurines, shoes, furniture, vehicles. Objects that demo poorly: scenes, flat images, humans (most mesh models aren't trained for them), thin structures.
|
||||
@@ -0,0 +1,89 @@
|
||||
# Persistent storage with Buckets
|
||||
|
||||
Spaces are stateless. All data is wiped on restart / rebuild. For state that must survive (user uploads, generations, dynamic feeds, logs, growing databases): mount an HF **Bucket** — S3-like object storage living at `hf://buckets/<ns>/<bucket>`.
|
||||
|
||||
Buckets are paid (per-TB storage). Check `whoami.canPay` and confirm with the user before creating one. Pricing + free tier: https://huggingface.co/storage.
|
||||
|
||||
Full docs: https://huggingface.co/docs/hub/storage-buckets.
|
||||
|
||||
## Create + attach
|
||||
|
||||
```bash
|
||||
hf buckets create <ns>/<bucket-name> # --private optional
|
||||
hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data
|
||||
```
|
||||
|
||||
After this, writes to `/data/` in the Space are durable. Reads come from the bucket via the Xet storage backend.
|
||||
|
||||
To make the bucket files publicly addressable: leave the bucket public. Public bucket files are served at `https://huggingface.co/buckets/<ns>/<bucket>/resolve/<path>` (HTTP 302 redirect to a signed CDN URL). The Space writes once and the public URL works forever — no streaming proxy needed.
|
||||
|
||||
## Write-durable, read-fast pattern
|
||||
|
||||
For a feed-style Space (e.g. a community jam where users save generations and browse a public timeline), don't re-scan disk on every request. Module-level disk scan → in-memory list → every write appends to both:
|
||||
|
||||
```python
|
||||
import os, json, uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
BUCKET_ID = "<ns>/<bucket-name>"
|
||||
BUCKET_URL = f"https://huggingface.co/buckets/{BUCKET_ID}/resolve"
|
||||
_feed = []
|
||||
|
||||
def _load_feed():
|
||||
root = "/data/songs"
|
||||
if not os.path.isdir(root):
|
||||
return
|
||||
for sid in os.listdir(root):
|
||||
meta = f"{root}/{sid}/meta.json"
|
||||
if os.path.isfile(meta):
|
||||
_feed.append(json.load(open(meta)))
|
||||
_feed.sort(key=lambda s: s["created_at"], reverse=True)
|
||||
|
||||
_load_feed() # one scan at startup
|
||||
|
||||
@app.api(name="save", time_limit=60)
|
||||
def save(audio_bytes: bytes, title: str):
|
||||
sid = uuid.uuid4().hex[:12]
|
||||
d = f"/data/songs/{sid}"; os.makedirs(d, exist_ok=True)
|
||||
open(f"{d}/audio.wav", "wb").write(audio_bytes)
|
||||
meta = {"id": sid, "title": title,
|
||||
"url": f"{BUCKET_URL}/songs/{sid}/audio.wav",
|
||||
"created_at": datetime.now(timezone.utc).isoformat()}
|
||||
json.dump(meta, open(f"{d}/meta.json", "w"))
|
||||
_feed.insert(0, meta) # cache stays current — no re-scan
|
||||
return meta
|
||||
|
||||
@app.api(name="feed", concurrency_limit=10)
|
||||
def feed(): return _feed[:50] # zero disk I/O
|
||||
```
|
||||
|
||||
Reference Space using this pattern: https://huggingface.co/spaces/victor/ace-step-jam
|
||||
|
||||
## Anti-pattern: bucket as model-weights cache
|
||||
|
||||
**Do NOT** `snapshot_download(..., local_dir="/data/weights")` and load checkpoints from there. Bucket I/O is S3-paced; reading a 22 GB `safetensors` from `/data` during `from_pretrained` stalls past any `@spaces.GPU` duration cap.
|
||||
|
||||
For model weights, let HF Hub re-download to local container disk on each cold start. With `HF_HUB_ENABLE_HF_TRANSFER=1` (set in the runtime by default) this is fast — typically much faster than streaming the same bytes through bucket I/O at request time.
|
||||
|
||||
Bucket I/O is fine for occasional metadata reads (the feed pattern above) or saving user information. It is *not* fine as the path your model loader streams gigabytes through every cold start.
|
||||
|
||||
## Cache redirects
|
||||
|
||||
`/home/user/.cache` is read-only on ZeroGPU. Redirect transient caches at the top of `app.py`, before any library import that uses them:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("HF_HOME", "/data/.cache/huggingface") # or /tmp on non-bucket Spaces
|
||||
os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
|
||||
os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")
|
||||
```
|
||||
|
||||
Missing redirections fail silently or at first matplotlib / transformers / diffusers import.
|
||||
|
||||
## Write access from the Space
|
||||
|
||||
The Space's `HF_TOKEN` secret needs write permission on the bucket. Set via Settings → Secrets in the Space UI, or `hf spaces secrets set <id> HF_TOKEN=<token>`.
|
||||
|
||||
## Security note
|
||||
|
||||
Public bucket files are publicly accessible forever at their resolve URL. **Don't write PII** to a public bucket. If you need durable but private storage (e.g. per-user history requiring an HF login), keep the bucket private and gate reads through your Space's own auth.
|
||||
@@ -0,0 +1,236 @@
|
||||
# Debugging and iteration
|
||||
|
||||
How to iterate cheaply, read logs, and smoke-test. Dev mode + SSH exists as a last resort (covered at the bottom).
|
||||
|
||||
## The rung ladder
|
||||
|
||||
Pick the cheapest update mechanism that fits the change. Going one rung too high wastes 30 s – 15 min per cycle.
|
||||
|
||||
| Rung | When | Command | Cost |
|
||||
|---|---|---|---|
|
||||
| 1. Hot-reload | Pure Python edit on a Gradio Space (SDK 6.1+), **no new deps** | `hf spaces hot-reload <id> -f app.py` | seconds, no rebuild |
|
||||
| 2. `hf upload` | Code-only change hot-reload can't handle (`gr.Server`, Streamlit, Docker entrypoint, non-Python file) | `hf upload <id> . --repo-type space --include '<file>'` | 30–90 s app restart |
|
||||
| 3. Full rebuild | `requirements.txt`, `Dockerfile`, README frontmatter, or hardware change | `hf upload <id> . --repo-type space --exclude "**/__pycache__/**" && hf spaces logs <id> --build --follow` | 1–15 min |
|
||||
|
||||
`hf upload` defaults to a **model** repo — always pass `--repo-type space`, or it uploads to (and silently creates) a model repo of the same name. Also `--exclude "**/__pycache__/**"` so local bytecode caches don't get committed into the Space.
|
||||
| 4. Factory reboot | Container in inconsistent state (broken pip env, etc.) | `hf spaces restart <id> --factory-reboot` | full rebuild + cold start |
|
||||
|
||||
### How hot-reload works
|
||||
|
||||
`hf spaces hot-reload <id> -f app.py` patches the running Python process in place via `jurigged` (vendored inside the `spaces` package), then commits the change to the repo with a hot-reload marker that tells the platform to skip its usual restart. Seconds, no rebuild.
|
||||
|
||||
Applies cleanly to function-body changes and new top-level symbols. Does **not** rerun module-level imports or one-time init — the model was loaded when `app.py` first ran and jurigged won't re-execute that load. New pip deps, README frontmatter, `Dockerfile`, and hardware changes need a full rebuild (rung 3).
|
||||
|
||||
Independent of dev mode. Marked experimental in the CLI. Requires Gradio SDK 6.1+.
|
||||
|
||||
### Footguns
|
||||
|
||||
- **Hot-reload poisons factory reboot.** A commit that's only a hot-reload leaves runtime metadata that's valid only while the hot process is alive. `--factory-reboot` on top of one can fail with `fatal: could not read Username for 'https://huggingface.co'`. Recovery: push any normal `hf upload` commit (even a one-line no-op) first, then restart.
|
||||
- **`runtime.sha` lags repo SHA on restart.** `hf upload` succeeds → repo updates → `hf spaces info` keeps reporting the *previous* commit's SHA under `runtime` for several minutes while the new container loads. Poll `runtime.sha`, not just `stage`, and don't issue another restart until it flips.
|
||||
- **Concurrent uploads or restart-while-uploading collide.** Wait for one to finish.
|
||||
- **"Let me try locally first"** for anything that depends on the Space's Python / torch / CUDA env. The Space environment is the only one that matters. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing.
|
||||
|
||||
## Reading logs
|
||||
|
||||
```bash
|
||||
hf spaces info <id> --expand runtime # stage at a glance
|
||||
hf spaces logs <id> --build --follow # build log, live
|
||||
hf spaces logs <id> --follow # run log, live
|
||||
hf spaces logs <id> --build --tail 500 # bigger window — default is small
|
||||
```
|
||||
|
||||
Find the **first** error in the build log, not the last. Cascading errors after the first are noise.
|
||||
|
||||
State machine (terminal states in bold):
|
||||
|
||||
```
|
||||
BUILDING → APP_STARTING → RUNNING
|
||||
↘ RUNTIME_ERROR
|
||||
↘ BUILD_ERROR
|
||||
↘ CONFIG_ERROR
|
||||
```
|
||||
|
||||
For stage-specific lookups, see [`known-errors.md`](known-errors.md).
|
||||
|
||||
## Smoke-test patterns
|
||||
|
||||
A Space isn't done until a `gradio_client` call against the live URL exercises the endpoint end-to-end. Four steps in order — keep `hf spaces logs <id> --follow` running in another terminal throughout, so any silent fallback (model snapping to a different size, missing optional dep, dtype downgrade) surfaces.
|
||||
|
||||
### A. Alive?
|
||||
|
||||
```bash
|
||||
hf spaces info <id> --expand runtime --format json \
|
||||
| python3 -c "import json,sys; r=json.load(sys.stdin)['runtime']; \
|
||||
print(r['stage'], r.get('hardware','?'))"
|
||||
# expect: RUNNING zero-a10g
|
||||
```
|
||||
|
||||
If `requested_hardware` is `cpu-basic` when you wanted GPU, your `--flavor` was rejected silently. Fix with `hf spaces settings <id> --hardware zero-a10g`.
|
||||
|
||||
### B. Logs clean post-boot?
|
||||
|
||||
```bash
|
||||
hf spaces logs <id> --tail 200
|
||||
```
|
||||
|
||||
Confirm the model finished loading, no import warnings, no "falling back to CPU" / dtype-downgrade messages, no failing health checks the platform forgave. Do this before calling the API — many silent failures (a config typo loading the wrong model, a missing optional dep, a one-time init that errored but didn't crash boot) are only visible here.
|
||||
|
||||
### C. API actually functions?
|
||||
|
||||
Default — sync `gr.Interface` / `gr.Blocks` / `gr.ChatInterface` / `gr.Server` `@app.api`:
|
||||
|
||||
```python
|
||||
from gradio_client import Client, handle_file
|
||||
import os
|
||||
|
||||
c = Client("<ns>/<name>", token=os.environ["HF_TOKEN"],
|
||||
httpx_kwargs={"timeout": 600}) # ≥ @spaces.GPU duration + 60s
|
||||
|
||||
print(c.view_api()) # discover endpoints — don't guess api_name
|
||||
|
||||
result = c.predict(
|
||||
handle_file("test.png"), # file inputs need handle_file()
|
||||
"short prompt",
|
||||
api_name="/generate", # matches @app.api(name=...) or the function name
|
||||
)
|
||||
```
|
||||
|
||||
**Streaming endpoints** (function uses `yield` or `TextIteratorStreamer`) — `.predict()` returns only the final value. Iterate chunks via `.submit()`:
|
||||
|
||||
```python
|
||||
job = c.submit("short prompt", api_name="/chat")
|
||||
for chunk in job: print(chunk, end="")
|
||||
# or job.result() for the final value
|
||||
```
|
||||
|
||||
**`gr.Server` custom `@app.get/post(...)` routes** don't appear in `view_api()`. Hit them with plain HTTP:
|
||||
|
||||
```python
|
||||
import httpx
|
||||
r = httpx.post(f"https://<subdomain>.hf.space/your_route",
|
||||
json={...}, timeout=600,
|
||||
headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"})
|
||||
```
|
||||
|
||||
**OAuth-gated Spaces** (`hf_oauth: true` + `gr.LoginButton`) — anonymous `Client` can't authenticate. Test interactively after sign-in, or capture a session token and pass via `httpx_kwargs={"headers": {...}}`.
|
||||
|
||||
**MCP server mode** (`launch(mcp_server=True)`) — different protocol. Use an MCP client.
|
||||
|
||||
### D. Output bytes AND logs look right?
|
||||
|
||||
HTTP 200 ≠ correct output. Sniff both the returned file and the run log emitted during the call.
|
||||
|
||||
```python
|
||||
head = open(path, "rb").read(16)
|
||||
# b'glTF...' → glb
|
||||
# b'\x89PNG' → png
|
||||
# b'\xff\xd8' → jpeg
|
||||
# b'RIFF...WEBP' → webp
|
||||
# b'RIFF...WAVE' → wav
|
||||
# head[4:8]==b'ftyp' → mp4
|
||||
# b'ply\n' → ply
|
||||
```
|
||||
|
||||
For text: non-empty, not all `<think>...</think>` (thinking-model leak), length reasonable. For images: returned dimensions match what was requested (some models snap to nearest preset).
|
||||
|
||||
Look at the tailed run log alongside — silent fallbacks (model snapping resolution, missing optional dep falling back to a slower path, dtype downgrade) only show up there.
|
||||
|
||||
### What NOT to do
|
||||
|
||||
- **Don't launch Playwright / headless browser** to verify backend logic. The Gradio UI calls the same API `gradio_client` does — one `predict` tests both.
|
||||
- **Don't build mock-mode + local-server harnesses** before pushing. Local-green ≠ Space-green.
|
||||
- **Don't smoke-test with full-budget inputs.** Smallest input that exercises the GPU code path — short prompt, small image, low step count. You're verifying wiring, not quality.
|
||||
|
||||
## Iterating on the Space, not locally
|
||||
|
||||
The Space env is the only one that matters: Python, torch, CUDA, file paths, env vars, gradio version, the `spaces` hijack all differ from your laptop.
|
||||
|
||||
Workflow:
|
||||
|
||||
1. Decide SDK + hardware. Write the smallest `app.py` / `Dockerfile` + `requirements.txt` + README frontmatter — just enough that the entry point loads.
|
||||
2. Push immediately. Don't build a Playwright / mock harness first.
|
||||
3. Once `RUNNING`: verify with `gradio_client` against the real Space. That's your test loop.
|
||||
4. Iterate via the cheapest rung.
|
||||
|
||||
`python3 -m py_compile app.py` is the maximum local check worth doing.
|
||||
|
||||
## Last resort: dev mode + SSH
|
||||
|
||||
Use only when:
|
||||
|
||||
- A failure is non-deterministic (device-side asserts, OOM under specific shapes, race conditions).
|
||||
- You need `CUDA_LAUNCH_BLOCKING=1` or `gdb` to localize a CUDA error.
|
||||
- You'd burn 4+ build cycles trying variations from outside.
|
||||
|
||||
Reading logs + grepping [`known-errors.md`](known-errors.md) + a tight `gradio_client` smoke loop solves the vast majority of issues. Dev mode is a heavy hammer.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **PRO / Team / Enterprise plan** — dev mode is a paid feature.
|
||||
2. **An SSH key registered on the user's HF profile.** Without this, SSH refuses the connection. If the user doesn't have one yet, they need to:
|
||||
- Generate a keypair locally: `ssh-keygen -t ed25519 -f ~/.ssh/hf_dev -N ''` (no passphrase keeps automation simple; user can pick differently if they prefer).
|
||||
- Add the **public** key (`~/.ssh/hf_dev.pub`) at https://huggingface.co/settings/keys.
|
||||
- Keep the **private** key (`~/.ssh/hf_dev`) on the machine they'll SSH from.
|
||||
3. **The Space must be in `RUNNING` or `RUNTIME_ERROR`** before dev mode lets you in — not `BUILD_ERROR`. If it's in build error, push a stub `app.py` that boots cleanly first (e.g. `import gradio as gr; gr.Interface(lambda: 'ok', None, 'text').launch()`), then enable dev mode.
|
||||
|
||||
### Enable
|
||||
|
||||
No `huggingface_hub` Python wrapper yet — use the REST endpoint:
|
||||
|
||||
```bash
|
||||
curl -s -X POST \
|
||||
-H "Authorization: Bearer $HF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
"https://huggingface.co/api/spaces/<ns>/<space>/dev-mode" \
|
||||
-d '{"enabled": true}'
|
||||
```
|
||||
|
||||
### SSH in
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/hf_dev -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
|
||||
<ns>-<space>@ssh.hf.space
|
||||
```
|
||||
|
||||
Username is `<namespace>-<space>` **lowercase**, with `-` replacing `/`. Dots in the Space name become hyphens too.
|
||||
|
||||
### Inside the VM
|
||||
|
||||
It's a normal container at `/home/user/app/`. You can edit files, `pip install`, run repros, call `@spaces.GPU`-decorated functions interactively (they get a real GPU window).
|
||||
|
||||
**`nvidia-smi` in the bare terminal will fail** with `NVML: Unknown Error`. Expected — ZeroGPU only exposes the real GPU inside `@spaces.GPU` calls. Don't assume the GPU is broken.
|
||||
|
||||
**Edits in `/home/user/app/` don't survive** a restart, sleep, or dev-mode-disable. Only commits persist.
|
||||
|
||||
### Smoke-test the fix inside the container
|
||||
|
||||
Before exiting dev mode, verify the fix actually works under a real GPU window:
|
||||
|
||||
```bash
|
||||
cat > /home/user/app/_devtest.py <<'PY'
|
||||
import spaces, torch
|
||||
from app import predict # or whatever your @spaces.GPU function is
|
||||
print(predict(<realistic-args>))
|
||||
PY
|
||||
python3 _devtest.py
|
||||
```
|
||||
|
||||
### Persist + exit
|
||||
|
||||
Commit + push from inside the container (`git config user.email / user.name` first; the HF git remote works). Then disable dev mode:
|
||||
|
||||
```bash
|
||||
curl -s -X POST -H "Authorization: Bearer $HF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
"https://huggingface.co/api/spaces/<ns>/<space>/dev-mode" \
|
||||
-d '{"enabled": false}'
|
||||
```
|
||||
|
||||
**Factory-reboot** to apply the pushed state (in dev mode the Space won't rebuild on commits):
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi
|
||||
HfApi(token=HF_TOKEN).restart_space("<ns>/<space>", factory_reboot=True)
|
||||
```
|
||||
|
||||
Then re-run the outside-the-container smoke test. Dev-mode success does **not** guarantee post-rebuild success — different image, different process tree.
|
||||
@@ -0,0 +1,200 @@
|
||||
# Gradio for Spaces
|
||||
|
||||
Patterns and quirks specific to running Gradio inside a Space. Assumes you're already comfortable with stock Gradio components and `gr.Blocks` / `gr.Interface`.
|
||||
|
||||
For deeper Gradio API guidance — components, layouts, event listeners, chatbots, the Gradio 5→6 migration — use the dedicated `huggingface-gradio` skill. Install it with `hf skills add huggingface-gradio` (add `--claude --global` to also install for Claude Code, user-level).
|
||||
|
||||
For ZeroGPU-specific decorator + worker semantics, see [`zerogpu.md`](zerogpu.md).
|
||||
|
||||
## Themes and layout
|
||||
|
||||
- Default theme preference: `gr.themes.Citrus()`. Alternatives: `gr.themes.Soft()` or no theme. Pick once and don't over-style.
|
||||
- For apps that don't need full-width, constrain with CSS so they're readable on 4K displays:
|
||||
```python
|
||||
CSS = """
|
||||
#col-container { max-width: 1100px; margin: 0 auto; }
|
||||
.dark .gradio-container { color: var(--body-text-color); }
|
||||
"""
|
||||
with gr.Blocks(theme=gr.themes.Citrus(), css=CSS) as demo: ...
|
||||
```
|
||||
Always include the `.dark .gradio-container` override with `gr.themes.Citrus()` — without it Citrus's dark mode renders dark text on a dark background (text inherits unset colors). The same fix is harmless (and worth keeping) under other themes.
|
||||
- Width-cap with `!important` if Gradio 6's own breakpoints fight you — target `main`, `.gradio-container`, and the inner fillable wrapper, otherwise the width caps but goes flush-left.
|
||||
|
||||
## Minimal layout most demos converge to
|
||||
|
||||
```python
|
||||
with gr.Row():
|
||||
prompt = gr.Textbox(show_label=False, placeholder="…", container=False, scale=4)
|
||||
run = gr.Button("Run", variant="primary", scale=1)
|
||||
output = gr.Image(...)
|
||||
with gr.Accordion("Advanced settings", open=False):
|
||||
...
|
||||
```
|
||||
|
||||
`container=False` on the inline Textbox removes the default outer border for a tighter look.
|
||||
|
||||
## `gr.Markdown` for the intro
|
||||
|
||||
Be succinct. Title, one-line description, a links section. Don't over-explain how it works — that's what the model card is for.
|
||||
|
||||
## `gr.Examples`
|
||||
|
||||
**Add `gr.Examples` whenever it makes sense** (the app takes user input and representative inputs exist). It's the first thing a visitor clicks and it doubles as an instant smoke test. Prefer examples that mirror the **official examples from the original repo / model card** — the prompts, images, and settings the authors showcase — over inventing your own.
|
||||
|
||||
Keep each example row to the **few variables a user actually changes** (the prompt, the input image). Give the handler **default values for everything else** (steps, guidance, seed, …) so an example row is `["a cat on a windowsill"]`, not a wall of knobs. Design the signature so the interesting inputs come first and the rest default:
|
||||
|
||||
```python
|
||||
@spaces.GPU(duration=60)
|
||||
def generate(prompt, seed=42, steps=28, guidance=5.0): # only `prompt` varies in examples
|
||||
...
|
||||
|
||||
gr.Examples(
|
||||
examples=[
|
||||
["a cat sitting on a windowsill"],
|
||||
["mountains at sunset, photorealistic"],
|
||||
],
|
||||
inputs=[prompt], # just the varied inputs; the rest use the handler defaults
|
||||
outputs=output,
|
||||
fn=generate,
|
||||
cache_examples=True,
|
||||
cache_mode="lazy",
|
||||
)
|
||||
```
|
||||
|
||||
Why these flags:
|
||||
|
||||
- `cache_examples=True` makes example clicks instant (no re-inference per visitor).
|
||||
- **`cache_mode="lazy"`** — caches on first user click for each example. **Required on ZeroGPU** (Gradio's default on ZeroGPU is already lazy via `GRADIO_CACHE_MODE=lazy`). Eager would pre-run every example at app startup, but ZeroGPU has no GPU attached at startup — it'd fail and burn the creator's daily quota.
|
||||
- `cache_examples=True` silently disables `run_on_click` / `run_examples_on_click`. If your app relies on click-only behavior, set `cache_examples=False`.
|
||||
|
||||
The cache key is the **example row's file path**, not a content hash. Regenerating an asset in place serves the stale cached output forever. If you replace example files, bump a `cache_version` marker or wipe `.gradio/cached_examples/<id>/`.
|
||||
|
||||
Hot-reload (rung 1) does **not** rebuild the cache. A `cache_version` bump needs a real commit + restart.
|
||||
|
||||
## Streaming and generators
|
||||
|
||||
`gr.Interface(fn=...)` and `.click(fn=...)` both accept generator functions. Each `yield` pushes a new value:
|
||||
|
||||
```python
|
||||
@spaces.GPU(duration=120)
|
||||
def generate(prompt):
|
||||
yield gr.update(value=None, label="Starting…")
|
||||
for k in range(num_steps):
|
||||
yield gr.update(value=preview(k), label=f"Step {k+1}/{num_steps}")
|
||||
yield gr.update(value=final, label="Done")
|
||||
```
|
||||
|
||||
Use `gr.update(label=...)` for status narration — feels like a status line without a separate component.
|
||||
|
||||
**Caveat**: `gr.Progress(track_tqdm=True)` and `yield` partial outputs fight each other — pick one streaming mechanism.
|
||||
|
||||
## Useful UX bits
|
||||
|
||||
- `progress=gr.Progress(track_tqdm=True)` in the GPU function gives a free tqdm-driven progress bar.
|
||||
- Pair a "Randomize seed" checkbox with the seed input, and write the actually-used seed back so users can re-run deterministically:
|
||||
```python
|
||||
randomize = gr.Checkbox(label="Randomize seed", value=True)
|
||||
seed = gr.Number(label="Seed", value=0, precision=0)
|
||||
|
||||
@spaces.GPU
|
||||
def gen(..., seed, randomize_seed):
|
||||
if randomize_seed:
|
||||
seed = random.randint(0, 2**31 - 1)
|
||||
seed = int(seed)
|
||||
yield ..., gr.update(value=seed) # write back into the Seed input
|
||||
...
|
||||
|
||||
run.click(gen, inputs=[..., seed, randomize], outputs=[..., seed])
|
||||
```
|
||||
|
||||
## Custom HTML components
|
||||
|
||||
Stock Gradio components cover 95% of cases. When they don't, `gr.HTML(...)` lets you build contextual custom UI without leaving the Gradio Space (no need to switch to Docker).
|
||||
|
||||
Guide: https://www.gradio.app/guides/custom-HTML-components
|
||||
Example with a 3D camera-angle picker that makes sense in context: https://huggingface.co/spaces/multimodalart/qwen-image-multiple-angles-3d-camera
|
||||
|
||||
Don't reach for this if a stock component covers the need.
|
||||
|
||||
## Custom frontends — `gr.Server`
|
||||
|
||||
For fully custom frontends with their own HTML/JS, while keeping Gradio's queue + GPU scheduling:
|
||||
|
||||
```python
|
||||
from gradio import Server
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
app = Server(title="my-app")
|
||||
|
||||
@spaces.GPU(duration=60)
|
||||
def _run_gpu(prompt): return inference(prompt)
|
||||
|
||||
@app.api(name="generate", concurrency_limit=1, time_limit=180)
|
||||
def generate(prompt: str) -> str:
|
||||
return _run_gpu(prompt) # @app.api wraps, doesn't stack with @spaces.GPU
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def homepage():
|
||||
return open("index.html").read()
|
||||
|
||||
demo = app # HF runtime expects `demo`
|
||||
if __name__ == "__main__":
|
||||
demo.launch(ssr_mode=False)
|
||||
```
|
||||
|
||||
Required for the custom `/` route to actually serve:
|
||||
|
||||
```bash
|
||||
hf spaces variables add <ns>/<name> --env GRADIO_SSR_MODE=false
|
||||
```
|
||||
|
||||
`launch(ssr_mode=False)` is ignored on HF — must be the env var.
|
||||
|
||||
Valid `@app.api` kwargs: `name`, `description`, `concurrency_limit`, `concurrency_id`, `queue`, `batch`, `max_batch_size`, `api_visibility`, `time_limit`, `stream_every`.
|
||||
|
||||
**Don't stack `@spaces.GPU` and `@app.api`** on the same function — silently breaks request flow. Keep them on separate functions.
|
||||
|
||||
**Two-ceiling coordination**: both `@spaces.GPU(duration=N)` and `@app.api(time_limit=M)` apply; the lower wins. Set `time_limit` to your max duration across modes — a too-low `time_limit` kills the request even if GPU duration would have allowed it.
|
||||
|
||||
Hot-reload (rung 1 in [`debugging.md`](debugging.md)) does **not** work with `gr.Server` — always `hf upload` or commit.
|
||||
|
||||
Reference Space: https://huggingface.co/spaces/huggingface-projects/rf-detr-realtime-webcam
|
||||
|
||||
## Slow-startup Gradio (big-model Spaces)
|
||||
|
||||
For Spaces that take 10–20 min to load weights:
|
||||
|
||||
- Set `startup_duration_timeout: 1h` in README frontmatter (default 30 min).
|
||||
- Disable SSR: `hf spaces variables add <id> --env GRADIO_SSR_MODE=false`. Otherwise the SSR health check times out before the app finishes loading.
|
||||
|
||||
## Expose the Space as an MCP server
|
||||
|
||||
Launch with `demo.launch(mcp_server=True)` (Gradio 5+) so every API endpoint is also exposed as an MCP tool — free, and makes the Space usable by agents. For it to be useful:
|
||||
|
||||
- **Every API-triggered function needs a docstring and type hints.** Each Gradio event handler is auto-exposed over the API; Gradio turns the signature into the MCP tool's input schema and the docstring into its description. A function without them still works but surfaces as an opaque, unusable tool.
|
||||
- Give the important endpoints stable names (`api_name="generate"` on the `.click(...)`, or `@app.api(name=...)`), so tool names don't churn.
|
||||
- MCP mode can pull in extra deps (`gradio[mcp]`); if `launch` complains, see the pin note in [`known-errors.md`](known-errors.md).
|
||||
|
||||
```python
|
||||
@spaces.GPU(duration=60)
|
||||
def generate(prompt: str, seed: int = 42) -> str:
|
||||
"""Generate an image from a text prompt.
|
||||
|
||||
Args:
|
||||
prompt: what to generate.
|
||||
seed: RNG seed for reproducibility.
|
||||
"""
|
||||
...
|
||||
|
||||
btn.click(generate, inputs=[prompt, seed], outputs=out, api_name="generate")
|
||||
demo.launch(mcp_server=True)
|
||||
```
|
||||
|
||||
## Don't
|
||||
|
||||
- `gr.Button(text="X")` — `text=` was removed; use `gr.Button("X")`.
|
||||
- `gr.Button(type="button")` — drop the kwarg.
|
||||
- `.click(..., _js=share_js)` — renamed to `js=`.
|
||||
- `.style(height=...)` — removed in gradio 4+.
|
||||
- `gr.ImageMask(brush_color=...)` — kwarg removed.
|
||||
- `demo.launch(mcp_server=True)` on gradio 4.x — only valid on 5+.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Community GPU grants
|
||||
|
||||
When a non-PRO user has a good use case for ZeroGPU (open research demo, hobbyist project, educational tool, institutional showcase) and doesn't want to subscribe, they can request a free community grant from Hugging Face.
|
||||
|
||||
## The flow
|
||||
|
||||
1. **Build the Space.** Create it as `--flavor cpu-basic` (the user is not PRO, so creating with `--flavor zero-a10g` will fail at `create_repo`). Code the app for ZeroGPU anyway — `import spaces`, `@spaces.GPU`, module-scope `.to("cuda")`. The Space will technically run on CPU until the grant is approved, but it'll be ready to switch over instantly.
|
||||
|
||||
In this mode you **can't iterate-with-real-inference** before the grant — the CPU Space won't actually run heavy compute. Get the app to BUILD cleanly and `RUNNING` (even if the runtime would OOM on real input), then submit.
|
||||
|
||||
2. **Submit a Community Tab discussion** on the Space. Title:
|
||||
|
||||
```
|
||||
Apply for a GPU community grant: <Personal|Company|Academic> project
|
||||
```
|
||||
|
||||
Pick the closest fit. Body:
|
||||
|
||||
```
|
||||
Description of the app: one paragraph on what it does + who it's for.
|
||||
Justification: one paragraph on why this should run on ZeroGPU
|
||||
(open-source, research, educational, etc.).
|
||||
```
|
||||
|
||||
If the user didn't give you a justification, a reasonable default is "Non-PRO wants to build a public ZeroGPU app — happy to provide more context if helpful."
|
||||
|
||||
3. **Wait.** Open and publicly-facing applications by researchers, tinkerers, and institutions are typically approved. Approval can take days.
|
||||
|
||||
4. **Once approved**, the Space is moved to ZeroGPU automatically — no code change needed. The user comes back and you can iterate / refine with real GPU access.
|
||||
|
||||
## When to suggest this
|
||||
|
||||
- User is not on PRO but their use case is a clear fit for ZeroGPU (a public ML demo, not a private tool).
|
||||
- The model fits in `large` (≤ 48 GB VRAM at the chosen precision).
|
||||
|
||||
## When NOT to suggest this
|
||||
|
||||
- Private / commercial / closed-source projects — push the user toward PRO instead.
|
||||
- The model genuinely needs dedicated paid hardware (huge LLM, vLLM/JAX/ONNX as main model with heavy init) — `canPay=True` users can use paid flavors directly.
|
||||
- The user is already PRO — they have ZeroGPU access; no grant needed.
|
||||
|
||||
## Posting the request programmatically
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi
|
||||
|
||||
api = HfApi(token="hf_...")
|
||||
api.create_discussion(
|
||||
repo_id="<ns>/<space>",
|
||||
repo_type="space",
|
||||
title="Apply for a GPU community grant: Personal project",
|
||||
description="<description and justification>",
|
||||
)
|
||||
```
|
||||
|
||||
The Community Tab must be enabled on the Space (default — keep it on).
|
||||
@@ -0,0 +1,85 @@
|
||||
# Inference Providers — when not to host the model
|
||||
|
||||
Some Spaces don't need a GPU at all. If the model is available through HF Inference Providers (Cerebras, Fireworks, Together, Replicate, OpenRouter, etc.), the Space can be a thin Gradio shell that proxies to a hosted endpoint:
|
||||
|
||||
- Zero VRAM, no `@spaces.GPU`, no model download.
|
||||
- Works for models too large to fit on ZeroGPU (120B+).
|
||||
- Hardware can be `cpu-basic` — no GPU at all.
|
||||
|
||||
## When to use this pattern
|
||||
|
||||
- **Stateless chat or text completion** with a big model.
|
||||
- **The user wants a public demo of a frontier-scale model** that obviously doesn't fit on a single 48 GB MIG.
|
||||
- **The user wants to ship something fast** without worrying about quantization / sharding.
|
||||
|
||||
## When NOT to use this pattern
|
||||
|
||||
- The model isn't available on any Inference Provider. Check with:
|
||||
```bash
|
||||
curl "https://huggingface.co/api/models/<ns>/<repo>?expand[]=inferenceProviderMapping"
|
||||
```
|
||||
- The Space needs **custom decoding** (special sampling, tool use, retrieval, anything stateful or interactive across calls).
|
||||
- The Space needs **multimodal** beyond what the provider exposes.
|
||||
- The user explicitly wants to own the inference stack (model loading, decoding, performance tuning).
|
||||
|
||||
For those, host the model yourself on ZeroGPU — see [`zerogpu.md`](zerogpu.md).
|
||||
|
||||
## Two billing modes
|
||||
|
||||
Choose based on who pays for inference.
|
||||
|
||||
### Mode A — Space creator pays (simple)
|
||||
|
||||
Set `HF_TOKEN` as a Space secret. The Space uses `InferenceClient` directly. Every visitor's call is billed to the Space creator's account.
|
||||
|
||||
```python
|
||||
import os, gradio as gr
|
||||
from huggingface_hub import InferenceClient
|
||||
|
||||
client = InferenceClient(api_key=os.environ["HF_TOKEN"], provider="fireworks-ai")
|
||||
|
||||
def chat(msg, history):
|
||||
return client.chat_completion(
|
||||
model="<org>/<model>",
|
||||
messages=[*history, {"role": "user", "content": msg}],
|
||||
max_tokens=512,
|
||||
).choices[0].message.content
|
||||
|
||||
gr.ChatInterface(chat).launch()
|
||||
```
|
||||
|
||||
Use when you want users to "just click and try it" — no sign-in friction. Cost is on you.
|
||||
|
||||
### Mode B — Visitor pays (recommended for public demos)
|
||||
|
||||
`gr.LoginButton` + `gr.load("models/...")` with `accept_token=button`. Each visitor signs in with their HF account; inference is billed to **their** account.
|
||||
|
||||
```python
|
||||
import gradio as gr
|
||||
|
||||
with gr.Blocks(fill_height=True) as demo:
|
||||
with gr.Sidebar():
|
||||
button = gr.LoginButton("Sign in")
|
||||
gr.load("models/<org>/<model>", accept_token=button, provider="fireworks-ai")
|
||||
demo.launch()
|
||||
```
|
||||
|
||||
README frontmatter needs:
|
||||
|
||||
```yaml
|
||||
hf_oauth: true
|
||||
hf_oauth_scopes:
|
||||
- inference-api
|
||||
```
|
||||
|
||||
This is the **recommended pattern for public demos** — sustainable cost-wise, and visitors get to use their own provider quotas (which most have paid for or get free).
|
||||
|
||||
## Hardware
|
||||
|
||||
`cpu-basic`. No GPU. Don't put `--flavor zero-a10g` — you'd waste a paid grant.
|
||||
|
||||
## Anti-pattern: `@spaces.GPU` wrapping a provider call
|
||||
|
||||
If you do use Inference Providers, do **not** wrap the call in `@spaces.GPU`. The decorator reserves a GPU slot on your Space for the full `duration=`, but the function does no GPU work — just an HTTP call out. You burn your own ZeroGPU quota for nothing.
|
||||
|
||||
A provider-proxy Space wants `cpu-basic` hardware and zero `@spaces.GPU`.
|
||||
@@ -0,0 +1,232 @@
|
||||
# Known errors
|
||||
|
||||
Check if this is a known issue before trying your own fix. Entries are keyed by the substring that actually appears in `runtime.errorMessage`, the build log, or a Python traceback — grep this file for the error you saw.
|
||||
|
||||
If you hit something not listed here and figure out a fix, please ask your human to PR it back so future runs benefit.
|
||||
|
||||
---
|
||||
|
||||
## Build / config errors
|
||||
|
||||
These come from the Space build pipeline before the app starts. Read with `hf spaces logs <id> --build --tail 500` — find the **first** error, not the last.
|
||||
|
||||
### `CONFIG_ERROR: torch version in requirements.txt is not compatible with ZeroGPU`
|
||||
|
||||
**Cause**: `requirements.txt` pins `torch==X.Y.Z` to a version outside the supported set (`2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`).
|
||||
**Fix**: Unpin torch (preferred — the runtime preinstalls the latest supported version), or pin to one of the supported values.
|
||||
|
||||
### `Cannot install … because these package versions have conflicting dependencies` / `ResolutionImpossible`
|
||||
|
||||
**Cause**: A dep conflicts with the Gradio SDK pinned by `sdk_version:` in README. Most commonly `pydantic`, `uvicorn`, `huggingface_hub`, or `jinja2` pinned to old values that the SDK no longer accepts.
|
||||
**Fix**: Unpin the offender. For `gradio[mcp]` specifically, `uvicorn>=0.31.1` and `pydantic>=2.11.10` are required.
|
||||
|
||||
### Build hangs in dependency resolution > 10 min
|
||||
|
||||
**Cause**: pip backtracking through a deep version space.
|
||||
**Fix**: Pin the conflicting transitive dep. The `--build` logs will show which one. Bump `startup_duration_timeout: 1h` in README frontmatter if heavy downloads are expected.
|
||||
|
||||
### `ModuleNotFoundError: No module named 'pkg_resources'`
|
||||
|
||||
**Cause**: setuptools 81 dropped `pkg_resources`; an old package's `setup.py` imports it.
|
||||
**Fix**: Bump or unpin the offender. Typical culprits: `deepspeed==0.15.x` (training-only — usually safe to drop from inference Spaces), `openai-whisper==20231117`.
|
||||
|
||||
### `400 Bad Request` from `/api/validate-yaml` during `create_repo` / `upload_file`
|
||||
|
||||
**Cause**: README frontmatter failed server validation. Most common: `short_description` over the (undocumented) character cap — target ≤ 60.
|
||||
**Fix**: Shorten `short_description`. Long descriptions go in the README body. Also double-check `colorFrom`/`colorTo` are one of `red|yellow|green|blue|indigo|purple|pink|gray`.
|
||||
|
||||
### `403 Forbidden` from `create_repo` with `space_hardware="zero-a10g"`
|
||||
|
||||
**Cause**: The user isn't on PRO / Team / Enterprise so the API rejects ZeroGPU at creation.
|
||||
**Fix**: Retry without `space_hardware=`. Keep `hardware:` out of README frontmatter (silently ignored anyway). The Space is created on CPU; point the user at PRO upgrade or [community grant](grants.md).
|
||||
|
||||
### `403 Forbidden` from `create_commit(..., create_pr=True)`
|
||||
|
||||
**Cause**: Upstream Space has Discussions disabled.
|
||||
**Fix**: Ask the maintainer to enable Discussions, or push directly if you have write access.
|
||||
|
||||
---
|
||||
|
||||
## Startup / RUNTIME_ERROR
|
||||
|
||||
These come from `hf spaces logs <id> --tail 500`.
|
||||
|
||||
### `RuntimeError: CUDA has been initialized before importing the spaces package`
|
||||
|
||||
**Cause**: Something triggered CUDA init in the main process before `import spaces`. Usually wrong import order; sometimes a third-party lib eagerly initializing CUDA at import time (e.g. `numba.cuda`).
|
||||
**Fix**: Reorder so `import spaces` is first. For numba-using stacks (NeMo, RAPIDS bits):
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("NUMBA_DISABLE_CUDA", "1")
|
||||
import spaces
|
||||
```
|
||||
|
||||
### `RuntimeError: No @spaces.GPU function detected during startup`
|
||||
|
||||
**Cause**: The function bound to `.click(fn=...)` / `.submit(...)` isn't decorated. Decorating an inner helper doesn't count — the startup scan only walks Gradio's registered handlers.
|
||||
**Fix**: Decorate the function Gradio binds. If that conflicts with another decorator, wrap explicitly:
|
||||
```python
|
||||
@spaces.GPU(duration=60)
|
||||
def gpu_inner(...): ...
|
||||
def gradio_handler(...): return gpu_inner(...)
|
||||
```
|
||||
(Or just decorate `gradio_handler` directly.)
|
||||
|
||||
### `ImportError: cannot import name 'HfFolder' from 'huggingface_hub'`
|
||||
|
||||
**Cause**: Old gradio (`4.44` and similar) imports `HfFolder` from `huggingface_hub`, which was removed in recent hub releases.
|
||||
**Fix**: Two options.
|
||||
- Pin `huggingface-hub==0.25.0` in `requirements.txt` (keeps old gradio happy).
|
||||
- Bump `sdk_version` in README to `5.x` or `6.x` (also fixes a lot of other API breaks).
|
||||
If a Gradio custom component locks the major (`gradio-image-prompter`, `gradio_litmodel3d`, …), install it with `--no-deps` so its `gradio<5.0` requirement doesn't bind.
|
||||
|
||||
### `ImportError: cannot import name 'is_traceable_wrapper_subclass' from 'torch.utils._python_dispatch'`
|
||||
|
||||
**Cause**: A dep with `torchaudio<2.1` / `torch<2` in its `setup.py` (e.g. `demucs`, `audiocraft`) downgraded torch silently. The build succeeded, the app booted, and `import spaces` then died on a missing torch symbol.
|
||||
**Fix**: Install the offender from `app.py` with `--no-deps` *before* `import spaces`:
|
||||
```python
|
||||
import subprocess, sys
|
||||
subprocess.run([sys.executable, "-m", "pip", "install", "--no-deps",
|
||||
"git+https://github.com/facebookresearch/demucs"], check=True)
|
||||
import spaces
|
||||
```
|
||||
List its actual runtime deps (`dora-search einops julius lameenc openunmix pyyaml tqdm` for demucs) yourself in `requirements.txt`.
|
||||
|
||||
### `_pickle.UnpicklingError: Weights only load failed`
|
||||
|
||||
**Cause**: `torch.load` weights-only default flipped to `True` in torch 2.6. Old checkpoints pickling numpy/object globals fail.
|
||||
**Fix**: For trusted upstream checkpoints, monkey-patch before the package import:
|
||||
```python
|
||||
import torch
|
||||
_orig = torch.load
|
||||
torch.load = lambda *a, **k: _orig(*a, **{**k, "weights_only": k.get("weights_only", False)})
|
||||
```
|
||||
|
||||
### Stuck at `ZeroGPU init – 10.0%` then 60 s timeout
|
||||
|
||||
**Cause**: A library called `cuInit` in the parent process, poisoning the fork (most often `numba.cuda` via NeMo). The actual `@spaces.GPU` body never starts.
|
||||
**Fix**: `os.environ.setdefault("NUMBA_DISABLE_CUDA", "1")` as the first line of `app.py`, before `import spaces`.
|
||||
|
||||
### `RUNTIME_ERROR` right after long `APP_STARTING`, logs sparse
|
||||
|
||||
**Cause**: Boot exceeded `startup_duration_timeout` (default 30 min). Big-model loads commonly trigger this.
|
||||
**Fix**: Bump `startup_duration_timeout: 1h` in README frontmatter. For Gradio 6 specifically, also set `GRADIO_SSR_MODE=false` via `hf spaces variables add <id> --env GRADIO_SSR_MODE=false` to dodge SSR health-check timeouts during slow boot.
|
||||
|
||||
### `RUNNING` but the public URL returns 404
|
||||
|
||||
**Cause**: The Space is private. Anonymous Client / browser hits return 404.
|
||||
**Fix**: Authenticate. `gradio_client.Client(space, token=os.environ["HF_TOKEN"])`. The kwarg is `token=`, not `hf_token=`.
|
||||
|
||||
### `workload was not healthy after 30 min`
|
||||
|
||||
**Cause**: Infra-side scheduling or a build that genuinely can't finish in time.
|
||||
**Fix**: Usually not actionable in code. Bump `startup_duration_timeout` if heavy downloads are expected; otherwise wait or report.
|
||||
|
||||
### Exit code 128 / containerd / scheduling failure
|
||||
|
||||
**Cause**: HF infra glitch.
|
||||
**Fix**: `hf spaces restart <id> --factory-reboot`. If it persists, retry later or report. Not fixable in code.
|
||||
|
||||
---
|
||||
|
||||
## Inference-time errors
|
||||
|
||||
These appear in `hf spaces logs <id> --follow` while a request is running.
|
||||
|
||||
### `ZeroGPU illegal duration`
|
||||
|
||||
**Cause**: `@spaces.GPU(duration=N)` is larger than the visitor's tier per-call cap.
|
||||
**Fix**: Lower `N`. Tier caps live in the [ZeroGPU docs](https://huggingface.co/docs/hub/spaces-zerogpu).
|
||||
|
||||
### `ZeroGPU quota exceeded (X requested vs Y left)`
|
||||
|
||||
**Cause**: The visitor's remaining quota < `requested duration`. The comparison is `requested vs remaining`, not `actual vs remaining` — a 10-second task left at the default 60 s blocks the user as soon as their remaining drops below 60 s.
|
||||
**Fix**: Lower `duration` to the realistic worst case. For input-dependent runtime, use a callable estimator.
|
||||
|
||||
### `RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at .../CUDACachingAllocator.cpp`
|
||||
|
||||
**Cause**: Allocator fragmentation under transient memory spikes (high-res pixel-space ops, large attention activations, SR models, video DiTs). Not a clean OOM.
|
||||
**Fix**: Set expandable segments at the **very top** of `app.py`, before any torch import:
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
|
||||
import spaces
|
||||
import torch
|
||||
```
|
||||
Usually a single-line fix that replaces lowering resolution or moving to `xlarge`.
|
||||
|
||||
### Call hangs forever on first `@spaces.GPU` entry
|
||||
|
||||
**Cause**: The decorated function returned a CUDA tensor. Unpickling it in the main process triggers `torch.cuda._lazy_init()`, which ZeroGPU blocks.
|
||||
**Fix**: Convert to CPU before returning: `return tensor.cpu()` or `.cpu().numpy()`. For `gr.State`, scrub before yielding.
|
||||
|
||||
### `PicklingError` at call entry
|
||||
|
||||
**Cause**: An argument crossing the fork boundary contains an unpicklable object — file handle, lock, lambda, closure, or `gr.SelectData`.
|
||||
**Fix**: Extract the picklable fields in a thin un-decorated wrapper, pass plain values to the `@spaces.GPU` function. For `gr.SelectData` specifically, pull out `evt.index[0]`, `evt.index[1]` etc. outside the decorator.
|
||||
|
||||
### `RecursionError` inside `gr.SelectData.__getattr__`
|
||||
|
||||
**Cause**: Same as above — `gr.SelectData` doesn't survive pickle.
|
||||
**Fix**: Same — extract its fields before crossing the boundary.
|
||||
|
||||
### `CUDA error … flash_fwd_launch_template.h: no kernel image is available for execution on the device` (or `:188: invalid argument`)
|
||||
|
||||
**Cause**: A Flash Attention 3 kernel was loaded — directly via `kernels-community/{flash-attn3,vllm-flash-attn3,sgl-flash-attn3}`, or indirectly via an old `xformers` wheel that auto-dispatched to FA3. FA3 has no Blackwell sm_120 build.
|
||||
**Fix**: Use `attn_implementation="sdpa"`, or `"flash_attention_2"` with the FA2 wheel from `multimodalart/zerogpu-blackwell-wheels`. For xformers, the prebuilt wheel from the same dataset auto-dispatches to FA2 — no monkey-patch.
|
||||
|
||||
### `NotImplementedError: sgl_flash_attn3 is only supported on sm80 and above with CUDA >= 12.3`
|
||||
|
||||
**Cause**: `kernels-community/sgl-flash-attn3` rejects sm_120 at runtime despite the error wording. Same root cause as the FA3 entry above.
|
||||
**Fix**: Same — SDPA or FA2.
|
||||
|
||||
### `ImportError: cannot import name 'flash_attn_varlen_func' from 'flash_attn'` / model insists on `attn_implementation="flash_attention_2"`
|
||||
|
||||
**Cause**: The model imports flash_attn at module top with no escape hatch, and the runtime doesn't ship it.
|
||||
**Fix**: Install the prebuilt `flash_attn-2.8.3-cp310-cp310-linux_x86_64.whl` from `multimodalart/zerogpu-blackwell-wheels`. Requires `python_version: "3.10"` in README (wheel is cp310 only). Real `flash_attn_2_cuda` satisfies xformers' `flash_attn_gpu` probe too.
|
||||
|
||||
For transformers `AutoModel`-style configs that aren't a hard import, swap `attn_implementation="flash_attention_2"` → `"sdpa"`. Torch-native, zero deps.
|
||||
|
||||
### `selective_scan_cuda.so undefined symbol` / `_torchaudio.abi3.so undefined symbol`
|
||||
|
||||
**Cause**: A direct-URL prebuilt CUDA wheel pinned to an old torch ABI (`cu12torch2.4cxx11abiFALSE`, etc.) — won't load on the current runtime.
|
||||
**Fix**: Drop the URL-pinned wheel from `requirements.txt`. Use a torch-current wheel from `multimodalart/zerogpu-blackwell-wheels`, kernels-community, or upstream's release page.
|
||||
|
||||
### `TypeError: 'dict' object is not hashable` inside `jinja2/utils.py:get`
|
||||
|
||||
**Cause**: Old gradio (4.44) + modern starlette / jinja2 cache clash.
|
||||
**Fix**: Either pin `jinja2<3.2` + `starlette<0.40` (keeps old gradio), or bump `sdk_version` past 5.0.
|
||||
|
||||
---
|
||||
|
||||
## Smoke-test / client errors
|
||||
|
||||
### `Client.__init__() got an unexpected keyword argument 'hf_token'`
|
||||
|
||||
**Cause**: Older `gradio_client` API used `hf_token=`; current uses `token=`.
|
||||
**Fix**: `Client(space, token=os.environ["HF_TOKEN"])`.
|
||||
|
||||
### `httpx.ReadTimeout` on `client.predict(...)`
|
||||
|
||||
**Cause**: Default timeout too small for the GPU duration.
|
||||
**Fix**: `Client(..., httpx_kwargs={"timeout": 600})`. Set this to at least your `@spaces.GPU(duration=N)` plus 60 s.
|
||||
|
||||
### `404` on what looks like a valid endpoint name
|
||||
|
||||
**Cause**: Streaming endpoint (function uses `yield`), or a custom `gr.Server` `@app.get/post` route that doesn't appear in `view_api()`.
|
||||
**Fix**: For streaming, use `client.submit(...).result()` (or iterate the job). For custom routes, use `httpx.post(base_url + "/route", ...)` directly — bypass `gradio_client`.
|
||||
|
||||
### Result file looks empty / dimensions wrong
|
||||
|
||||
**Cause**: HTTP 200 ≠ correct output. The model snapped to a different size, or the wrong endpoint was hit.
|
||||
**Fix**: Sniff the returned file's magic bytes (`glTF`, `\x89PNG`, `RIFF…WEBP`, `RIFF…WAVE`, `[4:8]==b"ftyp"`) and check returned dimensions match what was requested.
|
||||
|
||||
---
|
||||
|
||||
## Submitting new entries
|
||||
|
||||
If you hit an error not in this file and figure out the fix, please ask your human to PR it back to this skill. Format:
|
||||
|
||||
- A 1-line heading with the exact error substring an agent would grep for.
|
||||
- **Cause**: one sentence on what triggered it.
|
||||
- **Fix**: concrete commands or code. If the fix needs more than 5 lines of narrative, point to another reference file (e.g. [`debugging.md`](debugging.md), [`zerogpu.md`](zerogpu.md)) for depth.
|
||||
@@ -0,0 +1,169 @@
|
||||
# requirements.txt for Spaces
|
||||
|
||||
Rules for what to pin, what to leave alone, where to source CUDA wheels, and which torch-side-cars drift silently.
|
||||
|
||||
## What's preinstalled (do not list)
|
||||
|
||||
The Gradio SDK base image already installs these on every hardware tier — listing them in `requirements.txt` causes resolution failures or, worse, lets pip silently drift the runtime out of compatibility:
|
||||
|
||||
| Package | Pinning rules |
|
||||
|---|---|
|
||||
| `gradio` | Don't list. Locked by `sdk_version:` in README frontmatter; pinning here is ignored or breaks. |
|
||||
| `spaces` | Don't list. Platform-pinned; a user pin always loses. |
|
||||
| `huggingface_hub` | Don't list by default. Pin only as a workaround for old `gradio<5` that imports the removed `HfFolder` symbol (see [`known-errors.md`](known-errors.md)). |
|
||||
| `torch` | **Pinnable, but only within `{2.8.0, 2.9.1, 2.10.0, 2.11.0}`.** Anything outside causes `CONFIG_ERROR: torch version in requirements.txt is not compatible`. Default is to leave unpinned (runtime preinstalls 2.11), but pinning is appropriate when (a) a specific version is known-good for your model, (b) you're matching a CUDA-extension wheel's `torch2.X` tag, or (c) a dep would otherwise drag torch outside the supported set. When you pin torch, also pin `torchvision` / `torchaudio` to the matching minor — see the "Torch-family side-car drift" section below. |
|
||||
|
||||
## What to list
|
||||
|
||||
Everything you actually `import`, including the often-forgotten:
|
||||
|
||||
- `torchvision`, `torchaudio` — **not** preinstalled. Leave unpinned; pip resolves against the installed `torch` major.minor.
|
||||
- `accelerate` — needed whenever you use `device_map=`. Listing it also silences `low_cpu_mem_usage=False` warnings.
|
||||
- `sentencepiece` — required by most LLM tokenizers; rarely transitive.
|
||||
- `einops` — required by `flash_attn.layers.rotary` and many model repos.
|
||||
- Domain libs: `diffusers`, `transformers`, `safetensors`, `pillow`, `numpy`, etc.
|
||||
|
||||
If a research repo ships a Python package directory (`models/`, `pipeline/`, …), just upload the directory with the rest of the Space — the whole repo root is importable as `/home/user/app`. **Do not** try to reference local paths from `requirements.txt`.
|
||||
|
||||
## Pinning torch
|
||||
|
||||
ZeroGPU accepts only `2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`. Default is unpinned (runtime preinstalls the latest). Pinning is fine — and sometimes warranted — within that set:
|
||||
|
||||
- A specific torch is known-good for your model (numerics, attention kernel availability, etc.).
|
||||
- A direct-URL CUDA wheel encodes a `torch2.X` tag (see "Prebuilt CUDA wheels" below) — pin torch to match.
|
||||
- A dep's `setup.py` would otherwise downgrade torch outside the supported set.
|
||||
|
||||
`2.8.0` is the safest fallback for old requirements that refuse modern torch. `2.10.0` / `2.11.0` is the sweet spot for new code. When you pin torch, also pin `torchvision` / `torchaudio` to the matching minor — see the side-car drift section.
|
||||
|
||||
When a dep would silently downgrade torch (e.g. some forks of `demucs`, `audiocraft` pin `torchaudio<2.1`), install the offender from `app.py` with `--no-deps` rather than pinning torch around it:
|
||||
|
||||
```python
|
||||
import subprocess, sys
|
||||
subprocess.run([sys.executable, "-m", "pip", "install", "--no-deps",
|
||||
"git+https://github.com/facebookresearch/demucs"], check=True)
|
||||
import spaces # safe now — torch wasn't touched
|
||||
```
|
||||
|
||||
List the offender's real runtime deps yourself in `requirements.txt`.
|
||||
|
||||
## Torch-family side-car drift
|
||||
|
||||
`torchvision`, `torchaudio`, `torchcodec` are built against a specific `torch` major.minor. Listing them unpinned **usually** works, but two known drift patterns:
|
||||
|
||||
- `torchaudio==2.11.0` (and later) **dropped its `Requires-Dist: torch==X.Y.Z` line**. With torch pinned to 2.10, pip silently resolves torchaudio to 2.11.0 and the import fails on ABI mismatch.
|
||||
- `torchcodec` declares no torch dependency in PyPI metadata at all.
|
||||
|
||||
Verification after `pip install` or `uv lock --upgrade`:
|
||||
|
||||
```bash
|
||||
curl -s https://pypi.org/pypi/<pkg>/<version>/json \
|
||||
| python3 -c "import json,sys,re; rd=json.load(sys.stdin)['info'].get('requires_dist') or []; \
|
||||
print('\n'.join(x for x in rd if re.match(r'^torch(?![a-z])', x)) or '(no torch constraint)')"
|
||||
```
|
||||
|
||||
When PyPI is silent, fall back to the project's README compatibility table (torchcodec's lives at https://github.com/pytorch/torchcodec).
|
||||
|
||||
## Prebuilt CUDA wheels — the Blackwell wheels dataset
|
||||
|
||||
This is the **first** thing to reach for when a CUDA-extension package has no upstream wheel matching the ZeroGPU torch / CUDA / cxx11-abi cell. Prefer it over any runtime workaround (`pip install git+…` inside `@spaces.GPU`, committed stub packages, `sys.modules` injection, monkey-patch shims), all of which are slower, fragile, and eat `duration` budget. Canonical prebuilt sm_120 wheels:
|
||||
|
||||
> https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels
|
||||
|
||||
Wheels live at `wheels/<cell>/<wheel>`. A **cell** encodes torch × CUDA × Python as `pt<torch>-cu<cuda>-cp<pyver>`. Every cell ships the same seven packages:
|
||||
|
||||
`flash_attn` (two versions: `2.8.3` and `2.7.4.post1`), `xformers`, `pytorch3d`, `nvdiffrast`, `diff_gaussian_rasterization`, `torchmcubes`.
|
||||
|
||||
Current cells (12):
|
||||
|
||||
| torch | CUDA | Python cells available |
|
||||
|---|---|---|
|
||||
| 2.8.0 (`pt28`) | 12.8 | cp310, cp311, cp312 |
|
||||
| 2.9.1 (`pt291`) | 12.8 | cp310 |
|
||||
| 2.10.0 (`pt210`) | 12.8 / 13.0 | cp310 (cu128), cp312 (cu130) |
|
||||
| 2.11.0 (`pt211`) | 13.0 | cp312, cp313 |
|
||||
| 2.12.0 (`pt212`) | 13.0 | cp310, cp311, cp312, cp313 |
|
||||
|
||||
### Picking a cell
|
||||
|
||||
1. **Match `cp<pyver>` to your `python_version:`.** The wheels are cp-ABI-specific — a `cp310` wheel needs Python 3.10, `cp312` needs 3.12, etc. (`flash_attn` now ships cp310 **through** cp313, so this is a free choice, not a forced pin to 3.10 as in older versions of this doc.)
|
||||
2. **The wheels are torch-minor-tolerant.** The `flash_attn` / `xformers` filenames encode no torch version, so a `pt212-cu130-cp310` wheel runs fine on the live torch-2.11 cp310 runtime. `flash_attn` ships cp310 through cp313, so the Python choice is free. Pick the highest-torch cell for your Python version unless you've pinned an older torch — then match it (torch 2.8 → a `pt28-cu128-cp3XX` cell).
|
||||
3. **Copy the *exact* filename from the cell you pick.** The `xformers` build hash differs across cells (`0.0.34+3da0fc92…` on the cu130 / torch≥2.10 cells, `0.0.34+41531cee…` on the cu128 / torch 2.8–2.9 cells). Don't hardcode one filename across cells — list the cell (`hf download --repo-type dataset multimodalart/zerogpu-blackwell-wheels --include "wheels/<cell>/*"` or the Hub file browser) and copy what's there.
|
||||
|
||||
Reference by direct URL in `requirements.txt`:
|
||||
|
||||
```
|
||||
https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/<cell>/<wheel>
|
||||
```
|
||||
|
||||
### Per-package status
|
||||
|
||||
Each package's version is constant across cells; only the `cp`/torch/CUDA tags in the filename change. The "instead of" column is the runtime workaround to avoid — the wheel is the clean path.
|
||||
|
||||
| Package | Instead of | Version + caveats |
|
||||
|---|---|---|
|
||||
| `flash_attn` | committing a `flash_attn/` stub package; `sys.modules["flash_attn"] = …` injection | **2.8.3** (default) or **2.7.4.post1** (repos that pin `flash-attn<2.8`). Ships **cp310–cp313**. Needs `einops` for `flash_attn.layers.rotary`. Built `FLASH_ATTN_CUDA_ARCHS=120` (sm_120 only). This is FlashAttention-**2** — FA3/FA4 do **not** run on sm_120 (no TMEM); see [`zerogpu.md`](zerogpu.md) → Attention backends. Its real `flash_attn_2_cuda` also satisfies xformers' `flash_attn_gpu` probe. |
|
||||
| `xformers` | an MEA→SDPA monkey-patch shim; a Cutlass-force shim | **0.0.34** (`Requires: torch>=2.10`). **Build hash differs per cell** — copy the exact filename. Auto-dispatch picks FA2 (`fa2F`) on sm_120; classic Cutlass / FA3 reject sm_120 but auto-dispatch never selects them. |
|
||||
| `pytorch3d` | a runtime `pip install git+…pytorch3d.git` inside `@spaces.GPU` | **0.7.9**. Needs `numpy`, `iopath`, `fvcore` listed. No torch pin in metadata; loads cleanly on torch 2.11. |
|
||||
| `nvdiffrast` | a runtime build with `TORCH_CUDA_ARCH_LIST=12.0` | **0.4.0**. Needs `numpy`. `RasterizeGLContext` is a deprecation alias for `RasterizeCudaContext` — no headless-GL footgun. |
|
||||
| `diff_gaussian_rasterization` | a runtime build from `graphdeco-inria/diff-gaussian-rasterization.git` | **Upstream Inria API only** (returns 2-tuple `(color, radii)`). Does NOT match the ashawkey fork (4-tuple incl. alpha+depth) used by `ashawkey/LGM`, `dylanebert/LGM-mini`, etc. Forks need their own wheel. |
|
||||
| `torchmcubes` | a runtime `pip install git+…torchmcubes.git` | **0.1.0**. **sm_120 only** (no fatbin for older archs). Works on ZeroGPU / Blackwell; not portable to a dedicated T4 / L4 / A10G Space. |
|
||||
|
||||
### Pattern
|
||||
|
||||
Resolve the exact filenames from your chosen cell, then:
|
||||
|
||||
```
|
||||
# requirements.txt (cell = pt212-cu130-cp310 → needs python_version "3.10")
|
||||
numpy
|
||||
einops
|
||||
https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt212-cu130-cp310/flash_attn-2.8.3-cp310-cp310-linux_x86_64.whl
|
||||
https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt212-cu130-cp310/xformers-0.0.34+3da0fc92.d20260528-cp39-abi3-linux_x86_64.whl
|
||||
```
|
||||
|
||||
```yaml
|
||||
# README frontmatter — pin Python to match the wheel cell's cp tag
|
||||
python_version: "3.10"
|
||||
```
|
||||
|
||||
**Do not** install these from `@spaces.GPU` startup. A `subprocess.check_call` pip-install at first GPU acquire is strictly worse than the wheel URL — slower cold start, eats `duration` budget, breaks reproducibility, and the build sometimes exceeds the `@spaces.GPU(duration=1500)` cap.
|
||||
|
||||
### When you need a wheel that's not in the dataset
|
||||
|
||||
Three options, in preference order:
|
||||
|
||||
1. **kernels-community** — https://huggingface.co/kernels-community handles ABI matching for you. Often the simplest path; no version pinning needed.
|
||||
2. **Upstream wheel matrix** — e.g. flash-attention's releases page ships a fairly complete `cu12 / torch / Python` matrix at https://github.com/Dao-AILab/flash-attention/releases. Pin `torch==X.Y.Z` in `requirements.txt` to match the wheel's `torch2.X` tag.
|
||||
3. **Build it yourself and host on HF Hub.** Last resort — see [`debugging.md`](debugging.md) for the in-`@spaces.GPU` source-build pattern as a stopgap while a wheel is being built.
|
||||
|
||||
## Reading a CUDA wheel filename
|
||||
|
||||
```
|
||||
flash_attn-2.8.3+cu130torch2.12cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
|
||||
```
|
||||
|
||||
| Tag | Meaning |
|
||||
|---|---|
|
||||
| `cu130` | CUDA major version (13.0) |
|
||||
| `torch2.12` | torch major.minor the wheel was compiled against |
|
||||
| `cxx11abiFALSE` | C++ stdlib ABI choice (`TRUE` or `FALSE`) |
|
||||
| `cp310-cp310` | CPython version (3.10) |
|
||||
|
||||
ABI / symbol mismatches at any of these → `ImportError` on first import. Pin `torch` to match `torch2.X`. Set `python_version:` to match `cp3XX`.
|
||||
|
||||
## Don't pin `xformers`
|
||||
|
||||
Leave bare in `requirements.txt` (or use the prebuilt URL above). Pip picks the wheel matching your installed torch.
|
||||
|
||||
## Don't pin `spaces`
|
||||
|
||||
Even if a `uv export` produces it, exclude with `--no-emit-package spaces`. The platform always pins its own version.
|
||||
|
||||
## Specifically about Python version
|
||||
|
||||
Pinning `python_version:` is effectively required:
|
||||
|
||||
- ZeroGPU officially supports **3.10.13** and **3.12.12**.
|
||||
- The runtime default is 3.10.
|
||||
- Pinning to a `cp3XX` wheel matrix (e.g. `cp310` flash_attn wheel) forces matching Python.
|
||||
|
||||
Both `"3.12"` and `"3.12.12"` forms are accepted in YAML.
|
||||
@@ -0,0 +1,349 @@
|
||||
# ZeroGPU
|
||||
|
||||
Read this whenever the Space targets ZeroGPU (`zero-a10g` flavor). The SKILL.md's 3-rule summary is a starting point; this file covers the model in enough detail to debug and design.
|
||||
|
||||
For numerical limits (per-tier daily quota minutes, runs-per-day caps, current backing GPU, supported Python / torch versions): https://huggingface.co/docs/hub/spaces-zerogpu. Those values change over time and are deliberately kept out of this skill.
|
||||
|
||||
## The mental model
|
||||
|
||||
A ZeroGPU Space runs as **two processes**:
|
||||
|
||||
- **Main web process** — long-lived. Imports `app.py`, launches Gradio. Holds no VRAM and, after the startup "pack" step, no model weights in RAM either.
|
||||
- **GPU worker** — short-lived. Forked per `@spaces.GPU` request (or reused if warm). Eventually killed by the ZeroGPU scheduler when another Space needs the slot. Your code never kills its own worker.
|
||||
|
||||
`import spaces` monkey-patches `torch.cuda.*` in the main process so that `.to("cuda")` and `torch.cuda.is_available()` work at module scope **without** a real GPU attached. Module-level `model.to("cuda")` is intercepted: the tensor data physically stays in main-process RAM at this point, with a CUDA-presenting "fake" tensor registered alongside. At a startup "pack" step, the backend writes those original CPU tensors to disk via `O_DIRECT` and frees the RAM. After pack, main holds no weights anywhere.
|
||||
|
||||
When a `@spaces.GPU` call lands, the scheduler routes it to a worker:
|
||||
|
||||
- **Cold worker** — forked from the main process; torch is unpatched; real CUDA is initialized; weights are streamed disk → pinned host → VRAM via a double-buffered pipeline. This is the cold-start cost.
|
||||
- **Warm worker** — alive worker bound to the same slot; init is skipped; weights stay on VRAM from the previous call.
|
||||
|
||||
A warm worker eventually dies when another Space needs the slot. Occasional cold starts on a low-traffic Space are normal.
|
||||
|
||||
## The three rules
|
||||
|
||||
### 1. `import spaces` before any CUDA-touching import
|
||||
|
||||
```python
|
||||
import spaces # FIRST
|
||||
import torch # then this
|
||||
```
|
||||
|
||||
If something initializes CUDA before `import spaces`, the patch can't apply and you get `RuntimeError: CUDA has been initialized before importing the spaces package`. For libraries that eagerly init CUDA on import (e.g. `numba.cuda`, NeMo via numba), set the disable env *before* the import:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("NUMBA_DISABLE_CUDA", "1")
|
||||
import spaces
|
||||
```
|
||||
|
||||
### 2. Load models at module scope, `.to("cuda")` eagerly
|
||||
|
||||
```python
|
||||
pipe = DiffusionPipeline.from_pretrained("...", torch_dtype=torch.bfloat16).to("cuda")
|
||||
```
|
||||
|
||||
Do **not** lazy-load inside `@spaces.GPU`. The hijack is designed for module-level placement; deferring it puts tens of seconds of checkpoint I/O + dtype cast + GPU move inside every cold request.
|
||||
|
||||
Use the **string `"cuda"`** — never an integer device id. ZeroGPU re-allocates device ids per request, so `.to(0)`, `device_map={"": 0}`, `torch.cuda.set_device(0)` silently break.
|
||||
|
||||
For plain `from_pretrained` loads, use `.to("cuda")`, **not** `device_map="cuda"` (which routes through `accelerate.set_module_tensor_to_device` and calls `torch._C._cuda_init()` at load time, bypassing the hijack). The exception is loaders that are ZeroGPU-aware — notably the `bitsandbytes` quantization path; `from_pretrained(..., quantization_config=BitsAndBytesConfig(...))` works with `device_map="cuda"`.
|
||||
|
||||
**Preloading multiple variants** (e.g. base + refiner, image + video model) is fine as long as their combined VRAM fits. Load all of them sequentially at module scope into a dict, then key per request. Don't unload/reload between requests — that puts the load cost back on the user.
|
||||
|
||||
### 3. Decorate the function Gradio binds
|
||||
|
||||
ZeroGPU's startup scan walks Gradio's registered event handlers for `@spaces.GPU`-marked functions. If you decorate `inner_helper` but `click(fn=outer)` is what's wired up, you get `RuntimeError: No @spaces.GPU function detected during startup`. Always decorate the function passed to the event handler.
|
||||
|
||||
```python
|
||||
@spaces.GPU(duration=60)
|
||||
def generate(prompt):
|
||||
return pipe(prompt).images[0]
|
||||
|
||||
btn.click(fn=generate, inputs=prompt_box, outputs=image_out)
|
||||
```
|
||||
|
||||
## Sizing duration
|
||||
|
||||
`@spaces.GPU(duration=N)` means "reserve N seconds of GPU time." Two failure modes:
|
||||
|
||||
- **`ZeroGPU illegal duration`** — `N` exceeds the visitor's tier cap. Lowering `duration` is the only fix.
|
||||
- **`ZeroGPU quota exceeded`** — the visitor's remaining quota is less than `requested`. Compared as `requested vs remaining`, not `actual vs remaining` — so a 10-second task left at the default 60 s blocks the user as soon as their remaining drops below 60 s.
|
||||
|
||||
Smaller `duration` also ranks **higher** in the queue. Both reasons push toward declaring the realistic worst case, not a comfortable margin.
|
||||
|
||||
**Pick the value — don't guess.** A too-high duration deploys cleanly then errors on the first call; too-low silently truncates. Methodology:
|
||||
|
||||
1. Ship with a placeholder (e.g. 180 s).
|
||||
2. Instrument with `time.perf_counter()` and return the seconds in the response.
|
||||
3. Run 2–3 representative calls via `gradio_client`.
|
||||
4. Set `duration = round(measured_max × 1.4)`.
|
||||
|
||||
For input-dependent runtime, pass a **callable**:
|
||||
|
||||
```python
|
||||
def _estimate(prompt, steps, *args, **kwargs):
|
||||
# Swallow extras with *args, **kwargs — Gradio passes progress= positionally
|
||||
# and a strict signature will raise "takes 5 positional arguments but 6 were given"
|
||||
return min(240, 60 + int(steps * 3.5))
|
||||
|
||||
@spaces.GPU(duration=_estimate)
|
||||
def generate(prompt, steps, ..., progress=gr.Progress(track_tqdm=True)):
|
||||
...
|
||||
```
|
||||
|
||||
## Sizing memory: `large` vs `xlarge`
|
||||
|
||||
`size="large"` (default) is half the backing card (48 GB on Blackwell). `size="xlarge"` is the full card (96 GB) and costs **2× quota** per second — plus higher queue waits. Use `large` unless the workload genuinely OOMs.
|
||||
|
||||
Rough VRAM sizing:
|
||||
|
||||
| Mode | Memory rule | 7B | 27B | 70B |
|
||||
|------|------------|------|------|------|
|
||||
| bf16 | `params × 2` GB | 14 GB ✓ large | 54 GB → xlarge | 140 GB → quant + xlarge |
|
||||
| int8 | `params × 1` GB | 7 GB ✓ large | 27 GB ✓ large | 70 GB → xlarge |
|
||||
| 4-bit (NF4 / int4) | `params × ~0.55` GB | 4 GB ✓ large | 15 GB ✓ large | 40 GB ✓ large |
|
||||
|
||||
Numbers are for weights only; activations and KV cache add on top (significant for long context).
|
||||
|
||||
## Quantization
|
||||
|
||||
ZeroGPU supports two quantization stacks: **`bitsandbytes`** (drop-in for transformers, well-trodden) and **`torchao`** (torch-native, newer, smaller install). Pick by what your model's `from_pretrained` actually wires up; if both work, default to `bitsandbytes` for transformers LLMs and `torchao` for diffusers.
|
||||
|
||||
### bitsandbytes (NF4 / int8)
|
||||
|
||||
```python
|
||||
import spaces, torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
|
||||
|
||||
bnb = BitsAndBytesConfig(
|
||||
load_in_4bit=True,
|
||||
bnb_4bit_quant_type="nf4",
|
||||
bnb_4bit_use_double_quant=True,
|
||||
bnb_4bit_compute_dtype=torch.bfloat16,
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL_ID,
|
||||
quantization_config=bnb,
|
||||
device_map="cuda", # OK here — bnb's loader is ZeroGPU-aware
|
||||
dtype=torch.bfloat16,
|
||||
).eval()
|
||||
```
|
||||
|
||||
This is the one case where `device_map="cuda"` is **safe** on ZeroGPU at module scope (bitsandbytes' loader path intercepts cleanly). For non-bnb loads, stick to `.to("cuda")`.
|
||||
|
||||
`load_in_8bit=True` swaps the 4-bit block for int8 — same hijack-safe loader. Bigger but higher quality, no `compute_dtype` knob.
|
||||
|
||||
### torchao
|
||||
|
||||
```python
|
||||
import spaces, torch
|
||||
from diffusers import DiffusionPipeline
|
||||
from torchao.quantization import quantize_, Int8WeightOnlyConfig
|
||||
|
||||
pipe = DiffusionPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).to("cuda")
|
||||
quantize_(pipe.transformer, Int8WeightOnlyConfig()) # mutates in place
|
||||
```
|
||||
|
||||
`torchao` is more flexible (fine-grained per-module quantization, `Int4WeightOnlyConfig`, `Float8WeightOnlyConfig`, etc.) and works with diffusers' `from_pretrained(..., quantization_config=TorchAoConfig(...))` integration too. No CUDA build dependency — installs as a wheel.
|
||||
|
||||
### Attention backends
|
||||
|
||||
**Default to `attn_implementation="sdpa"`** — torch-native, works everywhere on sm_120, and is the right choice for the overwhelming majority of Spaces. Reach for a Flash-Attention backend only when the **upstream repo already references FA** (its config/model code defaults to or recommends `flash_attn`) — match what it expects rather than forcing SDPA. If FA then breaks on Blackwell, fall back to SDPA.
|
||||
|
||||
**Flash Attention 2** — when you do need it, use the prebuilt wheel at `multimodalart/zerogpu-blackwell-wheels` ([`requirements.md`](requirements.md)), cp310–cp313. Drop the wheel URL in `requirements.txt`; no monkey-patch, no runtime build. The wheel's real `flash_attn_2_cuda` also satisfies xformers' import-time probes.
|
||||
|
||||
**xformers** — same wheels dataset; auto-dispatch picks FA2 on sm_120 with no monkey-patch.
|
||||
|
||||
**FA2 is the ceiling on ZeroGPU Blackwell — FA3 and FA4 cannot run on sm_120.** The RTX PRO 6000 (sm_120) lacks the TMEM / `tcgen05` tensor-memory subsystem the FA3/FA4 kernels are built on; those kernels only exist for Hopper (sm_90a) and datacenter Blackwell (sm_100a). This is a **hardware** limit, not a packaging gap — no wheel or branch fixes it:
|
||||
|
||||
- `kernels-community/flash-attn3` (any revision, including `fake-ops-return-probs`), `vllm-flash-attn3`, `sgl-flash-attn3` all either fail to load (no matching build) or hard-fault at call time with `CUDA error: no kernel image is available for execution on the device` — which **kills the ZeroGPU worker** (surfaces as `GPU task aborted`); it does not fall back.
|
||||
- Verified empirically on the live runtime (`NVIDIA RTX PRO 6000 Blackwell … sm_120`, torch 2.11 / cu130) and confirmed upstream — every framework (vLLM, SGLang) falls back to FA2 on sm_120.
|
||||
|
||||
So on ZeroGPU the ladder is **SDPA (default) → FA2 (only if the repo uses FA)**. Don't wire in FA3/FA4 — a stray `config.json` `kernels` entry loading `flash-attn3` will abort the worker on first GPU call.
|
||||
|
||||
## Concurrency
|
||||
|
||||
Handlers run **concurrently by default**. Three rules:
|
||||
|
||||
1. **No mutable global state.** Handlers writing to a module-level dict / list race each other.
|
||||
2. **No fixed output paths.** Two concurrent calls writing to `output.png` clobber each other (and leak data across users). Use `tempfile.NamedTemporaryFile(suffix=...)`.
|
||||
3. **Read-only globals are safe** — models, tokenizers, configs loaded once and only read inside handlers.
|
||||
|
||||
## Process isolation and pickle
|
||||
|
||||
`@spaces.GPU` runs in a separate fork. Arguments and return values cross via pickle:
|
||||
|
||||
- **Only picklable objects** in/out. File handles, locks, lambdas, closures over unpicklable state → `PicklingError`.
|
||||
- **Never return CUDA tensors.** Unpickling in the main process triggers `torch.cuda._lazy_init()`, which ZeroGPU blocks → the call hangs. Convert to CPU first: `return tensor.cpu()` or `.cpu().numpy()`.
|
||||
- CPU tensors, numpy arrays, PIL Images, plain Python objects work fine.
|
||||
- `gr.SelectData` is a special case — its `__getattr__` recurses under pickle. Extract the fields you need (`evt.index[0]`, etc.) in a thin un-decorated wrapper, pass plain values to the `@spaces.GPU` function.
|
||||
|
||||
### `gr.State` across the fork
|
||||
|
||||
`gr.State` is pickled on every yield. The handler receives a **copy**:
|
||||
|
||||
- In-place mutations inside the fork are invisible to other handlers until you explicitly `yield` the mutated value back.
|
||||
- Yielding `gr.update()` for a state slot **skips** the update — other handlers continue to see pre-yield value.
|
||||
- For large state, minimize how often you yield it — ideally once at the end.
|
||||
- CUDA tensors inside state must be CPU-d before yielding (same `_lazy_init` issue).
|
||||
|
||||
## Generators and streaming
|
||||
|
||||
`@spaces.GPU` supports generator functions — first-class for progressive UI updates:
|
||||
|
||||
```python
|
||||
@spaces.GPU(duration=120)
|
||||
def generate(prompt):
|
||||
yield gr.update(value=None, label="Starting…")
|
||||
for step in range(num_steps):
|
||||
latent = step_fn(...)
|
||||
yield gr.update(value=preview(latent), label=f"Step {step+1}/{num_steps}")
|
||||
yield gr.update(value=final_image, label="Done")
|
||||
```
|
||||
|
||||
`gr.Progress(track_tqdm=True)` and `yield` compete with each other — pick one.
|
||||
|
||||
For streaming previews **inside** a diffusers `callback_on_step_end`, use a thread + queue inside the decorator (forks share threads):
|
||||
|
||||
```python
|
||||
@spaces.GPU(duration=180)
|
||||
def generate(prompt, num_steps):
|
||||
q = queue.Queue()
|
||||
DONE = object()
|
||||
def cb(pipe, step, t, kw):
|
||||
q.put((step, taef1_preview(kw["latents"])))
|
||||
return kw
|
||||
def run():
|
||||
out = pipeline(prompt=prompt, num_inference_steps=num_steps,
|
||||
callback_on_step_end=cb,
|
||||
callback_on_step_end_tensor_inputs=["latents"])
|
||||
q.put((DONE, out))
|
||||
threading.Thread(target=run, daemon=True).start()
|
||||
while True:
|
||||
idx, payload = q.get()
|
||||
if idx is DONE: break
|
||||
yield gr.update(value=payload, label=f"Step {idx+1}/{num_steps}")
|
||||
```
|
||||
|
||||
**Do not** use `ProcessPoolExecutor` / `multiprocessing.Pool` inside `@spaces.GPU` — the daemonic fork can't spawn children (`AssertionError: daemonic processes are not allowed to have children`). Threads only.
|
||||
|
||||
## Compilation (AoTI)
|
||||
|
||||
`torch.compile` (JIT) is **not supported** on ZeroGPU — the forked GPU worker can't host the compile daemon. The supported path is PyTorch **ahead-of-time inductor (AoTI)**, wrapped by the `spaces` package (torch 2.8+, `spaces` ≥ 0.50). Its real value: compile a graph **once, offline**, publish it to the Hub, and have the serving Space load it — so serving cold starts pay no compile cost.
|
||||
|
||||
### The `spaces` AoTI API
|
||||
|
||||
- `spaces.aoti_capture(module)` — context manager. Run one real forward pass inside it; it patches `module.forward` to record that call's `args`/`kwargs` and abort the run. Gives you real example inputs for export.
|
||||
- `spaces.aoti_compile(exported, inductor_configs=None)` → an in-memory `ZeroGPUCompiledModel`.
|
||||
- `spaces.aoti_compile_and_save(package_dir, exported, inductor_configs=None)` — compile and write `{package_dir}/root/package.pt2`.
|
||||
- `spaces.aoti_apply(compiled, module)` — swap `module.forward` for an in-memory compiled model, same process.
|
||||
- `spaces.aoti_blocks_load(module, repo_id, variant=None)` — the high-level loader. For a module exposing `_repeated_blocks` (most diffusers transformers), it downloads `{BlockClass}[.{variant}]/package.pt2` from `repo_id` and patches every matching block. Weights stay **runtime inputs**, so one compiled block-graph serves any checkpoint with the same block architecture (e.g. a base and its distilled/turbo variant share one graph).
|
||||
|
||||
### Recommended workflow: precompile the repeated block, load at runtime
|
||||
|
||||
Split into a **compile Space** (run by hand, occasionally) and the **serving Space**, connected by a Hub **model** repo (`aoti_blocks_load` downloads with the default `repo_type="model"`).
|
||||
|
||||
**Compile Space** — capture one repeated block's inputs, export, compile, save, upload:
|
||||
|
||||
```python
|
||||
import os, spaces, torch, tempfile, shutil
|
||||
from pathlib import Path
|
||||
from huggingface_hub import HfApi
|
||||
|
||||
pipe = DiffusionPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).to("cuda")
|
||||
block = pipe.transformer.transformer_blocks[0] # one representative repeated block
|
||||
BLOCK = type(block).__name__
|
||||
|
||||
@spaces.GPU(duration=1500)
|
||||
def compile_and_upload():
|
||||
with spaces.aoti_capture(block) as call: # capture the block's real inputs…
|
||||
pipe("a prompt", num_inference_steps=1) # …from the first block call of a 1-step run
|
||||
exported = torch.export.export(
|
||||
block, call.args, call.kwargs,
|
||||
dynamic_shapes=None, # start static; see note below
|
||||
strict=False,
|
||||
)
|
||||
tmp = Path(tempfile.mkdtemp())
|
||||
spaces.aoti_compile_and_save(tmp, exported) # -> tmp/root/package.pt2
|
||||
out = Path(tempfile.mkdtemp()); (out / BLOCK).mkdir()
|
||||
shutil.copy(tmp / "root" / "package.pt2", out / BLOCK / "package.pt2")
|
||||
HfApi(token=os.environ["HF_TOKEN"]).upload_folder(
|
||||
folder_path=str(out), repo_id=REPO, repo_type="model")
|
||||
```
|
||||
|
||||
The repo ends up as `{BlockClass}[.{variant}]/package.pt2`, optionally with a `config[.variant].json` listing custom `kernels` to fetch at load.
|
||||
|
||||
**Serving Space** — one line at module scope, with an eager fallback:
|
||||
|
||||
```python
|
||||
pipe = DiffusionPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).to("cuda")
|
||||
try:
|
||||
spaces.aoti_blocks_load(pipe.transformer, REPO) # variant="fp8da" if you saved a variant
|
||||
except Exception as e:
|
||||
print(f"AoTI load failed ({e!r}); running eager")
|
||||
```
|
||||
|
||||
Only the repeated block is compiled, so the artifact is small and the same graph patches all N layers; compilation (minutes) happens offline, never on a serving cold start.
|
||||
|
||||
### Footguns
|
||||
|
||||
- **Construct the block identically in both Spaces.** The compiled graph's constant FQNs must line up with the serving module. If the block chooses kernels/norms via env var or import probe (e.g. a triton fused RMSNorm vs the pure-torch path), pin the *same* choice in compile and serving.
|
||||
- **Dynamic shapes.** `dynamic_shapes=None` bakes in the one resolution you captured. To serve variable sequence length / image size, pass `torch.export.Dim(...)` for those axes (e.g. `{"hidden_states": {1: torch.export.Dim("seq")}}`) — this is per-model tuning.
|
||||
- **Data-dependent inputs don't export.** A block taking per-sample python int lists (some double-stream blocks) can make `torch.export` refuse to make them dynamic — those stay eager. Single-stream blocks taking only packed tensors export cleanly.
|
||||
- **Custom kernels.** Ship a `config[.variant].json` with a `kernels` list (`repo_id` + `revision`); `aoti_blocks_load` calls `kernels.get_kernel(...)` before loading so the op is registered. (FA3 still won't run on sm_120 — see Attention backends.)
|
||||
- **Simpler in-process variant.** For a one-off, skip the Hub round-trip: `spaces.aoti_apply(spaces.aoti_compile(exported), module)` compiles and applies in the same process — but you re-compile on every cold start.
|
||||
|
||||
### Reference Spaces
|
||||
|
||||
- `multimodalart/Boogu-Image-0.1-Edit-aoti-compile` (compile + upload) and `multimodalart/Boogu-Image` (serving via block loading) — the blocks pattern end to end.
|
||||
- `zerogpu-aoti/Qwen-Image`, `zerogpu-aoti/Wan2` — artifact repos showing the `{BlockClass}.{variant}/package.pt2` + `config.{variant}.json` layout.
|
||||
- Deeper background on inductor configs and dynamic shapes: https://huggingface.co/blog/zerogpu-aoti
|
||||
|
||||
## Local development
|
||||
|
||||
**Do NOT** wrap `import spaces` in `try/except` with a no-op fallback. Off-ZeroGPU, the `spaces` package is *already* a true no-op — the heavyweight behavior is gated on `SPACES_ZERO_GPU=1`, set only on ZeroGPU. `@spaces.GPU` returns the undecorated function unchanged elsewhere. The Gradio base image installs `spaces` on every hardware tier, so a duplicate onto T4 / A10G / CPU works without code changes too.
|
||||
|
||||
That said: **iterate ON the Space, not locally.** The Space environment (Python, torch, CUDA, drivers, env vars) differs from yours; passing local tests doesn't prove the Space works. Push early — even with the app not fully polished — and use the rung ladder ([`debugging.md`](debugging.md)) against the live URL.
|
||||
|
||||
## Allocator config for memory pressure
|
||||
|
||||
If your workload hits transient allocation spikes (high-res pixel-space ops, large attention activations, SR models, video DiTs) and you see:
|
||||
|
||||
```
|
||||
RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at .../CUDACachingAllocator.cpp
|
||||
```
|
||||
|
||||
set expandable segments at the **very top** of `app.py`, before any torch import:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
|
||||
import spaces
|
||||
import torch
|
||||
```
|
||||
|
||||
Often single-line fix for what looks like an OOM. See [`known-errors.md`](known-errors.md).
|
||||
|
||||
## Example caching
|
||||
|
||||
`gr.Examples` defaults on ZeroGPU:
|
||||
|
||||
- `cache_examples=True`
|
||||
- `cache_mode="lazy"` (eager would pre-run examples at startup, but no GPU is attached at startup)
|
||||
|
||||
Don't override to `cache_mode="eager"` on ZeroGPU — it will fail or burn the creator's daily quota. The cache is keyed by example **file path**, not content hash: regenerating an asset in place serves the stale cached output. Bump a `cache_version` constant if you replace example files.
|
||||
|
||||
## Real-time sessions
|
||||
|
||||
For real-time apps (webcam, audio streaming), the per-call fork model is too costly. ZeroGPU supports reusable "real-time sessions" — one GPU allocation amortized across many small requests. Reference Spaces:
|
||||
|
||||
- https://huggingface.co/spaces/diffusers/unofficial-SDXL-Turbo-i2i-t2i
|
||||
- https://huggingface.co/spaces/huggingface-projects/rf-detr-realtime-webcam
|
||||
|
||||
## When things go wrong
|
||||
|
||||
For specific error strings (CUDA init order, illegal duration, allocator asserts, PicklingError, returning CUDA tensors, …): [`known-errors.md`](known-errors.md). It covers the ZeroGPU-specific patterns alongside everything else and is the single error lookup for the skill.
|
||||
|
||||
When the log endpoint can't explain a failure (device-side asserts, OOM under specific shapes, race conditions in pickle), dev mode + SSH is the last-resort tool — see [`debugging.md`](debugging.md).
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
name: huggingface-tool-builder
|
||||
description: Use this skill when the user wants to build tool/scripts or achieve a task where using data from the Hugging Face API would help. This is especially useful when chaining or combining API calls or the task will be repeated/automated. This Skill creates a reusable script to fetch, enrich or process data.
|
||||
license: Apache-2.0 (modified; see UPSTREAMS.json)
|
||||
---
|
||||
|
||||
# Hugging Face API Tool Builder
|
||||
|
||||
Your purpose is now is to create reusable command line scripts and utilities for using the Hugging Face API, allowing chaining, piping and intermediate processing where helpful. You can access the API directly, as well as use the `hf` command line tool. Model and Dataset cards can be accessed from repositories directly.
|
||||
|
||||
## Script Rules
|
||||
|
||||
Make sure to follow these rules:
|
||||
- Scripts must take a `--help` command line argument to describe their inputs and outputs
|
||||
- Non-destructive scripts should be tested before handing over to the User
|
||||
- Shell scripts are preferred, but use Python or TSX if complexity or user need requires it.
|
||||
- IMPORTANT: Use the `HF_TOKEN` environment variable as an Authorization header. For example: `curl -H "Authorization: Bearer ${HF_TOKEN}" https://huggingface.co/api/`. This provides higher rate limits and appropriate authorization for data access.
|
||||
- Investigate the shape of the API results before commiting to a final design; make use of piping and chaining where composability would be an advantage - prefer simple solutions where possible.
|
||||
- Share usage examples once complete.
|
||||
|
||||
Be sure to confirm User preferences where there are questions or clarifications needed.
|
||||
|
||||
## Sample Scripts
|
||||
|
||||
Paths below are relative to this skill directory.
|
||||
|
||||
Reference examples:
|
||||
- `references/hf_model_papers_auth.sh` — uses `HF_TOKEN` automatically and chains trending → model metadata → model card parsing with fallbacks; it demonstrates multi-step API usage plus auth hygiene for gated/private content.
|
||||
- `references/find_models_by_paper.sh` — optional `HF_TOKEN` usage via `--token`, consistent authenticated search, and a retry path when arXiv-prefixed searches are too narrow; it shows resilient query strategy and clear user-facing help.
|
||||
- `references/hf_model_card_frontmatter.sh` — uses the `hf` CLI to download model cards, extracts YAML frontmatter, and emits NDJSON summaries (license, pipeline tag, tags, gated prompt flag) for easy filtering.
|
||||
|
||||
Baseline examples (ultra-simple, minimal logic, raw JSON output with `HF_TOKEN` header):
|
||||
- `references/baseline_hf_api.sh` — bash
|
||||
- `references/baseline_hf_api.py` — python
|
||||
- `references/baseline_hf_api.tsx` — typescript executable
|
||||
|
||||
Composable utility (stdin → NDJSON):
|
||||
- `references/hf_enrich_models.sh` — reads model IDs from stdin, fetches metadata per ID, emits one JSON object per line for streaming pipelines.
|
||||
|
||||
Composability through piping (shell-friendly JSON output):
|
||||
- `references/baseline_hf_api.sh 25 | jq -r '.[].id' | references/hf_enrich_models.sh | jq -s 'sort_by(.downloads) | reverse | .[:10]'`
|
||||
- `references/baseline_hf_api.sh 50 | jq '[.[] | {id, downloads}] | sort_by(.downloads) | reverse | .[:10]'`
|
||||
- `printf '%s\n' openai/gpt-oss-120b meta-llama/Meta-Llama-3.1-8B | references/hf_model_card_frontmatter.sh | jq -s 'map({id, license, has_extra_gated_prompt})'`
|
||||
|
||||
## High Level Endpoints
|
||||
|
||||
The following are the main API endpoints available at `https://huggingface.co`
|
||||
|
||||
```
|
||||
/api/datasets
|
||||
/api/models
|
||||
/api/spaces
|
||||
/api/collections
|
||||
/api/daily_papers
|
||||
/api/notifications
|
||||
/api/settings
|
||||
/api/whoami-v2
|
||||
/api/trending
|
||||
/oauth/userinfo
|
||||
```
|
||||
|
||||
## Accessing the API
|
||||
|
||||
The API is documented with the OpenAPI standard at `https://huggingface.co/.well-known/openapi.json`.
|
||||
|
||||
**IMPORTANT:** DO NOT ATTEMPT to read `https://huggingface.co/.well-known/openapi.json` directly as it is too large to process.
|
||||
|
||||
**IMPORTANT** Use `jq` to query and extract relevant parts. For example,
|
||||
|
||||
Command to Get All 160 Endpoints
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/.well-known/openapi.json" | jq '.paths | keys | sort'
|
||||
```
|
||||
|
||||
Model Search Endpoint Details
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/.well-known/openapi.json" | jq '.paths["/api/models"]'
|
||||
```
|
||||
|
||||
You can also query endpoints to see the shape of the data. When doing so constrain results to low numbers to make them easy to process, yet representative.
|
||||
|
||||
## Using the HF command line tool
|
||||
|
||||
The `hf` command line tool gives you further access to Hugging Face repository content and infrastructure.
|
||||
|
||||
```bash
|
||||
❯ hf --help
|
||||
Usage: hf [OPTIONS] COMMAND [ARGS]...
|
||||
|
||||
Hugging Face Hub CLI
|
||||
|
||||
Options:
|
||||
--help Show this message and exit.
|
||||
|
||||
Commands:
|
||||
auth Manage authentication (login, logout, etc.).
|
||||
buckets Commands to interact with buckets.
|
||||
cache Manage local cache directory.
|
||||
collections Interact with collections on the Hub.
|
||||
datasets Interact with datasets on the Hub.
|
||||
discussions Manage discussions and pull requests on the Hub.
|
||||
download Download files from the Hub.
|
||||
endpoints Manage Hugging Face Inference Endpoints.
|
||||
env Print information about the environment.
|
||||
extensions Manage hf CLI extensions.
|
||||
jobs Run and manage Jobs on the Hub.
|
||||
models Interact with models on the Hub.
|
||||
papers Interact with papers on the Hub.
|
||||
repos Manage repos on the Hub.
|
||||
skills Manage skills for AI assistants.
|
||||
spaces Interact with spaces on the Hub.
|
||||
sync Sync files between local directory and a bucket.
|
||||
upload Upload a file or a folder to the Hub.
|
||||
upload-large-folder Upload a large folder to the Hub.
|
||||
version Print information about the hf version.
|
||||
webhooks Manage webhooks on the Hub.
|
||||
```
|
||||
|
||||
The `hf` CLI command has replaced the now deprecated `huggingface-cli` command.
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Ultra-simple Hugging Face API example (Python).
|
||||
|
||||
Fetches a small list of models from the HF API and prints raw JSON.
|
||||
Uses HF_TOKEN for auth if the environment variable is set.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
|
||||
def show_help() -> None:
|
||||
print(
|
||||
"""Ultra-simple Hugging Face API example (Python)
|
||||
|
||||
Usage:
|
||||
baseline_hf_api.py [limit]
|
||||
baseline_hf_api.py --help
|
||||
|
||||
Description:
|
||||
Fetches a small list of models from the HF API and prints raw JSON.
|
||||
Uses HF_TOKEN for auth if the environment variable is set.
|
||||
|
||||
Examples:
|
||||
baseline_hf_api.py
|
||||
baseline_hf_api.py 5
|
||||
HF_TOKEN=your_token baseline_hf_api.py 10
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "--help":
|
||||
show_help()
|
||||
return 0
|
||||
|
||||
limit = sys.argv[1] if len(sys.argv) > 1 else "3"
|
||||
if not limit.isdigit():
|
||||
print("Error: limit must be a number", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
token = os.getenv("HF_TOKEN")
|
||||
headers = {"Authorization": f"Bearer {token}"} if token else {}
|
||||
url = f"https://huggingface.co/api/models?limit={limit}"
|
||||
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
sys.stdout.write(resp.read().decode("utf-8"))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
show_help() {
|
||||
cat << EOF
|
||||
Ultra-simple Hugging Face API example (Shell)
|
||||
|
||||
Usage:
|
||||
$0 [limit]
|
||||
$0 --help
|
||||
|
||||
Description:
|
||||
Fetches a small list of models from the HF API and prints raw JSON.
|
||||
Uses HF_TOKEN for auth if the environment variable is set.
|
||||
|
||||
Examples:
|
||||
$0
|
||||
$0 5
|
||||
HF_TOKEN=your_token $0 10
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" ]]; then
|
||||
show_help
|
||||
exit 0
|
||||
fi
|
||||
|
||||
LIMIT="${1:-3}"
|
||||
if ! [[ "$LIMIT" =~ ^[0-9]+$ ]]; then
|
||||
echo "Error: limit must be a number" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
headers=()
|
||||
if [[ -n "${HF_TOKEN:-}" ]]; then
|
||||
headers=(-H "Authorization: Bearer ${HF_TOKEN}")
|
||||
fi
|
||||
|
||||
curl -s "${headers[@]}" "https://huggingface.co/api/models?limit=${LIMIT}"
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env tsx
|
||||
|
||||
/**
|
||||
* Ultra-simple Hugging Face API example (TSX).
|
||||
*
|
||||
* Fetches a small list of models from the HF API and prints raw JSON.
|
||||
* Uses HF_TOKEN for auth if the environment variable is set.
|
||||
*/
|
||||
|
||||
const showHelp = () => {
|
||||
console.log(`Ultra-simple Hugging Face API example (TSX)
|
||||
|
||||
Usage:
|
||||
baseline_hf_api.tsx [limit]
|
||||
baseline_hf_api.tsx --help
|
||||
|
||||
Description:
|
||||
Fetches a small list of models from the HF API and prints raw JSON.
|
||||
Uses HF_TOKEN for auth if the environment variable is set.
|
||||
|
||||
Examples:
|
||||
baseline_hf_api.tsx
|
||||
baseline_hf_api.tsx 5
|
||||
HF_TOKEN=your_token baseline_hf_api.tsx 10
|
||||
`);
|
||||
};
|
||||
|
||||
const arg = process.argv[2];
|
||||
if (arg === "--help") {
|
||||
showHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const limit = arg ?? "3";
|
||||
if (!/^\d+$/.test(limit)) {
|
||||
console.error("Error: limit must be a number");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const token = process.env.HF_TOKEN;
|
||||
const headers: Record<string, string> = token
|
||||
? { Authorization: `Bearer ${token}` }
|
||||
: {};
|
||||
|
||||
const url = `https://huggingface.co/api/models?limit=${limit}`;
|
||||
|
||||
(async () => {
|
||||
const res = await fetch(url, { headers });
|
||||
|
||||
if (!res.ok) {
|
||||
console.error(`Error: ${res.status} ${res.statusText}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const text = await res.text();
|
||||
process.stdout.write(text);
|
||||
})();
|
||||
@@ -0,0 +1,230 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Find models associated with papers on Hugging Face
|
||||
# Usage: ./find_models_by_paper.sh [arXiv_id|search_term]
|
||||
# Optional: Set HF_TOKEN environment variable for private/gated models
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Help function
|
||||
show_help() {
|
||||
echo -e "${BLUE}Find models associated with papers on Hugging Face${NC}"
|
||||
echo ""
|
||||
echo -e "${YELLOW}Usage:${NC}"
|
||||
echo " $0 [OPTIONS] [search_term|arXiv_id]"
|
||||
echo ""
|
||||
echo -e "${YELLOW}Options:${NC}"
|
||||
echo " --help Show this help message"
|
||||
echo " --token Use HF_TOKEN environment variable (if set)"
|
||||
echo ""
|
||||
echo -e "${YELLOW}Environment:${NC}"
|
||||
echo " HF_TOKEN Optional: Hugging Face token for private/gated models"
|
||||
echo ""
|
||||
echo -e "${YELLOW}Examples:${NC}"
|
||||
echo " $0 1910.01108 # Search by arXiv ID"
|
||||
echo " $0 distilbert # Search by model name"
|
||||
echo " $0 transformer # Search by keyword"
|
||||
echo " HF_TOKEN=your_token $0 1910.01108 # Use authentication"
|
||||
echo ""
|
||||
echo -e "${YELLOW}Description:${NC}"
|
||||
echo "This script finds Hugging Face models that are associated with research papers."
|
||||
echo "It searches for models that have arXiv IDs in their tags or mentions papers in their metadata."
|
||||
echo ""
|
||||
echo -e "${YELLOW}Notes:${NC}"
|
||||
echo "• HF_TOKEN is optional for public models"
|
||||
echo "• Use HF_TOKEN for private repositories or gated models"
|
||||
echo "• HF_TOKEN enables higher rate limits for heavy usage"
|
||||
}
|
||||
|
||||
# Parse arguments
|
||||
USE_TOKEN=false
|
||||
POSITIONAL_ARGS=()
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--help)
|
||||
show_help
|
||||
exit 0
|
||||
;;
|
||||
--token)
|
||||
USE_TOKEN=true
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
show_help
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
POSITIONAL_ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
set -- "${POSITIONAL_ARGS[@]}"
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
echo -e "${RED}Error: Please provide a search term or arXiv ID${NC}"
|
||||
echo -e "Use ${YELLOW}$0 --help${NC} for usage information"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SEARCH_TERM="$1"
|
||||
|
||||
# Set up authentication header if HF_TOKEN is available
|
||||
if [[ -n "$HF_TOKEN" ]] && [[ "$USE_TOKEN" == true || -n "$HF_TOKEN" ]]; then
|
||||
AUTH_HEADER="-H \"Authorization: Bearer $HF_TOKEN\""
|
||||
echo -e "${BLUE}Using HF_TOKEN for authentication${NC}"
|
||||
else
|
||||
AUTH_HEADER=""
|
||||
if [[ -n "$HF_TOKEN" ]]; then
|
||||
echo -e "${YELLOW}HF_TOKEN found but not using it (add --token flag to use)${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check if the input looks like an arXiv ID (format: YYYY.NNNNN or YYYY.NNNNNNN)
|
||||
if [[ "$SEARCH_TERM" =~ ^[0-9]{4}\.[0-9]{4,7}$ ]]; then
|
||||
echo -e "${BLUE}Searching for models associated with arXiv paper: $SEARCH_TERM${NC}"
|
||||
SEARCH_QUERY="arxiv%3A$SEARCH_TERM"
|
||||
IS_ARXIV_SEARCH=true
|
||||
else
|
||||
echo -e "${BLUE}Searching for models related to: $SEARCH_TERM${NC}"
|
||||
SEARCH_QUERY="$SEARCH_TERM"
|
||||
IS_ARXIV_SEARCH=false
|
||||
fi
|
||||
|
||||
# Function to extract arXiv IDs from tags
|
||||
extract_arxiv_ids() {
|
||||
local tags="$1"
|
||||
echo "$tags" | jq -r '.[] | select(. | startswith("arxiv:")) | split(":")[1]' 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Function to get paper title from arXiv ID
|
||||
get_paper_title() {
|
||||
local arxiv_id="$1"
|
||||
# Try to get paper title from Hugging Face tags if available
|
||||
# This is a simplified approach - in practice, you might want to call arXiv API
|
||||
echo "Paper Title (arXiv:$arxiv_id)"
|
||||
}
|
||||
|
||||
# Search for models
|
||||
API_URL="https://huggingface.co/api/models"
|
||||
echo -e "${YELLOW}Searching Hugging Face API...${NC}"
|
||||
|
||||
# Build curl command with authentication if available
|
||||
CURL_CMD="curl -s $AUTH_HEADER \"$API_URL?search=$SEARCH_QUERY&limit=50\""
|
||||
echo -e "${BLUE}API Query: $API_URL?search=$SEARCH_QUERY&limit=50${NC}"
|
||||
|
||||
# Execute the API call
|
||||
if [[ -n "$HF_TOKEN" ]]; then
|
||||
RESPONSE=$(curl -s -H "Authorization: Bearer $HF_TOKEN" "$API_URL?search=$SEARCH_QUERY&limit=50" || true)
|
||||
else
|
||||
RESPONSE=$(curl -s "$API_URL?search=$SEARCH_QUERY&limit=50" || true)
|
||||
fi
|
||||
|
||||
# Check if we got a valid response
|
||||
if [[ -z "$RESPONSE" ]] || [[ "$RESPONSE" == "[]" ]]; then
|
||||
echo -e "${RED}No models found for search term: $SEARCH_TERM${NC}"
|
||||
|
||||
# If arXiv search failed, try without arxiv: prefix
|
||||
if [[ "$IS_ARXIV_SEARCH" == true ]]; then
|
||||
echo -e "${YELLOW}Trying broader search without arxiv: prefix...${NC}"
|
||||
SEARCH_QUERY="$SEARCH_TERM"
|
||||
IS_ARXIV_SEARCH=false
|
||||
|
||||
if [[ -n "$HF_TOKEN" ]]; then
|
||||
RESPONSE=$(curl -s -H "Authorization: Bearer $HF_TOKEN" "$API_URL?search=$SEARCH_QUERY&limit=50" || true)
|
||||
else
|
||||
RESPONSE=$(curl -s "$API_URL?search=$SEARCH_QUERY&limit=50" || true)
|
||||
fi
|
||||
|
||||
if [[ -z "$RESPONSE" ]] || [[ "$RESPONSE" == "[]" ]]; then
|
||||
echo -e "${RED}Still no results found. Try a different search term.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Process the results
|
||||
echo -e "${GREEN}Found models! Processing results...${NC}"
|
||||
|
||||
# Use jq to process the JSON response and find models with paper associations
|
||||
MODELS_WITH_PAPERS=$(echo "$RESPONSE" | jq -r '
|
||||
.[] |
|
||||
select(.id != null) |
|
||||
{
|
||||
id: .id,
|
||||
arxiv_tags: [.tags[] | select(. | startswith("arxiv:"))] | join("; "),
|
||||
downloads: (.downloads // 0),
|
||||
likes: (.likes // 0),
|
||||
task: (.pipeline_tag // "unknown"),
|
||||
library: (.library_name // "unknown")
|
||||
}
|
||||
| @base64' 2>/dev/null || true)
|
||||
|
||||
# Count total results
|
||||
TOTAL_MODELS=$(echo "$RESPONSE" | jq 'length' 2>/dev/null || echo "0")
|
||||
MODELS_WITH_PAPERS_COUNT=$(echo "$MODELS_WITH_PAPERS" | wc -l)
|
||||
|
||||
echo -e "${BLUE}Results Summary:${NC}"
|
||||
echo -e " Total models found: $TOTAL_MODELS"
|
||||
echo -e " Models with paper associations: $MODELS_WITH_PAPERS_COUNT"
|
||||
echo ""
|
||||
|
||||
if [[ -z "$MODELS_WITH_PAPERS" ]]; then
|
||||
# Show all models even if no paper associations found
|
||||
echo -e "${YELLOW}No explicit paper associations found. Showing all matching models:${NC}"
|
||||
echo "$RESPONSE" | jq -r '
|
||||
.[] |
|
||||
select(.id != null) |
|
||||
"📦 \(.id)
|
||||
Task: \(.pipeline_tag // "unknown")
|
||||
Downloads: \(.downloads // 0)
|
||||
Likes: \(.likes // 0)
|
||||
Library: \(.library_name // "unknown")
|
||||
---"
|
||||
' 2>/dev/null || echo "Failed to parse response"
|
||||
else
|
||||
# Show models with paper associations
|
||||
echo -e "${GREEN}Models with paper associations:${NC}"
|
||||
echo "$MODELS_WITH_PAPERS" | while read -r model_data; do
|
||||
if [[ -n "$model_data" ]]; then
|
||||
# Decode base64 and show formatted
|
||||
echo "$model_data" | base64 -d | jq -r '
|
||||
"📄 \(.id)
|
||||
arXiv: \(.arxiv_tags)
|
||||
Task: \(.task)
|
||||
Downloads: \(.downloads)
|
||||
Likes: \(.likes)
|
||||
Library: \(.library)
|
||||
---"
|
||||
' 2>/dev/null || echo "Failed to parse model data"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# Additional search tips
|
||||
echo ""
|
||||
echo -e "${BLUE}Search Tips:${NC}"
|
||||
echo "• Try searching with the full arXiv ID (e.g., 1910.01108)"
|
||||
echo "• Try searching with the paper title keywords"
|
||||
echo "• Try searching with the model name"
|
||||
echo "• Use HF_TOKEN for private models or higher rate limits"
|
||||
echo ""
|
||||
echo -e "${BLUE}Examples to try:${NC}"
|
||||
echo " $0 1910.01108 # DistilBERT paper"
|
||||
echo " $0 1810.04805 # BERT paper"
|
||||
echo " $0 1706.03762 # Attention is All You Need paper"
|
||||
echo " $0 roberta # RoBERTa models"
|
||||
echo " $0 transformer # Transformer models"
|
||||
echo " HF_TOKEN=your_token $0 1910.01108 # Use authentication"
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
show_help() {
|
||||
cat << 'USAGE'
|
||||
Stream model IDs on stdin, emit one JSON object per line (NDJSON).
|
||||
|
||||
Usage:
|
||||
hf_enrich_models.sh [MODEL_ID ...]
|
||||
cat ids.txt | hf_enrich_models.sh
|
||||
baseline_hf_api.sh 50 | jq -r '.[].id' | hf_enrich_models.sh
|
||||
|
||||
Description:
|
||||
Reads newline-separated model IDs and fetches basic metadata for each.
|
||||
Outputs NDJSON with id, downloads, likes, pipeline_tag, tags.
|
||||
Uses HF_TOKEN for auth if the environment variable is set.
|
||||
|
||||
Examples:
|
||||
hf_enrich_models.sh gpt2 distilbert-base-uncased
|
||||
baseline_hf_api.sh 50 | jq -r '.[].id' | hf_enrich_models.sh | jq -s 'sort_by(.downloads)'
|
||||
HF_TOKEN=your_token hf_enrich_models.sh microsoft/DialoGPT-medium
|
||||
USAGE
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" ]]; then
|
||||
show_help
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
echo "Error: jq is required but not installed" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
headers=()
|
||||
if [[ -n "${HF_TOKEN:-}" ]]; then
|
||||
headers=(-H "Authorization: Bearer ${HF_TOKEN}")
|
||||
fi
|
||||
|
||||
emit_error() {
|
||||
local model_id="$1"
|
||||
local message="$2"
|
||||
jq -cn --arg id "$model_id" --arg error "$message" '{id: $id, error: $error}'
|
||||
}
|
||||
|
||||
process_id() {
|
||||
local model_id="$1"
|
||||
|
||||
if [[ -z "$model_id" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
local url="https://huggingface.co/api/models/${model_id}"
|
||||
local response
|
||||
response=$(curl -s "${headers[@]}" "$url" 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$response" ]]; then
|
||||
emit_error "$model_id" "request_failed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if ! jq -e . >/dev/null 2>&1 <<<"$response"; then
|
||||
emit_error "$model_id" "invalid_json"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if jq -e '.error' >/dev/null 2>&1 <<<"$response"; then
|
||||
emit_error "$model_id" "not_found"
|
||||
return 0
|
||||
fi
|
||||
|
||||
jq -c --arg id "$model_id" '{
|
||||
id: (.id // $id),
|
||||
downloads: (.downloads // 0),
|
||||
likes: (.likes // 0),
|
||||
pipeline_tag: (.pipeline_tag // "unknown"),
|
||||
tags: (.tags // [])
|
||||
}' <<<"$response" 2>/dev/null || emit_error "$model_id" "parse_failed"
|
||||
}
|
||||
|
||||
if [[ $# -gt 0 ]]; then
|
||||
for model_id in "$@"; do
|
||||
process_id "$model_id"
|
||||
done
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ -t 0 ]]; then
|
||||
show_help
|
||||
exit 1
|
||||
fi
|
||||
|
||||
while IFS= read -r model_id; do
|
||||
process_id "$model_id"
|
||||
done
|
||||
@@ -0,0 +1,188 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
show_help() {
|
||||
cat << 'USAGE'
|
||||
Fetch Hugging Face model cards via the hf CLI and summarize frontmatter.
|
||||
|
||||
Usage:
|
||||
hf_model_card_frontmatter.sh [MODEL_ID ...]
|
||||
cat ids.txt | hf_model_card_frontmatter.sh
|
||||
|
||||
Description:
|
||||
Downloads README.md for each model via `hf download`, extracts YAML
|
||||
frontmatter, and emits one JSON object per line (NDJSON) with key fields.
|
||||
Uses HF_TOKEN if set (passed to the hf CLI).
|
||||
|
||||
Output fields:
|
||||
id, license, pipeline_tag, library_name, tags, language,
|
||||
new_version, has_extra_gated_prompt
|
||||
|
||||
Examples:
|
||||
hf_model_card_frontmatter.sh openai/gpt-oss-120b
|
||||
cat ids.txt | hf_model_card_frontmatter.sh | jq -s '.'
|
||||
hf_model_card_frontmatter.sh meta-llama/Meta-Llama-3-8B \
|
||||
| jq -s 'map({id, license, has_extra_gated_prompt})'
|
||||
USAGE
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" ]]; then
|
||||
show_help
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v hf >/dev/null 2>&1; then
|
||||
echo "Error: hf CLI is required but not installed" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "Error: python3 is required but not installed" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
token_args=()
|
||||
if [[ -n "${HF_TOKEN:-}" ]]; then
|
||||
token_args=(--token "$HF_TOKEN")
|
||||
fi
|
||||
|
||||
tmp_dir=$(mktemp -d)
|
||||
cleanup() {
|
||||
rm -rf "$tmp_dir"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
emit_error() {
|
||||
local model_id="$1"
|
||||
local message="$2"
|
||||
python3 - << 'PY' "$model_id" "$message"
|
||||
import json
|
||||
import sys
|
||||
|
||||
model_id = sys.argv[1]
|
||||
message = sys.argv[2]
|
||||
print(json.dumps({"id": model_id, "error": message}))
|
||||
PY
|
||||
}
|
||||
|
||||
parse_readme() {
|
||||
local model_id="$1"
|
||||
local readme_path="$2"
|
||||
|
||||
MODEL_ID="$model_id" README_PATH="$readme_path" python3 - << 'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
model_id = os.environ.get("MODEL_ID", "")
|
||||
readme_path = os.environ.get("README_PATH", "")
|
||||
|
||||
try:
|
||||
with open(readme_path, "r", encoding="utf-8") as f:
|
||||
lines = f.read().splitlines()
|
||||
except OSError:
|
||||
print(json.dumps({"id": model_id, "error": "readme_missing"}))
|
||||
sys.exit(0)
|
||||
|
||||
frontmatter = []
|
||||
in_block = False
|
||||
for line in lines:
|
||||
if line.strip() == "---":
|
||||
if in_block:
|
||||
break
|
||||
in_block = True
|
||||
continue
|
||||
if in_block:
|
||||
frontmatter.append(line)
|
||||
|
||||
if not frontmatter:
|
||||
print(json.dumps({"id": model_id, "error": "frontmatter_missing"}))
|
||||
sys.exit(0)
|
||||
|
||||
key = None
|
||||
out = {}
|
||||
|
||||
for line in frontmatter:
|
||||
stripped = line.strip()
|
||||
if not stripped or line.lstrip().startswith("#"):
|
||||
continue
|
||||
|
||||
if ":" in line and not line.lstrip().startswith("- "):
|
||||
key_candidate, value = line.split(":", 1)
|
||||
key_candidate = key_candidate.strip()
|
||||
value = value.strip()
|
||||
if key_candidate and all(c.isalnum() or c in "_-" for c in key_candidate):
|
||||
key = key_candidate
|
||||
if value in ("|", "|-", ">", ">-") or value == "":
|
||||
out[key] = None
|
||||
continue
|
||||
if value.startswith("[") and value.endswith("]"):
|
||||
items = [v.strip() for v in value.strip("[]").split(",") if v.strip()]
|
||||
out[key] = items
|
||||
else:
|
||||
out[key] = value
|
||||
continue
|
||||
|
||||
if line.lstrip().startswith("- ") and key:
|
||||
item = line.strip()[2:]
|
||||
if key not in out or out[key] is None:
|
||||
out[key] = []
|
||||
if isinstance(out[key], list):
|
||||
out[key].append(item)
|
||||
|
||||
result = {
|
||||
"id": model_id,
|
||||
"license": out.get("license"),
|
||||
"pipeline_tag": out.get("pipeline_tag"),
|
||||
"library_name": out.get("library_name"),
|
||||
"tags": out.get("tags", []),
|
||||
"language": out.get("language", []),
|
||||
"new_version": out.get("new_version"),
|
||||
"has_extra_gated_prompt": "extra_gated_prompt" in out,
|
||||
}
|
||||
|
||||
print(json.dumps(result))
|
||||
PY
|
||||
}
|
||||
|
||||
process_id() {
|
||||
local model_id="$1"
|
||||
|
||||
if [[ -z "$model_id" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
local safe_id
|
||||
safe_id=$(printf '%s' "$model_id" | tr '/' '_')
|
||||
local local_dir="$tmp_dir/$safe_id"
|
||||
|
||||
if ! hf download "$model_id" README.md --repo-type model --local-dir "$local_dir" "${token_args[@]}" >/dev/null 2>&1; then
|
||||
emit_error "$model_id" "download_failed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local readme_path="$local_dir/README.md"
|
||||
if [[ ! -f "$readme_path" ]]; then
|
||||
emit_error "$model_id" "readme_missing"
|
||||
return 0
|
||||
fi
|
||||
|
||||
parse_readme "$model_id" "$readme_path"
|
||||
}
|
||||
|
||||
if [[ $# -gt 0 ]]; then
|
||||
for model_id in "$@"; do
|
||||
process_id "$model_id"
|
||||
done
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ -t 0 ]]; then
|
||||
show_help
|
||||
exit 1
|
||||
fi
|
||||
|
||||
while IFS= read -r model_id; do
|
||||
process_id "$model_id"
|
||||
done
|
||||
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Hugging Face Model Papers Tool with Authentication
|
||||
# Fetches papers referenced by Hugging Face models using HF_TOKEN if available
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Help function
|
||||
show_help() {
|
||||
cat << EOF
|
||||
Hugging Face Model Papers Tool with Authentication
|
||||
|
||||
This tool fetches papers referenced by Hugging Face models.
|
||||
Supports authentication via HF_TOKEN environment variable.
|
||||
|
||||
Usage:
|
||||
$0 [OPTIONS]
|
||||
|
||||
Options:
|
||||
MODEL_ID Specific model to analyze (e.g., microsoft/DialoGPT-medium)
|
||||
--trending [N] Show papers for top N trending models (default: 5)
|
||||
--help Show this help message
|
||||
|
||||
Environment Variables:
|
||||
HF_TOKEN Hugging Face API token (optional, for private models)
|
||||
|
||||
Examples:
|
||||
# Get papers for a specific model
|
||||
$0 microsoft/DialoGPT-medium
|
||||
|
||||
# Get papers with authentication
|
||||
HF_TOKEN=your_token_here $0 your-private-model
|
||||
|
||||
# Get papers for top 3 trending models
|
||||
$0 --trending 3
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# Function to make authenticated API calls
|
||||
hf_api_call() {
|
||||
local url="$1"
|
||||
local headers=()
|
||||
|
||||
# Add authentication header if HF_TOKEN is set
|
||||
if [[ -n "${HF_TOKEN:-}" ]]; then
|
||||
headers+=(-H "Authorization: Bearer $HF_TOKEN")
|
||||
fi
|
||||
|
||||
curl -s "${headers[@]}" "$url" 2>/dev/null || echo '{"error": "Network error"}'
|
||||
}
|
||||
|
||||
# Function to extract papers from text
|
||||
extract_papers() {
|
||||
local text="$1"
|
||||
local title="$2"
|
||||
|
||||
echo "$title"
|
||||
|
||||
# Find ArXiv URLs
|
||||
local arxiv_urls=$(echo "$text" | grep -oE 'https?://arxiv\.org/[^[:space:]\])]+' | head -5)
|
||||
if [[ -n "$arxiv_urls" ]]; then
|
||||
echo "ArXiv Papers:"
|
||||
echo "$arxiv_urls" | sed 's/^/ • /'
|
||||
fi
|
||||
|
||||
# Find DOI URLs
|
||||
local doi_urls=$(echo "$text" | grep -oE 'https?://doi\.org/[^[:space:]\])]+' | head -3)
|
||||
if [[ -n "$doi_urls" ]]; then
|
||||
echo "DOI Papers:"
|
||||
echo "$doi_urls" | sed 's/^/ • /'
|
||||
fi
|
||||
|
||||
# Find arxiv IDs in format YYYY.NNNNN
|
||||
local arxiv_ids=$(echo "$text" | grep -oE 'arXiv:[0-9]{4}\.[0-9]{4,5}' | head -5)
|
||||
if [[ -n "$arxiv_ids" ]]; then
|
||||
echo "ArXiv IDs:"
|
||||
echo "$arxiv_ids" | sed 's/^/ • /'
|
||||
fi
|
||||
|
||||
# Check for paper mentions
|
||||
if echo "$text" | grep -qi "paper\|publication\|citation"; then
|
||||
local paper_mentions=$(echo "$text" | grep -i -A1 -B1 "paper\|publication" | head -6)
|
||||
if [[ -n "$paper_mentions" ]]; then
|
||||
echo "Paper mentions:"
|
||||
echo "$paper_mentions" | sed 's/^/ /'
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ -z "$arxiv_urls" && -z "$doi_urls" && -z "$arxiv_ids" ]]; then
|
||||
echo "No papers found in model card"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to get model papers
|
||||
get_model_papers() {
|
||||
local model_id="$1"
|
||||
|
||||
echo "=== $model_id ==="
|
||||
|
||||
# Get model info from API with authentication
|
||||
local api_url="https://huggingface.co/api/models/$model_id"
|
||||
local response=$(hf_api_call "$api_url")
|
||||
|
||||
if echo "$response" | grep -q '"error"'; then
|
||||
echo "Error: Could not fetch model '$model_id'"
|
||||
if [[ -z "${HF_TOKEN:-}" ]]; then
|
||||
echo "Note: This might be a private model. Try setting HF_TOKEN environment variable."
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Parse basic info
|
||||
local downloads=$(echo "$response" | jq -r '.downloads // 0')
|
||||
local likes=$(echo "$response" | jq -r '.likes // 0')
|
||||
echo "Downloads: $downloads | Likes: $likes"
|
||||
|
||||
# Get model card
|
||||
local card_url="https://huggingface.co/$model_id/raw/main/README.md"
|
||||
local card_content=$(curl -s "$card_url" 2>/dev/null || echo "")
|
||||
|
||||
if [[ -n "$card_content" ]]; then
|
||||
extract_papers "$card_content" "Papers from model card:"
|
||||
else
|
||||
echo "Could not fetch model card"
|
||||
fi
|
||||
|
||||
# Check tags for arxiv references
|
||||
local arxiv_tag=$(echo "$response" | jq -r '.tags[]' 2>/dev/null | grep arxiv || true)
|
||||
if [[ -n "$arxiv_tag" ]]; then
|
||||
echo "ArXiv from tags: $arxiv_tag"
|
||||
fi
|
||||
|
||||
echo
|
||||
}
|
||||
|
||||
# Function to get trending models
|
||||
get_trending_models() {
|
||||
local limit="${1:-5}"
|
||||
|
||||
echo "Fetching top $limit trending models..."
|
||||
|
||||
local trending_url="https://huggingface.co/api/trending?type=model&limit=$limit"
|
||||
local response=$(hf_api_call "$trending_url")
|
||||
|
||||
echo "$response" | jq -r '.recentlyTrending[] | .repoData.id' | head -"$limit" | while read -r model_id; do
|
||||
if [[ -n "$model_id" ]]; then
|
||||
get_model_papers "$model_id"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Main
|
||||
if [[ $# -eq 0 ]]; then
|
||||
echo "Error: No arguments provided"
|
||||
show_help
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$1" == "--help" ]]; then
|
||||
show_help
|
||||
exit 0
|
||||
elif [[ "$1" == "--trending" ]]; then
|
||||
if [[ -n "${2:-}" ]] && [[ "$2" =~ ^[0-9]+$ ]]; then
|
||||
get_trending_models "$2"
|
||||
else
|
||||
get_trending_models 5
|
||||
fi
|
||||
else
|
||||
get_model_papers "$1"
|
||||
fi
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user