diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 6687ebd31ea..3a4cd424a06 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -25,7 +25,8 @@ Changelog **Backward Breaking Changes** -- Move the Mistral Medium 3.5 checkpoint-mirror recipe from ``huggingface/models/nvidia/Mistral-Medium-3.5-128B-NVFP4/ptq/nvfp4-max-calib`` to ``huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib``, keying it by the canonical Hugging Face base model. Update any saved ``--recipe`` paths to the new location. +- Move the checkpoint-mirror recipe tier from ``huggingface/models///`` to the top-level ``models///`` (and the Step 3.5 Flash recipe from ``huggingface/step3p5/Step3.5-Flash/ptq/`` to ``models/stepfun-ai/Step-3.5-Flash/ptq/``), so instance-specific recipes sit beside ``general/`` and ``huggingface/`` instead of under the per-``model_type`` tree. Update any saved ``--recipe`` paths for these checkpoint recipes accordingly; the per-``model_type`` recipes under ``huggingface/`` are unchanged. +- Move the Mistral Medium 3.5 checkpoint-mirror recipe from ``models/nvidia/Mistral-Medium-3.5-128B-NVFP4/ptq/nvfp4-max-calib`` to ``models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib``, keying it by the canonical Hugging Face base model. Update any saved ``--recipe`` paths to the new location. - Remove the ``--auto_quantize_bits``, ``--auto_quantize_method``, ``--auto_quantize_score_size``, ``--auto_quantize_cost_model`` and ``--auto_quantize_active_moe_expert_ratio`` flags from ``examples/hf_ptq`` (deprecated in 0.46). Use an AutoQuantize ``--recipe`` from ``modelopt_recipes/general/auto_quantize/`` instead. Those recipes now also splice in the shared base ``cost_excluded_layers`` unit, which the removed CLI applied unconditionally, so a VL model keeps its vision tower and MTP layers out of the effective-bits denominator. On a VL model this changes the per-layer cost weights, so an existing ``--auto_quantize_checkpoint`` from an earlier release is rejected with "Use a different checkpoint path"; delete or repoint it to re-run the search. - Remove the ``examples/llm_ptq`` symlink and the ``examples/vlm_ptq`` forwarder (both deprecated in 0.46). Use ``examples/hf_ptq``, passing ``--vlm`` for vision-language models. - Remove the backward-compat ``--qformat`` / ``--quant_cfg`` short names ``int8_sq``, ``int8_wo``, ``w4a8_awq``, ``nvfp4_awq``, ``nvfp4_mse``, ``nvfp4_local_hessian``, ``fp8_pb_wo`` and ``fp8_pc_pt`` (deprecated in 0.45). Use the preset basename under ``modelopt_recipes/configs/ptq/presets/model/`` instead: ``int8_smoothquant``, ``int8_weight_only``, ``w4a8_awq_beta``, ``nvfp4_awq_lite``, ``nvfp4_w4a4_weight_mse_fp8_sweep``, ``nvfp4_w4a4_weight_local_hessian``, ``fp8_2d_blockwise_weight_only`` and ``fp8_per_channel_per_token``. The ``modelopt.recipe.presets.QFORMAT_ALIASES`` table and the ``aliases`` argument of ``load_quant_cfg_choices()`` are removed along with them. diff --git a/docs/source/guides/10_recipes.rst b/docs/source/guides/10_recipes.rst index 0bd4378219a..4ee7a413848 100644 --- a/docs/source/guides/10_recipes.rst +++ b/docs/source/guides/10_recipes.rst @@ -519,11 +519,14 @@ General PTQ recipes are model-agnostic and apply to any supported architecture: Model-specific recipes ---------------------- -Model-specific recipes are tuned for a particular Hugging Face ``model_type`` -(or a specific released model) and live under -``huggingface//[/]/``. See +Model-specific recipes come in two tiers: architecture recipes keyed by a +Hugging Face ``model_type`` under ``huggingface///``, and +checkpoint mirrors keyed by a model-hub path under +``models////``. See `modelopt_recipes/huggingface/README.md `_ -for the layout convention and recipe-lookup order. +and +`modelopt_recipes/models/README.md `_ +for the layout conventions and recipe-lookup order. .. list-table:: :header-rows: 1 @@ -531,7 +534,7 @@ for the layout convention and recipe-lookup order. * - Recipe path - Description - * - ``huggingface/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`` + * - ``models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only`` - NVFP4 MLP-only for Step 3.5 Flash MoE model * - ``huggingface/minimax_m3_vl/ptq/mxfp8_nvfp4_experts`` - MXFP8 language-model base with MSE-calibrated NVFP4 routed experts for MiniMax-M3 @@ -686,10 +689,14 @@ The ``modelopt_recipes/`` package is organized as follows: | +-- nvfp4_omlp_only-kv_fp8_cast.yaml | +-- nvfp4_omlp_only-kv_fp8.yaml | +-- nvfp4_weight_only-kv_fp8_cast.yaml - +-- huggingface/ # Model-specific recipes + +-- huggingface/ # Architecture-specific recipes (by model_type) | +-- / # see modelopt_recipes/huggingface/README.md | +-- / | +-- .yaml + +-- models/ # Checkpoint-specific recipes (by model-hub path) + | +-- // # see modelopt_recipes/models/README.md + | +-- / + | +-- .yaml +-- configs/ # Reusable config snippets (imported via $import) +-- numerics/ # Numeric format definitions | +-- fp8.yaml diff --git a/modelopt_recipes/README.md b/modelopt_recipes/README.md index b366e4cc670..46ecc61b813 100644 --- a/modelopt_recipes/README.md +++ b/modelopt_recipes/README.md @@ -42,13 +42,14 @@ huggingface/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast`. | Directory | What lives here | |-----------|-----------------| | `general/` | **Model-agnostic** recipes — a good starting point for any model. PTQ combos, speculative-decoding training, and distillation. | -| `huggingface//` | **Model-specific** recipes keyed by a HF `model_type`, optionally nested by released checkpoint. Use these first if your model has an entry. | -| `models//` | **Instance-specific** recipes that mirror a particular published checkpoint's quantization config. | +| `huggingface//` | **Architecture-specific** recipes keyed by a HF `model_type`; one recipe covers every checkpoint of that architecture. | +| `models///` | **Checkpoint-specific** recipes that mirror a particular published checkpoint, keyed by its model-hub path (e.g. `nvidia/Nemotron-3-Nano-4B-BF16`). | | `configs/` | Shared building blocks (`numerics/`, `ptq/units/`, `ptq/presets/`) that recipes compose from via `$import`. Not run directly. | -**Choosing where to look:** check `huggingface//` (then any nested -`/`) for your model first; if there's no entry, fall back to -`general/`. The presence of a model folder signals a recommended, tuned recipe. +**Choosing where to look:** check `models///` for your exact +checkpoint first, then `huggingface//` for its architecture; if +neither has an entry, fall back to `general/`. The presence of a model folder +signals a recommended, tuned recipe. --- @@ -77,9 +78,12 @@ how the model-specific recipes compare to the general ones and why they deviate. ## `models/` — checkpoint-specific recipes -These mirror a single **published checkpoint's** quantization config exactly — -a per-component mixed-precision scheme tuned to match a specific release. Browse -[`models/`](models/) for the available checkpoints. +These mirror a single **published checkpoint's** quantization config exactly — a +per-component mixed-precision scheme tuned to match a specific release. Each is +keyed by the checkpoint's **model-hub path** `/` (as on the +Hugging Face Hub, ModelScope, etc.). Browse [`models/`](models/) for the +available checkpoints; see [`models/README.md`](models/README.md) for the naming +convention. --- @@ -90,6 +94,6 @@ a per-component mixed-precision scheme tuned to match a specific release. Browse - **Tuned for a HF architecture** → `huggingface///`, with a `README.md` documenting the delta from the generic preset. Verify the exact `model_type` against the checkpoint's `config.json` before placing it. -- **Mirrors a specific released checkpoint** → `models//`. +- **Mirrors a specific released checkpoint** → `models///` (its model-hub path). - Share reused bodies via a `# modelopt-schema:`-tagged snippet and `$import` it; keep recipe wrappers thin. diff --git a/modelopt_recipes/huggingface/README.md b/modelopt_recipes/huggingface/README.md index 33ea80ad348..054493431e1 100644 --- a/modelopt_recipes/huggingface/README.md +++ b/modelopt_recipes/huggingface/README.md @@ -1,21 +1,23 @@ -# Model-specific recipes for Hugging Face models +# Architecture-specific recipes for Hugging Face models This folder holds model-optimization recipes (e.g. PTQ recipes) whose -behavior is tied to a **specific Hugging Face model architecture or model instance**. +behavior is tied to a **specific Hugging Face `model_type` (architecture)** — one +recipe covers every checkpoint of that architecture. Recipes tuned to a single +*published checkpoint* live in the sibling [`../models/`](../models/) tier +instead. ## Choosing a recipe -Built-in recipes live in two places: `modelopt_recipes/huggingface//` -for model-specific recipes and `modelopt_recipes/general/` for model-agnostic -ones. When deciding which to use: - -1. **Look in `huggingface//` first** for the target model's - Hugging Face `model_type`, and inside it for a nested - `/` folder if the recipe is tuned for one released - checkpoint rather than every checkpoint of that `model_type`. The - presence of a folder here signals that there is a recommended recipe - for that `model_type` or model instance. -2. **Fall back to `general/`** if no `/` folder applies. The +Built-in recipes live in three tiers — pick the most specific that applies: + +1. **[`../models///`](../models/)** first, if there is an entry + for your **exact** published checkpoint (keyed by its model-hub path). It + mirrors a validated, per-checkpoint scheme. +2. **`huggingface//`** for the target model's Hugging Face + `model_type` — an architecture-level recipe that applies to every checkpoint + of that `model_type`. The presence of a folder here signals a recommended + recipe for that architecture. +3. **Fall back to `general/`** if no `/` folder applies. The general recipes are a good starting point for any model — and the recommended starting point for a model architecture that does not yet have a model-specific entry. @@ -67,17 +69,13 @@ include the field name the snippet represents as a secondary suffix is its natural canonical home; other importers reference it by the same relative path under `modelopt_recipes/`. -### Per-family nested layout for specific model variants - -If a recipe is tuned for one specific released model rather than every -checkpoint under a `model_type`, nest the model name as an extra level: +### Checkpoint-specific recipes -```text -/ - / - / - .yaml -``` +If a recipe is tuned for one specific released checkpoint rather than every +checkpoint of a `model_type`, it does not belong here — it lives in the +top-level [`../models/`](../models/) tier, keyed by the checkpoint's model-hub +path `/`. See [`../models/README.md`](../models/README.md) for +that convention. ### Per-folder READMEs diff --git a/modelopt_recipes/models/README.md b/modelopt_recipes/models/README.md new file mode 100644 index 00000000000..b76934dd77e --- /dev/null +++ b/modelopt_recipes/models/README.md @@ -0,0 +1,96 @@ +# Recipes for specific model-hub checkpoints + +This folder holds model-optimization recipes (e.g. PTQ recipes) tuned for a +**specific published model instance** — one checkpoint released on a model hub +such as the [Hugging Face Hub](https://huggingface.co/), +[ModelScope](https://modelscope.cn/), or similar. Unlike +[`../huggingface/`](../huggingface/), which keys recipes by a transformers +`model_type` (an architecture shared by many checkpoints), a recipe here mirrors +**one checkpoint's** quantization scheme verbatim. + +## Folder structure + +Each instance is keyed by its **model-hub path** — the same `/` +you pass to `from_pretrained(...)` or find in the hub URL. The on-disk path +mirrors the hub path exactly: + +```text +modelopt_recipes/models/ + / # hub namespace / organization, e.g. nvidia, mistralai + / # hub model id, e.g. Nemotron-3-Nano-4B-BF16 + / # optimization workflow, e.g. ptq + .yaml + [..yaml] # optional $import snippet helpers (see below) + [README.md] # optional; describes what's checkpoint-specific +``` + +For example, the recipe for the hub checkpoint `nvidia/Nemotron-3-Nano-4B-BF16` +(`https://huggingface.co/nvidia/Nemotron-3-Nano-4B-BF16`) lives at +`models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/`. Because the folder path *is* the +hub path, you can go straight from a checkpoint id to its recipe — and back — +with no lookup table. + +`` is the optimization workflow the recipe targets (e.g. `ptq` for +post-training quantization). + +### Naming the `/` folders + +Use the checkpoint's exact hub `/`, including casing. When the +same weights are published on more than one hub (e.g. the Hugging Face Hub and +ModelScope) under the same `/`, a single folder serves them all. +When a recipe was tuned against one **canonical / base** checkpoint but also +applies to its mirrors, key it by that base model's id. + +## Choosing a recipe + +Prefer the most specific entry that applies to your model: + +1. **`models///`** — if there is an entry for your **exact** + checkpoint. It reproduces a validated, often per-component mixed-precision + scheme for that release; use it to match a published quantized checkpoint. +2. **[`huggingface//`](../huggingface/)** — an architecture-level + recipe that applies to every checkpoint of that `model_type`. +3. **[`general/`](../general/)** — model-agnostic recipes; a good starting point + for any model without a more specific entry. + +## Selecting a recipe at runtime + +Use the path relative to `modelopt_recipes/`: + +```text +--recipe models//// +``` + +or from Python: + +```python +from modelopt.recipe import load_recipe + +recipe = load_recipe("models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16") +``` + +## What belongs here + +A recipe earns a place here only when it mirrors **one specific released (or +planned) checkpoint** — a hand-mapped, usually per-layer or per-component +precision scheme tuned to match that exact release. If the tuning generalizes to +every checkpoint of an architecture, it belongs under +[`../huggingface//`](../huggingface/) instead; if it is +model-agnostic, it belongs under [`../general/`](../general/). See +[`../ptq.md`](../ptq.md) for what each checkpoint mirror does and how it compares +to its general baseline. + +## Sharing content across recipes + +When several recipes reuse the same body, extract it into a sibling **snippet** +file with a `# modelopt-schema:` header and `$import` it, keeping each recipe +wrapper thin. Name snippets so they are obviously not runnable recipes (e.g. +`..yaml`), and reference them by their path relative to +`modelopt_recipes/`. + +## Per-folder READMEs + +Each `/` folder may contain a short `README.md` describing exactly what is +checkpoint-specific — which layers deviate, the calibration used, and the +reference checkpoint it mirrors — so reviewers and users don't have to diff the +YAML against the generic presets to see the intent. diff --git a/modelopt_recipes/huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml b/modelopt_recipes/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml rename to modelopt_recipes/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml b/modelopt_recipes/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml diff --git a/modelopt_recipes/huggingface/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml b/modelopt_recipes/models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only.yaml similarity index 100% rename from modelopt_recipes/huggingface/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml rename to modelopt_recipes/models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only.yaml diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 53c4ef99504..eda4cb47768 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -3,10 +3,9 @@ This doc walks through the **PTQ quantization schemes** in two parts: the model-agnostic recipes under [`general/ptq/`](general/ptq/) (the recommended starting point for any model), and then the -[model-specific recipes](#model-specific-recipes-huggingface) under -`huggingface/` — per-`model_type` folders plus the -`huggingface/models///` tier — comparing each to its general -baseline and explaining why it deviates. +[model-specific recipes](#model-specific-recipes) — per-`model_type` folders +under `huggingface/` plus the checkpoint-mirror `models///` +tier — comparing each to its general baseline and explaining why it deviates. --- @@ -227,14 +226,14 @@ These can also be **stacked** when a single method isn't enough — e.g. `mse` + --- -## Model-specific recipes (`huggingface/`) +## Model-specific recipes The general recipes above are **model-agnostic**: they select layers by wildcard (`*mlp*`, `*self_attn*`, `*[kv]_bmm_quantizer`) and lean on the shared `default_disabled_quantizers` exclusions, so the same file works on any architecture whose module names follow the usual conventions. A recipe only earns a place under `huggingface//` or -`huggingface/models///` when a model has to **deviate** from +`models///` when a model has to **deviate** from that baseline. The deviations come in four kinds: | Kind | What changes vs. the general recipe | Examples | @@ -297,7 +296,7 @@ general recipes never enable output quantizers, and the pattern must stay scoped to GEMM outputs — a `DynamicQuantize` on non-GEMM outputs (embedding lookup, pooling) fails to compile in TensorRT. -A lighter case: **`step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`** is close to +A lighter case: **`models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only`** is close to `general/ptq/nvfp4_mlp_only` (NVFP4 on MoE/MLP weights+inputs, FP8 KV) but pinned to one released checkpoint and carrying instance-specific disables (`share_expert`, `moe.gate`, the conv1d branches). @@ -344,14 +343,14 @@ everything else matches the general recipe. ### Checkpoint mirrors — `models//` -The `huggingface/models/` tier reproduces a **single published (or planned) +The `models/` tier reproduces a **single published (or planned) checkpoint's** quant config verbatim: - **`models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib`** mirrors `nvidia/Mistral-Medium-3.5-128B-NVFP4`: decoder MLP layers 4–86 use NVFP4 W4A4, edge MLP layers 0–3 and 87 use FP8 W8A8, and all attention projections and the KV cache use FP8. It uses max calibration. -- **`Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse`** mirrors +- **`models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse`** mirrors `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` exactly — a hybrid **Mamba-MoE** with a hand-mapped, **per-component** precision scheme: - MoE routed experts → NVFP4 W4A4, `group_size 16`, **static** weight scales @@ -362,17 +361,17 @@ checkpoint's** quant config verbatim: `nvfp4-mse.yaml` uses MSE calibration with an FP8-scale sweep (matches the release); `nvfp4-max-calib.yaml` is the identical layer map under plain `max` calibration, kept for comparison. -- **`Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6`** follows the same Super-style +- **`models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6`** follows the same Super-style component map (routed experts NVFP4 W4A4 block-16; shared experts + Mamba `in/out_proj` + KV cache FP8; everything else BF16), but the routed-expert weights use **Four-over-Six (4/6)** NVFP4: an MSE search picks each weight's amax multiplier from `[1.0, 1.5]` (M=6 vs. M=4). Activations stay dynamic NVFP4 (not MSE-calibrated). -- **`Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6`** applies +- **`models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6`** applies Four-over-Six NVFP4 W4A16 to routed experts, shared experts, and the language model head; Mamba `in/out_proj` weights and inputs plus the KV cache use FP8, while attention remains BF16. -- **`Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16`** mirrors the GGUF **Q4_K_M** bit +- **`models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16`** mirrors the GGUF **Q4_K_M** bit allocation of the Nemotron-H hybrid, mapped onto NVFP4/FP8 **per layer**: Q4_K/Q5_0 linears → NVFP4 W4A4 (attention q/k/v/o kept uniform so export can fuse them), the Q6_K MLP `down_proj` layers → FP8 W8A8, embeddings → NVFP4 diff --git a/tests/unit/recipe/test_loader.py b/tests/unit/recipe/test_loader.py index 3dfb9906a54..cd791601ac5 100644 --- a/tests/unit/recipe/test_loader.py +++ b/tests/unit/recipe/test_loader.py @@ -169,7 +169,13 @@ def test_load_recipe_builtin_description(): "general/ptq/nvfp4_experts_only-kv_fp8", "general/ptq/nvfp4_experts_only-kv_fp8_cast", "general/ptq/nvfp4_experts_only-kv_fp8_layerwise", - "huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib", + "models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib", + "models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16", + "models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib", + "models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse", + "models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6", + "models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6", + "models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only", "general/ptq/nvfp4_mlp_only-kv_fp8", "general/ptq/nvfp4_mlp_only-novit-kv_fp8", "general/ptq/nvfp4_mlp_only-kv_fp8_cast", diff --git a/tests/unit/recipe/test_recipe_docs.py b/tests/unit/recipe/test_recipe_docs.py index adeb10dcee3..e2847d7f3d5 100644 --- a/tests/unit/recipe/test_recipe_docs.py +++ b/tests/unit/recipe/test_recipe_docs.py @@ -85,19 +85,22 @@ def test_general_ptq_recipe_count_in_ptq_md(): def test_every_model_specific_ptq_dir_is_mentioned(): - """Every model dir under huggingface/ with PTQ recipes must appear in ptq.md. + """Every model dir under huggingface/ or models/ with PTQ recipes must appear in ptq.md. The identifier checked is the directory containing the ptq/ folder — the - HF model_type (e.g. ``gemma4``), a nested checkpoint dir (e.g. - ``Step3.5-Flash``), or a models// leaf (e.g. - ``Nemotron-3-Nano-4B``). + HF ``model_type`` (e.g. ``gemma4``) or a models// leaf + (e.g. ``Step-3.5-Flash``, ``Nemotron-3-Nano-4B-BF16``). """ doc = _ptq_md_text() - hf_dir = RECIPES_DIR / "huggingface" + search_dirs = (RECIPES_DIR / "huggingface", RECIPES_DIR / "models") model_dirs = sorted( - {yaml_path.parent.parent.name for yaml_path in hf_dir.glob("**/ptq/*.yaml")} + { + yaml_path.parent.parent.name + for base in search_dirs + for yaml_path in base.glob("**/ptq/*.yaml") + } ) - assert model_dirs, "No model-specific PTQ recipes found under huggingface/" + assert model_dirs, "No model-specific PTQ recipes found under huggingface/ or models/" missing = [name for name in model_dirs if name not in doc] assert not missing, ( f"Model-specific PTQ recipe folders are missing from " diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml index 6ec49dd7a0a..d673ed3ca1b 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml @@ -29,7 +29,7 @@ pipeline: - --calib-size 32 environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse + - QUANT_CFG: models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 # MMLU + Export run as separate tasks; quantize.sh does quantize only. - RUN_MMLU: "false" @@ -52,7 +52,7 @@ pipeline: script: common/megatron_lm/export/export.sh environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse + - QUANT_CFG: models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - TP: "1" - PP: "4" diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml index dfe363da634..42150fa00ac 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml @@ -30,7 +30,7 @@ pipeline: - --calib-size 32 environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 + - QUANT_CFG: models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 # MMLU + Export run as separate tasks; quantize.sh does quantize only. - RUN_MMLU: "false" @@ -53,7 +53,7 @@ pipeline: script: common/megatron_lm/export/export.sh environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 + - QUANT_CFG: models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 - TP: "1" - PP: "12" diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml index e14e39a185f..210ffbf84d8 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml @@ -65,7 +65,7 @@ pipeline: --tp_size 1 --pp_size 1 --ep_size 1 - --recipe huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 + --recipe models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 --calib_batch_size 1 --calib_num_samples 1000 --seq_length 32768 diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml index 7f9556bb37d..cdb68388a6b 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml @@ -22,7 +22,7 @@ pipeline: --hf_model_name_or_path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --trust_remote_code --tp_size 1 - --recipe huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 + --recipe models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 --calib_batch_size 8 --calib_num_samples 256 --seq_length 512 diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml index 8519647b1e8..b04ca7cd2d2 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml @@ -53,7 +53,7 @@ pipeline: - --export-default-te-spec environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 + - QUANT_CFG: models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 - MLM_MODEL_CKPT: /cicd/megatron-lm-bf16/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-MCore - MLM_MODEL_SAVE: /cicd/megatron-lm/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-W4A16 - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16