From c5ecf7881e35c4f664e6120cd3bd3929bcd13c86 Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Thu, 20 Aug 2026 00:53:14 +0000 Subject: [PATCH 1/4] Move checkpoint-mirror recipes to top-level modelopt_recipes/models/ Separate Hugging Face Hub model instances from transformers `model_type` recipes by relocating the instance tier out of huggingface/ to a top-level modelopt_recipes/models/, matching the layout already documented in modelopt_recipes/README.md: - models/{mistralai,nvidia}//ptq/ (from huggingface/models/) - models/step3p5/Step3.5-Flash/ptq/ (from huggingface/step3p5/) huggingface/ keeps the per-model_type recipes unchanged. Update the recipe references that pointed at the old paths: the launcher QUANT_CFG/--recipe values, the loader/doc-consistency tests (the docs test now globs models/ too), docs/source/guides/10_recipes.rst and modelopt_recipes/ptq.md. Add a 0.47 backward-breaking CHANGELOG entry; released changelog entries keep their historical paths. Signed-off-by: Shengliang Xu --- CHANGELOG.rst | 3 ++- docs/source/guides/10_recipes.rst | 2 +- .../Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml | 0 .../Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml | 0 .../ptq/nvfp4-max-calib.yaml | 0 .../ptq/nvfp4-mse.yaml | 0 .../ptq/nvfp4-4o6.yaml | 0 .../ptq/w4a16_nvfp4_4o6.yaml | 0 .../step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml | 0 modelopt_recipes/ptq.md | 6 +++--- tests/unit/recipe/test_loader.py | 2 +- tests/unit/recipe/test_recipe_docs.py | 12 ++++++++---- .../megatron_lm_ptq.yaml | 4 ++-- .../megatron_lm_ptq.yaml | 4 ++-- .../mbridge_qad.yaml | 2 +- .../mbridge_quantize.yaml | 2 +- .../megatron_lm_qad.yaml | 2 +- 17 files changed, 22 insertions(+), 17 deletions(-) rename modelopt_recipes/{huggingface => }/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml (100%) rename modelopt_recipes/{huggingface => }/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml (100%) rename modelopt_recipes/{huggingface => }/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml (100%) rename modelopt_recipes/{huggingface => }/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml (100%) rename modelopt_recipes/{huggingface => }/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml (100%) rename modelopt_recipes/{huggingface => }/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml (100%) rename modelopt_recipes/{huggingface => models}/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml (100%) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 4580a045f39..16dfc660b95 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -24,7 +24,8 @@ Changelog **Backward Breaking Changes** -- Move the Mistral Medium 3.5 checkpoint-mirror recipe from ``huggingface/models/nvidia/Mistral-Medium-3.5-128B-NVFP4/ptq/nvfp4-max-calib`` to ``huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib``, keying it by the canonical Hugging Face base model. Update any saved ``--recipe`` paths to the new location. +- Move the checkpoint-mirror recipe tier from ``huggingface/models///`` to the top-level ``models///`` (and the Step 3.5 Flash recipe from ``huggingface/step3p5/Step3.5-Flash/ptq/`` to ``models/step3p5/Step3.5-Flash/ptq/``), so instance-specific recipes sit beside ``general/`` and ``huggingface/`` instead of under the per-``model_type`` tree. Drop the ``huggingface/`` prefix in any saved ``--recipe`` paths. +- Move the Mistral Medium 3.5 checkpoint-mirror recipe from ``models/nvidia/Mistral-Medium-3.5-128B-NVFP4/ptq/nvfp4-max-calib`` to ``models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib``, keying it by the canonical Hugging Face base model. Update any saved ``--recipe`` paths to the new location. - Remove the ``--auto_quantize_bits``, ``--auto_quantize_method``, ``--auto_quantize_score_size``, ``--auto_quantize_cost_model`` and ``--auto_quantize_active_moe_expert_ratio`` flags from ``examples/hf_ptq`` (deprecated in 0.46). Use an AutoQuantize ``--recipe`` from ``modelopt_recipes/general/auto_quantize/`` instead. Those recipes now also splice in the shared base ``cost_excluded_layers`` unit, which the removed CLI applied unconditionally, so a VL model keeps its vision tower and MTP layers out of the effective-bits denominator. On a VL model this changes the per-layer cost weights, so an existing ``--auto_quantize_checkpoint`` from an earlier release is rejected with "Use a different checkpoint path"; delete or repoint it to re-run the search. - Remove the ``examples/llm_ptq`` symlink and the ``examples/vlm_ptq`` forwarder (both deprecated in 0.46). Use ``examples/hf_ptq``, passing ``--vlm`` for vision-language models. - Remove the backward-compat ``--qformat`` / ``--quant_cfg`` short names ``int8_sq``, ``int8_wo``, ``w4a8_awq``, ``nvfp4_awq``, ``nvfp4_mse``, ``nvfp4_local_hessian``, ``fp8_pb_wo`` and ``fp8_pc_pt`` (deprecated in 0.45). Use the preset basename under ``modelopt_recipes/configs/ptq/presets/model/`` instead: ``int8_smoothquant``, ``int8_weight_only``, ``w4a8_awq_beta``, ``nvfp4_awq_lite``, ``nvfp4_w4a4_weight_mse_fp8_sweep``, ``nvfp4_w4a4_weight_local_hessian``, ``fp8_2d_blockwise_weight_only`` and ``fp8_per_channel_per_token``. The ``modelopt.recipe.presets.QFORMAT_ALIASES`` table and the ``aliases`` argument of ``load_quant_cfg_choices()`` are removed along with them. diff --git a/docs/source/guides/10_recipes.rst b/docs/source/guides/10_recipes.rst index 0bd4378219a..15b7626aecc 100644 --- a/docs/source/guides/10_recipes.rst +++ b/docs/source/guides/10_recipes.rst @@ -531,7 +531,7 @@ for the layout convention and recipe-lookup order. * - Recipe path - Description - * - ``huggingface/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`` + * - ``models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`` - NVFP4 MLP-only for Step 3.5 Flash MoE model * - ``huggingface/minimax_m3_vl/ptq/mxfp8_nvfp4_experts`` - MXFP8 language-model base with MSE-calibrated NVFP4 routed experts for MiniMax-M3 diff --git a/modelopt_recipes/huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml b/modelopt_recipes/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml rename to modelopt_recipes/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml b/modelopt_recipes/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6.yaml diff --git a/modelopt_recipes/huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml b/modelopt_recipes/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml similarity index 100% rename from modelopt_recipes/huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml rename to modelopt_recipes/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6.yaml diff --git a/modelopt_recipes/huggingface/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml b/modelopt_recipes/models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml similarity index 100% rename from modelopt_recipes/huggingface/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml rename to modelopt_recipes/models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 53c4ef99504..78dcd7bdea8 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -5,7 +5,7 @@ model-agnostic recipes under [`general/ptq/`](general/ptq/) (the recommended starting point for any model), and then the [model-specific recipes](#model-specific-recipes-huggingface) under `huggingface/` — per-`model_type` folders plus the -`huggingface/models///` tier — comparing each to its general +`models///` tier — comparing each to its general baseline and explaining why it deviates. --- @@ -234,7 +234,7 @@ The general recipes above are **model-agnostic**: they select layers by wildcard `default_disabled_quantizers` exclusions, so the same file works on any architecture whose module names follow the usual conventions. A recipe only earns a place under `huggingface//` or -`huggingface/models///` when a model has to **deviate** from +`models///` when a model has to **deviate** from that baseline. The deviations come in four kinds: | Kind | What changes vs. the general recipe | Examples | @@ -344,7 +344,7 @@ everything else matches the general recipe. ### Checkpoint mirrors — `models//` -The `huggingface/models/` tier reproduces a **single published (or planned) +The `models/` tier reproduces a **single published (or planned) checkpoint's** quant config verbatim: - **`models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib`** mirrors diff --git a/tests/unit/recipe/test_loader.py b/tests/unit/recipe/test_loader.py index 3dfb9906a54..cc3a2ff1f7c 100644 --- a/tests/unit/recipe/test_loader.py +++ b/tests/unit/recipe/test_loader.py @@ -169,7 +169,7 @@ def test_load_recipe_builtin_description(): "general/ptq/nvfp4_experts_only-kv_fp8", "general/ptq/nvfp4_experts_only-kv_fp8_cast", "general/ptq/nvfp4_experts_only-kv_fp8_layerwise", - "huggingface/models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib", + "models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib", "general/ptq/nvfp4_mlp_only-kv_fp8", "general/ptq/nvfp4_mlp_only-novit-kv_fp8", "general/ptq/nvfp4_mlp_only-kv_fp8_cast", diff --git a/tests/unit/recipe/test_recipe_docs.py b/tests/unit/recipe/test_recipe_docs.py index adeb10dcee3..43a3558432c 100644 --- a/tests/unit/recipe/test_recipe_docs.py +++ b/tests/unit/recipe/test_recipe_docs.py @@ -85,7 +85,7 @@ def test_general_ptq_recipe_count_in_ptq_md(): def test_every_model_specific_ptq_dir_is_mentioned(): - """Every model dir under huggingface/ with PTQ recipes must appear in ptq.md. + """Every model dir under huggingface/ or models/ with PTQ recipes must appear in ptq.md. The identifier checked is the directory containing the ptq/ folder — the HF model_type (e.g. ``gemma4``), a nested checkpoint dir (e.g. @@ -93,11 +93,15 @@ def test_every_model_specific_ptq_dir_is_mentioned(): ``Nemotron-3-Nano-4B``). """ doc = _ptq_md_text() - hf_dir = RECIPES_DIR / "huggingface" + search_dirs = (RECIPES_DIR / "huggingface", RECIPES_DIR / "models") model_dirs = sorted( - {yaml_path.parent.parent.name for yaml_path in hf_dir.glob("**/ptq/*.yaml")} + { + yaml_path.parent.parent.name + for base in search_dirs + for yaml_path in base.glob("**/ptq/*.yaml") + } ) - assert model_dirs, "No model-specific PTQ recipes found under huggingface/" + assert model_dirs, "No model-specific PTQ recipes found under huggingface/ or models/" missing = [name for name in model_dirs if name not in doc] assert not missing, ( f"Model-specific PTQ recipe folders are missing from " diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml index 6ec49dd7a0a..d673ed3ca1b 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/megatron_lm_ptq.yaml @@ -29,7 +29,7 @@ pipeline: - --calib-size 32 environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse + - QUANT_CFG: models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 # MMLU + Export run as separate tasks; quantize.sh does quantize only. - RUN_MMLU: "false" @@ -52,7 +52,7 @@ pipeline: script: common/megatron_lm/export/export.sh environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse + - QUANT_CFG: models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - TP: "1" - PP: "4" diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml index dfe363da634..42150fa00ac 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/megatron_lm_ptq.yaml @@ -30,7 +30,7 @@ pipeline: - --calib-size 32 environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 + - QUANT_CFG: models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 # MMLU + Export run as separate tasks; quantize.sh does quantize only. - RUN_MMLU: "false" @@ -53,7 +53,7 @@ pipeline: script: common/megatron_lm/export/export.sh environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 + - QUANT_CFG: models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 - TP: "1" - PP: "12" diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml index e14e39a185f..210ffbf84d8 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml @@ -65,7 +65,7 @@ pipeline: --tp_size 1 --pp_size 1 --ep_size 1 - --recipe huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 + --recipe models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 --calib_batch_size 1 --calib_num_samples 1000 --seq_length 32768 diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml index 7f9556bb37d..cdb68388a6b 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_quantize.yaml @@ -22,7 +22,7 @@ pipeline: --hf_model_name_or_path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --trust_remote_code --tp_size 1 - --recipe huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 + --recipe models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 --calib_batch_size 8 --calib_num_samples 256 --seq_length 512 diff --git a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml index 8519647b1e8..b04ca7cd2d2 100644 --- a/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml +++ b/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml @@ -53,7 +53,7 @@ pipeline: - --export-default-te-spec environment: - MLM_MODEL_CFG: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 - - QUANT_CFG: huggingface/models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 + - QUANT_CFG: models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 - MLM_MODEL_CKPT: /cicd/megatron-lm-bf16/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-MCore - MLM_MODEL_SAVE: /cicd/megatron-lm/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-W4A16 - HF_MODEL_CKPT: /hf-local/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 From bf056a0087cfe548655bba56db280ebcc76e3f5e Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Thu, 20 Aug 2026 01:04:52 +0000 Subject: [PATCH 2/4] Document the top-level models/ recipe tier (hub-path keyed) Add modelopt_recipes/models/README.md describing the checkpoint-mirror tier: recipes for a specific published model instance on a model hub (Hugging Face Hub, ModelScope, etc.), keyed by the hub path / so the on-disk path mirrors the hub path. Update the surrounding docs now that instances live in the top-level models/ tier and huggingface/ is purely per-model_type: rework huggingface/README.md to point checkpoint-tuned recipes at ../models/ (dropping the old nested // layout), and refresh modelopt_recipes/README.md, ptq.md and docs/source/guides/10_recipes.rst to describe the two tiers. Signed-off-by: Shengliang Xu --- docs/source/guides/10_recipes.rst | 11 +-- modelopt_recipes/README.md | 22 +++--- modelopt_recipes/huggingface/README.md | 42 ++++++----- modelopt_recipes/models/README.md | 96 ++++++++++++++++++++++++++ modelopt_recipes/ptq.md | 11 ++- 5 files changed, 141 insertions(+), 41 deletions(-) create mode 100644 modelopt_recipes/models/README.md diff --git a/docs/source/guides/10_recipes.rst b/docs/source/guides/10_recipes.rst index 15b7626aecc..cf511014adc 100644 --- a/docs/source/guides/10_recipes.rst +++ b/docs/source/guides/10_recipes.rst @@ -519,11 +519,14 @@ General PTQ recipes are model-agnostic and apply to any supported architecture: Model-specific recipes ---------------------- -Model-specific recipes are tuned for a particular Hugging Face ``model_type`` -(or a specific released model) and live under -``huggingface//[/]/``. See +Model-specific recipes come in two tiers: architecture recipes keyed by a +Hugging Face ``model_type`` under ``huggingface///``, and +checkpoint mirrors keyed by a model-hub path under +``models////``. See `modelopt_recipes/huggingface/README.md `_ -for the layout convention and recipe-lookup order. +and +`modelopt_recipes/models/README.md `_ +for the layout conventions and recipe-lookup order. .. list-table:: :header-rows: 1 diff --git a/modelopt_recipes/README.md b/modelopt_recipes/README.md index b366e4cc670..46ecc61b813 100644 --- a/modelopt_recipes/README.md +++ b/modelopt_recipes/README.md @@ -42,13 +42,14 @@ huggingface/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast`. | Directory | What lives here | |-----------|-----------------| | `general/` | **Model-agnostic** recipes — a good starting point for any model. PTQ combos, speculative-decoding training, and distillation. | -| `huggingface//` | **Model-specific** recipes keyed by a HF `model_type`, optionally nested by released checkpoint. Use these first if your model has an entry. | -| `models//` | **Instance-specific** recipes that mirror a particular published checkpoint's quantization config. | +| `huggingface//` | **Architecture-specific** recipes keyed by a HF `model_type`; one recipe covers every checkpoint of that architecture. | +| `models///` | **Checkpoint-specific** recipes that mirror a particular published checkpoint, keyed by its model-hub path (e.g. `nvidia/Nemotron-3-Nano-4B-BF16`). | | `configs/` | Shared building blocks (`numerics/`, `ptq/units/`, `ptq/presets/`) that recipes compose from via `$import`. Not run directly. | -**Choosing where to look:** check `huggingface//` (then any nested -`/`) for your model first; if there's no entry, fall back to -`general/`. The presence of a model folder signals a recommended, tuned recipe. +**Choosing where to look:** check `models///` for your exact +checkpoint first, then `huggingface//` for its architecture; if +neither has an entry, fall back to `general/`. The presence of a model folder +signals a recommended, tuned recipe. --- @@ -77,9 +78,12 @@ how the model-specific recipes compare to the general ones and why they deviate. ## `models/` — checkpoint-specific recipes -These mirror a single **published checkpoint's** quantization config exactly — -a per-component mixed-precision scheme tuned to match a specific release. Browse -[`models/`](models/) for the available checkpoints. +These mirror a single **published checkpoint's** quantization config exactly — a +per-component mixed-precision scheme tuned to match a specific release. Each is +keyed by the checkpoint's **model-hub path** `/` (as on the +Hugging Face Hub, ModelScope, etc.). Browse [`models/`](models/) for the +available checkpoints; see [`models/README.md`](models/README.md) for the naming +convention. --- @@ -90,6 +94,6 @@ a per-component mixed-precision scheme tuned to match a specific release. Browse - **Tuned for a HF architecture** → `huggingface///`, with a `README.md` documenting the delta from the generic preset. Verify the exact `model_type` against the checkpoint's `config.json` before placing it. -- **Mirrors a specific released checkpoint** → `models//`. +- **Mirrors a specific released checkpoint** → `models///` (its model-hub path). - Share reused bodies via a `# modelopt-schema:`-tagged snippet and `$import` it; keep recipe wrappers thin. diff --git a/modelopt_recipes/huggingface/README.md b/modelopt_recipes/huggingface/README.md index 33ea80ad348..71d270c5c4b 100644 --- a/modelopt_recipes/huggingface/README.md +++ b/modelopt_recipes/huggingface/README.md @@ -1,21 +1,23 @@ # Model-specific recipes for Hugging Face models This folder holds model-optimization recipes (e.g. PTQ recipes) whose -behavior is tied to a **specific Hugging Face model architecture or model instance**. +behavior is tied to a **specific Hugging Face `model_type` (architecture)** — one +recipe covers every checkpoint of that architecture. Recipes tuned to a single +*published checkpoint* live in the sibling [`../models/`](../models/) tier +instead. ## Choosing a recipe -Built-in recipes live in two places: `modelopt_recipes/huggingface//` -for model-specific recipes and `modelopt_recipes/general/` for model-agnostic -ones. When deciding which to use: - -1. **Look in `huggingface//` first** for the target model's - Hugging Face `model_type`, and inside it for a nested - `/` folder if the recipe is tuned for one released - checkpoint rather than every checkpoint of that `model_type`. The - presence of a folder here signals that there is a recommended recipe - for that `model_type` or model instance. -2. **Fall back to `general/`** if no `/` folder applies. The +Built-in recipes live in three tiers — pick the most specific that applies: + +1. **[`../models///`](../models/)** first, if there is an entry + for your **exact** published checkpoint (keyed by its model-hub path). It + mirrors a validated, per-checkpoint scheme. +2. **`huggingface//`** for the target model's Hugging Face + `model_type` — an architecture-level recipe that applies to every checkpoint + of that `model_type`. The presence of a folder here signals a recommended + recipe for that architecture. +3. **Fall back to `general/`** if no `/` folder applies. The general recipes are a good starting point for any model — and the recommended starting point for a model architecture that does not yet have a model-specific entry. @@ -67,17 +69,13 @@ include the field name the snippet represents as a secondary suffix is its natural canonical home; other importers reference it by the same relative path under `modelopt_recipes/`. -### Per-family nested layout for specific model variants - -If a recipe is tuned for one specific released model rather than every -checkpoint under a `model_type`, nest the model name as an extra level: +### Checkpoint-specific recipes -```text -/ - / - / - .yaml -``` +If a recipe is tuned for one specific released checkpoint rather than every +checkpoint of a `model_type`, it does not belong here — it lives in the +top-level [`../models/`](../models/) tier, keyed by the checkpoint's model-hub +path `/`. See [`../models/README.md`](../models/README.md) for +that convention. ### Per-folder READMEs diff --git a/modelopt_recipes/models/README.md b/modelopt_recipes/models/README.md new file mode 100644 index 00000000000..b76934dd77e --- /dev/null +++ b/modelopt_recipes/models/README.md @@ -0,0 +1,96 @@ +# Recipes for specific model-hub checkpoints + +This folder holds model-optimization recipes (e.g. PTQ recipes) tuned for a +**specific published model instance** — one checkpoint released on a model hub +such as the [Hugging Face Hub](https://huggingface.co/), +[ModelScope](https://modelscope.cn/), or similar. Unlike +[`../huggingface/`](../huggingface/), which keys recipes by a transformers +`model_type` (an architecture shared by many checkpoints), a recipe here mirrors +**one checkpoint's** quantization scheme verbatim. + +## Folder structure + +Each instance is keyed by its **model-hub path** — the same `/` +you pass to `from_pretrained(...)` or find in the hub URL. The on-disk path +mirrors the hub path exactly: + +```text +modelopt_recipes/models/ + / # hub namespace / organization, e.g. nvidia, mistralai + / # hub model id, e.g. Nemotron-3-Nano-4B-BF16 + / # optimization workflow, e.g. ptq + .yaml + [..yaml] # optional $import snippet helpers (see below) + [README.md] # optional; describes what's checkpoint-specific +``` + +For example, the recipe for the hub checkpoint `nvidia/Nemotron-3-Nano-4B-BF16` +(`https://huggingface.co/nvidia/Nemotron-3-Nano-4B-BF16`) lives at +`models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/`. Because the folder path *is* the +hub path, you can go straight from a checkpoint id to its recipe — and back — +with no lookup table. + +`` is the optimization workflow the recipe targets (e.g. `ptq` for +post-training quantization). + +### Naming the `/` folders + +Use the checkpoint's exact hub `/`, including casing. When the +same weights are published on more than one hub (e.g. the Hugging Face Hub and +ModelScope) under the same `/`, a single folder serves them all. +When a recipe was tuned against one **canonical / base** checkpoint but also +applies to its mirrors, key it by that base model's id. + +## Choosing a recipe + +Prefer the most specific entry that applies to your model: + +1. **`models///`** — if there is an entry for your **exact** + checkpoint. It reproduces a validated, often per-component mixed-precision + scheme for that release; use it to match a published quantized checkpoint. +2. **[`huggingface//`](../huggingface/)** — an architecture-level + recipe that applies to every checkpoint of that `model_type`. +3. **[`general/`](../general/)** — model-agnostic recipes; a good starting point + for any model without a more specific entry. + +## Selecting a recipe at runtime + +Use the path relative to `modelopt_recipes/`: + +```text +--recipe models//// +``` + +or from Python: + +```python +from modelopt.recipe import load_recipe + +recipe = load_recipe("models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16") +``` + +## What belongs here + +A recipe earns a place here only when it mirrors **one specific released (or +planned) checkpoint** — a hand-mapped, usually per-layer or per-component +precision scheme tuned to match that exact release. If the tuning generalizes to +every checkpoint of an architecture, it belongs under +[`../huggingface//`](../huggingface/) instead; if it is +model-agnostic, it belongs under [`../general/`](../general/). See +[`../ptq.md`](../ptq.md) for what each checkpoint mirror does and how it compares +to its general baseline. + +## Sharing content across recipes + +When several recipes reuse the same body, extract it into a sibling **snippet** +file with a `# modelopt-schema:` header and `$import` it, keeping each recipe +wrapper thin. Name snippets so they are obviously not runnable recipes (e.g. +`..yaml`), and reference them by their path relative to +`modelopt_recipes/`. + +## Per-folder READMEs + +Each `/` folder may contain a short `README.md` describing exactly what is +checkpoint-specific — which layers deviate, the calibration used, and the +reference checkpoint it mirrors — so reviewers and users don't have to diff the +YAML against the generic presets to see the intent. diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 78dcd7bdea8..1ba6f4405dd 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -3,10 +3,9 @@ This doc walks through the **PTQ quantization schemes** in two parts: the model-agnostic recipes under [`general/ptq/`](general/ptq/) (the recommended starting point for any model), and then the -[model-specific recipes](#model-specific-recipes-huggingface) under -`huggingface/` — per-`model_type` folders plus the -`models///` tier — comparing each to its general -baseline and explaining why it deviates. +[model-specific recipes](#model-specific-recipes) — per-`model_type` folders +under `huggingface/` plus the checkpoint-mirror `models///` +tier — comparing each to its general baseline and explaining why it deviates. --- @@ -227,7 +226,7 @@ These can also be **stacked** when a single method isn't enough — e.g. `mse` + --- -## Model-specific recipes (`huggingface/`) +## Model-specific recipes The general recipes above are **model-agnostic**: they select layers by wildcard (`*mlp*`, `*self_attn*`, `*[kv]_bmm_quantizer`) and lean on the shared @@ -297,7 +296,7 @@ general recipes never enable output quantizers, and the pattern must stay scoped to GEMM outputs — a `DynamicQuantize` on non-GEMM outputs (embedding lookup, pooling) fails to compile in TensorRT. -A lighter case: **`step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`** is close to +A lighter case: **`models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`** is close to `general/ptq/nvfp4_mlp_only` (NVFP4 on MoE/MLP weights+inputs, FP8 KV) but pinned to one released checkpoint and carrying instance-specific disables (`share_expert`, `moe.gate`, the conv1d branches). From 001e93d7511c6261b4ecea2b2604713a80fc79fd Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Thu, 20 Aug 2026 01:24:26 +0000 Subject: [PATCH 3/4] Rename Step 3.5 Flash recipe to its stepfun-ai/Step-3.5-Flash hub path Key the checkpoint by its canonical Hugging Face repo id (https://huggingface.co/stepfun-ai/Step-3.5-Flash): rename the org folder step3p5 -> stepfun-ai and the model folder Step3.5-Flash -> Step-3.5-Flash so the on-disk path mirrors the model-hub path exactly, per the models// convention. Update the recipe-path references in docs/source/guides/10_recipes.rst, modelopt_recipes/ptq.md, the unreleased 0.47 CHANGELOG entry, and the recipe-docs test docstring. Released CHANGELOG entries keep their historical paths. The Step3p5* architecture class names (trust_remote_code modeling code) are unaffected. Signed-off-by: Shengliang Xu --- CHANGELOG.rst | 2 +- docs/source/guides/10_recipes.rst | 2 +- .../Step-3.5-Flash}/ptq/nvfp4-mlp-only.yaml | 0 modelopt_recipes/ptq.md | 2 +- tests/unit/recipe/test_recipe_docs.py | 5 ++--- 5 files changed, 5 insertions(+), 6 deletions(-) rename modelopt_recipes/models/{step3p5/Step3.5-Flash => stepfun-ai/Step-3.5-Flash}/ptq/nvfp4-mlp-only.yaml (100%) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 16dfc660b95..61064269e8b 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -24,7 +24,7 @@ Changelog **Backward Breaking Changes** -- Move the checkpoint-mirror recipe tier from ``huggingface/models///`` to the top-level ``models///`` (and the Step 3.5 Flash recipe from ``huggingface/step3p5/Step3.5-Flash/ptq/`` to ``models/step3p5/Step3.5-Flash/ptq/``), so instance-specific recipes sit beside ``general/`` and ``huggingface/`` instead of under the per-``model_type`` tree. Drop the ``huggingface/`` prefix in any saved ``--recipe`` paths. +- Move the checkpoint-mirror recipe tier from ``huggingface/models///`` to the top-level ``models///`` (and the Step 3.5 Flash recipe from ``huggingface/step3p5/Step3.5-Flash/ptq/`` to ``models/stepfun-ai/Step-3.5-Flash/ptq/``), so instance-specific recipes sit beside ``general/`` and ``huggingface/`` instead of under the per-``model_type`` tree. Drop the ``huggingface/`` prefix in any saved ``--recipe`` paths. - Move the Mistral Medium 3.5 checkpoint-mirror recipe from ``models/nvidia/Mistral-Medium-3.5-128B-NVFP4/ptq/nvfp4-max-calib`` to ``models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib``, keying it by the canonical Hugging Face base model. Update any saved ``--recipe`` paths to the new location. - Remove the ``--auto_quantize_bits``, ``--auto_quantize_method``, ``--auto_quantize_score_size``, ``--auto_quantize_cost_model`` and ``--auto_quantize_active_moe_expert_ratio`` flags from ``examples/hf_ptq`` (deprecated in 0.46). Use an AutoQuantize ``--recipe`` from ``modelopt_recipes/general/auto_quantize/`` instead. Those recipes now also splice in the shared base ``cost_excluded_layers`` unit, which the removed CLI applied unconditionally, so a VL model keeps its vision tower and MTP layers out of the effective-bits denominator. On a VL model this changes the per-layer cost weights, so an existing ``--auto_quantize_checkpoint`` from an earlier release is rejected with "Use a different checkpoint path"; delete or repoint it to re-run the search. - Remove the ``examples/llm_ptq`` symlink and the ``examples/vlm_ptq`` forwarder (both deprecated in 0.46). Use ``examples/hf_ptq``, passing ``--vlm`` for vision-language models. diff --git a/docs/source/guides/10_recipes.rst b/docs/source/guides/10_recipes.rst index cf511014adc..c1ab515ce52 100644 --- a/docs/source/guides/10_recipes.rst +++ b/docs/source/guides/10_recipes.rst @@ -534,7 +534,7 @@ for the layout conventions and recipe-lookup order. * - Recipe path - Description - * - ``models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`` + * - ``models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only`` - NVFP4 MLP-only for Step 3.5 Flash MoE model * - ``huggingface/minimax_m3_vl/ptq/mxfp8_nvfp4_experts`` - MXFP8 language-model base with MSE-calibrated NVFP4 routed experts for MiniMax-M3 diff --git a/modelopt_recipes/models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml b/modelopt_recipes/models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only.yaml similarity index 100% rename from modelopt_recipes/models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml rename to modelopt_recipes/models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only.yaml diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 1ba6f4405dd..1740974f923 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -296,7 +296,7 @@ general recipes never enable output quantizers, and the pattern must stay scoped to GEMM outputs — a `DynamicQuantize` on non-GEMM outputs (embedding lookup, pooling) fails to compile in TensorRT. -A lighter case: **`models/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only`** is close to +A lighter case: **`models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only`** is close to `general/ptq/nvfp4_mlp_only` (NVFP4 on MoE/MLP weights+inputs, FP8 KV) but pinned to one released checkpoint and carrying instance-specific disables (`share_expert`, `moe.gate`, the conv1d branches). diff --git a/tests/unit/recipe/test_recipe_docs.py b/tests/unit/recipe/test_recipe_docs.py index 43a3558432c..e2847d7f3d5 100644 --- a/tests/unit/recipe/test_recipe_docs.py +++ b/tests/unit/recipe/test_recipe_docs.py @@ -88,9 +88,8 @@ def test_every_model_specific_ptq_dir_is_mentioned(): """Every model dir under huggingface/ or models/ with PTQ recipes must appear in ptq.md. The identifier checked is the directory containing the ptq/ folder — the - HF model_type (e.g. ``gemma4``), a nested checkpoint dir (e.g. - ``Step3.5-Flash``), or a models// leaf (e.g. - ``Nemotron-3-Nano-4B``). + HF ``model_type`` (e.g. ``gemma4``) or a models// leaf + (e.g. ``Step-3.5-Flash``, ``Nemotron-3-Nano-4B-BF16``). """ doc = _ptq_md_text() search_dirs = (RECIPES_DIR / "huggingface", RECIPES_DIR / "models") From 0f334292c3a44a6ea9ead7fb5ef0ee36f8185e5b Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Thu, 20 Aug 2026 16:46:13 +0000 Subject: [PATCH 4/4] Address PR review: polish docs and broaden models/ smoke test - CHANGELOG: scope the migration note to the moved checkpoint recipes (the per-model_type recipes under huggingface/ did not move). [CodeRabbit] - 10_recipes.rst: add the models/ branch to the repo-layout tree. [CodeRabbit] - test_loader.py: smoke-test all seven models/ checkpoint recipes, not just the Mistral one. [CodeRabbit] - ptq.md: prefix the bare Nemotron checkpoint-mirror bullets with models/nvidia/ so every path is copy-pasteable. [Claude] - huggingface/README.md: retitle to "Architecture-specific recipes" now that it covers only per-model_type recipes. [Claude] Signed-off-by: Shengliang Xu --- CHANGELOG.rst | 2 +- docs/source/guides/10_recipes.rst | 6 +++++- modelopt_recipes/huggingface/README.md | 2 +- modelopt_recipes/ptq.md | 8 ++++---- tests/unit/recipe/test_loader.py | 6 ++++++ 5 files changed, 17 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 61064269e8b..304faa1f14b 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -24,7 +24,7 @@ Changelog **Backward Breaking Changes** -- Move the checkpoint-mirror recipe tier from ``huggingface/models///`` to the top-level ``models///`` (and the Step 3.5 Flash recipe from ``huggingface/step3p5/Step3.5-Flash/ptq/`` to ``models/stepfun-ai/Step-3.5-Flash/ptq/``), so instance-specific recipes sit beside ``general/`` and ``huggingface/`` instead of under the per-``model_type`` tree. Drop the ``huggingface/`` prefix in any saved ``--recipe`` paths. +- Move the checkpoint-mirror recipe tier from ``huggingface/models///`` to the top-level ``models///`` (and the Step 3.5 Flash recipe from ``huggingface/step3p5/Step3.5-Flash/ptq/`` to ``models/stepfun-ai/Step-3.5-Flash/ptq/``), so instance-specific recipes sit beside ``general/`` and ``huggingface/`` instead of under the per-``model_type`` tree. Update any saved ``--recipe`` paths for these checkpoint recipes accordingly; the per-``model_type`` recipes under ``huggingface/`` are unchanged. - Move the Mistral Medium 3.5 checkpoint-mirror recipe from ``models/nvidia/Mistral-Medium-3.5-128B-NVFP4/ptq/nvfp4-max-calib`` to ``models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib``, keying it by the canonical Hugging Face base model. Update any saved ``--recipe`` paths to the new location. - Remove the ``--auto_quantize_bits``, ``--auto_quantize_method``, ``--auto_quantize_score_size``, ``--auto_quantize_cost_model`` and ``--auto_quantize_active_moe_expert_ratio`` flags from ``examples/hf_ptq`` (deprecated in 0.46). Use an AutoQuantize ``--recipe`` from ``modelopt_recipes/general/auto_quantize/`` instead. Those recipes now also splice in the shared base ``cost_excluded_layers`` unit, which the removed CLI applied unconditionally, so a VL model keeps its vision tower and MTP layers out of the effective-bits denominator. On a VL model this changes the per-layer cost weights, so an existing ``--auto_quantize_checkpoint`` from an earlier release is rejected with "Use a different checkpoint path"; delete or repoint it to re-run the search. - Remove the ``examples/llm_ptq`` symlink and the ``examples/vlm_ptq`` forwarder (both deprecated in 0.46). Use ``examples/hf_ptq``, passing ``--vlm`` for vision-language models. diff --git a/docs/source/guides/10_recipes.rst b/docs/source/guides/10_recipes.rst index c1ab515ce52..4ee7a413848 100644 --- a/docs/source/guides/10_recipes.rst +++ b/docs/source/guides/10_recipes.rst @@ -689,10 +689,14 @@ The ``modelopt_recipes/`` package is organized as follows: | +-- nvfp4_omlp_only-kv_fp8_cast.yaml | +-- nvfp4_omlp_only-kv_fp8.yaml | +-- nvfp4_weight_only-kv_fp8_cast.yaml - +-- huggingface/ # Model-specific recipes + +-- huggingface/ # Architecture-specific recipes (by model_type) | +-- / # see modelopt_recipes/huggingface/README.md | +-- / | +-- .yaml + +-- models/ # Checkpoint-specific recipes (by model-hub path) + | +-- // # see modelopt_recipes/models/README.md + | +-- / + | +-- .yaml +-- configs/ # Reusable config snippets (imported via $import) +-- numerics/ # Numeric format definitions | +-- fp8.yaml diff --git a/modelopt_recipes/huggingface/README.md b/modelopt_recipes/huggingface/README.md index 71d270c5c4b..054493431e1 100644 --- a/modelopt_recipes/huggingface/README.md +++ b/modelopt_recipes/huggingface/README.md @@ -1,4 +1,4 @@ -# Model-specific recipes for Hugging Face models +# Architecture-specific recipes for Hugging Face models This folder holds model-optimization recipes (e.g. PTQ recipes) whose behavior is tied to a **specific Hugging Face `model_type` (architecture)** — one diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 1740974f923..eda4cb47768 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -350,7 +350,7 @@ checkpoint's** quant config verbatim: `nvidia/Mistral-Medium-3.5-128B-NVFP4`: decoder MLP layers 4–86 use NVFP4 W4A4, edge MLP layers 0–3 and 87 use FP8 W8A8, and all attention projections and the KV cache use FP8. It uses max calibration. -- **`Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse`** mirrors +- **`models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse`** mirrors `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` exactly — a hybrid **Mamba-MoE** with a hand-mapped, **per-component** precision scheme: - MoE routed experts → NVFP4 W4A4, `group_size 16`, **static** weight scales @@ -361,17 +361,17 @@ checkpoint's** quant config verbatim: `nvfp4-mse.yaml` uses MSE calibration with an FP8-scale sweep (matches the release); `nvfp4-max-calib.yaml` is the identical layer map under plain `max` calibration, kept for comparison. -- **`Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6`** follows the same Super-style +- **`models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6`** follows the same Super-style component map (routed experts NVFP4 W4A4 block-16; shared experts + Mamba `in/out_proj` + KV cache FP8; everything else BF16), but the routed-expert weights use **Four-over-Six (4/6)** NVFP4: an MSE search picks each weight's amax multiplier from `[1.0, 1.5]` (M=6 vs. M=4). Activations stay dynamic NVFP4 (not MSE-calibrated). -- **`Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6`** applies +- **`models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6`** applies Four-over-Six NVFP4 W4A16 to routed experts, shared experts, and the language model head; Mamba `in/out_proj` weights and inputs plus the KV cache use FP8, while attention remains BF16. -- **`Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16`** mirrors the GGUF **Q4_K_M** bit +- **`models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16`** mirrors the GGUF **Q4_K_M** bit allocation of the Nemotron-H hybrid, mapped onto NVFP4/FP8 **per layer**: Q4_K/Q5_0 linears → NVFP4 W4A4 (attention q/k/v/o kept uniform so export can fuse them), the Q6_K MLP `down_proj` layers → FP8 W8A8, embeddings → NVFP4 diff --git a/tests/unit/recipe/test_loader.py b/tests/unit/recipe/test_loader.py index cc3a2ff1f7c..cd791601ac5 100644 --- a/tests/unit/recipe/test_loader.py +++ b/tests/unit/recipe/test_loader.py @@ -170,6 +170,12 @@ def test_load_recipe_builtin_description(): "general/ptq/nvfp4_experts_only-kv_fp8_cast", "general/ptq/nvfp4_experts_only-kv_fp8_layerwise", "models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib", + "models/nvidia/Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16", + "models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-max-calib", + "models/nvidia/Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse", + "models/nvidia/Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6", + "models/nvidia/Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6", + "models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only", "general/ptq/nvfp4_mlp_only-kv_fp8", "general/ptq/nvfp4_mlp_only-novit-kv_fp8", "general/ptq/nvfp4_mlp_only-kv_fp8_cast",