Sync with Microsoft ONNX Runtime - 18072026#1205
Open
ai-fw-intg wants to merge 19 commits into
Open
Conversation
…icrosoft#29475) For per-axis QuantizeLinear/DequantizeLinear, the scale and zero_point tensors are 1-D. WebNN's quantizeLinear/dequantizeLinear requires them to have the same rank as the input (it broadcasts per-dimension and does not left-pad ranks). The reshape was skipped when the quantization axis was the last dimension, leaving a rank-1 scale against a higher-rank input and triggering "the rank of scale is not equal to the rank of input". Drop the `axis != input_rank - 1` guard so the reshape runs for all per-axis cases; the existing math already yields the correct shape (e.g. input [512, 1024], scale [1024], axis 1 -> scale [1, 1024]).
…e via cpuinfo (microsoft#29621) On Linux/aarch64, `PosixEnv::GetL2CacheSize()` returns `sysconf(_SC_LEVEL2_CACHE_SIZE)`, but glibc's aarch64 sysconf backend does not implement the cache-size queries and returns 0. That makes the `l2_cache_size_ > 0` gate in MultiHeadAttention always false, so the CPU FlashAttention path is silently disabled and the kernel falls back to materializing the full attention score tensor (multi-GiB at long sequence lengths, and OOM on small ARM instances). GroupQueryAttention divides by the same 0 and ends up with a degenerate tile size of 1. This queries cpuinfo for the L2 cache size before falling back to sysconf/sysctl, mirroring the existing `cpuinfo_available_` usage in the same file for physical-core count and thread affinity. cpuinfo reads sizes from sysfs cacheinfo and works on aarch64. The change is guarded by `ORT_USE_CPUINFO` and only returns early on a positive result, so x86-64 (working sysconf) and Windows (separate `WindowsEnv` implementation) are unaffected. Fixes microsoft#29613.
### Description This updates the Windows BinSkim-compliant build flags so `/Qspectre` builds also link against the MSVC Spectre-mitigated CRT/STL static libraries. `/Qspectre` only affects ONNX Runtime's own object files; BinSkim BA2024 can still report violations when the default non-Spectre `libcmt.lib`, `libcpmt.lib`, or `libvcruntime.lib` are linked into the final binary. ### Motivation and Context Release validation reported BinSkim BA2024 (`EnableSpectreMitigations`) warnings for `onnxruntime.dll` even when ORT was built with `--use_binskim_compliant_compile_flags`. The warning identified MSVC runtime and STL static libraries as the non-mitigated modules. This change makes the build option select the Spectre-mitigated MSVC library directory when it is available from the Visual Studio toolset. ### Key Changes - Adds `get_msvc_spectre_lib_dir()` to locate `%VCToolsInstallDir%\lib\spectre\<arch>` for the target Windows architecture. - Appends a quoted `/LIBPATH:<spectre-lib-dir>` linker flag whenever Windows BinSkim flags enable `/Qspectre` and AddressSanitizer is not enabled. - Emits a warning when the Spectre-mitigated MSVC libraries cannot be found, with guidance to install the Visual Studio "C++ Spectre-mitigated libs" component. - Preserves the existing ASAN behavior because ASAN libraries do not have Spectre-mitigated variants. ### Testing - `python3 -m ruff check tools/ci_build/build.py` - `python3 -m ruff format --check tools/ci_build/build.py` `lintrunner -a tools/ci_build/build.py` was also attempted. It found the repository config and applied no file changes, but the local environment could not execute the Ruff lintrunner adapters because `python` is not available on PATH; the direct `python3 -m ruff` checks above passed. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
…#29255) ### Description `GridSample` sizes its output from the grid, so an input with a zero-size spatial dimension skips the empty-output early return and reaches interpolation with an invalid spatial extent. This change validates that every input spatial dimension is non-empty in both the CPU and CUDA kernels, returning a clear error for such malformed inputs. ### Changes - CPU `GridSample::Compute`: reject any zero-size input spatial dimension with a descriptive error. - CUDA `GridSample::ComputeInternal`: same validation, layout-aware for NCHW and NHWC, covering both 4-D (H, W) and 5-D (D, H, W) inputs. Messages are consistent across both EPs. - Add an expect-failure test (zero-size spatial dim, `padding_mode=border`) scoped to the CPU and CUDA execution providers, which are the kernels that carry the guard. ### Motivation Improves input validation and error diagnostics for malformed `GridSample` inputs. Covers both the CPU and CUDA execution providers; no behavior change for valid inputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…18 / MLAS path) (microsoft#29629) ### Summary Fixes wrong `AveragePool` output on the CPU EP when `ceil_mode=1` **and** `count_include_pad=1` for **opset 7–18** (the float MLAS path). This is the direct fix for the CPU-EP repro in pytorch/pytorch#183528. ### Root cause `Pool<float, AveragePool>::Compute` routes float `AveragePool` opset 7–18 to MLAS. With `count_include_pad=1`, MLAS divides every window by the **full kernel size**. Under `ceil_mode=1` the output grid gains a trailing window whose extent runs past `input + pad_tail` (phantom cells). MLAS's full-kernel divisor counts those phantom cells, producing an average that is too small on the boundary windows. The opset-19 reference functor (`AveragePool{1,2,3}DTask` in `pool_functors.h`) already handles this correctly: it clamps the window end to `input + pad_tail` and divides by `1 + (end - start - 1)/dilation`, excluding the phantom cells. ### Fix Dispatch-fallback, **zero MLAS edits**: 1. Extract the opset-19 reference average-pool loop into a shared free function `ComputeAveragePoolReference<T>(context, pool_attrs, tp)` (reads strides from `PoolAttributes`, `p=0`), and make `AveragePoolV19<T>::Compute` a thin wrapper over it — one canonical loop, no copy-paste drift. 2. In `Pool<float, AveragePool>::Compute`, route **only** the buggy combo `ceil_mode == 1 && count_include_pad && !global_pooling` to the reference loop. Every other case (`ceil_mode=0`, `count_include_pad=0`/exclude-pad, global pooling, all MaxPool) keeps the fast MLAS path unchanged. `global_pooling` is excluded because it has no ceil/pads (already correct) and leaves `strides[]` unpopulated; `ComputeAveragePoolReference` `ORT_ENFORCE`s that precondition. ### Perf Zero impact on the common path. The reference loop runs only for `ceil_mode=1 && count_include_pad=1` (rare); the guard is a cheap early branch. ### Relationship to microsoft#16752 ORT PR microsoft#16752 fixed this behavior for **opset ≥ 19** only (via the new v19 reference functor). Opset 7–18 float still went through MLAS and remained wrong. This PR closes that gap by reusing the same already-correct functor for the 7–18 dispatch fallback. ### Tests Added to `pool_op_test.cc` (CPU-focused; GPU/other EPs excluded so CI stays green): - `AveragePool_18_ceil_count_include_pad_1d` — opset-18 clone of the existing v19 test; the direct #183528 repro. - `AveragePool_18_ceil_count_include_pad_2d` — the `arange(1,17).reshape(1,1,4,4)`, k3/s2/pad1 case → `[1.556, 3.333, 2.0, 6.333, 11.0, 6.0, 4.5, 7.5, 4.0]`. - `AveragePool_18_ceil_count_include_pad_3d` — exercises the 3D functor path. - `AveragePool_18_ceil_count_exclude_pad_2d` — `count_include_pad=0` no-regression guard (stays on MLAS). Expected values are derived from the CPU v19 reference (equivalently, hand-computed per the ONNX spec). Full `PoolTest` suite passes locally. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…#29687) This pull request updates the build and CI configuration for CUDA-related workflows and the main CMake options. The main changes are the addition of new CMake build flags to enable CUDA quantization preprocessing, improved formatting for build flags, and a change in the default for the CUDA quant preprocess build option. These updates improve clarity, make it easier to customize builds, and ensure that the CUDA quant preprocess module is only built when explicitly requested. **Build configuration changes:** * Added the `--cmake_extra_defines onnxruntime_BUILD_CUDA_QUANT_PREPROCESS=ON` flag to the CUDA build jobs in `.github/workflows/linux_cuda_ci.yml` and `.github/workflows/linux_cuda_plugin_ci.yml`, enabling the CUDA quantization preprocessing module during CI builds. [[1]](diffhunk://#diff-04806013a5e7991ba5606145885d9b0fcd99a7df1f3bb96a2d38fc724ccd9b2aL32-R46) [[2]](diffhunk://#diff-04806013a5e7991ba5606145885d9b0fcd99a7df1f3bb96a2d38fc724ccd9b2aL114-R138) [[3]](diffhunk://#diff-64cd92765a9461e73a80c6b0401fe1960170834725a7e8a2ea153aff6d8f8388R45) * Added the `--cmake_extra_defines onnxruntime_QUICK_BUILD=ON` and `--cmake_extra_defines onnxruntime_USE_FPA_INTB_GEMM=OFF` flags to the CUDA no-cudnn build job for faster builds and to disable a specific GEMM implementation. [[1]](diffhunk://#diff-4e310144ab53bd9b6e48c7ceba29ad2c310724645278a28b85f7ba3a453c4980L35-R47) [[2]](diffhunk://#diff-04806013a5e7991ba5606145885d9b0fcd99a7df1f3bb96a2d38fc724ccd9b2aL114-R138) **Formatting and maintainability:** * Reformatted long `extra_build_flags` strings in workflow YAML files to use multi-line lists for improved readability and easier maintenance. [[1]](diffhunk://#diff-04806013a5e7991ba5606145885d9b0fcd99a7df1f3bb96a2d38fc724ccd9b2aL32-R46) [[2]](diffhunk://#diff-04806013a5e7991ba5606145885d9b0fcd99a7df1f3bb96a2d38fc724ccd9b2aL114-R138) [[3]](diffhunk://#diff-4e310144ab53bd9b6e48c7ceba29ad2c310724645278a28b85f7ba3a453c4980L35-R47) **CMake option default change:** * Changed the default value of the `onnxruntime_BUILD_CUDA_QUANT_PREPROCESS` CMake option from `ON` to `OFF` in `cmake/CMakeLists.txt`, so the CUDA quantization preprocessing module is only built when explicitly enabled.
…ft#29658) ### Description When ONNX Runtime is built with the CUDA execution provider as a plugin (`onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`), the EP-API op-kernel adapter (`ep::adapter::KernelImpl::PrePackWeightImpl`) received a valid `OrtAllocator*` from the framework but discarded it and forwarded a **null** `AllocatorPtr{}` into the wrapped kernel's `PrePack()`. Any CUDA kernel that pre-packs a constant weight (e.g. `MatMulNBits`, `Conv`, `GroupQueryAttention`, quantized MoE) then allocated scratch through that null allocator and crashed during session initialization: ``` IAllocator::ValidateAllocator(const T&) [with T = std::shared_ptr<onnxruntime::IAllocator>] allocator != nullptr was false ``` This surfaced end to end as an ONNX Runtime GenAI `og.Model(...)` failure on a gpt-oss-20b (`MatMulNBits`) model when the CUDA EP was loaded as a plugin: the trivial init session succeeded, but the first real model session crashed while pre-packing quantized weights. ### Key Changes | File | Change | |---|---| | `include/onnxruntime/ep/adapter/op_kernel.h` | `PrePackWeightImpl` now wraps the incoming `OrtAllocator*` and forwards a valid `AllocatorPtr` to `OpKernel::PrePack` instead of a null `AllocatorPtr{}`. | | `include/onnxruntime/ep/adapter/allocator.h` | Add a **non-owning** `IAllocatorWrappingOrtAllocator(OrtAllocator*)` constructor. The framework owns the pre-pack allocator, so the wrapper must not take ownership (an owning `Ort::Allocator` would release it on destruction). Calls now go through the raw `OrtAllocator*` function pointers directly, preserving the `Reserve`→`Alloc` (version ≥ 18) and `GetStats`/`AllocOnStream` (version ≥ 23) fallbacks. | | `onnxruntime/test/python/transformers/test_cuda_plugin_ep.py` | Add `test_registration_matmul_nbits_prepack`: builds a fp16 `MatMulNBits` model with a runtime-prepacked (`weight_prepacked=0`) quantized weight, so weight pre-packing (`MatMulNBits::PrePack_B` → `IAllocator::MakeUniquePtr(alloc, ...)`) runs during session creation. This crashed before the fix and now passes. | ### Motivation and Context The pre-pack allocator is provided and owned by the framework for the duration of the `PrePack` call. The legacy (in-tree) CUDA EP received it correctly; only the plugin op-kernel adapter dropped it. The non-owning wrapper matches the lifetime contract used elsewhere in the adapter (e.g. `KernelInfoGetAllocator`) and keeps the CUDA-EP-as-plugin build behaviorally identical to the in-tree EP. ### Testing - New `test_registration_matmul_nbits_prepack` in `test_cuda_plugin_ep.py` passes on a CUDA-EP-as-plugin build (`ORT_TEST_CUDA_PLUGIN_EP=1`) and skips gracefully when the device lacks fpA_intB GEMM support. It re-raises (fails) if the `allocator != nullptr` assertion recurs. - Verified the model that originally reproduced the crash now creates and runs end to end through the CUDA plugin EP (gpt-oss-20b, `MatMulNBits`), including a 100-sample MMLU sanity run. - Existing `test_cuda_plugin_ep.py` registration tests continue to pass.
Enable reshape fusion for models with symbolic (dynamic) dimensions, covering patterns found in MobileNetV2, vision encoders, and SLMs. Changes: - Match_Shape: allow dimension matching by symbolic dim_param, not just static values. Needed for SLMs (e.g. Qwen) where `position_ids` takes `Shape` from `input_ids` — same `dim_param` names, no static values. - Match_Shape: allow matching when only the gathered dimension agrees between `Shape` source and reshape root. Needed for vision encoders (e.g. moondream2) where `Shape` is taken from pre-projection tensor but `Reshape` operates on post-projection output. - Match_Shape: add `GlobalAveragePool` passthrough — accept `Shape` from pre-pool input when gathering batch dim. Needed for MobileNetV2. - ReshapeHelper: handle zero-dim shapes (`allow_zero=true`) by computing unknown dim from non-zero dimension product only. Needed for MobileNetV2 reshape fusion output. - Add corresponding unit tests.
### Description This adds native (non-LUT) 2-bit weight CompInt8 kernels to MLAS for AVX2 and AVX-VNNI, BlkLen 32/64/128, modeled on the AVX-512 kernels from microsoft#29064 and the ARM64 kernels from microsoft#29466. On these hosts a W2 MatMulNBits currently has the LUT path (opt-in, and only when N is a multiple of 128) or the fp32 dequant + SGEMM fallback. The packed layout is untouched. The block-group packer, scale layout and BlkSum machinery in sqnbitgemm_kernel_avx512_2bit.{h,cpp} are portable scalar C++ and the ARM64 kernels already reuse them, so this only adds the 256-bit compute kernels and the dispatch wiring. The four W2 entries are populated in MlasSQNBitGemmDispatchAvx2 and MlasSQNBitGemmDispatchAvx2vnni with BlkLen routing forwarders like the AVX-512 ones, and A quantization reuses the QuantizeARow_CompInt8_avx2 those tables already register. No cmake changes (the kernels are header-only and the avx2 source list already carries -mavxvnni where the compiler supports it) and no operator changes. Per-node routing: | Host | LUT mode | before | after | |---|---|---|---| | AVX2 / AVX-VNNI | off (default) | fp32 dequant + SGEMM | native W2 kernel | | AVX2 / AVX-VNNI | on, N % 128 == 0 | LUT kernel | LUT kernel (unchanged) | | AVX2 / AVX-VNNI | on, N % 128 != 0 | fp32 dequant + SGEMM | native W2 kernel | Tile shapes are per BlkLen and I picked them by measuring. BlkLen 32 and 64 use an R2xC4 main tile (one B block-group load and unpack shared across two rows) with an R1xC4 tail for an odd trailing row. BlkLen 128 stays R1xC4: the R2 variant measured 3 to 5% slower there, which tracks with register pressure, since a 128-byte group needs four B registers live plus eight accumulators and that does not fit sixteen YMM. M=1 always takes the R1 path, so decode is unaffected by the tiling choice either way. Hosts with AVX2 but no AVX-VNNI use the vpmaddubsw + vpmaddwd fallback, guarded the same way as the existing int8 kernels. Testing: - new direct-kernel tests in test_sqnbitgemm_2bit_gemm.cpp mirroring the AVX-512 set, four per BlkLen, with and without zero points. The non-VNNI variants guard on Avx2Supported_ so they also run on the AVX-512 CI hosts and cover the maddubs path there; the VNNI variants gate on the AVX2-VNNI dispatch being the active one. - the existing MatMul2Bits operator tests exercise the new path automatically on AVX2 hosts at accuracy_level 4. - validated the kernels against a float64 reference over 120+ cases (N tails, K tails, odd M, both dot paths) on packing produced by the production 3-call pack sequence. ### Motivation and Context microsoft#29064 closed this gap on AVX-512 and microsoft#29466 on ARM64, but AVX2/AVX-VNNI without AVX-512 covers most client x86 (Alder Lake through Arrow Lake, plus Zen 1-3 on the plain AVX2 path) and those hosts still land on dequant + SGEMM by default. Kernel-level numbers from a Core Ultra 5 225 (Arrow Lake, AVX2 + AVX-VNNI, no AVX-512), single thread, interleaved arms, min of 9 rounds on rdtsc: | BlkLen | cycles/MAC at prefill (shipped tile) | R2xC4 vs R1xC4 at prefill | |---|---|---| | 32 | ~0.048 | R2 5 to 6% faster (shipped) | | 64 | ~0.031 | R2 11 to 15% faster (shipped) | | 128 | ~0.027 | R2 3 to 5% slower (kept R1) | These are tile-level microbenchmarks, not end-to-end model numbers; my dev box has no MSVC so my validation is kernel-level and the MSVC build rides on CI. Happy to run whatever end-to-end comparison you want on top of this, and happy to restructure the tiles if you would rather keep all three BlkLens on the same shape.
…t naming (microsoft#28896) ### Description This PR adds Windows ARM64 support to CUDA plugin packaging and updates related build/packaging logic for correctness and consistency across architectures. In response to review feedback, it also: - Corrects CMake comments to match actual Windows ARM64 CUDA toolkit search behavior. - Prioritizes architecture-specific cuDNN DLL search paths before generic fallback paths. - Aligns Windows ARM64 Python artifact naming with the ARM64 CUDA toolkit version. - Extends packaging metadata with per-platform CUDA version fields (including win-arm64) and updates metadata-reading template validation/exports accordingly. ### Motivation and Context Windows ARM64 CUDA plugin packaging requires architecture-aware handling for toolkit/cuDNN discovery and artifact metadata. These updates prevent cross-architecture ambiguity (especially for CUDA 13.x win-arm64 vs x64), improve downstream artifact selection reliability, and keep metadata semantics consistent with produced artifacts. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
- Add an Intel-specific F16 MatMul path built on the 8x16x16
subgroup-matrix config (Xe2/Xe3). A and B are loaded directly from
global memory with no prepacking, and results are written through
workgroup scratch with bounds-checked stores, so M and N may be any size
while only K must be a multiple of 16. The impl is created in
MatMul::PrePackInternal and dispatched from Compute when the device
reports the required config.
- The tile shape and split-K factor are chosen per call rather than
fixed: TileM in {8,16,32,64}, TileN in {16,32,64}, and split-K to top up
occupancy, sized to the device's resident subgroups. The WGSL kernel is
parameterized by sg_mat_count_m/n and split_k, emitting the needed
subgroup-matrix accumulators via generation-time guards.
- An optional offline-tuned config table (generated from an autotuner
sweep) can override the heuristic; selection precedence is tuned table >
heuristic.
- Adds MatMul subgroup-matrix unit tests covering tile selections and
partial/large non-aligned dims.
…crosoft#28786) ### Description Adds MLAS half-precision GEMM and convolution support for Arm64 KleidiAI paths. This change: - Adds public MLAS HalfGemm backend-native packed-B APIs. - Adds public MLAS HalfConv prepare, execute, and packed weights+bias APIs. - Wires KleidiAI HalfGemm and HalfConv overrides into the MLAS platform dispatch. - Adds SME/SME2 FP16 HGEMM and FP16 IMATMUL ukernel selection. - Adds MLAS unit coverage for packed-B behavior, selector opt-out, zero-K handling, HalfConv prepare behaviour, and varied HalfGemm shapes/bias cases. ### Motivation and Context This is the second MR in the FP16 split and introduces the MLAS API and Arm64 backend plumbing needed for accelerated FP16 CPU kernels. The later CPU operator changes can use these MLAS HalfGemm and HalfConv entry points without carrying KleidiAI-specific details at the operator layer. The backend selector support also preserves an explicit fallback path when KleidiAI should not be used for a given call. Note: This PR is the second offshoot from [28487](microsoft#28487) --------- Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com> Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> Co-authored-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com>
### Description This patch updates Dawn to the latest version. ### Motivation and Context WebGPU Immediate Data and "subgroup-size-control" extension will be supported after the upgrade of Dawn.
### Description Copies the Java setup step from sibling files `windows_webgpu.yml`, `windows_x64_release_xnnpack.yml`, `windows_x64_release_build_x64_release.yml` to `windows_qnn_x64.yml`. This explicitly ensures a compatible Java version, rather than relying on the JDK already installed on the self-hosted runner. Otherwise there may be breakages when switching the runner. ### Motivation and Context I believe this will fix the failures in CI for the Windows x64 QNN CI Pipeline. I'm seeing it fail in both an unrelated PR: microsoft#29728 https://github.com/microsoft/onnxruntime/actions/runs/29509983398/job/87780661649?pr=29728 and in commits to main: https://github.com/microsoft/onnxruntime/actions/runs/29546572913/job/87780090142 https://github.com/microsoft/onnxruntime/actions/runs/29499730505/job/87625355397 This is assuming the chain of causation: 1. microsoft#29731 switches the runner pool 2. The `windows_qnn_x64.yml` pipeline finds JDK 8 already on the runner (previous runner pool had JDK 11) 3. The Spotless Gradle plugin v7.2.1 isn't compatible with JDK 8 4. onnxruntime Java project can't be configured 5. Build fails This new workflow action hopefully installs a compatible JDK & all will be well. microsoft#29753
…r. (microsoft#29634) ### Description <!-- Describe your changes. --> Fix structure of EpLibraryPlugin::Load so that we call Unload on any error and not just an exception. Currently we directly return on error so the Unload is never hit. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Cleanup failed load fully.
…rosoft#29622) ## Enable fpA_intB GEMM in CUDA builds and add configurable options ### Summary This PR turns the CUDA fpA_intB (weight-only, FP activation × int weight) MatMulNBits path on by default in CUDA builds, replaces the ambiguous `ORT_FPA_INTB_GEMM` bitmask with a simple on/off flag, and adds session-config keys so the path and its autotuning buckets can be controlled per session. It also makes the CUTLASS tactic profiler CUDA-graph safe and configurable. ### Motivation The fpA_intB kernels were previously gated behind `onnxruntime_USE_FPA_INTB_GEMM=OFF` and an `ORT_FPA_INTB_GEMM` integer bitmask (`0x01=all`, `0x02=GEMV`, `0x04=int4`, `0x08=int8`). The bitmask was ambiguous (e.g. `6` could read as "int4 + GEMV" or "GEMV for int4 and int8") and the GEMM/GEMV kernels actually share one weight layout, so splitting them was never valid. Shipping the kernels by default and exposing a plain enable flag plus per-session config makes the feature usable and tunable without rebuilding. ### Key Changes #### Build enablement | File | Change | |------|--------| | [cmake/CMakeLists.txt](cmake/CMakeLists.txt) | `onnxruntime_USE_FPA_INTB_GEMM` becomes a `cmake_dependent_option`, defaulting **ON** when `onnxruntime_USE_CUDA` is enabled (OFF otherwise). | | [tools/ci_build/github/linux/build_cuda_c_api_package.sh](tools/ci_build/github/linux/build_cuda_c_api_package.sh), [build_linux_python_package.sh](tools/ci_build/github/linux/build_linux_python_package.sh), [build_tensorrt_c_api_package.sh](tools/ci_build/github/linux/build_tensorrt_c_api_package.sh) | Flip packaging builds from `onnxruntime_USE_FPA_INTB_GEMM=OFF` to `ON`. | #### Option simplification (bitmask → boolean) - Removed `kFpAIntBGemmOption_All/Gemv/Int4/Int8` and the old bitmask parsing. - Added `ParseFpAIntBEnabled`: `""` / `"0"` / `"off"` → disabled; any other value → enabled (numeric non-zero still works for back-compat). - The enable flag now only governs nodes **without** prepacked weights. A prepacked weight is already stored in the fpA_intB layout, so the choice was fixed at export time; the constructor forces the path on for prepacked nodes and only `ORT_ENFORCE`s that the shape/hardware actually support it. - GEMV is no longer independently toggleable: it is enabled whenever supported, since GEMM and GEMV share the same weight layout. #### New session-config keys (EP-agnostic, config wins over env) | Config key | Env fallback | Meaning | |------------|--------------|---------| | `ep.cuda.fpa_intb_gemm` | `ORT_FPA_INTB_GEMM` | Enable/disable the fpA_intB path (`0`/`off` vs `1`/`on`). | | `ep.cuda.fpa_intb_profile_m` | `ORT_FPA_INTB_PROFILE_M` | Comma-separated initial profile-M buckets (e.g. `"1,8,64,512"`); empty uses the default bucket set. | These are read by both the built-in CUDA EP and the CUDA plugin EP via `OpKernelInfo::GetConfigOptions()`. #### Profiler: CUDA-graph-safe, in-memory autotuning - Added `getBestConfigOrProfile()` for lazy single-bucket profiling outside CUDA-graph capture; during capture the kernel falls back to a pure lookup (`getBestConfig`) because profiling launches kernels, records/synchronizes events, and allocates scratch — all illegal during capture. - Added configurable profile-M buckets: `ParseProfileMList`, `setProfileMOverride`, `getProfileMBuckets`, plus `kEnvProfileM` and `kDefaultProfileMaxM` (default max M lowered to `2048`). - Clearer error when an M bucket was not profiled before capture ("run a warmup inference outside capture first"). #### Docs - [docs/contrib_ops/cuda/matmul_nbits.md](docs/contrib_ops/cuda/matmul_nbits.md): `ORT_FPA_INTB_GEMM` documented as int/string on/off; clarified prepacked-weight strictness. ### Testing Notes - New: [onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py](onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py) — exercises the prepacked fpA_intB path and the boolean/numeric back-compat values of the enable flag. - To verify locally (CUDA, SM ≥ 75): - Build with CUDA (fpA_intB now defaults ON): `./build.sh --use_cuda ...` - Run: `python -m pytest onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py` - Sanity-check config override: set `ep.cuda.fpa_intb_gemm=0` on a non-prepacked node and confirm the path is skipped; confirm a prepacked node still forces the path on.
…soft#29537) ## Motivation MLAS has hand-optimized QNBitGemm kernels for x86 (AVX2/AVX512), ARM (NEON), and LoongArch, but none for RISC-V — on RISC-V the dispatch was null, so MlasIsQNBitGemmAvailable returned false and MatMulNBits fell back to the generic scalar path, leaving the RISC-V Vector unit idle for the hottest kernel in quantized-LLM inference. This PR adds a native RISC-V Vector (RVV) implementation of the QNBitGemm dispatch so weight-quantized matmul runs on the vector unit. ## What's implemented All compute modes the operator selects among, for rv64gcv: - 4-bit weights × fp32 activations (SQNBIT_CompFp32) — M=1 GEMV + dequant-to-SGEMM - 4-bit weights × int8 activations (SQNBIT_CompInt8) — quantize-A + int8×int4 kernel - 8-bit weights × int8 activations (SQNBIT_CompInt8, BlkSum path) - 4-/8-bit weights × fp16 activations (HQNBIT_CompFp16) — requires the Zvfh extension (rv64gcv_zvfh, MLAS_USE_RVV_ZVFH) plus the packing / quantization / workspace-sizing plumbing each mode needs (including the 3-call PrePack protocol). ## Design notes - The packed-B / block-sum / dequant-buffer layouts are private to this dispatch (produced and consumed only by these kernels), so plain layouts are used — natural for RVV's scalable vectors. - The SQ8 prepack unit test asserts a per-arch B layout (#ifdef ARM64 / #else x86); added a MLAS_TARGET_RISCV64 branch describing the RVV layout, following the existing per-arch pattern. - fp16 kernels accumulate in fp32 (fp16 → vfwcvt → fp32 FMA → fp16) for accuracy. ## Files New: - onnxruntime/core/mlas/lib/riscv64/qnbitgemm_kernel_rvv.cpp — SQ4/SQ8 kernels, packing, dispatch object - onnxruntime/core/mlas/lib/riscv64/hqnbitgemm_kernel_rvv.cpp — fp16 (Zvfh) dequant + GEMM - onnxruntime/test/mlas/unittest/test_qnbitgemm_rvv_fp16.cpp — RISC-V fp16 e2e test (HQ4 + HQ8) Modified: - cmake/onnxruntime_mlas.cmake — add the two sources (RVV / Zvfh source lists + - onnxruntime/core/mlas/lib/mlasi.h — extern decl of the dispatch - onnxruntime/core/mlas/lib/platform.cpp — assign QNBitGemmDispatch on RISC-V - onnxruntime/test/mlas/unittest/test_sq8bitgemm.cpp — RISC-V layout branch in the SQ8 prepack test ## Performance (RVV, VLEN=256, N=256, K=2048) The K-reduction runs at LMUL=4; profiling showed the kernels are widening-multip), so cutting per-chunk instruction overhead is the lever. - SQ4 CompInt8 (4-bit): ~3.4× vs the naive per-sub-block version (2.2 → 7.6 GOP/s) - SQ8 CompInt8 (8-bit): ~1.2× - fp16 GEMM tiled to reuse the B vfwcvt across rows ## Testing Built and run on real RISC-V hardware (K3 board, rv64gcv_zvfh, VLEN=256) via onnxruntime_mlas_test: - SQNBitGemm* : SQ8Bit* : BlockQ4* : BlockQ8* : RvvFp16* → 13,526 / 13,526 pass - Broader suite run passed with 0 failures as far as it ran (Activation → all QGemm); the run was only bounded by the very slow threaded large-GEMM stress tests, unrelated to this change. ## Build Enable fp16 with -Donnxruntime_USE_RVV_ZVFH=ON. The 4-/8-bit int8/fp32 paths build with the existing onnxruntime_USE_RVV.
Implement the MatMulBnb4 quantized matmul operator for the WebGPU execution provider, supporting both FP4 (quant_type=0) and NF4 (quant_type=1) blockwise 4-bit weight dequantization. ### Description <!-- Describe your changes. --> Add WebGPU execution provider support for the MatMulBnb4 contrib operator, enabling blockwise 4-bit quantized matmul (bitsandbytes-style FP4 and NF4) to run on WebGPU. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Models quantized with bitsandbytes 4-bit weights fell back to CPU when running on WebGPU, causing significant slowdowns, and cannot use graph capture feature. Adding a native WebGPU kernel keeps these quantized MatMulBnb4 on the GPU, avoiding costly EP fallback and CPU/GPU data transfers. marqo-fashionSigLIP-text-bnb4: ~30% faster marqo-fashionSigLIP-vision-bnb4: ~20% faster
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
July 17, 2026 20:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.