Conversation
runwangdl
force-pushed
the
gap9-ne16
branch
12 times, most recently
from
April 14, 2026 08:54
4edb011 to
748707a
Compare
Mirrors the Siracusa_w_neureka pattern. NE16Platform extends GAP9Platform with engines=[NE16Engine, GAP9ClusterEngine]; NE16Deployer extends GAP9Deployer (reuses ClDma transformers via GAP9Bindings). New Target: Deeploy/Targets/NE16/ (Platform, Engine, Bindings, Parsers, Tiler, Deployer, Templates, TileConstraints, TopologyOptimizationPasses). The _weightEncode function is ported from pulp-nnx/test/Ne16Weight.py (single CIN_SUBTILE=16 mode, no 1x1 vs 3x3 split). ConvTemplate subtile constants set per ne16_task_defs.h (output 3x3, weight stride bytes PW=16 DW/Dense=144). New test infrastructure: - DeeployTest/deeployRunner_tiled_gap9_w_ne16.py - DeeployTest/test_gap9_ne16_tiled_config.py (PW/DW/Dense RQ Conv) DeeployTest wiring: - testUtils/platformMapping.py: register GAP9_w_NE16 in the platforms list, mapPlatform, setupMemoryPlatform, mapDeployer. - testMVP.py: include GAP9_w_NE16 in the EngineColoringDeployerWrapper branch (without it NE16AdjustWeightMemoryLayoutPass never fires and parsing backtracks to exhaustion). - testUtils/core/execution.py: build the GAP9 SDK 'image' target for GAP9_w_NE16 too (so chip.soc.mram.bin is produced before gvsoc run). - CMakeLists.txt, DeeployTest/CMakeLists.txt: accept GAP9_w_NE16 alongside GAP9 in the platform branches. - TargetLibraries/GAP9/CMakeLists.txt: for GAP9_w_NE16 platform, add_subdirectory on pulp-nnx with USE_NE16=ON and link it into deeploygap9. Fix: Deeploy/Targets/PULPOpen/Templates/FloatGemmTemplate.py referenced an undefined symbol float32_tPtr from Deeploy.AbstractDataTypes; define it locally via PointerClass(float32_t) to unblock the import chain reached by NE16Platform. Verified on gvsoc gap9.evk: PW 1x1 RQ (Regular_RQ): 0/1152 errors, 901917 cycles DW 3x3 RQ (DW_2D_RQ): 0/1280 errors, 27339 cycles (--enable-3x3) Dense 3x3 (Regular_2D_RQ): 0/6372 errors, 244595 cycles (--enable-3x3)
- Add NE16 linear layer kernels, including a topology pass, NE16 templates, parsers, tile constraints, and bindings - The topology pass recognizes NE16-compatible GEMM layers, adjusts the weight layout for the NE16, and converts the requant shift/scale to the NE16 format - The template detects whether the input is signed; if so, it adds a +128 offset to the input during C runtime and compensates via the bias - Add GAP9 SDK-based Dequant/Quant templates using CNN_Copy.c kernels, replacing the generic templates - Add a generic DequantQuantMergePass that folds adjacent Dequant→Quant pairs into identity or RequantShift - Add a GAP9-specific TopologyOptimizer (GAP9Optimizer) to replace PULPOptimizer Bug fixes: - Add output signedness check in QuantChecker - Fix L3 DMA template (add proper casts) and remove the blocking L3 DMA hack - Isolate dory memory functions from other libraries in CMakeLists so they compile with -Og while compute kernels compile with -O3 - Disable PULPAddRequantMergePass due to incorrect pattern matching when Add has multiple consumers Co-authored-by: runwangdl <samanthawangdl@gmail.com>
runwangdl
force-pushed
the
gap9-ne16
branch
2 times, most recently
from
April 14, 2026 10:50
b8087fc to
b3f40e5
Compare
- TargetLibraries/GAP9/CMakeLists.txt: rename CNN_Libraries_NE16 → CNN_Libraries_HWPE (the actual gap9-sdk path); skip SDK CNN_BasicKernels_NE16.c source for GAP9_w_NE16 platform (it uses the pulp-nnx ne16 stack, so the SDK NE16 kernels are not needed). - Deeploy/Targets/NE16/Platform.py: instantiate the GAP9ClusterEngine with a trimmed includeList (no CNN_BasicKernels_NE16.h / ne16_utils.h / CNN_Copy.h) so the generated Network.c does not pull in the SDK NE16 header alongside pulp-nnx ne16_task_defs.h — the NE16_REG_* macros are defined in both and trigger -Werror redefs.
ghcr.io/pulp-platform/deeploy-gap9:* is hosted in pulp-platform's private GitHub Container Registry. Only upstream's self-hosted runners have credentials to pull it; on fork CI runs (ubuntu-latest) the docker pull fails with 'Error response from daemon: denied' and the whole job is reported as failure. Guard the select-env entry of all three gap9 workflows (ci-platform-gap9.yml, -tiled.yml, -w-ne16-tiled.yml) so they SKIP cleanly on forks instead of FAILING. Upstream behaviour is unchanged.
QuantChecker.checkOutputType (added by the NE16-Linear PR) requires opSigned == outputTypeSigned. Existing Generic and PULPOpen bindings only registered the signed-int8 output variant, so any Quant pattern with signed=0 (e.g. 4-bit unsigned quantization in Models/Transformer_DeepQuant) had no candidate and parsing exhausted backtracking. Add uint8 output to BasicQuantBindings and uint8 input to BasicDequantBindings in both Targets/Generic/Bindings.py and Targets/PULPOpen/Bindings.py. Verified: Models/Transformer_DeepQuant network gen now succeeds for both Generic and Siracusa platforms.
The Snitch FP32 GEMM/TransB-5000 build OOMs the GitHub-hosted runner
('std::bad_alloc' from the C compiler driver) when 4 pytest-xdist
workers compile in parallel. Two workers leave enough headroom on
the standard 7-GB runner.
(Pre-existing flake; surfaced as a hard fail in CI runs that happen
to land both heavy FP32 GEMM compilations on adjacent workers.)
The generated Network.c includes CNN_BasicKernels_NE16.h (from the GAP9 SDK autotiler CNN_Libraries_HWPE directory), but this path was missing from the cmake include directories, causing build failures on plain GAP9.
KerConv_NE16_T.Pad is declared as v4u (unsigned) in the GAP9 SDK but
the template was using (v4s){0,0,0,0} (signed), causing a compilation
error on GCC with -Werror.
Runs each NE16 conv kernel with --profileTiling after the normal test suite to collect cycle counts from gvsoc.
profileTiling generates code calling getCycles() in Network.c but the header declaring it was not included. Add CycleCounter.h to both GAP9 and NE16 platform include lists, and expose the GAP9 inc/ directory to the network target so the header is found at compile time.
GCC 7.1.1 has LTO linking bugs with the GAP9 SDK PMSIS library. The profiling step needs a clean rebuild with LTO disabled to avoid conflicts with the cached LTO-enabled build from the test step.
The --enable-3x3 flag was parsed by the runner script but never passed to generateNetwork.py, so NE16Engine.enable3x3 was always False. DW 3x3 and Dense 3x3 convolutions silently fell back to the PULP cluster instead of dispatching to NE16. Add the flag and set it on the engine.
Mirrors ci-platform-gap9-tiled.yml's coverage — the NE16 workflow only had kernel jobs, so the new MobileNetV1 (MLPerf/VisualWakeWords) model tests weren't running on upstream's runners.
…lows) The NE16 CI runs on ecab71e and 7f6ce26 tripped on git's "dubious ownership" check for TargetLibraries/CMSIS/third_party/CMSIS-NN during submodule fetch, killing the job before any pytest started. The CMSIS-NN submodule SHA hasn't changed since 822dd32 (where this workflow last passed), so this is purely a runner workspace-ownership issue. Add the same Mark workspace as safe step that every other platform's _runner-*.yml already uses (siracusa, snitch, gap9, gap9-tiled, chimera, generic, cortexm, mempool, siracusa-neureka-tiled, ...).
The pass was added during the NE16 Linear PR integration (6c8ae2b) and matches every Gemm/RequantizedGemm node without checking the engine attribute, so cluster-bound GEMMs (e.g. MLPerf AnomalyDetection's 10 Gemm+RQ layers — they never run on NE16) had their mul/bias rewritten into NE16 scale/scale_n/shift-diff layout. The cluster pulp_nn_linear kernel then consumed the rewritten constants under its original integer contract and produced ±1 mismatches versus the int8 reference outputs. Mirror the existing NE16AdjustWeightMemoryLayoutPass: bail out for nodes whose engine attr isn't "NE16". Pure-GAP9 cluster Gemms keep Deeploy's Generic + PULPGEMMRequantMergePass layout (including the bias += div/2 rounding compensation), matching the reference. gvsoc gap9.evk (Models/MLPerf/AnomalyDetection L1=64000): - before: 33/640 errors (all ±1), Runtime 89110 cycles - after: 0/640 errors, Runtime 79332 cycles - devel base 3b011bb (where bug doesn't exist): 0/640, 78500 cycles gap9_tiled L2 single-buffer models goes from 9/11 → 10/11 pass. The remaining failure (MLPerf/ImageClassification, parser backtracking on a standalone RequantShift node) is unrelated to GEMM and pre-dates this fix.
GAP9Platform's loweringPasses had PULPAddRequantMergePass commented out without an explanation. Without it, RequantShifts that follow an Add (residual quantize step in ResNet-like blocks) never get folded into a RequantizedAdd, so they survive standalone into the backend with their float32 scalar mul/add intact. The PULP RequantShift bindings only accept int32 mul/add, so parser-side type checking rejects every binding and the whole graph backtracks. MLPerf/ImageClassification (int8 ResNet, 14 RequantShifts, 3 Adds) was the visible victim — at the devel base 3b011bb it passed because PULPAddRequantMergePass was active there. Restore the pass to the lowering chain alongside its conv/gemm/matmul peers. The model now parses, builds and runs cleanly: - Models/MLPerf/ImageClassification on gvsoc gap9.evk: 0 / 10 errors, 1 365 882 cycles (devel base: 0 / 10 errors, 1 368 399 cycles — bit-equivalent) Also tighten RQSSplitPass.\_split_rqs_fun: when it duplicates the mul/add constants for each downstream RQ, cast their values to int32 if they are integer-valued. Source ONNX often stores RequantShift's scalar mul/add as float32 because they get folded into Conv/Gemm bias later. With the new split, they survive standalone, so type-checking needs them to already be int32. This is a defense-in-depth fix — PULPAddRequantMergePass alone already unblocks the failing model. gap9_tiled and models and singlebuffer and l2: 11 / 11 pass (was 9 / 11 before, then 10 / 11 after the GEMM engine-check fix).
Two FP32 kernel tests regressed since the NE16 Linear PR integration: 1. FP32 GEMM/Regular: the new GAP9_NE16GEMMInt32Mapper was prepended to the plain Gemm op binding list. It uses the same GEMMParser class as the FloatGEMM and GEMMDequant mappers, and Deeploy keys candidate bindings by parser class — so listing the NE16Int32 mapper first masked the other two. FP32 inputs failed the NE16Int32 type check and there was no backtrack to the FloatGEMM mapper. Drop GAP9_NE16GEMMInt32Mapper from the Gemm op list (the int8/uint8 path is already covered by RequantizedGemm with its own NE16 mapper). 2. FP32 Reshape/SkipConnection: MatMulAddMergePass fused MatMul+Add into a Gemm with alpha=beta=1 and no transA/transB. The MatMul inputs in this test don't share Gemm semantics, so the merge produced wrong outputs (16/16 errors). Devel base doesn't include this pass in the GAP9 lowering chain and the test passes there. Disable it for GAP9 to match. gvsoc gap9.evk: - FP32 GEMM/Regular: 0/1024 errors, 28987 cycles - FP32 Reshape/SkipConnection: 0/16 errors, 8343 cycles - devel base: 0/1024, 28k-ish; 0/16, 8020 cycles — same shape. gap9_tiled suite: 96 passed / 1 failed (was 92/5). Remaining failure (Models/CCT/FP32/CCT_2_32_32_128) also fails at 3b011bb devel base — pre-existing, unrelated to this PR.
…atform#193) * add support for Ceil operator for Generic target * add support for Clip operator for Generic target * move math.h out from Generic DeeployBasicMath.h to avoid conflicts with other targets * add support for Floor operator for Generic target * add support for Sub operator for Generic target * add support for Exp operator for Generic target * add support for Sigmoid operator for Generic target * add support for Swish operator for Generic target * add support for HardSigmoid and HardSwish operator for Generic target * in Swish parser make alpha an optional attribute * add support for Instance and Group Normalization operators for Generic target * add support for AveragePool, GlobalAveragePool, and GlobalMaxPool operators for Generic target * add computation of operation to new supported layers * fix minor type issue in AveragePool kernel * update CHANGELOG * minor fixes (implementing coderubbit suggestions) * minor fixes (address review comments) * update dates in the headers * minor fix in Clip Parser about min and max default values * forcing empty commit
…ta (pulp-platform#196) Makes GAP9 tests with `--defaultMemLevel L3` pass on the **board**. They already passed on L2 and on GVSoC; on the board every output was wrong while L2/GVSoC matched the reference.
* XDNA2 Platform Beta Support * Add XDNA container * First attempt at generating MLIR code with Deeploy * Generate tiled code but too much logic is in the Template * Move data movement in passes. Template represent for loop and aquire/release * Template is agnostic of tiling and data movement that are handled by code transformation passes * Add CI on self hosted runner * Remove unecessary install * Add cleanup step before checkout to fix permission * aie import is optional to not enforce mlir-aie and llvm-aie package installation * Decouple xdna requirements from dev requirements * Add general todos for future refactoring * Free output tasks in RT sequence * Add -v flag for XDNA platform only. * Pin llvm-aie version * Add remainder loop to Add kernel * Use workdir to easily discard build-time files * Bump mlir-aie to v1.3.2 and use llvm-aie nightly * Add docker action for XDNA * Use exernal_function instead of link_with to respect new norm from MLIR-AIE repo * Remove redundent mapping * Update Changelog * Use the xdna docker image from ghcr instead of local image * Rename MLIRDataTypes into MLIRAIETypes
* Remove /opt/xilinx binding * Fix XDNA dockerfile * Update changelog
* fix bugs that prevent Neureka to be selected as an engine to accelerate conv layers - in parser, weights are contrained to have 3 dimensions. That is correct for PW only. DW and Dense require 4 dimensions. The fix allows weights with 4 dimensions for DW and Dense. - in tiler, again only 3-dim weights for PW is supported. Add support for 4-dim weights for DW and Dense. - in test-runner, arguments for neureka are ignored. Add support for `enable-3x3` and `neureka-wmem` arguments. * add enable gvsoc trace for Neureka * fix out-channels bug and add a Conv/Regular_3x3 test * fix Conv DW parser * make Conv/Regular_3x3_RQ test smaller to ease debug * fix Neureka DW weight encoding and parsing * fix tiling constraints for Dense and DW * fix tiling for PW when batch size is greater than one (GEMM/Batch_RQ test) * fix Neureka tile contraints * refactor Neureka reshape PW function * add guards for GEMM to PW transformation that check the dimension of the bias is coherent with requantshift add vector * fix typo in PW conv parser * add contraints on weight tensor's non-tiled dimensions for Dense and DW convolutions * add DW lowering pass to fix miniMobileNetv2 parsing error * fix input channels tiling in DW Conv * set _live flag to True for global IO Buffers. The flag avoid that aliases of input/output (such as a no-op reshape view) is deallocated * remove experimental enable3x3 flag because now Neureka's 3x3 mode is totally supported * fix CHANGELOG due to rebase * minor refactor (addressing coderabbit review) * fix bug in Neureka's DW weight tile constraint and serialization, add DW and Dense 3x3 tests, general tile constraint refactor
…lp-platform#194) * Add SoCDAML Exercise and update README * Add SoCDAML Part III: hands-on lab for adding a new int8 operator (iLeakyReLU) * Move Part III lab to docs/tutorials/introduction.md * Apply pre-commit fixes (SPDX headers, yapf/clang-format/shfmt) * [docs] Add instructions to build container from scratch * [fix] Forward --neureka-wmem and --enable-3x3 from the runner to the generator * [fix] Make Part III deploy.sh robust to Platform.py import rewrapping * [fix] Use addTileSizeDivisibleConstraint for the Step 6a SIMD alignment constraint * [docs] Fix Part III flags, paths and imports, and document the scratch cache exports * [docs] Fix dangling SoCDAML.md reference * [docs] Add changelog entries for the SoCDAML tutorial fixes * [docs] Fix duplicated article in the Part III intro paragraph * [docs] Fix profiling sentence grammar in the --profileTiling section * [docs] Correct the Singularity installation step count to six * [docs] Add language identifiers to the Part III code fences * [fix] Include the positive int8 boundary in the iLeakyReLU fixtures * [docs] Match the skeleton README SIMD row to what the file leaves as TODO * [fix] Make deploy.sh undo remove only its own test artifacts * [fix] Keep ccache and pip caches on scratch through --cleanenv * [docs] Compress the SoCDAML changelog entries * [docs] Drop unreferenced image copies already present under docs/_static * [fix] Reject iLeakyReLU attributes the SIMD kernel cannot honour * [docs] Add the missing PR-list and Added entries for the Part III lab * [fix] Use GELUChecker so iLeakyReLU output keeps its signedness * [fix] Make the deploy.sh recovery command work from the script's own directory * [fix] Store the iLeakyReLU fixtures as int8 under the ONNX tensor names * [docs] Document the bind-source failure when the cache variables are unset * [build] Ignore the fixtures students generate from the Part III skeleton * [fix] Partition iLeakyReLU SIMD work by vector so no element is dropped * [fix] Stop forwarding --enable-3x3, removed from the generator in pulp-platform#188 * [fix] Apply the Part III core-library edits as a git patch * [fix] Drop the redundant --neureka-wmem forwarding fixed upstream in pulp-platform#188 * [fix] Keep the copied files when the core patch cannot be reverted * [fix] Scope deploy.sh recovery advice to the patched files only --------- Co-authored-by: victor.jung <jungvi@iis.ee.ethz.ch> Co-authored-by: viv-eth <vivianep@iis.ee.ethz.ch>
…platform#153) * Add FP32 operators for MicroLlama on Snitch (untiled) Add support for FP32 operators required by MicroLlama model: - RMSNorm: Fused RMS normalization - HardSwish: Activation function - Div: Element-wise division - Mul: Element-wise multiplication - MatMul: Matrix multiplication - Add: Element-wise addition (FP32 support) - Reshape, Transpose, Concat, Gather: Shape operations Components added: - Generic: Parsers, TypeCheckers, Layers, Bindings - Snitch Templates: FloatAdd, FloatDiv, FloatHardSwish, FloatMul, FloatRMSNorm, FloatMatMul, Reshape, Transpose, Gather - Snitch Kernels: C implementations for all FP32 operators - Test data: Hardswish, RMSNorm_fused kernels, microLlama_fp32_1 model This enables running MicroLlama FP32 model on Snitch in untiled mode: python testRunner_snitch.py -t Tests/Models/microLlama/microLlama_fp32_1 * Add tiling support for MicroLlama on Snitch Add SnitchTiledPlatform with TileConstraints for FP32 operators: - FloatDivTileConstraint: Division tiling with scalar broadcast - FloatMulTileConstraint: Multiplication tiling with scalar broadcast - ReshapeTileConstraint: Pass-through tiling for reshape Updates: - SnitchClusterTiling with tiled code transformation passes - Tiler.py with new tile constraints registration - platformMapping.py adds Snitch_tiled platform - testRunner_tiled_snitch.py for tiled model testing - CI workflows for both untiled and tiled Snitch * Fix Snitch tiled platform by unifying mapping to use TilingReadyBindings Previously the Snitch platform had separate tiled/untiled mappings (SnitchMapping vs SnitchTiledMapping), but mapPlatform() only returned SnitchPlatform with the untiled mapping, causing tiled tests to fail with missing tileConstraint. Unified to a single mapping using TilingReadyBindings for all operators, matching the upstream pattern used by PULPOpen.
The cluster slave stacks are sized by SLAVESTACKSIZE, which deeploytest.c
defined unconditionally at 3800 B. With 8 cores that reserves ~32 KB of the
128 KB L1 TCDM before the tiling arena starts, so --l1 values above ~98000
either fail to allocate ("Allocation failed for allocator 2") or, worse,
produce a tile layout that overruns L1 and shows up only as a DMA
out-of-bound trace at run time.
Two edits are needed and either one alone is a silent no-op: the GAP9
CMakeLists never turned the -D cache variable into a compile definition, and
the unconditional #define in the source shadowed the command-line one. This
mirrors what DeeployTest/Platforms/GAP9 already does downstream.
Measured on gvsoc with -D SLAVESTACKSIZE=512: the L1 arena grows from 98,176
to ~121,000 bytes, dense 64x64 @32x32 and 16x16 @64x64 at --l1 115000 go from
'Allocation failed' to running, and every previously-passing config keeps its
result bit-exact while getting ~1000 cycles faster.
The hoisted tile-control tables (numTiles / DMA cmd / size / dims / offsets) are read-only lookup tables the cluster controller uses to drive the tiling loop and program DMAs -- not bulk tile data. Emitting them at the innermost tile memory put them in the GAP9 L1 TCDM next to the cluster master stack, which both wastes scarce L1 and lets a deep master-stack write clobber a table entry, turning a DMA cmd into a garbage code pointer. Only the L2->L1 pass is redirected; the L3->L2 pass keeps its tables in L2 (== self.memory, unchanged), and platforms that do not tile into a level named "L1" are unaffected. Tile *data* buffers still go to L1. Verified on gvsoc: dense 32x32 @32x32 stays bit-exact at 64.77 MAC/cycle.
NE16 retires 32 output channels per pass, so a channel tile that is not a multiple of 32 leaves the remaining lanes idle for the whole tile. Nothing expressed that preference, and the solver was free to pick whatever fit -- Ko=3 and Ko=56 have both been observed in generated code. GAP9's AutoTiler gives its solver the same preference; see CNN_Generators_NE16.c, 'OutTileCons = CannotTileChannels ? OutFeat : 32' and 'InTileCons = Mode16?8:16', on the shared path that covers dense, depthwise and pointwise alike. Added as a PerformanceHint rather than a hard constraint, matching the SDK's degradation to full OutFeat when channels cannot be tiled: shapes with C < 32 or a tight L1 budget must stay tileable. Measured on gvsoc (dense, --l1 115000): peak goes 64.92 -> 75.99 MAC/cycle, 47% of the 162 MAC/cycle theoretical peak and ~2.2x what the GAP9 SDK reaches on MobileNetV1. Configs that previously could not be tiled at all now run, e.g. 8x8 @96x96 at 10.98 MAC/cycle with bit-exact results. Note: some newly-reachable multi-tile configs expose a pre-existing per-tile boundary bug (error count scales with tile count, invariant to L1 value and arena placement); that is tracked separately and is not introduced here.
The 90KB cap existed because the cluster slave stacks were pinned at 3800 B per core, so ~32KB of the 128KB L1 was gone before the tiling arena started while the tiler was still told it had the full budget. With SLAVESTACKSIZE overridable (see the GAP9 CMake/deeploytest change) a 1280 B slave stack lifts the measured ceiling from 90KB to 110KB. Measured on gvsoc, bit-exact in both cases: 90000 / stack 3800 -> 860,577 cycles (8.70 MAC/cycle) 110000 / stack 1280 -> 830,146 cycles (9.02 MAC/cycle) 121000 and 128000 still fail to allocate; the exact ceiling is not bisected. Single-buffer is unaffected -- it is not L1-bound on this model: 128000 to 131000 with stacks of 3800/1280/1024 all give an identical 794,080 cycles (9.43 MAC/cycle), which remains the best configuration.
The PULPOpen deployer already ends its lowering chain with the transpose clean-up (TransposeSplit / TransposeMerge / TransposeConstOpt / TransposeNoPermOpt). NE16Deployer then appends its own passes, so NE16OptimizationPass -- which inserts layout transposes of its own via _appendTranspose -- runs *after* that clean-up and its transposes are never folded. Two consecutive NE16 convs therefore end up separated by a HWC->CHW followed by a CHW->HWC, an identity pair that survives all the way to codegen, each half carrying its own tiling loop and L2<->L1 DMA round trip. Re-run the same clean-up chain after NE16OptimizationPass. MLPerf VisualWakeWords (MobileNetV1, 7.49 MMAC) on gvsoc, CI config (single-buffer, L2, l1=128000), bit-exact before and after: transpose cluster forks 232 -> 16 tiling loops 114 -> 62 NE16 dispatches 56 -> 56 (compute untouched) cycles 794,080 -> 537,450 (1.48x, -32.3%) MAC/cycle 9.43 -> 13.94 For reference the GAP9 SDK reaches 10.3 MAC/cycle on the closest model size.
# Conflicts: # DeeployTest/conftest.py # DeeployTest/testUtils/platformMapping.py # DeeployTest/test_platforms.py
…s buffer serializeTilingSolution always emits the weight tile as HyperRectangle((COffset, 0, ...), (CSize,) + weightShape[1:]) -- only the output-channel dimension is tiled, the NE16-encoded tail is always moved whole. addGeometricalConstraint only pinned weightOutChannelVar, leaving that tail free for the solver to shrink, so it could reserve less L1 than the DMA actually writes. For 64x64 dense the weight tail is (cinMajor=4, bits=8, H*W*cinMinorBytes=18) and the solver picked 16 for the last dim: 32*4*8*16 = 16384 B reserved, 32*4*8*18 = 18432 B transferred. The extra 2048 B ran straight over the mul and add requant parameters, which the allocator had placed immediately after the weight buffer, so the corresponding 2048 B of output came out wrong. The signature matched exactly: the wrong outputs formed two contiguous 1024 B runs 32768 B apart (one output tile's worth), the error count scaled with the tile count and was invariant to the --l1 value and to arena placement, and an untiled run was always correct. Measured on gvsoc at --l1 115000, all bit-exact after the fix: dense 64/64 @32x32 2047 errors -> 0 (74.33 MAC/cycle) dense 16/16 @64x64 8097 errors -> 0 (26.61 MAC/cycle) dense 4/4 @128x128 10121 errors -> 0 ( 4.34 MAC/cycle) dense 32/32 @32x32 (control) 0 errors, 145405 cycles unchanged
The 110000 set earlier was measured while the weight tile's NE16-encoded tail was under-reserved, so the solver had slack that only existed by overrunning the requant parameters. With that fixed the same budget no longer allocates. Measured on gvsoc, bit-exact: 100000 -> 567,140 cycles 110000 -> 'Allocation failed for allocator 2' Single-buffer remains the better configuration for this model at 536,521 cycles (13.96 MAC/cycle).
yapf drops a stray blank line in NE16PointwiseConstraint and clang-format rewraps an over-long comment in deeploytest.c. Both were introduced by the preceding NE16 commits.
Collects the primary sources for NE16 (the pulp-platform/ne16 and rbe repos, pulp-nnx, the GVSoC behavioural model, DORY's Ne16PerfModel, and the SDK's own tiler constraints), derives the 162 MAC/cycle 8-bit peak from the 9x9x16 array, and states the three alignment rules (Ci%16, Co%32, Ho/Wo%3) with the hardware reason for each. It then documents what we changed and why -- folding the redundant layout transposes, pinning the weight tile's NE16-encoded tail, the TP_OUT=32 tile preference, and making SLAVESTACKSIZE overridable -- with the measured effect of each, and closes with the open items (signed activations, activation fusion, narrow layers). The point the numbers make: a full network runs at ~1/5 of a well-shaped single layer and both are far from 162, because MobileNetV1 0.25x has 8/16/32 channel layers against a 16/32-wide datapath. Beyond the alignment rules the remaining wins are in removing non-compute work, which is where all of our 1.48x came from -- the NE16 dispatch count did not change at all.
NE16's nine columns retire one 3x3 output patch per pass, so an output tile
whose H or W is not a multiple of 3 wastes part of the patch on its border
pass. GAP9's AutoTiler passes the same spatial PreferedTileSize of 3 to its
solver; with the TP_OUT=32 channel hint added earlier, that completes the
Ki=16 / Ko=32 / spatial=3 set (Ki needs no hint here -- the input channel
dimension is already hard-pinned to its maximum so partial sums never arise).
The priority matters more than the rule. PerformanceHint tries *higher*
priorities first, so this is registered at 0: below the take-the-whole-
dimension hint (1) and below the channel alignment (2). Splitting the spatial
dimensions costs a 2-row/2-col halo re-fetch per extra tile while splitting
output channels costs nothing, so when L1 is tight the spatial alignment has
to give way.
Measured on the double-buffered DW_2D_RQ kernel, which is L1-tight:
no spatial hint 11,926 cycles
spatial hint at priority 2 19,391 cycles (tile shrunk to reach a multiple
of 3, so more tiles, more halo)
spatial hint at priority 0 11,926 cycles
All four NE16 CI jobs pass; VisualWakeWords is unchanged at 536,521 cycles.
…w modulo Supersedes the raw `outVar % 3 == 0` hints added in b775bee. Those constrained *every* tile including the remainder to be a multiple of 3, which cuts a large part of the feasible set away and pushes the solver to smaller body tiles -- more tiles, more halo re-fetch. addTileSizeDivisibleConstraint expresses what was actually meant: the body tile is a multiple of the modulo and the border tile is whatever is left over (TilerModel.py:225 -- tileSizeVar == mulMulVar * modulo, dim == mulVar * tileSizeVar + addVar, addVar <= tileSizeVar). NE16PWConv2DTileConstraint already had exactly this, inherited from the N-EUREKA constraints; the b775bee hints were stacked on top of a correct implementation. Dense and depthwise now follow the same shape, including the guard that takes the whole dimension when it is smaller than the hardware granularity. Depthwise deliberately gets the channel alignment only. Aligning its spatial dimensions is measured harmful: the double-buffered DW_2D_RQ kernel goes 11,926 -> 19,391 cycles (+63%). Depthwise has no input-channel reuse to amortise the halo re-fetch of an extra spatial split against, so the alignment costs more than the partially-filled border pass it avoids. All four NE16 CI jobs pass; VisualWakeWords single-buffer unchanged.
a47e812 made SLAVESTACKSIZE overridable but nothing ever set it, so CI kept the SDK default of 3800 B per core and the ~32 KB of L1 it reserves. The knob existed and was never turned. Every convolution in these tests runs on NE16, so the cluster cores only orchestrate and need very little stack. 512 B is measured to work; 256 B works too and is no faster. Measured on gvsoc, bit-exact in every case: models single-buffer 536,521 -> 536,489 (unchanged -- not L1-bound: this model is one tile per layer) models double-buffer 567,140 -> 535,364 (-5.6%; double-buffering needs two of every tile, so it was L1-bound and the freed ~23 KB is spent on bigger tiles) kernels single-buffer 7,558 -> 7,518 kernels double-buffer 11,926 -> 11,891 Double-buffered VisualWakeWords is now the faster of the two configurations. Note this is a per-test override, not a change to the default in DeeployTest/Platforms/GAP9/src/deeploytest.c. That file is shared with the plain GAP9 platform, whose convolutions run on the cluster: _fill_zero alone has a 1016 B stack frame, and undersizing the stack there corrupts a return address silently (the core ends up fetching from 0x0 or 0x10).
Two defects, both only triggered once the tiler splits a depthwise layer's output along channels (16 channels became a 14 + 2 split at --l1 115000). 1. computeInputCube (PULPOpen/TileConstraints/ConvTileConstraint.py) builds the input tile with the channel range hard-coded to (offset 0, size ch_im_in). That is right for dense, whose input channels are pinned to their full extent and never tiled, and wrong for depthwise, where each output channel is produced from exactly one input channel: an output tile covering channels [COffset, COffset + CSize) has to read precisely that slice. Uncorrected, the second tile read from offset 0 and the first over-read past its own tile. Fixed locally in the depthwise constraint rather than in the shared helper, so the dense and pointwise paths are untouched. 2. The depthwise weights are bit-serialised by _weightEncode(depthwise=True) into one packed block interleaving up to NE16_SUBTILE_INPUT_CHANNEL=16 parallel output channels, and serializeTilingSolution loads that whole block, from offset 0, for every tile. There is no byte offset that selects a sub-range of an interleaved block, so a channel tile is only ever paired with the right filters when it starts at channel 0: the 14 + 2 split computed channels 14..15 with the filters of channels 0..1. Keep the output channels untiled; a layer that no longer fits L1 is split spatially instead. Diagnosed by reducing the failing output indices modulo the tile geometry in CHW. Defect 1 alone left channels 0-13 correct and channels 14-15 wholly wrong, which is what pointed at the weight block. Measured on gvsoc at --l1 115000, all bit-exact after the fix: dw 16/16 @64x64 16,172 errors -> 0 (1.65 -> 1.96 MAC/cycle) dw 8/8 @96x96 32,652 errors -> 0 (1.29 -> 1.64) dw 4/4 @128x128 28,943 errors -> 0 (1.02 -> 1.53) dw 16/16 @32x32 (control) 0 errors, 57,017 cycles unchanged dw 32/32 @32x32 (control) 0 errors, 120,146 cycles unchanged The speedups are a side effect: not splitting channels leaves NE16 fuller. If the no-tiling constraint ever makes a shape infeasible, the better fix is to allow channel tiles that are multiples of 16 -- one whole packed block each -- and offset the weight cube by COffset // 16 blocks. All four NE16 CI jobs pass; VisualWakeWords unchanged at 536,489 cycles.
…transfers
MCHAN encodes the transfer length in the low MCHAN_TRANSFER_LEN_SIZE (17 on
GAP9) bits of cmd, so a command can carry at most 2**17 - 1 bytes. The guard
was
mchanTransferSizeBits = math.ceil(math.log2(mchanTransferSize))
assert mchanTransferSizeBits <= 17
which accepts a size of exactly 131072: log2 is 17.0, so the assertion passes,
and then (mchanFlags << 17) + 131072 carries 1 << 17 straight into mchanFlags
bit 0 -- the direction flag. The transfer silently runs the wrong way. Two
depthwise shapes with an activation of exactly 131072 B hit this and reported a
runtime DMA out-of-bound rather than an assertion failure.
Bound is now 2**17 - 1, and anything above it is issued as several
mchan_transfer_push_* calls under the same transfer id -- 1D chunked by bytes,
2D chunked by outer rows. That needs no new machinery: the generated code
already pushes several descriptors per id and waits once (input, weight, mul and
add share one transfer_input). Chunk counts are decided at code-generation time,
where math.prod(shape) is known. A 2D row that on its own exceeds the limit now
raises a clear error instead of producing silently wrong code.
Verified: the chunked form appears in the generated Network.c and splits 131072
into 131071 + 1; dw 16/16 @128x128 at --l1 32000 still runs in 1,810,221 cycles,
byte-identical to before, so transfers under the limit are untouched. All four
NE16 CI jobs pass.
Note this does not by itself make dw 32ch@64x64, 16ch@96x96 or 16ch@128x128 run
at --l1 115000; those fail for a separate, still-unidentified reason whose
boundary sits between --l1 32000 (passes) and 60000 (fails).
For context: TrainDeeploy 071e6e0 widened length_1d_copy to uint32_t on the
PULPOpen dory_dma path, which is a different path -- 17 bits here is a hardware
register width. That commit's own guard, assert length_2d_copy <= 2**17, carries
the same off-by-one.
b13bc2f fixed the wrong-filters bug by forbidding depthwise channel tiling outright. That is correct but heavier than it needs to be, and it does not cover layers wider than one packed block: with 64 channels the weights occupy four blocks, yet the single untiled tile still loaded only the first one, so dw 64/64 @32x32 produced 49,941 wrong outputs. _weightEncode(depthwise=True) lays the weights out as (cout=1, cinMajor=ceil(C/16), Bits*H*W*cinMinorBytes): one packed block per group of NE16_SUBTILE_INPUT_CHANNEL=16 output channels, with the 16 channels' bits interleaved *inside* a block. The interleaving is what cannot be sliced -- the blocks themselves can. So constrain channel tiles to whole blocks and hand each tile the blocks that actually hold its filters: blockStart = COffset // 16 blockStop = ceil((COffset + CSize) / 16) This is strictly better than the blanket ban: correct for any channel count, and wide depthwise layers can tile channels again instead of being forced to split spatially. Measured on gvsoc at --l1 115000: dw 64/64 @32x32 49,941 errors -> 0 (2.00 MAC/cycle) dw 16/16 @64x64 0 errors, 1.96 MAC/cycle (unchanged) dw 32/32 @96x96 0 errors, 1.71 MAC/cycle (unchanged) dw 16/16 @32x32 0 errors, 57,017 cycles (control, unchanged) All four NE16 CI jobs pass: kernels 7 + 2, models 536,489 and 535,201 cycles.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the NE16 neural engine as an accelerator Engine on top of the existing GAP9 platform, registered as a new composite platform
GAP9_w_NE16that mirrors theSiracusa_w_neurekapattern.Added
Deeploy/Targets/NE16/— full Target: Platform/Engine/Bindings/Parsers/Tiler/Deployer/Templates/TileConstraints/TopologyOptimizationPasses.NE16PlatformextendsGAP9Platformwithengines=[NE16Engine, GAP9ClusterEngine];NE16DeployerextendsGAP9Deployer._weightEncodeported frompulp-nnx/test/Ne16Weight.py(single CIN_SUBTILE=16 mode).DeeployTest/deeployRunner_tiled_gap9_w_ne16.py+DeeployTest/test_gap9_ne16_tiled_config.py— runner + kernel test config.DeeployTest/test_platforms.py— pytest functionstest_gap9_w_ne16_tiled_kernels_l2_{single,double}bufferunder markergap9_w_ne16_tiled..github/workflows/{ci-platform-gap9-w-ne16-tiled.yml,_runner-gap9-w-ne16-tiled.yml}— CI jobs (single + double buffer L2).TargetLibraries/GAP9/CMakeLists.txt—add_subdirectory(pulp-nnx)withUSE_NE16=ONforGAP9_w_NE16.Changed
DeeployTest/testUtils/platformMapping.py— registerGAP9_w_NE16in names/mapPlatform/setupMemoryPlatform/mapDeployer.DeeployTest/testMVP.py— wrap deployer withEngineColoringDeployerWrapperforGAP9_w_NE16(without it NE16 nodes never get an engine color and parsing fails).DeeployTest/testUtils/core/execution.py— append the GAP9 SDKimagebuild target forGAP9_w_NE16(sochip.soc.mram.binis produced beforegvsoc run).CMakeLists.txt,DeeployTest/CMakeLists.txt— acceptGAP9_w_NE16alongsideGAP9in the platform branches.Deeploy/Targets/NE16/Templates/ConvTemplate.py— NE16 subtile constants perne16_task_defs.h:CIN_SUBTILE16, output3, weight strided0 = 3*3*weight_d0_stride_mode8 = 18for DW/Dense (PWqw * weight_d0_stride = 16). Emit top-levelne16_task_tfields (weight_d0_stride,qw,subtile_output_channel,kernel_shape,depthwise) that the HW reads at dispatch time.Deeploy/Targets/NE16/TopologyOptimizationPasses/Passes.py— DW weight layout: after Deeploy's NHWC→NCHW transpose, swap axes 0/1 once more so_weightEncodesees the standard(cout, 1, H, W)layout and produces the correct(1, 1, packed_bytes)single-block output expected by the NE16 HW.Deeploy/Targets/NE16/TileConstraints/NE16DepthwiseConstraint.py— DW weight is a single packed block (not per-cout); constrainweightOutChannelVar == Maxand reuse the sameHyperRectangle((0,0,0), weightShape)for every output-channel tile.Deeploy/Targets/NE16/Parsers.py— drop thegroup == shape[1]check inNE16DWConv2DParser(invalid under the post-encode rank-3 layout).Fixed
Deeploy/Targets/PULPOpen/Templates/FloatGemmTemplate.py— work around a pre-existingImportError: cannot import name 'float32_tPtr' from 'Deeploy.AbstractDataTypes'by defining it locally viaPointerClass(float32_t).Test plan
Run on gvsoc
gap9.evkinsideghcr.io/pulp-platform/deeploy-gap9:devel. All verified dispatches (ne16_nnx_dispatchappears in generated Network.c for NE16-routed nodes):Kernels/Integer/Conv/PW_2D_RQ/Regular_RQKernels/Integer/Conv/PW_2D_RQ/Regular_RQKernels/Integer/Conv/PW_2DKernels/Integer/Conv/DW_2D_RQKernels/Integer/Conv/DW_2D_RQKernels/Integer/Conv/StriddedPadded_2D_RQKernels/Integer/Conv/PW_2D_RQ/Regular_RQKernels/Integer/Conv/DW_2D_RQFollow-up (out of scope):
PW_2D_RQ/Unsigned_RQuses int8 input.Ne16TestConf.pyonly supports uint8 and NE16 HAL doesn't expose a signed-input conf0 flag; proper support needs sign-propagation (shift int8 → uint8 + adjust weight_offset).Tests/Kernels/Integer/Conv/today (Regular_2D_RQis 8×8); coverage is via the model path once the remaining tiling-system edge cases are resolved.PR Merge Checklist
develcommit and pointing todevel.CHANGELOG.mdfile has been updated.