Skip to content

test(gap9): run the fine-tuning experiments in CI - #66

Open
runwangdl wants to merge 1 commit into
develfrom
feat/ci-training-experiments
Open

test(gap9): run the fine-tuning experiments in CI#66
runwangdl wants to merge 1 commit into
develfrom
feat/ci-training-experiments

Conversation

@runwangdl

Copy link
Copy Markdown
Owner

What

19 training cases in a job of their own, gap9-training-experiments, covering four families of parameter-efficient fine-tuning.

family model Mcyc / step
channel-wise ResNet8SplitBlock 42.66
channel-wise MobileNetV1SplitAlt 38.53
qlora CCT 53.89
qlora MobileNetV1 42.28
lora ResNet8 (PEFT) 56.52
lora MobileNetV1 51.13
layerwise ResNet8 last1 / 2 / 4 / 6 16.23 / 19.71 / 29.58 / 38.64
layerwise MobileNetV1 last1 … last24 18.73 / 22.14 / 24.83 / 26.60 / 28.72 / 32.92 / 36.60 / 39.70 / 44.28

Every one was run end-to-end on gvsoc against this commit and reported Errors: 0. The cycle counts above are from those runs and are recorded next to each entry in the config.

What is deliberately absent

Two candidates were tried on this same tree and fail, so they are not registered. The reason sits beside the table so a later fix has something to check against:

model failure
ResNet8_QLORA faults on device — cluster PE7 LSU Invalid access
ResNet8_PEFT_QLORA tiler assert in minimizeRectangle: per-channel transfers with the channel index written into the batch offset

The second is the same defect that a channel split whose halves are read twice runs into, which is why the channel-wise graphs here are built to read each half once. It is worth fixing on its own.

Why a separate job

gap9-training-tiled-l3-singlebuffer already runs close to an hour. Adding 19 cases to it would roughly double that, and an experiment going red would gate the mainline configurations. The new job runs in parallel and gates nothing.

Overrides

The per-model overrides carry what the verification runs actually used — CC_STACK_SIZE=4096, SLAVESTACKSIZE=512 — rather than the tuned values of the base models these graphs derive from (MobileNetV1 is tuned to cc_stack 8192). Registering settings nobody exercised would put a configuration into CI that was never measured.

Assets

Existing convention only: network.onnx, inputs.npz, outputs.npz per training directory plus the optimizer's network.onnx. Intermediate build output is not included. 22 MB total; check-added-large-files passes.

🤖 Generated with Claude Code

Adds 19 training cases in a job of their own, gap9-training-experiments, covering
four families:

  channel-wise  every convolution halved along a channel axis, one half trained
  qlora         int8 weight storage with an fp32 compute type
  lora          low-rank adapters
  layerwise     the last N weighted layers trained, N swept

Every case was run end-to-end on gvsoc against this commit and reported "Errors: 0";
the cycle counts in the config are from those runs.

    channel-wise   ResNet8SplitBlock       42.66    MobileNetV1SplitAlt   38.53
    qlora          CCT                     53.89    MobileNetV1           42.28
    lora           ResNet8 PEFT            56.52    MobileNetV1           51.13
    layerwise      ResNet8   last1/2/4/6   16.23 / 19.71 / 29.58 / 38.64
                   MobileNetV1 last1..24   18.73 / 22.14 / 24.83 / 26.60 / 28.72 /
                                           32.92 / 36.60 / 39.70 / 44.28

Two candidates were tried on this same tree and are NOT registered, with the reason
recorded beside the table so a later fix has something to check against:

  ResNet8_QLORA        faults on device, cluster PE7 LSU "Invalid access"
  ResNet8_PEFT_QLORA   tiler assert in minimizeRectangle -- per-channel transfers
                       with the channel index written into the batch offset, the
                       same defect a channel split with a doubly-read half hits

Its own job rather than more entries in l3-singlebuffer: that job already runs close
to an hour, and an experiment going red should not gate the mainline configurations.

The per-model overrides carry the settings the verification runs actually used
(CC_STACK_SIZE=4096, SLAVESTACKSIZE=512), not the tuned values of the base models
these graphs derive from. Registering settings nobody exercised would mean CI runs a
configuration that was never measured.

Assets follow the existing convention -- network.onnx, inputs.npz and outputs.npz per
training directory plus the optimizer's network.onnx, nothing else. 22 MB.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant