Skip to content

fix(training): route ConvGrad outputs by shape, not by position - #67

Open
runwangdl wants to merge 1 commit into
develfrom
fix/tiler-channel-offset
Open

fix(training): route ConvGrad outputs by shape, not by position#67
runwangdl wants to merge 1 commit into
develfrom
fix/tiler-channel-offset

Conversation

@runwangdl

Copy link
Copy Markdown
Owner

What

SplitConvGradPass read a fused ConvGrad's outputs positionally — outputs[0] as dX, outputs[1] as dW — and its docstring said so explicitly: "1 output (dX only): ConvGradX".

That assumption does not hold. ORT emits dX only when something downstream needs it. A trainable layer whose input needs no gradient — the first trainable layer of a network — emits dW alone. The positional read takes that lone dW to be dX and wraps it in a ConvGradX.

How it surfaces

Nothing downstream recovers. Conv2DGradXParser binds it as grad_in; the tiler slices it along its output channels; ConvGradXTileConstraintBase reads those cube offsets as batch offsets and derives a dY cube at (k, 0, 0, 0) on a tensor whose batch extent is 1:

AssertionError: Rectangle offset should be zero when the dimensions are the same.
Received rectangle HyperRectangle(offset=(1, 0, 0, 0), dims=(1, 16, 1, 1))
and reference shape (1, 16, 24, 24)

The assertion fires five layers away from the cause, which is why it reads as a tiler bug.

Fix

dX carries X's shape and dW carries W's, which tells them apart whatever order they arrive in — so classify by shape. Where shapes are unavailable the positional reading is kept, so graphs without shape annotations behave exactly as before. A node with no dX no longer produces a ConvGradX at all.

Silent mis-binding today

ResNet8 already contains such a node (layer1_0_conv1_b, dW only). It does not assert, because its shapes never reach that branch — so that graph has been binding a weight gradient as an input gradient all along, without any symptom.

Validation

result
channel-split MobileNetV1 assertion above → clean deploy, planned L3 = 3246.8 KB
pytest -m "gap9_tiled and training and singlebuffer" 19 passed, 0 failed
pytest -m deeploy_internal 44 failures before and after (40 test_dmas.py, 4 test_deeploy_internal.py), all pre-existing

DeeployTest/test_split_convgrad.py builds the graphs by hand, runs in 0.16 s with no ONNX asset or backend, and covers dW-only, dX-only, both orders, the bias case, and which operands each split node reads. Two of its five tests fail without this change.

🤖 Generated with Claude Code

SplitConvGradPass read a fused ConvGrad's outputs positionally -- outputs[0] as dX,
outputs[1] as dW -- and its docstring stated "1 output (dX only): ConvGradX".

ORT emits dX only when something downstream needs it. A trainable layer whose input
needs no gradient, i.e. the first trainable layer of a network, emits dW alone. The
positional read takes that lone dW to be dX and wraps it in a ConvGradX node.

Nothing downstream can recover. Conv2DGradXParser binds it as `grad_in`, the tiler
slices it along its output channels, and ConvGradXTileConstraintBase reads those cube
offsets as BATCH offsets, deriving a dY cube at offset (k, 0, 0, 0) on a tensor whose
batch extent is 1:

    AssertionError: Rectangle offset should be zero when the dimensions are the same.
    Received rectangle HyperRectangle(offset=(1, 0, 0, 0), dims=(1, 16, 1, 1))
    and reference shape (1, 16, 24, 24)

dX carries X's shape and dW carries W's, which tells them apart whatever order they
arrive in, so classify by shape. Where shapes are unavailable the positional reading
is kept, so graphs without shape annotations behave exactly as before.

Also stops emitting a ConvGradX at all when the node has no dX.

Found on a channel-split MobileNetV1 whose halves are each read twice: three of its
ConvGrad nodes emit dW alone. ResNet8 has one too (layer1_0_conv1_b) -- it does not
assert only because its shapes do not reach the branch, so that graph has been
mis-binding the same way silently.

Validation:
  * the channel-split MobileNetV1 goes from the assertion above to a clean deploy,
    planned L3 = 3246.8 KB
  * pytest -m "gap9_tiled and training and singlebuffer": 19 passed, 0 failed
  * pytest -m deeploy_internal: 44 failures before and after (40 test_dmas.py,
    4 test_deeploy_internal.py), all pre-existing
  * DeeployTest/test_split_convgrad.py builds the graphs by hand, runs in 0.16 s, and
    two of its five tests fail without this change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant