fix(training): route ConvGrad outputs by shape, not by position - #67
Open
runwangdl wants to merge 1 commit into
Open
fix(training): route ConvGrad outputs by shape, not by position#67runwangdl wants to merge 1 commit into
runwangdl wants to merge 1 commit into
Conversation
SplitConvGradPass read a fused ConvGrad's outputs positionally -- outputs[0] as dX,
outputs[1] as dW -- and its docstring stated "1 output (dX only): ConvGradX".
ORT emits dX only when something downstream needs it. A trainable layer whose input
needs no gradient, i.e. the first trainable layer of a network, emits dW alone. The
positional read takes that lone dW to be dX and wraps it in a ConvGradX node.
Nothing downstream can recover. Conv2DGradXParser binds it as `grad_in`, the tiler
slices it along its output channels, and ConvGradXTileConstraintBase reads those cube
offsets as BATCH offsets, deriving a dY cube at offset (k, 0, 0, 0) on a tensor whose
batch extent is 1:
AssertionError: Rectangle offset should be zero when the dimensions are the same.
Received rectangle HyperRectangle(offset=(1, 0, 0, 0), dims=(1, 16, 1, 1))
and reference shape (1, 16, 24, 24)
dX carries X's shape and dW carries W's, which tells them apart whatever order they
arrive in, so classify by shape. Where shapes are unavailable the positional reading
is kept, so graphs without shape annotations behave exactly as before.
Also stops emitting a ConvGradX at all when the node has no dX.
Found on a channel-split MobileNetV1 whose halves are each read twice: three of its
ConvGrad nodes emit dW alone. ResNet8 has one too (layer1_0_conv1_b) -- it does not
assert only because its shapes do not reach the branch, so that graph has been
mis-binding the same way silently.
Validation:
* the channel-split MobileNetV1 goes from the assertion above to a clean deploy,
planned L3 = 3246.8 KB
* pytest -m "gap9_tiled and training and singlebuffer": 19 passed, 0 failed
* pytest -m deeploy_internal: 44 failures before and after (40 test_dmas.py,
4 test_deeploy_internal.py), all pre-existing
* DeeployTest/test_split_convgrad.py builds the graphs by hand, runs in 0.16 s, and
two of its five tests fail without this change
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
SplitConvGradPassread a fusedConvGrad's outputs positionally —outputs[0]as dX,outputs[1]as dW — and its docstring said so explicitly: "1 output (dX only): ConvGradX".That assumption does not hold. ORT emits dX only when something downstream needs it. A trainable layer whose input needs no gradient — the first trainable layer of a network — emits dW alone. The positional read takes that lone dW to be dX and wraps it in a
ConvGradX.How it surfaces
Nothing downstream recovers.
Conv2DGradXParserbinds it asgrad_in; the tiler slices it along its output channels;ConvGradXTileConstraintBasereads those cube offsets as batch offsets and derives a dY cube at(k, 0, 0, 0)on a tensor whose batch extent is 1:The assertion fires five layers away from the cause, which is why it reads as a tiler bug.
Fix
dX carries X's shape and dW carries W's, which tells them apart whatever order they arrive in — so classify by shape. Where shapes are unavailable the positional reading is kept, so graphs without shape annotations behave exactly as before. A node with no dX no longer produces a
ConvGradXat all.Silent mis-binding today
ResNet8already contains such a node (layer1_0_conv1_b, dW only). It does not assert, because its shapes never reach that branch — so that graph has been binding a weight gradient as an input gradient all along, without any symptom.Validation
pytest -m "gap9_tiled and training and singlebuffer"pytest -m deeploy_internaltest_dmas.py, 4test_deeploy_internal.py), all pre-existingDeeployTest/test_split_convgrad.pybuilds the graphs by hand, runs in 0.16 s with no ONNX asset or backend, and covers dW-only, dX-only, both orders, the bias case, and which operands each split node reads. Two of its five tests fail without this change.🤖 Generated with Claude Code