Add fuse_gelu pass to fuse erf-based GELU into a Gelu op - #320
Open
take-cheeze wants to merge 1 commit into
Open
Add fuse_gelu pass to fuse erf-based GELU into a Gelu op#320take-cheeze wants to merge 1 commit into
take-cheeze wants to merge 1 commit into
Conversation
Mirror onnxslim's GELU fusion: match the exact GELU decomposition 0.5 * x * (1 + erf(x / sqrt(2))), spelled as Div -> Erf -> Add -> Mul -> Mul, and rewrite it to a single standard-domain Gelu operator. Operand ordering of the two commutative multiplications (and of the addition) is accepted in either direction; the non-commutative division requires x to be the numerator. The intermediate nodes must be single-use so the leftover decomposition becomes dead code. Constants are matched whether they are initializers or Constant nodes. Since the standard-domain Gelu op was introduced in opset 20, the fusion is gated on opset >= 20 to keep the resulting model valid. Tests cover both operand orderings, the low-opset no-op case, and a real-world graph produced by torch.onnx (exported at opset 17 as the erf decomposition, version-converted to 20, then fused). The torch test is gated on torch being importable and skips gracefully if the exporter is unavailable or emits a different structure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EEN9xE3D2ePRzBwU1fsu3F Signed-off-by: take-cheeze <takechi101010@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirror onnxslim's GELU fusion: match the exact GELU decomposition 0.5 * x * (1 + erf(x / sqrt(2))), spelled as Div -> Erf -> Add -> Mul -> Mul, and rewrite it to a single standard-domain Gelu operator.
Operand ordering of the two commutative multiplications (and of the addition) is accepted in either direction; the non-commutative division requires x to be the numerator. The intermediate nodes must be single-use so the leftover decomposition becomes dead code. Constants are matched whether they are initializers or Constant nodes. Since the standard-domain Gelu op was introduced in opset 20, the fusion is gated on opset >= 20 to keep the resulting model valid.
Tests cover both operand orderings, the low-opset no-op case, and a real-world graph produced by torch.onnx (exported at opset 17 as the erf decomposition, version-converted to 20, then fused). The torch test is gated on torch being importable and skips gracefully if the exporter is unavailable or emits a different structure.
Claude-Session: https://claude.ai/code/session_01EEN9xE3D2ePRzBwU1fsu3F