Flatten narrow inter-stage varying arrays for the fxc backend

[bkaradzic-microsoft]

Fixes a D3D11/fxc compilation hang on Babylon.js cascaded shadow maps.

The problem

SPIRV-Cross emits an array-typed member in the HLSL interface struct for an array-typed varying:

struct SPIRV_Cross_Input
{
    float vDepthMetric0[4] : TEXCOORD5;
};

fxc turns that into an indexable input register range, and rejects the range unless every register in it shares a write mask. A narrow (fewer than 4 component) element type can never satisfy that, because each register uses only .x:

error X8000: D3D11 Internal Compiler Error: Invalid Bytecode: Masks (and if pixel
shader, also interpolation mode) on all input registers in an index range must be
identical. Input register [1] does not match with others in the index range from 0 to 3.

fxc does not simply fail here, it then hangs, so D3D11 shader compilation never returns.

Babylon.js cascaded shadow maps are the trigger. lightFragmentDeclaration.fx declares:

varying float vDepthMetric{X}[SHADOWCSMNUM_CASCADES{X}];

Its companion varying vec4 vPositionFromLight{X}[...] is unaffected, since 4-component elements do give every register an identical mask.

The fix

A new glslang traverser rewrites each narrow array-typed inter-stage varying, before SPIR-V generation, into one varying per element plus a plain (non-varying) global array that all existing references are repointed at. The interface then contains no array, so fxc emits no indexable range:

// before
out float vDepthMetric0[4];
...
vDepthMetric0[i] = depth;

// after
out float vDepthMetric0_0;
out float vDepthMetric0_1;
out float vDepthMetric0_2;
out float vDepthMetric0_3;
float vDepthMetric0[4];
...
vDepthMetric0[i] = depth;
// appended to end of main():
vDepthMetric0_0 = vDepthMetric0[0];
vDepthMetric0_1 = vDepthMetric0[1];
// ...

Keeping the global array is what preserves dynamic indexing, which the naive rewrite to per-element varyings cannot do — lightFragment.fx selects the cascade with an index computed per fragment at runtime, not a literal:

https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/core/src/Shaders/ShadersInclude/lightFragment.fx#L283

Routing through a global also keeps writes performed by helper functions working, since they observe the global rather than a copy. Element-wise copies bridge the two forms: prepended to fragment main, appended to vertex main.

Locations

No special handling is needed. BabylonNative sets explicit locations only on vertex inputs and never calls mapIO, so inter-stage varyings reach SPIRV-Cross undecorated and SPIRV-Cross assigns them itself via get_vacant_location(), walking the varying list sorted by name. Both stages see the same names and sort identically, so they agree — before and after this change.

Scope

• Applied only on the DXBC path. dxc accepts the indexable range, so the DXIL path is untouched.
• Deliberately skips vertex inputs (owned by AssignLocationsAndNamesToVertexVaryings*) and fragment outputs (render targets).
• Only narrow (getVectorSize() < 4), sized, 1-D, non-struct, non-matrix, non-builtin arrays qualify.

Early returns

Vertex main() is rejected with a clear error if it can return before reaching the appended copies, since those would be jumped over and stale varyings emitted. No Babylon.js shader does this today; the check is there so it fails the build instead of mis-rendering.

Testing

The comprehensive GLSL cross-compilation test grows a scalar varying array read back through a runtime-derived index. Verified both directions on Win32 D3D11:

• with the pass — 21/21 unit tests pass, CompileComprehensiveGLSL completes in ~0.6s
• with the pass commented out — reproduces the X8000 error above and then hangs (>180s), matching the fxc behaviour described

Full enabled Playground validation sweep is unchanged at 301/302. The single failure (Outline) is an unrelated pre-existing Unsupported alpha mode: -1 that reproduces without this change.

Note

This supersedes BabylonJS/SPIRV-Cross#14, which fixed the same issue inside SPIRV-Cross. Doing it here avoids carrying a patch on a third-party library.

[Copilot]

Pull request overview

This PR adds a DXBC-only glslang AST rewrite that eliminates narrow (scalar/vec2/vec3) inter-stage varying arrays from the generated SPIR-V/HLSL interface to avoid an fxc DX11 compiler hang triggered by Babylon.js cascaded shadow maps.

Changes:

• Introduces a new traverser (FlattenNarrowVaryingArrays) that replaces each qualifying varying array with per-element varyings plus a plain global array, bridged by element-wise copies inserted into main().
• Wires the new pass into the DXBC compiler pipeline.
• Extends the comprehensive GLSL cross-compilation unit test to include a dynamically indexed scalar varying array, and updates the generated JS dist artifact accordingly.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
Plugins/ShaderCompiler/Source/ShaderCompilerTraversers.h	Declares the new FlattenNarrowVaryingArrays traverser and documents the fxc hang rationale/scope.
Plugins/ShaderCompiler/Source/ShaderCompilerTraversers.cpp	Implements the narrowing-array flattening logic, including main() copy insertion and vertex early-return guard.
Plugins/ShaderCompiler/Source/ShaderCompilerDXBC.cpp	Enables the new pass on the DXBC (fxc) compilation path.
Apps/UnitTests/JavaScript/src/tests.shaderCompilation.comprehensiveGLSL.ts	Adds a scalar varying array that is read via a runtime-derived index in the fragment shader.
Apps/UnitTests/JavaScript/dist/tests.shaderCompilation.comprehensiveGLSL.js	Updates the checked-in generated build output to match the TS test change.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.