Build: enable /OPT:REF for MSVC RelWithDebInfo

[bkaradzic-microsoft]

RelWithDebInfo is an optimized configuration, but on MSVC it currently keeps every unreferenced function that made it into the static libraries.

The reason is that MSVC turns /OPT:REF and /OPT:ICF off whenever /DEBUG is passed, and CMake's default RelWithDebInfo link flags are /debug /INCREMENTAL. So nothing gets stripped.

Turning incremental linking off and /OPT:REF on fixes that:

Playground.exe, RelWithDebInfo	Size
before	11,212,800 bytes
after	4,562,432 bytes
	−59.3%

Notes

• /OPT:ICF is deliberately left off. It measured zero additional bytes on top of /OPT:REF, and folding identical functions makes stack traces ambiguous — the opposite of what RelWithDebInfo is for.
• /INCREMENTAL:NO is required, not incidental: incremental linking is incompatible with /OPT:REF.
• Debug is untouched. Release and MinSizeRel already get /OPT:REF implicitly because they do not link with /DEBUG.
• Debuggability is unaffected — /DEBUG stays on and the PDB is still produced.

Verification

Confirmed through the generated Playground.vcxproj that OptimizeReferences is true for RelWithDebInfo only, and that EnableCOMDATFolding stays false in every configuration.

The stripped binary was then run through the full visual test suite headless, to confirm nothing that is actually reachable got removed:

Run complete. ran=304 passed=304 failed=0 missingRef=0 skipped=416
Playground: Finished in 4m 11.952s. (exit 0)

The flags are applied with add_link_options() and a $<$<CONFIG:RelWithDebInfo>:...> generator expression, so CMake owns the tokenization and folds them into the structured project properties rather than raw AdditionalOptions text:

	Debug	Release	MinSizeRel	RelWithDebInfo
OptimizeReferences	—	—	—	true
LinkIncremental	true	false	false	false

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adjust MSVC RelWithDebInfo linker behavior so unreferenced code is stripped (without enabling ICF), reducing binary size while keeping /DEBUG and PDB generation.

Changes:

• Overrides MSVC RelWithDebInfo linker flags to force /INCREMENTAL:NO and /OPT:REF.
• Removes any existing incremental/OPT:REF settings from the three RelWithDebInfo linker flag variables before appending the desired flags.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ryantrem]

Does this slow down the builds then, with incremental linking disabled?

[bkaradzic-microsoft]

Good question — I measured it rather than guessing, and the short answer is: a little, but you don't actually get a choice, and clean builds are unaffected.

Incremental relink of Playground.exe, RelWithDebInfo, warm .ilk, touching one .cpp and rebuilding:

	run 1	run 2	run 3	avg
master (/INCREMENTAL)	3.5s	3.4s	3.4s	3.43s
this PR (/INCREMENTAL:NO /OPT:REF)	4.5s	4.0s	4.1s	4.20s

So about +0.8s per edit-relink cycle, on a link that also drops the binary from 11,060,224 to 4,407,296 bytes.

The important part: /INCREMENTAL:NO is not a choice I made, it is what the linker does anyway. Incremental linking is incompatible with /OPT:REF. I tested this directly by applying /OPT:REF without /INCREMENTAL:NO:

LINK : warning LNK4075: ignoring '/INCREMENTAL' due to '/OPT:REF' specification

and the resulting binary was byte-for-byte the same size (4,407,296). So the flag only silences LNK4075 on every link of every target — it costs nothing extra. The real trade is /OPT:REF vs incremental linking, and you cannot have both.

Clean builds and CI are unaffected: they link once either way, and compilation dominates.

If the edit-relink cost is unwelcome for local iteration, two easy outs — happy to do either if you'd prefer:

1. Put it behind an option, e.g. BABYLON_NATIVE_OPT_REF defaulting to ON, so anyone iterating locally can turn it off.
2. Drop it from RelWithDebInfo and leave the size win to Release/MinSizeRel, which already get /OPT:REF implicitly since they don't link with /DEBUG. That loses the win where it's most useful though, since RelWithDebInfo is what CI builds and what ships in the size-sensitive scenarios.

My preference is to leave it as is: 0.8s on an incremental relink seems a fair price for a 60% smaller binary, and RelWithDebInfo is the config where the bloat actually matters.