Skip to content

Decouple kMaxUserCallDepth's guard from callStack_.size() - #65

Merged
revarbat merged 1 commit into
mainfrom
fix-callstack-depth-guard-conflation
Aug 1, 2026
Merged

Decouple kMaxUserCallDepth's guard from callStack_.size()#65
revarbat merged 1 commit into
mainfrom
fix-callstack-depth-guard-conflation

Conversation

@revarbat

@revarbat revarbat commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Fixes a real regression found while verifying 0.13.2 against Anklet.scad end-to-end: ERROR: Recursion too deep while calling function 'ident'.
  • enterUserCall's own depth guard checked callStack_.size() directly — but callStack_ also grows from cheap, zero-native-cost compiled module-call pushes (skipDepthGuard=true, Op::CallModule via pushBracketedModuleFrame), which cost no real native stack at all (serviced by driveVm's own heap-based loop). Those pushes still counted toward the SAME threshold a later genuinely interpreted call (skipDepthGuard=false) is guarded by — the exact same class of bug driveVmNativeDepth_ already exists to avoid for the other guard site.
  • Fix: a new nativeUserCallDepth_ counter, incremented/decremented only for skipDepthGuard=false pushes, checked instead of callStack_.size().
  • New regression test proves it, and is confirmed to actually fail without the fix (verified by temporarily reverting it locally).
  • Bumps to 0.13.3.

Known follow-up still outstanding (not fixed here)

Anklet.scad now gets past translate/multmatrix/color-wrapped recursion (0.13.2) and past the ident() false trip (this PR), but still doesn't fully render: children() (BOSL2's own children-forwarding mechanism) is the next and last remaining native-reentry source. Unlike the constructs Op::PushBuiltinWrap covers, children() forwards a runtime-varying set of children (resolved dynamically via ctx.childrenNodes/childrenCallerCtx per caller, not statically known from the AST) — so it can't reuse the same "compile children inline" trick. Fixing it needs its own design pass extending the explicit-stack VM to handle a runtime-varying splice target.

Test plan

  • Full oscad_eval_tests suite green (714/714), both OSCAD_BYTECODE_VM=0 and =1.
  • New test confirmed to fail without the fix (reverted locally, reproduced the exact "Recursion too deep while calling function" error), passes with it.

🤖 Generated with Claude Code

…2 ident() false trip)

enterUserCall's own depth guard checked callStack_.size() directly --
but callStack_ also grows from cheap, zero-native-cost compiled
module-call pushes (skipDepthGuard=true, Op::CallModule via
pushBracketedModuleFrame), which cost no real native stack at all
(serviced by driveVm's own heap-based loop). Those pushes still count
toward the SAME threshold a later genuinely interpreted call
(skipDepthGuard=false) is guarded by.

Caught for real: Anklet.scad's own ambient callStack_ depth (mostly
cheap compiled Op::CallModule pushes from its BOSL2 attachable() chain)
reached the mid-30s, tripping kMaxUserCallDepth=30 for BOSL2's plain
`ident()` function call -- even though the REAL native C++ nesting at
that point was nowhere near it. This is the exact same class of bug
driveVmNativeDepth_ already exists to avoid for the other guard site
(bytecode_vm.cpp) -- this is that same fix, applied here: a new
nativeUserCallDepth_ counter, incremented/decremented only for
skipDepthGuard=false pushes, checked instead of callStack_.size().

New regression test (DeepCompiledModuleRecursionDoesNotFalselyTripThe
InterpretedFunctionGuard) proves it: 100 levels of cheap compiled
module recursion followed by one genuinely-interpreted function call
(forced via a fast-continue breakpoint) no longer trips the guard --
confirmed to actually fail without the fix by temporarily reverting it.

Bumps to 0.13.3, since Anklet.scad (fixed for its translate/multmatrix/
color chain in 0.13.2) needs this too to fully render.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@revarbat
revarbat merged commit fbe5c76 into main Aug 1, 2026
3 checks passed
@revarbat
revarbat deleted the fix-callstack-depth-guard-conflation branch August 1, 2026 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant