Skip to content

Give closure calls the same ABI as direct calls - #47

Merged
wtholliday merged 2 commits into
mainfrom
fix-closure-sret-abi
Aug 27, 2026
Merged

Give closure calls the same ABI as direct calls#47
wtholliday merged 2 commits into
mainfrom
fix-closure-sret-abi

Conversation

@wtholliday

Copy link
Copy Markdown
Collaborator

Fixes #44.

Problem

The direct-call paths in src/stack_codegen.rs and src/vm_codegen.rs allocate an output slot for pointer-returned types, pass its address as an extra first argument, and push that address as the call's result. They also coerce &T params to addresses (translate_lvalue) and wrap sized arrays as slices.

The CallClosure paths did none of that. A closure or function pointer returning a struct passed one argument too few and consumed a result that was never produced:

struct P { x: i32, y: i32 }
mk(a: i32) -> P { var p: P; p.x = a; p.y = a * 2; p }
main { var g = mk; var p = g(3); assert(p.x == 3) }
backend before after
jit ok ok
asm segfault (139) ok
stack segfault (139) ok
vm panic src/vm.rs:1290, "attempt to subtract with overflow" ok

Fix

Fold both CallClosure sites in each backend into a single translate_closure_call that mirrors the direct-call ABI:

  • sret output pointer as the extra first argument when returns_via_pointer(ret_ty), with the arg count bumped and the output storage's address used as the call's result (sret callees return void, so the float/void bridging only applies on the non-sret path);
  • Reference params passed by address;
  • sized arrays wrapped as slices where the callee expects one.

Callee param types come from the solved Type::Func domain of the function expression.

Also: f32x4 returns in the VM backend

vm_codegen's returns_via_pointer was missing Type::Float32x4, so f32x4 returns used a "return the address of the callee's frame slot in r0, caller memcpys after the frame pops" convention. Direct calls survived that by luck; closure calls read zeros on both vm and asm. Adding Float32x4 puts it on sret like every other pointer-represented type, matching what the stack backend already did.

Tests

New golden tests, run against all four backends:

  • tests/cases/lambdas/closure_abi.lyte — struct, array, and tuple returns through a function pointer, a function-typed parameter, and a lambda; &i32 param; sized array coerced to a slice param.
  • tests/cases/simd/f32x4_closure_return.lyte — f32x4 returned through a function pointer and a function-typed parameter.

cargo test --workspace passes: 380 lib tests plus golden_tests_jit / golden_tests_vm / golden_tests_asm / golden_tests_stack. cargo fmt --check diff count is unchanged (all pre-existing, in files this PR doesn't touch).

🤖 Generated with Claude Code

https://claude.ai/code/session_01DeNzkFkzMbdH3XPzitDmVa

Fixes #44.

The direct-call paths in stack_codegen and vm_codegen allocate an output
slot for pointer-returned types, pass its address as an extra first
argument, and use it as the call's result. They also coerce `&T` params
to addresses and wrap sized arrays as slices. The CallClosure paths did
none of that, so a closure or function pointer returning a struct passed
one argument too few and consumed a result that was never produced —
segfault on the stack and asm backends, a subtract overflow in the VM.

Fold both CallClosure sites in each backend into a single
translate_closure_call that mirrors the direct-call ABI.

Also add Float32x4 to vm_codegen's returns_via_pointer. It was returning
f32x4 as the address of the callee's frame slot, which the caller then
copied from after the frame had been popped; direct calls survived that
by luck, closure calls read zeros. This puts it on sret like every other
pointer-represented type, matching the stack backend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DeNzkFkzMbdH3XPzitDmVa
Follow-up to the closure-call ABI fix, addressing review findings.

Check reference arguments on indirect calls. `reference_arg_positions`
and `check_slice_aliasing` both bailed out unless the callee was an
`Expr::Id` resolving to a `Decl::Func`, so the closure path accepted
what the direct path rejects: `var f = addto; f(v, v)` aliased two
borrowed parameters, and `var b = bump; b(v + 1)` passed a non-lvalue
where an address was expected — `translate_lvalue` fell through to
`translate_expr` and the value was used as an address (segfault on
asm/stack, panic on vm, verifier failure on jit). Both checks now fall
back to the callee expression's solved `Type::Func` domain, the same
source the backends use to decide which arguments to pass by address.

Route function-typed globals through the fat-pointer call path. The
`is_local_var` test only consulted `self.variables`, so a global holding
a closure fell into the direct-call path and trapped with a call stack
overflow on vm/asm/stack. Replaced with `holds_fat_pointer`, which also
accepts function-typed globals while leaving extern functions — which
live in globals memory too — on the direct-call path.

Factor `closure_param_types` out in the VM backend, matching the stack
backend, so the `Type::Func` → `Type::Tuple` param extraction has one
definition instead of three.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DeNzkFkzMbdH3XPzitDmVa
@wtholliday
wtholliday merged commit b8b8d7b into main Aug 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stack/VM: closures and function pointers with struct (sret) return types crash

1 participant