Give closure calls the same ABI as direct calls - #47
Merged
Conversation
Fixes #44. The direct-call paths in stack_codegen and vm_codegen allocate an output slot for pointer-returned types, pass its address as an extra first argument, and use it as the call's result. They also coerce `&T` params to addresses and wrap sized arrays as slices. The CallClosure paths did none of that, so a closure or function pointer returning a struct passed one argument too few and consumed a result that was never produced — segfault on the stack and asm backends, a subtract overflow in the VM. Fold both CallClosure sites in each backend into a single translate_closure_call that mirrors the direct-call ABI. Also add Float32x4 to vm_codegen's returns_via_pointer. It was returning f32x4 as the address of the callee's frame slot, which the caller then copied from after the frame had been popped; direct calls survived that by luck, closure calls read zeros. This puts it on sret like every other pointer-represented type, matching the stack backend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DeNzkFkzMbdH3XPzitDmVa
This was referenced Aug 27, 2026
Follow-up to the closure-call ABI fix, addressing review findings. Check reference arguments on indirect calls. `reference_arg_positions` and `check_slice_aliasing` both bailed out unless the callee was an `Expr::Id` resolving to a `Decl::Func`, so the closure path accepted what the direct path rejects: `var f = addto; f(v, v)` aliased two borrowed parameters, and `var b = bump; b(v + 1)` passed a non-lvalue where an address was expected — `translate_lvalue` fell through to `translate_expr` and the value was used as an address (segfault on asm/stack, panic on vm, verifier failure on jit). Both checks now fall back to the callee expression's solved `Type::Func` domain, the same source the backends use to decide which arguments to pass by address. Route function-typed globals through the fat-pointer call path. The `is_local_var` test only consulted `self.variables`, so a global holding a closure fell into the direct-call path and trapped with a call stack overflow on vm/asm/stack. Replaced with `holds_fat_pointer`, which also accepts function-typed globals while leaving extern functions — which live in globals memory too — on the direct-call path. Factor `closure_param_types` out in the VM backend, matching the stack backend, so the `Type::Func` → `Type::Tuple` param extraction has one definition instead of three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DeNzkFkzMbdH3XPzitDmVa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #44.
Problem
The direct-call paths in
src/stack_codegen.rsandsrc/vm_codegen.rsallocate an output slot for pointer-returned types, pass its address as an extra first argument, and push that address as the call's result. They also coerce&Tparams to addresses (translate_lvalue) and wrap sized arrays as slices.The
CallClosurepaths did none of that. A closure or function pointer returning a struct passed one argument too few and consumed a result that was never produced:jitasmstackvmsrc/vm.rs:1290, "attempt to subtract with overflow"Fix
Fold both
CallClosuresites in each backend into a singletranslate_closure_callthat mirrors the direct-call ABI:returns_via_pointer(ret_ty), with the arg count bumped and the output storage's address used as the call's result (sret callees return void, so the float/void bridging only applies on the non-sret path);Referenceparams passed by address;Callee param types come from the solved
Type::Funcdomain of the function expression.Also: f32x4 returns in the VM backend
vm_codegen'sreturns_via_pointerwas missingType::Float32x4, so f32x4 returns used a "return the address of the callee's frame slot in r0, caller memcpys after the frame pops" convention. Direct calls survived that by luck; closure calls read zeros on bothvmandasm. AddingFloat32x4puts it on sret like every other pointer-represented type, matching what the stack backend already did.Tests
New golden tests, run against all four backends:
tests/cases/lambdas/closure_abi.lyte— struct, array, and tuple returns through a function pointer, a function-typed parameter, and a lambda;&i32param; sized array coerced to a slice param.tests/cases/simd/f32x4_closure_return.lyte— f32x4 returned through a function pointer and a function-typed parameter.cargo test --workspacepasses: 380 lib tests plusgolden_tests_jit/golden_tests_vm/golden_tests_asm/golden_tests_stack.cargo fmt --checkdiff count is unchanged (all pre-existing, in files this PR doesn't touch).🤖 Generated with Claude Code
https://claude.ai/code/session_01DeNzkFkzMbdH3XPzitDmVa