amd64: make the ELF front end work on x86-64 binaries - #263
Open
o2alexanderfedin wants to merge 1 commit into
Open
amd64: make the ELF front end work on x86-64 binaries#263o2alexanderfedin wants to merge 1 commit into
o2alexanderfedin wants to merge 1 commit into
Conversation
elfconv builds for x86 (ECV_X86=1), CI publishes an :amd64 image, --arch accepts amd64, the
ELF loader classifies it, and Remill has had amd64 semantics for years. Lifting an x86-64 ELF
still aborted on the first function, because the ELF FRONT END is AArch64-only in four places.
Upstream CI cannot catch this: the AMD64 arm runs `ctest`, which is Remill's own
run-amd64_avx instruction-semantics tests. Nothing in it lifts an ELF.
Each fix below was measured by re-running the lift and observing it get further.
1. lifter/TraceManager.cpp — __wrap_main discovery.
The AArch64 code finds main by matching LITERAL AArch64 encodings (nop 1f 20 03 d5,
bti c 5f 24 03 d5, b as (byte & 0xfc) == 0x14) on a fixed 4-byte stride. x86-64 is
variable-length, so the stride is meaningless and those bytes can never match; it fell
through to elfconv_runtime_error. The AArch64 block is now guarded by ELFCONV_AARCH64_BUILD
and an amd64 branch reads main out of _start instead. It is a different problem, not the
same one: AArch64 needs a function SYNTHESISED because __libc_start_call_main reaches main
through an unnamed trampoline, while the System V ABI passes main as the first argument to
__libc_start_main, so _start materialises its address (48 c7 c7 imm32 / 48 8d 3d rel32 /
bf imm32) and main is an ordinary named function. Not finding it is non-fatal on amd64
because there is nothing to synthesise.
2. backend/remill/lib/BC/Util.cpp — FindIndirectBrAddress register naming.
Matched any register whose name startswith("X"), i.e. AArch64 X0..X30. amd64 GPRs are
RAX/RBX/.../R8..R15 and match nothing; measured, the block for _init loads NEXT_PC and RAX.
Worse, amd64 HAS X-prefixed registers — XMM0..XMM15 — and a prefix match would silently
select a vector register as a branch target, which is worse than the abort. Now an exact
GPR-name match, with the original prefix rule kept as a FALLBACK so AArch64 behaviour
(XZR included) is bit-identical.
3. Util.cpp — memory-indirect branches, which AArch64 does not have.
x86-64 branches through memory (jmp *disp(%rip)); every PLT stub does. Measured IR:
%9 = load i64, ptr %NEXT_PC / store %9 -> PC / %10 = add %9, 7 / store %10 -> NEXT_PC
call @jmpi<Mn<uint64_t>>(runtime_manager, state, i64 5001240, ptr %NEXT_PC)
The destination is NOT a value in the block: 5001240 is the GOT slot address and JMPI
resolves it at RUNTIME, writing through the NEXT_PC pointer. So the target is NEXT_PC read
AFTER the semantic call, which is why the load is appended rather than searched for. Safe
because both call sites terminate the block only after this returns. Runs only where the
code previously called abort().
4. lifter/TraceManager.cpp — .plt entry splitting.
The AArch64 scan looks for a `br` terminator 4 bytes at a time. On amd64 it never
terminates an entry, so it emitted ONE function spanning the whole .plt — measured,
fn_plt_401020 covering all 384 bytes of a 24-entry table. amd64 PLT entries are 16 bytes by
ABI (classic push/jmp or the IBT endbr64 form); sh_entsize is not relied on because some
linkers leave it 0. If the size is not a multiple of 16 the bytes are copied without
claiming a function shape, which is better than inventing one.
5. lifter/TraceManager.cpp — AddRestDisasmFunc LOG(FATAL).
Aborted whenever no function started after the address, which is reachable on the first
real amd64 binary rather than a bug. Now bounded by the end of the containing code section
— the same rule the symbol walk already applies to the last function it finds. The FATAL is
KEPT for the case it was written for: an address in no code section at all.
NOT A REGRESSION ON AArch64, and this was measured rather than reasoned about. Same static
glibc hello, lifted in the :arm64 image before and after: 900 functions detected both times,
bitcode BYTE-IDENTICAL at sha256 ec2dc6f94af1ec7a0df56869ceff64f04577bbff6f01a9c6fadd6003cac5f8c4
(4,388,108 bytes), wasm identical at 5,654,547 bytes.
WHAT STILL DOES NOT WORK, stated plainly. A static glibc x86-64 binary now lifts many
functions and then fails on a control-flow target of 0x927371, which lies in no section of an
image spanning ~0x400000-0x4c6000. Upstream of that are FIVE unsupported instruction forms —
NOP_MEMv_0F1F_32 (55), NOP_MEMv_0F1F_16 (27), MOVZX_GPR32_MEMw (6), MOVSXD_GPR64_MEMd (4),
FADD_MEMmem32real (1) — which are reported and stepped over rather than aborting. x86-64 is
therefore NOT yet end-to-end; these five fixes are necessary and not sufficient.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the ELF front end work on x86-64 binaries. Today an amd64 ELF aborts before any bitcode
is produced, and the gap is not instruction semantics — Remill has had amd64 for years, CI
publishes an
:amd64image,--archacceptsx86, amd64, and the loader already classifiesBinaryArch::ARCH_AMD64. The gap is entry-point and PLT discovery inlifter/TraceManager.cpp,which matches literal AArch64 encodings on a fixed 4-byte stride:
x86-64 has variable-length instructions, so that stride is meaningless and those bytes can
never match. Measured with two toolchains as a control —
gcc -static -O2(entry0x401670)and
clang-16 -static -O2(entry0x401590) — bothSIGABRT, same message, no.bc, no.wasm.What this changes
mainfrom_startby decoding the real forms(
48 c7 c7 imm32,48 8d 3d rel32,bf imm32) instead of the AArch64nop/b/noptriple.brterminator.
AddRestDisasmFuncbounded by the containing code section, so an address past the lastknown function no longer runs off the end.
IsIndirectBranchTargetRegister(backend/remill/lib/BC/Util.cpp) given the exact amd64GPR set. It previously tested
startswith("X"), which only means anything on AArch64. Thelegacy prefix test is preserved so AArch64 behaviour is unchanged.
NEXT_PCinstead of failing to resolve.Evidence
ninja test_dependencies && cteston the:arm64image: 2/2 pass.gives byte-identical bitcode, sha256
ec2dc6f9…at 900 functions, and byte-identical wasm.Notes
README.mdlists "advance x86-64 instruction support" under Contributing; this is thefront-end half of that, not the semantics half.
This touches
lifter/TraceManager.cpp, which the ELF-loader PR also touches — if both arewanted, one will need a trivial rebase on the other. Happy to reorder or split further.