diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 350fbce6..de9edf2c 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -2,24 +2,34 @@ name: Build ePass on: push: - branches: [ "main", "dev" ] - -env: - BUILD_TYPE: Release + branches: ["main", "dev"] + pull_request: + branches: ["main", "dev"] jobs: - build: + rust-core: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 - - - name: Install Dependency - run: sudo apt install -y libbpf-dev - - - name: Configure CMake - run: cd core && make configure - - - name: Build - run: cd core && make - + - uses: actions/checkout@v4 + + - name: Install system dependencies + run: | + sudo apt-get update + sudo apt-get install -y \ + clang \ + libbpf-dev \ + libelf-dev \ + pkg-config \ + zlib1g-dev + + - name: Install Rust toolchain + uses: dtolnay/rust-toolchain@stable + + - name: Build core-rs + run: cargo build --release --workspace + working-directory: core-rs + + - name: Test core-rs + run: cargo test --release --workspace + working-directory: core-rs diff --git a/.gitignore b/.gitignore index 2673917d..6fe0c6f5 100644 --- a/.gitignore +++ b/.gitignore @@ -53,3 +53,6 @@ modules.order Module.symvers Mkfile.old dkms.conf +/core-rs/target +/paper +/core-rs/epass-ir/tests/epir diff --git a/.gitmodules b/.gitmodules index 84ed8167..be2f335f 100644 --- a/.gitmodules +++ b/.gitmodules @@ -4,9 +4,6 @@ [submodule "third-party/ePass-falcolib"] path = third-party/ePass-falcolib url = git@github.com:OrderLab/ePass-falcolib.git -[submodule "ePass-kernel"] - path = ePass-kernel - url = git@github.com:OrderLab/ePass-kernel.git [submodule "third-party/ePass-libbpf"] path = third-party/ePass-libbpf url = git@github.com:OrderLab/ePass-libbpf.git diff --git a/README.md b/README.md index eadc5c66..3f8cd934 100644 --- a/README.md +++ b/README.md @@ -2,81 +2,121 @@ [![Build ePass](https://github.com/OrderLab/ePass/actions/workflows/build.yml/badge.svg)](https://github.com/OrderLab/ePass/actions/workflows/build.yml) -ePass is an in-kernel LLVM-like compiler framework that introduces an SSA-based intermediate representation (IR) for eBPF programs. It provides a lifter that lifts eBPF bytecode to ePass IR, a pass runner that runs user-defined passes, and a code generator that compiles IR to eBPF bytecode. Users could write flexible passes using our LLVM-like APIs to analyze and manipulate the IR. -ePass also provides an in-kernel supervisor that cooperates ePass core with the verifier to improve its flexibility (i.e. reduce false rejections) and safety (i.e. reduce false acceptance at runtime). It could also be used in userspace for testing. +ePass is an SSA-based compiler framework for eBPF programs. It lifts eBPF +bytecode into an LLVM-like IR, runs configurable IR passes, and lowers IR back to +eBPF bytecode for kernel verifier/JIT consumption. -## Key Features +The actively developed core is the Rust userspace library under `core-rs/`. The +old C implementation under `deprecated/core/` remains as historical reference. -- **IR-based compilation**: Converts BPF programs to an SSA-based intermediate representation for code rewriting -- **Flexible passes**: ePass core provides various APIs to analyze and manipulate the IR, allowing users to write flexible passes including static analyzing, runtime checks, and optimization. -- **Verifier aware**: ePass works with the existing verifier. The verifier is better for static verification while ePass focuses more on code rewriting and runtime verification. -- **User-friendly debugging**: ePass supports compiling to both kernel and userspace for easier debugging. +## Current architecture -> ⚠️ Warning: ePass is under active development and we are improving its usability and safety for production use. We welcome any suggestions and feedback. Feel free to open issues or [contact us](#contact-and-citation). +```text +eBPF bytecode + -> lift + -> ePass SSA IR + -> pass manager + -> register allocation + codegen + -> eBPF bytecode + -> kernel verifier +``` -## Design Goals +Key components: -- Flexible passes for diverse use cases -- Working with existing verifier instead of replacing its -- Keeping kernel safety -- Support both userspace and kernel +- `core-rs/epass-ir`: pure userspace Rust library. No kernel headers and no + libbpf dependency. +- `core-rs/epasstool`: CLI for dump-format and ELF-object workflows. ELF support + uses libbpf only in the CLI. +- `third-party/ePass-libbpf`: patched libbpf that can call the Rust C ABI + (`epass_run`) before loading programs. +- `third-party/ePass-bpftool`: bpftool built against the patched libbpf for + verifier testing. -## Prerequisites +## Features -- **clang >= 17** -- **Ninja** (optional, for faster compilation) -- **libbpf** +- SSA IR with arena allocation and typed handles (`InsnId`, `BbId`). +- eBPF bytecode lifter with CFG discovery and SSA construction. +- Pass manager with default-enabled and optional passes, pass-specific options, + pass-owned ordering, and post-pass validation. +- Built-in passes: `const_prop`, `phi`, `optimize_ir`, optional `dump_ir`. +- Register allocation and codegen back to eBPF bytecode. +- Parseable `.epir` IR text dump/load format for debugging. +- `IrBuilder` and CFG editing utilities for pass authors. +- C ABI for patched libbpf integration. -## Project Components +## Build -- `ePass core`: the core compiler framework, including a userspace CLI -- `ePass kernel`: Linux kernel 6.5 with ePass core built-in, along with the kernel component and kernel passes -- `ePass libbpf`: libbpf with ePass support for userspace ePass testing +Rust core and CLI: -There are some testing projects including `bpftool`, `xdp-tools`, `falcolib` in `third-party`. They depend on `ePass libbpf`. +```bash +cd core-rs +cargo build --release +cargo test --release +``` -### ePass Overview +Patched libbpf and bpftool for verifier testing: -![Overview](./docs/overview.png) +```bash +cd third-party/ePass-libbpf/src +make -j -### ePass Core +cd ../../ePass-bpftool/src +make -j +``` -![Core Architecture](./docs/core_design.png) +## CLI quick start -#### ePass Built-in Passes (Selected demo passes) +```bash +cd core-rs -- Instruction counter pass: runtime instruction limit check -- MSan pass: memory sanitizer for stack memory access -- Masking pass: fix a false-rejection due to lack of type -- Helper validation pass: validate helper function arguments to avoid CVE -- Code compaction pass: an optimization pass to reduce code size +# Rewrite an ELF object's selected BPF program and emit dump-format output. +./target/release/epasstool read -s prog -F log -o out.txt ../test/output/progs_simple1.o -## Quick Start +# Process dump-format input: one packed u64 per line. +./target/release/epasstool read -F log -o out.txt prog.txt -There are two ways to use ePass. The first way is to build a linux kernel with ePass builtin, which is used for production. Users could specify ePass options when calling the `BPF` system call. See [Kernel Testing](docs/KERNEL_TESTING.md). +# Print without rewriting. +./target/release/epasstool print --gopt print_dump ../test/output/progs_simple1.o -The second way is to build ePass in userspace and testing programs without changing the kernel, which is used mainly for testing. Users could specify ePass options via environment variable and use `ePass libbpf`. Programs will be modified in userspace before sending to the kernel. See [Userspace Testing](docs/USERSPACE_TESTING.md). +# Dump lifted IR before normal passes. +./target/release/epasstool read -P --popt 'dump_ir(/tmp/prog.epir)' -s prog ../test/output/progs_simple1.o -We recommend users trying ePass in userspace before switching to the ePass kernel version! +# Load IR directly instead of lifting bytecode, then run passes and codegen. +./target/release/epasstool read --gopt load_ir=/tmp/prog.epir -F log -o out.txt dummy.txt +``` -## Testing +## libbpf / verifier testing -See [Testing](./docs/TESTING.md). +The patched libbpf runs ePass when enabled by environment variable: -## Development and Contribution +```bash +sudo LIBBPF_ENABLE_EPASS=1 \ + LIBBPF_EPASS_GOPT='verbose=1' \ + third-party/ePass-bpftool/src/bpftool prog load test.o /sys/fs/bpf/test +``` -See [Development](./docs/CONTRIBUTION_GUIDE.md). +The rewritten bytecode is submitted to the kernel verifier. Use `bpftool prog +show pinned ...` to compare `xlated` byte sizes and confirm the modified program +was loaded. -## Roadmap and Future Work +## Documentation -- Support bpf-to-bpf calls -- Support loadng ePass IR to kernel -- Support compiling ePass IR to machine code directly in JIT +- [Rust core architecture](docs/CORE_RS.md) +- [Pass manager and pass options](docs/PASS_MANAGER.md) +- [Writing passes](docs/WRITING_PASSES.md) +- [IR text format](docs/IR_TEXT.md) +- [IR builder and instruction construction](docs/CREATE_INSTRUCTION.md) +- [Userspace / libbpf testing](docs/USERSPACE_TESTING.md) +- [Testing](docs/TESTING.md) -## Contact and Citation +## Status -Feel free to open an issue for question, bug report or feature request! You could also email . +- Core lifter, IR, pass framework, codegen, IR text load/dump, and verifier + testing path are active in `core-rs`. +- Several old C demo/instrumentation passes (MSan, insn counter, masking, + helper validation, etc.) are not yet fully ported. +- bpf-to-bpf calls and atomic memory ops remain unsupported/limited. -## Acknowledgement +## Contact -ePass is sponsored by [OrderLab](https://orderlab.io/) from University of Michigan. +Feel free to open an issue or email . diff --git a/core-rs/Cargo.lock b/core-rs/Cargo.lock new file mode 100644 index 00000000..b362da65 --- /dev/null +++ b/core-rs/Cargo.lock @@ -0,0 +1,178 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "cc" +version = "1.2.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" + +[[package]] +name = "epass-ir" +version = "0.2.0" +dependencies = [ + "indexmap", + "smallvec", + "thiserror", +] + +[[package]] +name = "epasstool" +version = "0.2.0" +dependencies = [ + "epass-ir", + "libbpf-sys", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown", +] + +[[package]] +name = "libbpf-sys" +version = "1.7.0+v1.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a109478760b2900aa2a6f2087e9d0de1d9c535b1758602af2845d5d2ccfaed7c" +dependencies = [ + "cc", + "nix", + "pkg-config", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", +] + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" diff --git a/core-rs/Cargo.toml b/core-rs/Cargo.toml new file mode 100644 index 00000000..9a556050 --- /dev/null +++ b/core-rs/Cargo.toml @@ -0,0 +1,11 @@ +[workspace] +resolver = "2" +members = ["epass-ir", "epasstool"] + +[workspace.package] +version = "0.2.0" +edition = "2021" +license = "GPL-2.0-only" + +[profile.release] +opt-level = 3 diff --git a/core-rs/README.md b/core-rs/README.md new file mode 100644 index 00000000..40284c75 --- /dev/null +++ b/core-rs/README.md @@ -0,0 +1,203 @@ +# ePass Rust Core + +`core-rs` is the actively-developed userspace ePass compiler core. It is a +from-scratch Rust re-architecture of the original C `core/` implementation. + +The core library (`epass-ir`) has no kernel or libbpf dependency. It exposes an +SSA IR, a pass manager, a register-allocating code generator, an IR text +serializer/parser, and a small C ABI used by patched libbpf. + +## Workspace layout + +```text +core-rs/ +├── epass-ir/ # pure Rust compiler library +└── epasstool/ # CLI; uses epass-ir and libbpf-sys for ELF I/O +``` + +## Pipeline + +```text +eBPF bytecode + -> lift + -> SSA IR + -> pass manager + -> codegen prep + -> liveness + register allocation + -> normalization/emission + -> eBPF bytecode +``` + +The default pass pipeline is pass-owned and ordered by pass registration: + +```text +const_prop -> phi -> optimize_ir +``` + +Optional passes, such as `dump_ir`, are enabled with `--popt` and order +themselves. For example: + +```bash +epasstool read -P --popt 'dump_ir(/tmp/prog.epir)' prog.o +``` + +runs: + +```text +dump_ir -> const_prop -> phi -> optimize_ir +``` + +## Main modules + +| Module | Responsibility | +|--------|----------------| +| `bytecode` | `BpfInsn` and BPF opcode constants; packed `u64` encode/decode | +| `ir` | Arena IR (`Function`, `BasicBlock`, `Insn`, `Value`), `IrBuilder`, CFG utilities, printer, `.epir` text I/O | +| `lift` | eBPF bytecode -> SSA IR with CFG discovery and SSA construction | +| `cfg` | Reachable block layout and end-block computation | +| `check` | IR verifier; includes def-use, branch, and phi predecessor checks | +| `pass` | Pass trait, pass option parsing, pass-owned ordering, postprocess | +| `passes` | Builtins: `dump_ir`, `const_prop`, `phi`, `optimize_ir` | +| `cg` | Code generation: liveness, interference, RA, spilling, SSA-out, emission | +| `pipeline` | End-to-end driver (`autorun`, `run_passes_only`) | +| `ffi` | C ABI (`epass_run`) for patched libbpf | + +## Build and test + +```bash +cargo build --release +cargo test --release +``` + +## CLI examples + +```bash +# Rewrite an ELF BPF program and dump rewritten bytecode. +./target/release/epasstool read -s prog -F log -o out.txt ../test/output/progs_simple1.o + +# Process dump-format text input. +./target/release/epasstool read -F log -o out.txt prog.txt + +# Print a program. +./target/release/epasstool print --gopt print_dump ../test/output/progs_simple1.o + +# Dump lifted IR before other passes. +./target/release/epasstool read -P --popt 'dump_ir(/tmp/prog.epir)' -s prog ../test/output/progs_simple1.o + +# Load IR directly, bypassing lift, then run passes/codegen. +./target/release/epasstool read --gopt load_ir=/tmp/prog.epir -F log -o out.txt dummy.txt + +# Disable a disableable default pass. +./target/release/epasstool read --popt '!const_prop' -s prog ../test/output/progs_simple1.o +``` + +## Global options (`--gopt`) + +Comma-separated: + +- `verbose=` +- `disable_coalesce` +- `print_bpf` +- `print_dump` +- `print_detail` +- `print_bpf_detail` +- `no_prog_check` +- `printonly` +- `dotgraph` +- `load_ir=` + +## Pass options (`--popt`) + +Pass options do not determine order. They enable/disable/configure passes; each +pass owns its ordering. + +Syntax: + +```text +pass +pass(arg) +!pass +``` + +Examples: + +```text +dump_ir(/tmp/a.epir) +dump_ir(path=/tmp/a.epir) +!const_prop +optimize_ir(no_dead_elim) +``` + +`phi` is not disableable. + +## Library usage + +```rust +use epass_ir::{autorun, default_passes, Env, Opts, BpfInsn}; + +let mut env = Env::new(Opts::default(), program /* Vec */); +let passes = default_passes(); +autorun(&mut env, &passes)?; +let rewritten = env.insns; +``` + +Dump/load IR: + +```rust +let text = epass_ir::dump_ir(&func); +let func = epass_ir::load_ir_str(&text)?; +``` + +Build IR in passes: + +```rust +use epass_ir::ir::{IrBuilder, InsertPos, Value, AluOp, BinOp}; + +let mut b = IrBuilder::before_terminator(func, bb); +let x = b.bin(BinOp::Add, AluOp::Alu64, Value::const64(1), Value::const64(2)); +``` + +## Validation + +Useful checks: + +```bash +cargo test --release + +# Generate EPIR test corpus +./target/release/epasstool read -P --popt 'dump_ir(/tmp/prog.epir)' -s prog ../test/output/progs_simple1.o + +# Verifier path through patched libbpf/bpftool +sudo LIBBPF_ENABLE_EPASS=1 third-party/ePass-bpftool/src/bpftool prog load test.o /sys/fs/bpf/test +``` + +## More documentation + +- [Rust core architecture](../docs/CORE_RS.md) +- [Pass manager](../docs/PASS_MANAGER.md) +- [Writing passes](../docs/WRITING_PASSES.md) +- [Instruction construction / `IrBuilder`](../docs/CREATE_INSTRUCTION.md) +- [IR text format](../docs/IR_TEXT.md) +- [Testing](../docs/TESTING.md) + +## Status + +Implemented: + +- lifter and SSA IR; +- pass manager with pass-owned ordering; +- `const_prop`, `phi`, `optimize_ir`, optional `dump_ir`; +- `.epir` dump/load; +- `IrBuilder` and CFG/BB utilities; +- register allocation with post-spill fallback; +- C ABI for libbpf. + +Not yet fully ported from old C core: + +- MSan; +- instruction counter; +- masking; +- helper validation; +- div-by-zero instrumentation; +- code compaction demo pass; +- verifier-dependent kernel passes. diff --git a/core/bpftests/.gitignore b/core-rs/bpftests/.gitignore similarity index 100% rename from core/bpftests/.gitignore rename to core-rs/bpftests/.gitignore diff --git a/core/bpftests/1mloop.c b/core-rs/bpftests/1mloop.c similarity index 100% rename from core/bpftests/1mloop.c rename to core-rs/bpftests/1mloop.c diff --git a/core/bpftests/Makefile b/core-rs/bpftests/Makefile similarity index 100% rename from core/bpftests/Makefile rename to core-rs/bpftests/Makefile diff --git a/core/bpftests/alu64.c b/core-rs/bpftests/alu64.c similarity index 100% rename from core/bpftests/alu64.c rename to core-rs/bpftests/alu64.c diff --git a/core/bpftests/asm.c b/core-rs/bpftests/asm.c similarity index 100% rename from core/bpftests/asm.c rename to core-rs/bpftests/asm.c diff --git a/core/bpftests/compact_opt.c b/core-rs/bpftests/compact_opt.c similarity index 100% rename from core/bpftests/compact_opt.c rename to core-rs/bpftests/compact_opt.c diff --git a/core/bpftests/complex.c b/core-rs/bpftests/complex.c similarity index 100% rename from core/bpftests/complex.c rename to core-rs/bpftests/complex.c diff --git a/core/bpftests/empty.c b/core-rs/bpftests/empty.c similarity index 100% rename from core/bpftests/empty.c rename to core-rs/bpftests/empty.c diff --git a/core/bpftests/exit.c b/core-rs/bpftests/exit.c similarity index 100% rename from core/bpftests/exit.c rename to core-rs/bpftests/exit.c diff --git a/core/bpftests/falco b/core-rs/bpftests/falco similarity index 100% rename from core/bpftests/falco rename to core-rs/bpftests/falco diff --git a/core/bpftests/goto.c b/core-rs/bpftests/goto.c similarity index 100% rename from core/bpftests/goto.c rename to core-rs/bpftests/goto.c diff --git a/core/bpftests/localcall.c b/core-rs/bpftests/localcall.c similarity index 100% rename from core/bpftests/localcall.c rename to core-rs/bpftests/localcall.c diff --git a/core/bpftests/log/test1.txt b/core-rs/bpftests/log/test1.txt similarity index 100% rename from core/bpftests/log/test1.txt rename to core-rs/bpftests/log/test1.txt diff --git a/core/bpftests/log/test2.txt b/core-rs/bpftests/log/test2.txt similarity index 100% rename from core/bpftests/log/test2.txt rename to core-rs/bpftests/log/test2.txt diff --git a/core/bpftests/loop1.c b/core-rs/bpftests/loop1.c similarity index 100% rename from core/bpftests/loop1.c rename to core-rs/bpftests/loop1.c diff --git a/core/bpftests/loop2.c b/core-rs/bpftests/loop2.c similarity index 100% rename from core/bpftests/loop2.c rename to core-rs/bpftests/loop2.c diff --git a/core/bpftests/loop3.c b/core-rs/bpftests/loop3.c similarity index 100% rename from core/bpftests/loop3.c rename to core-rs/bpftests/loop3.c diff --git a/core/bpftests/loop3r.c b/core-rs/bpftests/loop3r.c similarity index 100% rename from core/bpftests/loop3r.c rename to core-rs/bpftests/loop3r.c diff --git a/core/bpftests/lsm.c b/core-rs/bpftests/lsm.c similarity index 100% rename from core/bpftests/lsm.c rename to core-rs/bpftests/lsm.c diff --git a/core/bpftests/map1.c b/core-rs/bpftests/map1.c similarity index 100% rename from core/bpftests/map1.c rename to core-rs/bpftests/map1.c diff --git a/core/bpftests/mask.c b/core-rs/bpftests/mask.c similarity index 100% rename from core/bpftests/mask.c rename to core-rs/bpftests/mask.c diff --git a/core/bpftests/mem1.c b/core-rs/bpftests/mem1.c similarity index 100% rename from core/bpftests/mem1.c rename to core-rs/bpftests/mem1.c diff --git a/core/bpftests/mem2.c b/core-rs/bpftests/mem2.c similarity index 100% rename from core/bpftests/mem2.c rename to core-rs/bpftests/mem2.c diff --git a/core/bpftests/msan.c b/core-rs/bpftests/msan.c similarity index 100% rename from core/bpftests/msan.c rename to core-rs/bpftests/msan.c diff --git a/core/bpftests/neg.c b/core-rs/bpftests/neg.c similarity index 100% rename from core/bpftests/neg.c rename to core-rs/bpftests/neg.c diff --git a/core/bpftests/output/.gitignore b/core-rs/bpftests/output/.gitignore similarity index 100% rename from core/bpftests/output/.gitignore rename to core-rs/bpftests/output/.gitignore diff --git a/core/bpftests/product.c b/core-rs/bpftests/product.c similarity index 100% rename from core/bpftests/product.c rename to core-rs/bpftests/product.c diff --git a/core/bpftests/readmap.c b/core-rs/bpftests/readmap.c similarity index 100% rename from core/bpftests/readmap.c rename to core-rs/bpftests/readmap.c diff --git a/core/bpftests/ringbuf.c b/core-rs/bpftests/ringbuf.c similarity index 100% rename from core/bpftests/ringbuf.c rename to core-rs/bpftests/ringbuf.c diff --git a/core/bpftests/simple1.c b/core-rs/bpftests/simple1.c similarity index 100% rename from core/bpftests/simple1.c rename to core-rs/bpftests/simple1.c diff --git a/core/bpftests/simple2.c b/core-rs/bpftests/simple2.c similarity index 100% rename from core/bpftests/simple2.c rename to core-rs/bpftests/simple2.c diff --git a/core/bpftests/simple_product.c b/core-rs/bpftests/simple_product.c similarity index 100% rename from core/bpftests/simple_product.c rename to core-rs/bpftests/simple_product.c diff --git a/core/bpftests/spillconst.c b/core-rs/bpftests/spillconst.c similarity index 100% rename from core/bpftests/spillconst.c rename to core-rs/bpftests/spillconst.c diff --git a/core/bpftests/stackmem.c b/core-rs/bpftests/stackmem.c similarity index 83% rename from core/bpftests/stackmem.c rename to core-rs/bpftests/stackmem.c index 1b387544..a87a5716 100644 --- a/core/bpftests/stackmem.c +++ b/core-rs/bpftests/stackmem.c @@ -6,10 +6,9 @@ char LICENSE[] SEC("license") = "GPL"; SEC("tracepoint/syscalls/sys_enter_mount") int prog(void *ctx) { - - volatile int a; + volatile int a; bpf_printk("r10 - 8: %p: %d\n", &a, a); - long* r10 = (long*)((int*)(&a) + 1); + long *r10 = (long *)((int *)(&a) + 1); bpf_printk("r10: %p: %d\n", r10, *r10); // Invalid bpf_printk("entry_prog: tail call failed\n"); diff --git a/core/bpftests/str.c b/core-rs/bpftests/str.c similarity index 100% rename from core/bpftests/str.c rename to core-rs/bpftests/str.c diff --git a/core/bpftests/tailcall.c b/core-rs/bpftests/tailcall.c similarity index 94% rename from core/bpftests/tailcall.c rename to core-rs/bpftests/tailcall.c index 2e2e16d9..db62d7e2 100644 --- a/core/bpftests/tailcall.c +++ b/core-rs/bpftests/tailcall.c @@ -19,7 +19,7 @@ int entry_prog(void *ctx) __u32 index = 1; // int a[5] = { 1, 2, 3, 4, 5 }; - volatile int a = 100; + volatile int a = 100; bpf_printk("p1 %p\n", &a); // Tail call to index 1 @@ -33,7 +33,7 @@ int entry_prog(void *ctx) SEC("tracepoint/syscalls/sys_enter_mount") int next_prog(void *ctx) { - volatile int a; + volatile int a; bpf_printk("p2 %p\n", &a); bpf_printk("p2 %d\n", a); return XDP_PASS; diff --git a/core/bpftests/tailcall_us.c b/core-rs/bpftests/tailcall_us.c similarity index 100% rename from core/bpftests/tailcall_us.c rename to core-rs/bpftests/tailcall_us.c diff --git a/core/bpftests/test_asm_easy.c b/core-rs/bpftests/test_asm_easy.c similarity index 100% rename from core/bpftests/test_asm_easy.c rename to core-rs/bpftests/test_asm_easy.c diff --git a/core/bpftests/test_spill.c b/core-rs/bpftests/test_spill.c similarity index 100% rename from core/bpftests/test_spill.c rename to core-rs/bpftests/test_spill.c diff --git a/core/bpftests/uninit1.c b/core-rs/bpftests/uninit1.c similarity index 100% rename from core/bpftests/uninit1.c rename to core-rs/bpftests/uninit1.c diff --git a/core-rs/epass-ir/Cargo.toml b/core-rs/epass-ir/Cargo.toml new file mode 100644 index 00000000..145c99e2 --- /dev/null +++ b/core-rs/epass-ir/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "epass-ir" +version.workspace = true +edition.workspace = true +license.workspace = true +description = "ePass: an SSA-based intermediate representation and compiler framework for eBPF programs (userspace)." + +[lib] +# `rlib` lets the Rust `epasstool` use this crate normally; `staticlib`/`cdylib` +# produce C-linkable `libepass_ir.a` / `libepass_ir.so` for the libbpf integration. +crate-type = ["rlib", "staticlib", "cdylib"] + +[dependencies] +indexmap = "2" +smallvec = "1" +thiserror = "2" + +[dev-dependencies] diff --git a/core-rs/epass-ir/examples/roundtrip.rs b/core-rs/epass-ir/examples/roundtrip.rs new file mode 100644 index 00000000..35fe9426 --- /dev/null +++ b/core-rs/epass-ir/examples/roundtrip.rs @@ -0,0 +1,35 @@ +//! Dev helper: read a dump-format file (u64 per line), run autorun, print the +//! resulting dump to stdout. Used to cross-check against the C `epass` tool. + +use std::io::Read; + +use epass_ir::{autorun, default_passes, logfmt, Env, Opts}; + +fn main() { + let path = std::env::args().nth(1).expect("usage: roundtrip "); + let mut text = String::new(); + std::fs::File::open(&path) + .expect("open") + .read_to_string(&mut text) + .expect("read"); + let prog = logfmt::parse_dump(&text); + let mut opts = Opts::default(); + opts.verbose = std::env::var("V").ok().and_then(|v| v.parse().ok()).unwrap_or(0); + let mut env = Env::new(opts, prog); + let passes = default_passes(); + match autorun(&mut env, &passes) { + Ok(()) => { + if env.opts.verbose > 0 { + eprint!("{}", env.log_buffer()); + } + print!("{}", logfmt::to_dump(&env.insns)) + } + Err(e) => { + if env.opts.verbose > 0 { + eprint!("{}", env.log_buffer()); + } + eprintln!("error: {e}"); + std::process::exit(1); + } + } +} diff --git a/core-rs/epass-ir/include/epass.h b/core-rs/epass-ir/include/epass.h new file mode 100644 index 00000000..039319bf --- /dev/null +++ b/core-rs/epass-ir/include/epass.h @@ -0,0 +1,75 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * ePass C ABI. + * + * Minimal C interface to the userspace ePass eBPF compiler (Rust `epass-ir`). + * Link against `libepass_ir.a` (or `.so`). + * + * Typical use (e.g. from libbpf): + * + * int err; + * epass_result *r = epass_run(insns, insn_cnt, getenv("LIBBPF_EPASS_GOPT"), &err); + * if (!r) { handle error code `err`; } + * else { + * const struct bpf_insn *out = (const void *)epass_result_insns(r); + * size_t out_cnt = epass_result_insn_cnt(r); + * bpf_program__set_insns(prog, out, out_cnt); + * epass_result_free(r); + * } + */ +#ifndef _EPASS_H +#define _EPASS_H + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/* + * An eBPF instruction, bit-compatible with the kernel/libbpf `struct bpf_insn`. + * `regs` packs dst_reg (low nibble) and src_reg (high nibble). + */ +struct epass_insn { + uint8_t code; + uint8_t regs; + int16_t off; + int32_t imm; +}; + +/* Opaque result handle. */ +typedef struct epass_result epass_result; + +/* + * Run the ePass pipeline (lift -> passes -> codegen) on a program. + * + * `insns` : pointer to `insn_cnt` instructions (struct bpf_insn or epass_insn; + * the layouts are identical). + * `gopt` : NUL-terminated global-option string (comma-separated), or NULL. + * Recognized keys: verbose=, disable_coalesce, no_prog_check, + * print_bpf, print_dump, print_detail, printonly, dotgraph. + * `out_err` : if non-NULL, receives 0 on success or a negative errno on failure. + * + * Returns a result handle on success, or NULL on failure. + */ +epass_result *epass_run(const struct epass_insn *insns, size_t insn_cnt, + const char *gopt, int *out_err); + +/* Pointer to the rewritten instruction array (valid until epass_result_free). */ +const struct epass_insn *epass_result_insns(const epass_result *r); + +/* Number of rewritten instructions. */ +size_t epass_result_insn_cnt(const epass_result *r); + +/* NUL-terminated log text from the run (valid until epass_result_free). */ +const char *epass_result_log(const epass_result *r); + +/* Free a result handle returned by epass_run(). */ +void epass_result_free(epass_result *r); + +#ifdef __cplusplus +} +#endif + +#endif /* _EPASS_H */ diff --git a/core-rs/epass-ir/src/bytecode.rs b/core-rs/epass-ir/src/bytecode.rs new file mode 100644 index 00000000..365d8bcf --- /dev/null +++ b/core-rs/epass-ir/src/bytecode.rs @@ -0,0 +1,196 @@ +//! eBPF instruction representation and the BPF ISA opcode constants. +//! +//! The in-memory encoding of [`BpfInsn`] matches the kernel's `struct bpf_insn`: +//! a little-endian `u64` laid out as `code:8 | dst_reg:4 | src_reg:4 | off:16 | imm:32`. +//! This lets us round-trip the "dump" format (one `u64` per instruction) exactly +//! like the C tool does. + +/// Instruction classes (low 3 bits of `code`). +pub mod class { + pub const LD: u8 = 0x00; + pub const LDX: u8 = 0x01; + pub const ST: u8 = 0x02; + pub const STX: u8 = 0x03; + pub const ALU: u8 = 0x04; + pub const JMP: u8 = 0x05; + pub const JMP32: u8 = 0x06; + pub const ALU64: u8 = 0x07; +} + +/// Size modifiers for load/store (bits 3-4 of `code`). +pub mod size { + pub const W: u8 = 0x00; // word, 32-bit + pub const H: u8 = 0x08; // half-word, 16-bit + pub const B: u8 = 0x10; // byte, 8-bit + pub const DW: u8 = 0x18; // double word, 64-bit +} + +/// Mode modifiers for load/store (bits 5-7 of `code`). +pub mod mode { + pub const IMM: u8 = 0x00; + pub const ABS: u8 = 0x20; + pub const IND: u8 = 0x40; + pub const MEM: u8 = 0x60; + pub const MEMSX: u8 = 0x80; + pub const ATOMIC: u8 = 0xc0; +} + +/// Source operand selector (bit 3 of `code` for ALU/JMP). +pub mod src { + pub const K: u8 = 0x00; // use 32-bit immediate + pub const X: u8 = 0x08; // use source register +} + +/// ALU / JMP operation codes (high 4 bits of `code`). +pub mod op { + // ALU + pub const ADD: u8 = 0x00; + pub const SUB: u8 = 0x10; + pub const MUL: u8 = 0x20; + pub const DIV: u8 = 0x30; + pub const OR: u8 = 0x40; + pub const AND: u8 = 0x50; + pub const LSH: u8 = 0x60; + pub const RSH: u8 = 0x70; + pub const NEG: u8 = 0x80; + pub const MOD: u8 = 0x90; + pub const XOR: u8 = 0xa0; + pub const MOV: u8 = 0xb0; + pub const ARSH: u8 = 0xc0; + pub const END: u8 = 0xd0; + + // JMP + pub const JA: u8 = 0x00; + pub const JEQ: u8 = 0x10; + pub const JGT: u8 = 0x20; + pub const JGE: u8 = 0x30; + pub const JSET: u8 = 0x40; + pub const JNE: u8 = 0x50; + pub const JSGT: u8 = 0x60; + pub const JSGE: u8 = 0x70; + pub const CALL: u8 = 0x80; + pub const EXIT: u8 = 0x90; + pub const JLT: u8 = 0xa0; + pub const JLE: u8 = 0xb0; + pub const JSLT: u8 = 0xc0; + pub const JSLE: u8 = 0xd0; +} + +/// Endianness sub-codes for `BPF_END` (carried in `src`). +pub mod end { + pub const TO_LE: u8 = 0x00; + pub const TO_BE: u8 = 0x08; +} + +/// `src_reg` value marking an ePass `ecall` extension instruction. +pub const EPASS_CALL: u8 = 6; + +pub const MAX_BPF_REG: usize = 11; +pub const BPF_REG_0: u8 = 0; +pub const BPF_REG_1: u8 = 1; +pub const BPF_REG_6: u8 = 6; +pub const BPF_REG_10: u8 = 10; + +/// The maximum number of arguments an eBPF helper call accepts (r1..=r5). +pub const MAX_FUNC_ARG: usize = 5; + +#[inline] +pub const fn class_of(code: u8) -> u8 { + code & 0x07 +} +#[inline] +pub const fn op_of(code: u8) -> u8 { + code & 0xf0 +} +#[inline] +pub const fn src_of(code: u8) -> u8 { + code & 0x08 +} +#[inline] +pub const fn size_of(code: u8) -> u8 { + code & 0x18 +} +#[inline] +pub const fn mode_of(code: u8) -> u8 { + code & 0xe0 +} + +/// A single eBPF instruction, bit-compatible with the kernel `struct bpf_insn`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct BpfInsn { + pub code: u8, + pub dst_reg: u8, + pub src_reg: u8, + pub off: i16, + pub imm: i32, +} + +impl BpfInsn { + pub const fn new(code: u8, dst_reg: u8, src_reg: u8, off: i16, imm: i32) -> Self { + Self { + code, + dst_reg, + src_reg, + off, + imm, + } + } + + /// Decode a `bpf_insn` from its packed little-endian `u64` representation. + pub fn from_u64(raw: u64) -> Self { + let code = (raw & 0xff) as u8; + let regs = ((raw >> 8) & 0xff) as u8; + let dst_reg = regs & 0x0f; + let src_reg = (regs >> 4) & 0x0f; + let off = ((raw >> 16) & 0xffff) as u16 as i16; + let imm = ((raw >> 32) & 0xffff_ffff) as u32 as i32; + Self { + code, + dst_reg, + src_reg, + off, + imm, + } + } + + /// Encode this instruction into its packed little-endian `u64` representation. + pub fn to_u64(self) -> u64 { + let regs = (self.dst_reg & 0x0f) | ((self.src_reg & 0x0f) << 4); + (self.code as u64) + | ((regs as u64) << 8) + | ((self.off as u16 as u64) << 16) + | ((self.imm as u32 as u64) << 32) + } + + /// `true` if this is the second slot of a 64-bit immediate load (all-zero `code`). + pub fn is_imm64_continuation(self) -> bool { + self.code == 0 + } +} + +/// Combine the `imm` of two consecutive instruction slots into a 64-bit immediate. +pub fn join_imm64(low_imm: i32, high_imm: i32) -> i64 { + let low = (low_imm as u32) as u64; + ((high_imm as i64) << 32) | (low as i64) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn roundtrip_u64() { + let insn = BpfInsn::new(class::ALU64 | op::ADD | src::X, 1, 2, -3, 0x1234_5678); + assert_eq!(BpfInsn::from_u64(insn.to_u64()), insn); + } + + #[test] + fn decode_known_layout() { + // code=0x07, dst=1, src=2, off=0, imm=0 -> add64 r1, r2 + let raw = 0x0000_0000_0000_2107u64; + let insn = BpfInsn::from_u64(raw); + assert_eq!(insn.code, 0x07); + assert_eq!(insn.dst_reg, 1); + assert_eq!(insn.src_reg, 2); + } +} diff --git a/core-rs/epass-ir/src/cfg.rs b/core-rs/epass-ir/src/cfg.rs new file mode 100644 index 00000000..c5e77cb7 --- /dev/null +++ b/core-rs/epass-ir/src/cfg.rs @@ -0,0 +1,119 @@ +//! CFG postprocessing: compute the reachable-block chain layout and end blocks. +//! +//! ePass lays out basic blocks as *chains* so that conditional-jump +//! fallthrough targets (`bb1`) are physically adjacent, which the normalizer +//! relies on. This mirrors the C `add_reach` / `find_chain_head` logic. + +use crate::env::Env; +use crate::error::Result; +use crate::internal; +use crate::ir::{BbId, Function, InsnKind}; + +/// Recompute `reachable_bbs` (chain layout) and `end_bbs`. +pub fn finalize(env: &mut Env, func: &mut Function) -> Result<()> { + let _ = env; + let order = reachable_chain_layout(func)?; + func.reachable_bbs = order; + func.end_bbs = func + .reachable_bbs + .iter() + .copied() + .filter(|&bb| func.bb(bb).succs.is_empty()) + .collect(); + Ok(()) +} + +/// Walk from a block back to the head of its fallthrough chain. +fn find_chain_head(func: &Function, mut bb: BbId) -> Result { + loop { + let preds = &func.bb(bb).preds; + if preds.is_empty() { + return Ok(bb); + } + let mut chain_pred: Option = None; + for &p in preds.iter() { + let succs = &func.bb(p).succs; + match succs.len() { + 1 => { + // A `ja` edge is not a fallthrough chain edge. + let is_ja = func + .bb(p) + .last() + .map(|l| matches!(func.insn(l).kind, InsnKind::Ja)) + .unwrap_or(false); + if !is_ja { + if chain_pred.is_some() { + return Err(internal!("multiple chain predecessors")); + } + chain_pred = Some(p); + } + } + 2 => { + // Only the first successor (fallthrough) chains. + if func.bb(p).succs[0] == bb { + if chain_pred.is_some() { + return Err(internal!("multiple chain predecessors")); + } + chain_pred = Some(p); + } + } + _ => return Err(internal!("block has >2 successors")), + } + } + match chain_pred { + None => return Ok(bb), + Some(p) => bb = p, + } + } +} + +fn reachable_chain_layout(func: &Function) -> Result> { + let n = func.all_bbs.len(); + let mut visited = vec![false; func.bb_arena_len()]; + let mut layout = Vec::new(); + let mut todo = vec![func.entry]; + let mut head = 0; + + let _ = n; + while head < todo.len() { + let mut bb = todo[head]; + head += 1; + if visited[bb.index()] { + continue; + } + bb = find_chain_head(func, bb)?; + if visited[bb.index()] { + continue; + } + // Walk this chain. + loop { + visited[bb.index()] = true; + layout.push(bb); + let succs = &func.bb(bb).succs; + match succs.len() { + 0 => break, + 1 => { + let is_ja = func + .bb(bb) + .last() + .map(|l| matches!(func.insn(l).kind, InsnKind::Ja)) + .unwrap_or(false); + if is_ja { + todo.push(succs[0]); + break; + } else { + bb = succs[0]; + } + } + 2 => { + let s0 = succs[0]; + let s1 = succs[1]; + todo.push(s1); + bb = s0; + } + _ => return Err(internal!(">2 successors, invalid CFG")), + } + } + } + Ok(layout) +} diff --git a/core-rs/epass-ir/src/cg/alloc.rs b/core-rs/epass-ir/src/cg/alloc.rs new file mode 100644 index 00000000..f526c157 --- /dev/null +++ b/core-rs/epass-ir/src/cg/alloc.rs @@ -0,0 +1,369 @@ +//! Register allocation core: maximum-cardinality search, pre-spilling, the +//! spill transform, greedy coloring, copy coalescing, and SSA-out (phi removal). + +use crate::env::Env; +use crate::error::Result; +use crate::ir::insn::InsnKind; +use crate::ir::value::VrType; +use crate::ir::{Function, InsnId, InsertPos, Value}; +use crate::{internal, invalid}; + +use super::prepare::{create_alloc, ensure_extra}; +use super::{liveness, CgState, RA_COLORS}; + +/// Reset state at the start of an RA iteration. +pub fn clean_iteration(func: &Function, cg: &mut CgState) { + liveness::clean(func, cg); +} + +/// Maximum cardinality search: produce a simplicial elimination order (SEO). +fn mcs(cg: &mut CgState) { + cg.seo.clear(); + let mut remaining: Vec = cg.all_var.clone(); + // Reset lambda. + for &v in &remaining { + cg.extra_mut(v).lambda = 0; + } + while !remaining.is_empty() { + // Pick the vertex with maximum lambda (first wins ties, matching C `>=`). + let mut max_l = 0u32; + let mut max_i = remaining[0]; + for &v in &remaining { + let l = cg.extra(v).lambda; + if l >= max_l { + max_l = l; + max_i = v; + } + } + cg.seo.push(max_i); + let adj = cg.extra(max_i).adj.clone(); + for u in adj { + if remaining.contains(&u) { + cg.extra_mut(u).lambda += 1; + } + } + remaining.retain(|&x| x != max_i); + } +} + +/// Pre-spill pass: find maximal cliques exceeding `RA_COLORS` and select the +/// highest-weight virtual registers to spill. Returns the spill set. +pub fn pre_spill(env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result> { + let _ = (env, func); + mcs(cg); + + // Build the maximal cliques (eps[i]) along the SEO. + let seo = cg.seo.clone(); + let mut eps: Vec> = Vec::with_capacity(seo.len()); + for &v in &seo { + cg.extra_mut(v).w += 1; + } + // Reset w then recompute per the clique structure. + for &v in &seo { + cg.extra_mut(v).w = 0; + } + for (i, &v) in seo.iter().enumerate() { + let mut q = vec![v]; + cg.extra_mut(v).w += 1; + let adj = cg.extra(v).adj.clone(); + for u in adj { + // u is in the clique if it appears earlier in the SEO. + if seo[..i].contains(&u) { + q.push(u); + cg.extra_mut(u).w += 1; + } + } + eps.push(q); + } + + // Greedily spill the heaviest VR from any oversized clique. + let mut to_spill: Vec = Vec::new(); + loop { + let over = eps.iter().position(|c| c.len() > RA_COLORS); + let Some(idx) = over else { break }; + // Choose the max-weight virtual register in this clique. + let mut max_w = 0u32; + let mut max_i: Option = None; + for &v in &eps[idx] { + let e = cg.extra(v); + if e.w >= max_w && !e.nonvr && !e.spilled_once { + max_w = e.w; + max_i = Some(v); + } + } + let chosen = max_i.ok_or_else(|| internal!("oversized clique with no spillable VR"))?; + to_spill.push(chosen); + for c in &mut eps { + c.retain(|&x| x != chosen); + } + } + Ok(to_spill) +} + +/// Spill every selected value everywhere (spill-everywhere): allocate a stack +/// slot, store the def, and reload at each use. +pub fn spill( + env: &mut Env, + func: &mut Function, + cg: &mut CgState, + to_spill: &[InsnId], +) -> Result<()> { + let _ = env; + for &v in to_spill { + if matches!(func.insn(v).kind, InsnKind::Call { .. } | InsnKind::AllocArray { .. }) { + return Err(internal!("attempted to spill a call/allocarray")); + } + let users = func.insn(v).users.clone(); + + let alloc = if matches!(func.insn(v).kind, InsnKind::Alloc { .. }) { + v + } else { + let entry = func.entry; + let alloc = create_alloc(cg, func, entry, VrType::B64, InsertPos::FrontAfterPhi); + // store alloc, v (placed right after v's def) + let store = func.build_store_at(v, alloc, Value::Insn(v), InsertPos::Back); + ensure_extra(cg, func, store); + alloc + }; + + // Finalize the stack slot. + let off = cg.new_spill(8); + { + let e = cg.extra_mut(alloc); + e.finalized = true; + e.vr_pos.allocated = true; + e.vr_pos.spilled = off; + e.vr_pos.spilled_size = 8; + } + + if !matches!(func.insn(v).kind, InsnKind::Alloc { .. }) { + for user in users { + spill_one_use(func, cg, user, alloc, v)?; + } + // The def now has a tiny live range (def -> store); never re-spill it. + cg.extra_mut(v).spilled_once = true; + } + } + Ok(()) +} + +/// Rewrite a single user of a spilled value to reload from the stack. +fn spill_one_use( + func: &mut Function, + cg: &mut CgState, + user: InsnId, + alloc: InsnId, + v: InsnId, +) -> Result<()> { + match func.insn(user).kind.clone() { + // A store of the spilled value into its own slot: nothing to do. + InsnKind::Store + if func.insn(user).values.first() == Some(&Value::Insn(user)) => + { + Ok(()) + } + InsnKind::Phi => { + // Reload at the end of each predecessor block contributing v. + let entries = func.insn(user).phi.clone(); + for (idx, entry) in entries.iter().enumerate() { + if entry.value == Value::Insn(v) { + let load = func.build_load_bb(entry.bb, alloc, InsertPos::BackBeforeJmp); + ensure_extra(cg, func, load); + // Update the phi operand. + func.remove_use(entry.value, user); + func.insn_mut(user).phi[idx].value = Value::Insn(load); + func.add_use(Value::Insn(load), user); + } + } + Ok(()) + } + _ => { + let load = func.build_load_at(user, alloc, InsertPos::Front); + ensure_extra(cg, func, load); + func.change_value(user, Value::Insn(v), Value::Insn(load)); + Ok(()) + } + } +} + +/// Greedy coloring along the SEO (optimal for chordal graphs). +/// +/// Returns `Ok(None)` when every value was colored. On a non-chordal graph the +/// MCS order is not a perfect elimination order, so greedy coloring can run out +/// of registers at a node even when the graph is K-colorable; in that case this +/// returns `Ok(Some(v))` naming the value `v` it could not place, so the caller +/// can post-spill and re-run register allocation (matching the "post-spilling" +/// fallback of Pereira & Palsberg). Any value colored before the failure keeps +/// its assignment, but the caller is expected to reset state via a fresh RA +/// iteration (`clean_iteration`) before retrying. +pub fn coloring(env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result> { + let _ = (env, func); + let seo = cg.seo.clone(); + for v in seo { + if cg.extra(v).vr_pos.allocated { + continue; + } + let mut used = [false; RA_COLORS]; + let adj = cg.extra(v).adj.clone(); + for a in adj { + let ae = cg.extra(a); + if ae.vr_pos.allocated && ae.vr_pos.spilled == 0 { + used[ae.vr_pos.alloc_reg as usize] = true; + } + } + let mut assigned = false; + for (i, &u) in used.iter().enumerate() { + if !u { + let e = cg.extra_mut(v); + e.vr_pos.allocated = true; + e.vr_pos.alloc_reg = i as u8; + assigned = true; + break; + } + } + if !assigned { + return Ok(Some(v)); + } + } + Ok(None) +} + +/// Choose a value to post-spill when greedy coloring cannot place `failed`. +/// +/// Prefer `failed` itself; if it is not spillable (e.g. it is the already-minimal +/// def range left behind by an earlier spill), fall back to its highest-degree +/// spillable interference neighbor, which relieves the same pressure point. +pub fn pick_spill_victim(func: &Function, cg: &CgState, failed: InsnId) -> Result { + let spillable = |v: InsnId| -> bool { + let e = cg.extra(v); + !e.nonvr + && !e.spilled_once + && e.vr_pos.spilled == 0 + && !matches!( + func.insn(v).kind, + InsnKind::Call { .. } + | InsnKind::Ecall + | InsnKind::AllocArray { .. } + | InsnKind::Alloc { .. } + ) + }; + if spillable(failed) { + return Ok(failed); + } + let mut best: Option = None; + let mut best_deg = 0usize; + for &a in &cg.extra(failed).adj { + if spillable(a) { + let deg = cg.extra(a).adj.len(); + if best.is_none() || deg > best_deg { + best = Some(a); + best_deg = deg; + } + } + } + best.ok_or_else(|| { + crate::error::Error::RegAlloc( + "coloring failed and no spillable value is available".into(), + ) + }) +} + +/// Best-effort copy coalescing for assign / store / load chains. +pub fn coalesce(env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + let _ = env; + let insns: Vec<_> = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .collect(); + for id in insns { + match func.insn(id).kind.clone() { + InsnKind::Assign => { + if let Value::Insn(src) = func.insn(id).values[0] { + try_coalesce(cg, id, src); + } + } + InsnKind::Store => { + let v0 = func.insn(id).values[0]; + let v1 = func.insn(id).values[1]; + if let (Value::Insn(alloc), Value::Insn(src)) = (v0, v1) { + try_coalesce(cg, alloc, src); + } + } + InsnKind::Load => { + if let Value::Insn(alloc) = func.insn(id).values[0] { + try_coalesce(cg, id, alloc); + } + } + _ => {} + } + } + Ok(()) +} + +/// Try to give `v1` and `v2` the same register if it does not break coloring. +fn try_coalesce(cg: &mut CgState, v1: InsnId, v2: InsnId) { + let (s1, s2) = (cg.extra(v1).vr_pos.spilled, cg.extra(v2).vr_pos.spilled); + let (r1, r2) = (cg.extra(v1).vr_pos.alloc_reg, cg.extra(v2).vr_pos.alloc_reg); + if s1 != 0 || s2 != 0 || r1 == r2 { + return; + } + // Colors used by both neighborhoods. + let mut used = [false; RA_COLORS]; + for v in [v1, v2] { + let adj = cg.extra(v).adj.clone(); + for a in adj { + let ae = cg.extra(a); + if ae.vr_pos.allocated && ae.vr_pos.spilled == 0 { + used[ae.vr_pos.alloc_reg as usize] = true; + } + } + } + let f1 = cg.extra(v1).finalized; + let f2 = cg.extra(v2).finalized; + if f1 { + if !used[r1 as usize] { + cg.extra_mut(v2).vr_pos.alloc_reg = r1; + } + } else if f2 { + if !used[r2 as usize] { + cg.extra_mut(v1).vr_pos.alloc_reg = r2; + } + } else if let Some(free) = (0..RA_COLORS).find(|&i| !used[i]) { + cg.extra_mut(v1).vr_pos.alloc_reg = free as u8; + cg.extra_mut(v2).vr_pos.alloc_reg = free as u8; + } +} + +/// SSA-out: replace each phi with copies in its predecessor blocks. +pub fn remove_phi(env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + let _ = env; + let phis: Vec = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .filter(|&id| matches!(func.insn(id).kind, InsnKind::Phi)) + .collect(); + + for phi in phis { + let vrpos = cg.extra(phi).vr_pos; + if vrpos.spilled != 0 { + return Err(invalid!("phi cannot be spilled")); + } + let entries = func.insn(phi).phi.clone(); + for entry in entries { + // assign(entry.value) in the predecessor block, sharing phi's reg. + let assign = func.build_assign_bb(entry.bb, entry.value, InsertPos::BackBeforeJmp); + ensure_extra(cg, func, assign); + cg.extra_mut(assign).vr_pos = vrpos; + cg.extra_mut(assign).finalized = true; + func.remove_use(entry.value, phi); + } + func.insn_mut(phi).phi.clear(); + let reg = cg.regs[vrpos.alloc_reg as usize]; + func.replace_all_uses(phi, Value::Insn(reg)); + func.erase_insn(phi); + cg.extra.remove(&phi); + } + Ok(()) +} diff --git a/core-rs/epass-ir/src/cg/liveness.rs b/core-rs/epass-ir/src/cg/liveness.rs new file mode 100644 index 00000000..146452e3 --- /dev/null +++ b/core-rs/epass-ir/src/cg/liveness.rs @@ -0,0 +1,261 @@ +//! SSA liveness analysis and interference-graph construction. +//! +//! Follows the path-exploration scheme of the Pereira & Palsberg chordal-graph +//! allocator: for each SSA value, walk backwards from each use to the def, +//! marking live-in/out and recording interferences with the live values that +//! are killed along the way. + +use std::collections::HashSet; + +use crate::bytecode::{BPF_REG_0, BPF_REG_6}; +use crate::env::Env; +use crate::error::Result; +use crate::ir::insn::InsnKind; +use crate::ir::{BbId, Function, InsnId}; + +use super::{CgState, RA_COLORS}; + +/// Reset per-iteration liveness/interference data. +pub fn clean(func: &Function, cg: &mut CgState) { + cg.all_var.clear(); + let regs = cg.regs; + for reg in regs { + let e = cg.extra_mut(reg); + e.adj.clear(); + e.live_in.clear(); + e.live_out.clear(); + e.lambda = 0; + e.w = 0; + cg.all_var.push(reg); + } + for &bb in &func.reachable_bbs { + for &id in &func.bb(bb).insns { + let e = cg.extra_mut(id); + e.adj.clear(); + e.live_in.clear(); + e.live_out.clear(); + e.lambda = 0; + e.w = 0; + if !e.finalized { + e.vr_pos.allocated = false; + } + } + } +} + +fn push_unique(v: &mut Vec, id: InsnId) { + if !v.contains(&id) { + v.push(id); + } +} + +/// Liveness analysis: populate `live_in`/`live_out` for every statement. +pub fn analyze(env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + let _ = env; + // Collect all virtual registers (instructions with a non-finalized dst). + let mut vars: Vec = Vec::new(); + for &bb in &func.reachable_bbs { + for &id in &func.bb(bb).insns { + let e = cg.extra(id); + if e.dst == Some(id) && !e.finalized { + vars.push(id); + } + } + } + for &v in &vars { + push_unique(&mut cg.all_var, v); + let users = func.insn(v).users.clone(); + let mut visited: HashSet = HashSet::new(); + for s in users { + if matches!(func.insn(s).kind, InsnKind::Phi) { + // For a phi user, liveness propagates from the predecessor block + // corresponding to each operand equal to `v`. + let phi = func.insn(s).phi.clone(); + for entry in phi { + if entry.value == crate::ir::Value::Insn(v) { + live_out_at_block(func, cg, &mut visited, entry.bb, v); + } + } + } else { + live_in_at_statement(func, cg, &mut visited, s, v); + } + } + } + Ok(()) +} + +fn live_out_at_block( + func: &Function, + cg: &mut CgState, + visited: &mut HashSet, + n: BbId, + v: InsnId, +) { + if visited.contains(&n) { + return; + } + visited.insert(n); + if let Some(last) = func.bb(n).last() { + live_out_at_statement(func, cg, visited, last, v); + } else { + let preds = func.bb(n).preds.clone(); + for p in preds { + live_out_at_block(func, cg, visited, p, v); + } + } +} + +fn live_out_at_statement( + func: &Function, + cg: &mut CgState, + visited: &mut HashSet, + s: InsnId, + v: InsnId, +) { + push_unique(&mut cg.extra_mut(s).live_out, v); + let dst = cg.extra(s).dst; + match dst { + Some(d) => { + if d != v { + // `s` kills `d` (not `v`); `v` and `d` interfere, keep going up. + make_conflict(func, cg, v, d); + live_in_at_statement(func, cg, visited, s, v); + } + // If `d == v`, this statement defines `v`: stop propagating upward. + } + None => { + // `s` has no destination (no kill): keep propagating. + live_in_at_statement(func, cg, visited, s, v); + } + } +} + +fn live_in_at_statement( + func: &Function, + cg: &mut CgState, + visited: &mut HashSet, + s: InsnId, + v: InsnId, +) { + push_unique(&mut cg.extra_mut(s).live_in, v); + match func.prev_insn(s) { + None => { + let preds = func.insn(s).parent_bb; + let preds = func.bb(preds).preds.clone(); + for p in preds { + live_out_at_block(func, cg, visited, p, v); + } + } + Some(prev) => live_out_at_statement(func, cg, visited, prev, v), + } +} + +/// Record an interference edge between two values, resolving pre-colored / +/// pre-spilled values to their physical registers (or dropping if spilled). +fn make_conflict(func: &Function, cg: &mut CgState, v1: InsnId, v2: InsnId) { + if v1 == v2 { + return; + } + let r1 = match resolve_conflict_node(cg, v1) { + Some(r) => r, + None => return, + }; + let r2 = match resolve_conflict_node(cg, v2) { + Some(r) => r, + None => return, + }; + if r1 == r2 { + return; + } + let _ = func; + push_unique(&mut cg.extra_mut(r1).adj, r2); + push_unique(&mut cg.extra_mut(r2).adj, r1); +} + +/// Map a value to its interference node: itself, or its physical register if +/// pre-colored. Returns `None` for pre-spilled values (no register conflict). +fn resolve_conflict_node(cg: &CgState, v: InsnId) -> Option { + let e = cg.extra(v); + if e.vr_pos.allocated { + if e.vr_pos.spilled != 0 { + None + } else { + Some(cg.regs[e.vr_pos.alloc_reg as usize]) + } + } else { + Some(v) + } +} + +/// Build the interference graph: physical-register clique, phi conflicts, +/// caller-saved constraints across calls, and ALU operand constraints. +pub fn build_interference(env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + let _ = env; + + // All physical registers conflict pairwise. + for i in 0..RA_COLORS { + for j in (i + 1)..RA_COLORS { + let (a, b) = (cg.regs[i], cg.regs[j]); + make_conflict(func, cg, a, b); + } + } + + let bbs = func.reachable_bbs.clone(); + for bb in bbs { + let insns = func.bb(bb).insns.clone(); + for id in insns { + match func.insn(id).kind.clone() { + InsnKind::Phi => { + let phi = func.insn(id).phi.clone(); + for entry in phi { + phi_conflict_at_block(func, cg, entry.bb, id); + } + } + InsnKind::Call { .. } | InsnKind::Ecall => { + // Values live across the call conflict with caller-saved R0..R5. + let live_in = cg.extra(id).live_in.clone(); + let live_out: HashSet = cg.extra(id).live_out.iter().copied().collect(); + for v in live_in { + if live_out.contains(&v) { + for r in BPF_REG_0..BPF_REG_6 { + let reg = cg.regs[r as usize]; + make_conflict(func, cg, reg, v); + } + } + } + } + _ => { + if func.insn(id).is_bin_alu() { + // a = ALU b c : dst conflicts with the second operand. + if let crate::ir::Value::Insn(c) = func.insn(id).values[1] { + make_conflict(func, cg, id, c); + } + } + } + } + } + } + Ok(()) +} + +/// A phi value conflicts with the live-out (or live-in for jumps) of each +/// predecessor block's terminator. +fn phi_conflict_at_block(func: &Function, cg: &mut CgState, n: BbId, v: InsnId) { + if let Some(last) = func.bb(n).last() { + let set = if func.insn(last).is_jmp() { + cg.extra(last).live_in.clone() + } else { + cg.extra(last).live_out.clone() + }; + for u in set { + if u != v { + make_conflict(func, cg, u, v); + } + } + } else { + let preds = func.bb(n).preds.clone(); + for p in preds { + phi_conflict_at_block(func, cg, p, v); + } + } +} diff --git a/core-rs/epass-ir/src/cg/mod.rs b/core-rs/epass-ir/src/cg/mod.rs new file mode 100644 index 00000000..902adc98 --- /dev/null +++ b/core-rs/epass-ir/src/cg/mod.rs @@ -0,0 +1,191 @@ +//! Code generator: SSA-based register allocation (chordal-graph coloring) and +//! lowering of the IR back to eBPF bytecode. +//! +//! Algorithm reference: Pereira & Palsberg, "Register Allocation via the +//! Coloring of Chordal Graphs" (APLAS 2005). The register-allocation state for +//! each instruction lives in a side-table ([`CgState::extra`]) keyed by +//! [`InsnId`], keeping the IR node itself stage-agnostic. + +mod alloc; +mod liveness; +mod norm; +mod prepare; + +use std::collections::HashMap; + +use crate::env::{Env, Timer}; +use crate::error::Result; +use crate::ir::value::VrPos; +use crate::ir::{Function, InsnId}; +use crate::{internal, log_debug}; + +/// Number of allocatable physical registers (R0..R9). +pub const RA_COLORS: usize = 10; + +/// Per-instruction register-allocation metadata. +#[derive(Debug, Clone)] +pub struct CgExtra { + /// The instruction whose allocation represents this value's location + /// (`Some(self)` for value-producing insns, `None` for void insns). + pub dst: Option, + /// Liveness: values live-in / live-out at this statement. + pub live_in: Vec, + pub live_out: Vec, + /// Interference-graph adjacency. + pub adj: Vec, + /// MCS weight (lambda) and clique weight (w). + pub lambda: u32, + pub w: u32, + /// Whether `vr_pos` is fixed (pre-colored / pre-spilled). + pub finalized: bool, + pub vr_pos: VrPos, + /// A non-virtual pseudo (physical register / stack pointer). + pub nonvr: bool, + /// Whether this value has already been spilled-everywhere (its def now has + /// a tiny live range; never re-select it for spilling). + pub spilled_once: bool, +} + +impl CgExtra { + fn new(dst: Option) -> Self { + CgExtra { + dst, + live_in: Vec::new(), + live_out: Vec::new(), + adj: Vec::new(), + lambda: 0, + w: 0, + finalized: false, + vr_pos: VrPos::default(), + nonvr: false, + spilled_once: false, + } + } +} + +/// Code-generation state for a function. +pub struct CgState { + /// Per-instruction RA metadata, keyed by instruction id. + pub extra: HashMap, + /// Physical-register pseudo-instructions R0..R9. + pub regs: [InsnId; RA_COLORS], + /// Simplicial elimination order (from MCS). + pub seo: Vec, + /// All virtual registers in the interference graph. + pub all_var: Vec, + /// Current (negative) stack offset for spills/arrays. + pub stack_offset: i32, +} + +impl CgState { + pub fn extra(&self, id: InsnId) -> &CgExtra { + self.extra.get(&id).expect("missing CG extra") + } + pub fn extra_mut(&mut self, id: InsnId) -> &mut CgExtra { + self.extra.get_mut(&id).expect("missing CG extra") + } + + /// Allocate `size` bytes of stack and return the new (negative) offset. + pub fn new_spill(&mut self, size: u32) -> i32 { + self.stack_offset -= size as i32; + self.stack_offset + } +} + +/// Compile a function: run CG prep, register allocation, SSA-out, and lower to +/// bytecode (written into `env.insns`). +pub fn compile(env: &mut Env, func: &mut Function) -> Result<()> { + let timer = Timer::start(); + + // The normal pass pipeline runs optimize_ir before codegen. CG prep starts + // from the post-pass IR and only performs lowering required for allocation. + let mut cg = prepare::init_cg(env, func)?; + + prepare::change_call(env, func, &mut cg)?; + log_ir(env, func, "after change_call"); + + prepare::change_fun_arg(env, func, &mut cg)?; + log_ir(env, func, "after change_fun_arg"); + + prepare::spill_array(env, func, &mut cg)?; + prepare::spill_const(env, func, &mut cg)?; + log_ir(env, func, "after spill prep"); + + // Register-allocation fixpoint. Each iteration: + // 1. (re)build liveness + interference, + // 2. pre-spill any clique larger than the register count, and + // 3. greedily color along the MCS order. + // + // On a chordal graph this converges in one pass (pre-spilling guarantees a + // K-colorable graph, Pereira & Palsberg Theorem 2). The RA interference + // graph here is *not* chordal — the caller-saved / two-address / phi + // constraint edges break the SSA dominance property the chordality proof + // relies on — so the MCS order is not a perfect elimination order and greedy + // coloring can fail even when the graph is K-colorable. When that happens we + // post-spill the offending value (paper's "post-spilling" fallback) and + // re-run. Each value is spilled at most once, so this terminates. + let max_iter = 64; + let mut iteration = 0; + loop { + if iteration > max_iter { + return Err(internal!("register allocation did not converge")); + } + log_debug!(env, "----- RA iteration {} -----\n", iteration); + alloc::clean_iteration(func, &mut cg); + liveness::analyze(env, func, &mut cg)?; + liveness::build_interference(env, func, &mut cg)?; + + let to_spill = alloc::pre_spill(env, func, &mut cg)?; + log_debug!(env, "RA iteration {} pre-spills {}\n", iteration, to_spill.len()); + if !to_spill.is_empty() { + alloc::spill(env, func, &mut cg, &to_spill)?; + iteration += 1; + continue; + } + + // No oversized clique: attempt to color. On a non-chordal graph this can + // still fail; if so, post-spill the failing value and retry. + match alloc::coloring(env, func, &mut cg)? { + None => break, + Some(failed) => { + let victim = alloc::pick_spill_victim(func, &cg, failed)?; + log_debug!( + env, + "RA iteration {} coloring failed at %{}, post-spilling %{}\n", + iteration, + failed.0, + victim.0 + ); + alloc::spill(env, func, &mut cg, &[victim])?; + iteration += 1; + } + } + } + log_debug!(env, "RA converged in {} iterations\n", iteration); + + log_ir(env, func, "after coloring"); + + if !env.opts.disable_coalesce { + alloc::coalesce(env, func, &mut cg)?; + log_ir(env, func, "after coalescing"); + } + + let offset = cg.stack_offset; + prepare::add_stack_offset(func, offset); + + alloc::remove_phi(env, func, &mut cg)?; + log_ir(env, func, "after SSA-out"); + + norm::normalize_and_emit(env, func, &mut cg)?; + + env.cg_time_ns += timer.elapsed_ns(); + Ok(()) +} + +fn log_ir(env: &mut Env, func: &Function, msg: &str) { + if env.opts.verbose >= 2 { + log_debug!(env, "----- CG: {} -----\n", msg); + let txt = crate::ir::print::print_function(func); + log_debug!(env, "{}", txt); + } +} diff --git a/core-rs/epass-ir/src/cg/norm.rs b/core-rs/epass-ir/src/cg/norm.rs new file mode 100644 index 00000000..abd7d071 --- /dev/null +++ b/core-rs/epass-ir/src/cg/norm.rs @@ -0,0 +1,667 @@ +//! Normalization and bytecode emission. +//! +//! After register allocation, each value has a concrete location (register, +//! stack slot, or constant). This stage rewrites IR instructions into a +//! two-address, machine-encodable form, resolves builtin constants and jump +//! targets, and finally synthesizes `bpf_insn`s into `env.insns`. + +use std::collections::HashMap; + +use crate::bytecode::{self as bc, BpfInsn, BPF_REG_0, BPF_REG_10}; +use crate::env::Env; +use crate::error::Result; +use crate::ir::insn::{BinOp, Cond, EndKind, InsnKind}; +use crate::ir::value::{AluOp, VrPos, VrType}; +use crate::ir::{BbId, Function, InsnId, Value}; +use crate::{internal, invalid}; + +use super::CgState; + +/// A concrete operand location after RA. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Loc { + Reg(u8), + Stack(i32), + Const { v: i64, alu: AluOp }, + Undef, +} + +/// Resolve a value to its concrete location. +fn loc_of(v: Value) -> Loc { + match v { + Value::Const { v, ty, .. } => Loc::Const { v, alu: ty }, + Value::VrPos(p) | Value::FlattenDst(p) => loc_of_pos(p), + Value::Undef => Loc::Undef, + Value::Insn(_) => Loc::Undef, // must have been flattened already + } +} + +fn loc_of_pos(p: VrPos) -> Loc { + if !p.allocated { + Loc::Undef + } else if p.spilled != 0 { + Loc::Stack(p.spilled) + } else { + Loc::Reg(p.alloc_reg) + } +} + +/// The destination position finalized for an instruction. +fn dst_pos(cg: &CgState, func: &Function, id: InsnId) -> VrPos { + let _ = func; + match cg.extra.get(&id).and_then(|e| e.dst) { + Some(d) => cg.extra(d).vr_pos, + None => VrPos::default(), + } +} + +/// Flatten all instruction-valued operands to their finalized positions. +fn flatten(func: &mut Function, cg: &CgState) { + let bbs = func.reachable_bbs.clone(); + for bb in bbs { + let insns = func.bb(bb).insns.clone(); + for id in insns { + // Propagate alloc widths to load/store. + // (Typed load/store carry their alloc's width; raw ones already do.) + let n = func.insn(id).values.len(); + for i in 0..n { + if let Value::Insn(def) = func.insn(id).values[i] { + let pos = cg.extra(def).vr_pos; + func.insn_mut(id).values[i] = Value::FlattenDst(pos); + } + } + // Raw address bases. + match &mut func.insn_mut(id).kind { + InsnKind::LoadRaw { addr, .. } | InsnKind::StoreRaw { addr, .. } => { + if let Value::Insn(_) = addr.value { + // resolved below using a fresh borrow + } + } + _ => {} + } + if let InsnKind::LoadRaw { addr, .. } | InsnKind::StoreRaw { addr, .. } = + func.insn(id).kind.clone() + { + if let Value::Insn(def) = addr.value { + let pos = cg.extra(def).vr_pos; + if let InsnKind::LoadRaw { addr, .. } | InsnKind::StoreRaw { addr, .. } = + &mut func.insn_mut(id).kind + { + addr.value = Value::FlattenDst(pos); + } + } + } + } + } +} + +/// Replace builtin constants with their computed values. +fn replace_builtin_consts(func: &mut Function) { + let bbs = func.reachable_bbs.clone(); + for bb in bbs { + let cnt = bb_insn_cnt(func, bb); + let insns = func.bb(bb).insns.clone(); + for id in insns { + let n = func.insn(id).values.len(); + for i in 0..n { + if let Value::Const { v, builtin, .. } = &mut func.insn_mut(id).values[i] { + use crate::ir::value::BuiltinConst::*; + match *builtin { + BbInsnCnt => { + *v = cnt as i64; + *builtin = None; + } + BbInsnCriticalCnt => { + *v = cnt as i64; + *builtin = None; + } + None => {} + } + } + } + } + } +} + +fn bb_insn_cnt(func: &Function, bb: BbId) -> u32 { + func.bb(bb) + .insns + .iter() + .filter(|&&id| { + !matches!( + func.insn(id).kind, + InsnKind::Alloc { .. } | InsnKind::AllocArray { .. } + ) + }) + .count() as u32 +} + +// ---- emission ---- + +/// One emitted machine instruction (possibly wide). +#[derive(Debug, Clone, Copy)] +struct Emitted { + insn: BpfInsn, + wide: bool, + imm64: i64, + /// Filled during relocation for jumps. + is_ja: bool, + is_cond: bool, + target1: Option, // ja target / cond fallthrough is implicit + target2: Option, // cond taken target +} + +impl Emitted { + fn simple(insn: BpfInsn) -> Self { + Emitted { + insn, + wide: false, + imm64: 0, + is_ja: false, + is_cond: false, + target1: None, + target2: None, + } + } + fn wide(dst: u8, src: u8, imm64: i64, code: u8) -> Self { + Emitted { + insn: BpfInsn::new(code, dst, src, 0, (imm64 & 0xffff_ffff) as i32), + wide: true, + imm64, + is_ja: false, + is_cond: false, + target1: None, + target2: None, + } + } +} + +fn alu_class(alu: AluOp) -> u8 { + if alu == AluOp::Alu64 { + bc::class::ALU64 + } else { + bc::class::ALU + } +} + +fn binop_code(op: BinOp) -> u8 { + match op { + BinOp::Add => bc::op::ADD, + BinOp::Sub => bc::op::SUB, + BinOp::Mul => bc::op::MUL, + BinOp::Div => bc::op::DIV, + BinOp::Or => bc::op::OR, + BinOp::And => bc::op::AND, + BinOp::Lsh => bc::op::LSH, + BinOp::Arsh => bc::op::ARSH, + BinOp::Rsh => bc::op::RSH, + BinOp::Mod => bc::op::MOD, + BinOp::Xor => bc::op::XOR, + } +} + +fn cond_code(c: Cond) -> u8 { + match c { + Cond::Eq => bc::op::JEQ, + Cond::Ne => bc::op::JNE, + Cond::Gt => bc::op::JGT, + Cond::Ge => bc::op::JGE, + Cond::Lt => bc::op::JLT, + Cond::Le => bc::op::JLE, + Cond::Sgt => bc::op::JSGT, + Cond::Sge => bc::op::JSGE, + Cond::Slt => bc::op::JSLT, + Cond::Sle => bc::op::JSLE, + } +} + +/// Swap a condition when operands are reversed (for `const reg`). +fn swap_cond(c: Cond) -> Cond { + match c { + Cond::Gt => Cond::Lt, + Cond::Lt => Cond::Gt, + Cond::Ge => Cond::Le, + Cond::Le => Cond::Ge, + Cond::Sgt => Cond::Slt, + Cond::Slt => Cond::Sgt, + Cond::Sge => Cond::Sle, + Cond::Sle => Cond::Sge, + Cond::Eq => Cond::Eq, + Cond::Ne => Cond::Ne, + } +} + +fn vr_size(t: VrType) -> u8 { + match t { + VrType::B8 => bc::size::B, + VrType::B16 => bc::size::H, + VrType::B32 => bc::size::W, + VrType::B64 => bc::size::DW, + VrType::Unknown => bc::size::DW, + } +} + +/// Emit a `dst = imm` move (32- or 64-bit). +fn emit_load_const(out: &mut Vec, dst: u8, v: i64, alu: AluOp) { + if alu == AluOp::Alu64 { + // 64-bit immediate -> wide load. + out.push(Emitted::wide(dst, 0, v, bc::class::LD | bc::mode::IMM | bc::size::DW)); + } else { + out.push(Emitted::simple(BpfInsn::new( + bc::class::ALU64 | bc::op::MOV | bc::src::K, + dst, + 0, + 0, + v as i32, + ))); + } +} + +/// Emit `dst = src` register move. +fn emit_reg_move(out: &mut Vec, dst: u8, src: u8) { + if dst == src { + return; + } + out.push(Emitted::simple(BpfInsn::new( + bc::class::ALU64 | bc::op::MOV | bc::src::X, + dst, + src, + 0, + 0, + ))); +} + +/// Load a value into `dst` register, materializing const/stack as needed. +fn emit_into_reg(out: &mut Vec, dst: u8, loc: Loc) -> Result<()> { + match loc { + Loc::Reg(r) => emit_reg_move(out, dst, r), + Loc::Const { v, alu } => emit_load_const(out, dst, v, alu), + Loc::Stack(off) => { + // dst = *(u64 *)(r10 + off) + out.push(Emitted::simple(BpfInsn::new( + bc::class::LDX | bc::mode::MEM | bc::size::DW, + dst, + BPF_REG_10, + off as i16, + 0, + ))); + } + Loc::Undef => return Err(internal!("undefined operand in emission")), + } + Ok(()) +} + +/// Emit all machine instructions for one IR instruction. +fn emit_insn( + func: &Function, + cg: &CgState, + id: InsnId, + out: &mut Vec, +) -> Result<()> { + let insn = func.insn(id); + let dpos = dst_pos(cg, func, id); + let dreg = dpos.alloc_reg; + let dst_stack = dpos.allocated && dpos.spilled != 0; + + match &insn.kind { + InsnKind::Alloc { .. } | InsnKind::AllocArray { .. } => {} // no code + InsnKind::Assign => { + let v0 = loc_of(insn.values[0]); + emit_assign(out, dpos, v0)?; + } + InsnKind::Bin { op } => { + if dst_stack { + return Err(internal!("ALU destination spilled (unsupported)")); + } + emit_bin(out, dreg, *op, insn.alu_op, loc_of(insn.values[0]), loc_of(insn.values[1]))?; + } + InsnKind::Neg => { + emit_into_reg_if_needed(out, dreg, loc_of(insn.values[0]))?; + out.push(Emitted::simple(BpfInsn::new( + alu_class(insn.alu_op) | bc::op::NEG | bc::src::K, + dreg, + 0, + 0, + 0, + ))); + } + InsnKind::End { kind, swap_width } => { + emit_into_reg_if_needed(out, dreg, loc_of(insn.values[0]))?; + let sub = match kind { + EndKind::ToBe => bc::end::TO_BE, + EndKind::ToLe => bc::end::TO_LE, + }; + out.push(Emitted::simple(BpfInsn::new( + bc::class::ALU | bc::op::END | sub, + dreg, + 0, + 0, + *swap_width as i32, + ))); + } + InsnKind::LoadRaw { vr_type, addr } => { + if dst_stack { + return Err(internal!("loadraw destination spilled (unsupported)")); + } + emit_load_raw(out, dreg, *vr_type, addr)?; + } + InsnKind::StoreRaw { vr_type, addr } => { + emit_store_raw(out, *vr_type, addr, loc_of(insn.values[0]))?; + } + InsnKind::LoadImmExtra { extra, imm64 } => { + let code = bc::class::LD | bc::mode::IMM | bc::size::DW; + let mut e = Emitted::wide(dreg, extra.to_src_reg(), *imm64, code); + e.insn.src_reg = extra.to_src_reg(); + out.push(e); + } + InsnKind::Ret => { + if !insn.values.is_empty() { + emit_into_reg(out, BPF_REG_0, loc_of(insn.values[0]))?; + } + out.push(Emitted::simple(BpfInsn::new(bc::class::JMP | bc::op::EXIT, 0, 0, 0, 0))); + } + InsnKind::Call { fid } => { + out.push(Emitted::simple(BpfInsn::new( + bc::class::JMP | bc::op::CALL, + 0, + 0, + 0, + *fid, + ))); + } + InsnKind::Ja => { + let mut e = Emitted::simple(BpfInsn::new(bc::class::JMP | bc::op::JA, 0, 0, 0, 0)); + e.is_ja = true; + e.target1 = insn.bb1; + out.push(e); + } + InsnKind::CondJmp { cond } => { + emit_cond_jmp(out, *cond, insn.alu_op, loc_of(insn.values[0]), loc_of(insn.values[1]), insn.bb2)?; + } + InsnKind::Throw => { + // Lowered earlier in a full pipeline; emit exit as a safe fallback. + out.push(Emitted::simple(BpfInsn::new(bc::class::JMP | bc::op::EXIT, 0, 0, 0, 0))); + } + InsnKind::Store => { + // store , ==> assign val into the alloc's position. + let alloc_pos = match insn.values[0] { + Value::FlattenDst(p) | Value::VrPos(p) => p, + _ => return Err(internal!("store target is not an allocated slot")), + }; + let src = loc_of(insn.values[1]); + emit_assign(out, alloc_pos, src)?; + } + InsnKind::Load => { + // %x = load ==> assign the alloc's position into %x. + let src = loc_of(insn.values[0]); + emit_assign(out, dpos, src)?; + } + InsnKind::GetElemPtr => { + return Err(internal!("getelemptr survived to emission (unsupported)")); + } + InsnKind::Phi => return Err(internal!("phi survived to emission")), + InsnKind::Reg { .. } | InsnKind::FunctionArg { .. } => {} + InsnKind::Ecall => return Err(internal!("ecall survived to emission")), + } + Ok(()) +} + +fn emit_assign(out: &mut Vec, dpos: VrPos, src: Loc) -> Result<()> { + let dst_stack = dpos.allocated && dpos.spilled != 0; + if dst_stack { + // store src on stack at dpos.spilled + match src { + Loc::Reg(r) => out.push(Emitted::simple(BpfInsn::new( + bc::class::STX | bc::mode::MEM | bc::size::DW, + BPF_REG_10, + r, + dpos.spilled as i16, + 0, + ))), + Loc::Const { v, .. } => out.push(Emitted::simple(BpfInsn::new( + bc::class::ST | bc::mode::MEM | bc::size::DW, + BPF_REG_10, + 0, + dpos.spilled as i16, + v as i32, + ))), + Loc::Stack(_) => return Err(internal!("stack-to-stack assign (unsupported)")), + Loc::Undef => return Err(internal!("undef assign source")), + } + } else { + emit_into_reg(out, dpos.alloc_reg, src)?; + } + Ok(()) +} + +fn emit_into_reg_if_needed(out: &mut Vec, dreg: u8, src: Loc) -> Result<()> { + match src { + Loc::Reg(r) if r == dreg => Ok(()), + _ => emit_into_reg(out, dreg, src), + } +} + +fn emit_bin( + out: &mut Vec, + dreg: u8, + op: BinOp, + alu: AluOp, + v0: Loc, + v1: Loc, +) -> Result<()> { + // Normalize so v0 lands in dreg, then apply op with v1. + // Commutative: if v1 is in dreg, swap. + let (v0, v1) = if op.is_commutative() { + if let Loc::Reg(r) = v1 { + if r == dreg { + (v1, v0) + } else { + (v0, v1) + } + } else { + (v0, v1) + } + } else { + (v0, v1) + }; + + emit_into_reg_if_needed(out, dreg, v0)?; + let code = binop_code(op); + match v1 { + Loc::Reg(r) => out.push(Emitted::simple(BpfInsn::new( + alu_class(alu) | code | bc::src::X, + dreg, + r, + 0, + 0, + ))), + Loc::Const { v, .. } => { + if op == BinOp::Add && v == 0 { + return Ok(()); // no-op + } + out.push(Emitted::simple(BpfInsn::new( + alu_class(alu) | code | bc::src::K, + dreg, + 0, + 0, + v as i32, + ))) + } + Loc::Stack(off) => { + // Reload into a scratch is unsupported here; spill prep should + // prevent stack operands for ALU. Treat as error. + let _ = off; + return Err(internal!("ALU operand on stack (unsupported)")); + } + Loc::Undef => return Err(internal!("undef ALU operand")), + } + Ok(()) +} + +fn emit_load_raw( + out: &mut Vec, + dreg: u8, + vr_type: VrType, + addr: &crate::ir::value::AddrValue, +) -> Result<()> { + let size = vr_size(vr_type); + match loc_of(addr.value) { + Loc::Reg(base) => out.push(Emitted::simple(BpfInsn::new( + bc::class::LDX | size | bc::mode::MEM, + dreg, + base, + addr.offset, + 0, + ))), + other => return Err(internal!("loadraw base not a register: {other:?}")), + } + Ok(()) +} + +fn emit_store_raw( + out: &mut Vec, + vr_type: VrType, + addr: &crate::ir::value::AddrValue, + val: Loc, +) -> Result<()> { + let size = vr_size(vr_type); + let base = match loc_of(addr.value) { + Loc::Reg(b) => b, + other => return Err(internal!("storeraw base not a register: {other:?}")), + }; + match val { + Loc::Reg(r) => out.push(Emitted::simple(BpfInsn::new( + bc::class::STX | size | bc::mode::MEM, + base, + r, + addr.offset, + 0, + ))), + Loc::Const { v, .. } => out.push(Emitted::simple(BpfInsn::new( + bc::class::ST | size | bc::mode::MEM, + base, + 0, + addr.offset, + v as i32, + ))), + other => return Err(internal!("storeraw value unsupported: {other:?}")), + } + Ok(()) +} + +fn emit_cond_jmp( + out: &mut Vec, + cond: Cond, + alu: AluOp, + v0: Loc, + v1: Loc, + target: Option, +) -> Result<()> { + let jmp_class = if alu == AluOp::Alu64 { + bc::class::JMP + } else { + bc::class::JMP32 + }; + // Ensure v0 is a register; if v0 is const and v1 is reg, swap+invert. + let (cond, v0, v1) = match (v0, v1) { + (Loc::Reg(_), _) => (cond, v0, v1), + (Loc::Const { .. }, Loc::Reg(_)) => (swap_cond(cond), v1, v0), + _ => return Err(internal!("conditional jump needs at least one register operand")), + }; + let dst = match v0 { + Loc::Reg(r) => r, + _ => return Err(internal!("cond jmp dst not a register")), + }; + let mut e = match v1 { + Loc::Reg(r) => Emitted::simple(BpfInsn::new( + jmp_class | cond_code(cond) | bc::src::X, + dst, + r, + 0, + 0, + )), + Loc::Const { v, .. } => Emitted::simple(BpfInsn::new( + jmp_class | cond_code(cond) | bc::src::K, + dst, + 0, + 0, + v as i32, + )), + _ => return Err(internal!("cond jmp src unsupported")), + }; + e.is_cond = true; + e.target2 = target; + out.push(e); + Ok(()) +} + +/// Full normalization + emission driver. +pub fn normalize_and_emit(env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + flatten(func, cg); + replace_builtin_consts(func); + + // Emit per block, recording each block's starting machine position. + let mut block_emits: Vec<(BbId, Vec<(InsnId, Vec)>)> = Vec::new(); + let bbs = func.reachable_bbs.clone(); + for bb in bbs { + let mut insn_emits = Vec::new(); + let insns = func.bb(bb).insns.clone(); + for id in insns { + let mut out = Vec::new(); + emit_insn(func, cg, id, &mut out)?; + insn_emits.push((id, out)); + } + block_emits.push((bb, insn_emits)); + } + + // Compute machine positions for each block and instruction. + let mut block_pos: HashMap = HashMap::new(); + let mut pos = 0usize; + for (bb, insn_emits) in &block_emits { + block_pos.insert(*bb, pos); + for (_, emits) in insn_emits { + for e in emits { + pos += if e.wide { 2 } else { 1 }; + } + } + } + let total = pos; + if total >= 1_000_000 { + return Err(invalid!("program too large after code generation")); + } + + // Relocate jumps and synthesize. + let mut insns = vec![BpfInsn::default(); total]; + let mut cur = 0usize; + for (_bb, insn_emits) in &block_emits { + for (_, emits) in insn_emits { + for e in emits { + let mut machine = e.insn; + if e.is_ja { + let target = e.target1.ok_or_else(|| internal!("ja without target"))?; + let tpos = *block_pos.get(&target).unwrap(); + machine.off = (tpos as i64 - cur as i64 - 1) as i16; + } + if e.is_cond { + let target = e.target2.ok_or_else(|| internal!("cond jmp without target"))?; + let tpos = *block_pos.get(&target).unwrap(); + machine.off = (tpos as i64 - cur as i64 - 1) as i16; + } + insns[cur] = machine; + if e.wide { + insns[cur + 1] = BpfInsn { + imm: (e.imm64 >> 32) as i32, + ..Default::default() + }; + cur += 2; + } else { + cur += 1; + } + } + } + } + + env.insns = insns; + let _ = (cg, func); + Ok(()) +} diff --git a/core-rs/epass-ir/src/cg/prepare.rs b/core-rs/epass-ir/src/cg/prepare.rs new file mode 100644 index 00000000..4a870121 --- /dev/null +++ b/core-rs/epass-ir/src/cg/prepare.rs @@ -0,0 +1,254 @@ +//! Code-generation preparation: initialize RA metadata, lower calls and +//! function arguments to pre-colored copies, pre-spill arrays and +//! non-encodable constants, and fold the final stack offset into raw offsets. + +use std::collections::HashMap; + +use crate::bytecode::{BPF_REG_0, MAX_FUNC_ARG}; +use crate::env::Env; +use crate::error::Result; +use crate::ir::insn::InsnKind; +use crate::ir::value::{ConstKind, VrPos, VrType}; +use crate::ir::{Function, InsertPos, Value}; + +use super::{CgExtra, CgState, RA_COLORS}; + +/// Initialize per-instruction CG metadata and the physical-register pseudos. +pub fn init_cg(_env: &mut Env, func: &mut Function) -> Result { + // Create R0..R9 pseudo-instructions (pre-colored, non-virtual). + let mut regs = [crate::ir::InsnId(0); RA_COLORS]; + for (i, slot) in regs.iter_mut().enumerate() { + *slot = func.create_reg_pseudo(i as u8); + } + + let mut extra: HashMap = HashMap::new(); + + // Every real instruction gets a CgExtra; its `dst` is itself unless void. + for &bb in &func.reachable_bbs { + for &id in &func.bb(bb).insns { + let dst = if func.insn(id).is_void() { None } else { Some(id) }; + extra.insert(id, CgExtra::new(dst)); + } + } + + // Physical registers: pre-colored, finalized, non-virtual. + for (i, ®) in regs.iter().enumerate() { + let mut e = CgExtra::new(Some(reg)); + e.vr_pos = VrPos { + allocated: true, + alloc_reg: i as u8, + spilled: 0, + spilled_size: 0, + }; + e.nonvr = true; + e.finalized = true; + extra.insert(reg, e); + } + + // Stack pointer pseudo. + { + let mut e = CgExtra::new(Some(func.sp)); + e.vr_pos = VrPos::stack_ptr(); + e.nonvr = true; + e.finalized = true; + extra.insert(func.sp, e); + } + + Ok(CgState { + extra, + regs, + seo: Vec::new(), + all_var: Vec::new(), + stack_offset: 0, + }) +} + +/// Lower each `call`: move arguments into pre-colored R1..R5, and the result +/// out of R0. +pub fn change_call(_env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + let calls: Vec<_> = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .filter(|&id| matches!(func.insn(id).kind, InsnKind::Call { .. } | InsnKind::Ecall)) + .collect(); + + for call in calls { + // Arguments -> assign into R1..R5 (pre-colored), inserted before the call. + let args: Vec = func.insn(call).values.to_vec(); + for (i, arg) in args.iter().enumerate() { + func.remove_use(*arg, call); + let assign = func.build_assign_at(call, *arg, InsertPos::Front); + pre_color(cg, func, assign, (i + 1) as u8); + } + func.insn_mut(call).values.clear(); + + // Result: if used, copy R0 into a fresh assign and rewrite users. + let has_users = !func.insn(call).users.is_empty(); + cg.extra_mut(call).dst = None; + if has_users { + let r0 = cg.regs[BPF_REG_0 as usize]; + let assign = func.build_assign_at(call, Value::Insn(r0), InsertPos::Back); + ensure_extra(cg, func, assign); + func.replace_all_uses(call, Value::Insn(assign)); + } + } + Ok(()) +} + +/// Replace uses of function-argument pseudos with copies from R1..R5. +pub fn change_fun_arg(_env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + for i in 0..MAX_FUNC_ARG { + let arg = func.args[i]; + if func.insn(arg).users.is_empty() { + continue; + } + let reg = cg.regs[i + 1]; + let entry = func.entry; + let assign = func.build_assign_bb(entry, Value::Insn(reg), InsertPos::FrontAfterPhi); + ensure_extra(cg, func, assign); + func.replace_all_uses(arg, Value::Insn(assign)); + } + Ok(()) +} + +/// Assign fixed stack slots to `allocarray` instructions. +pub fn spill_array(_env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + let arrays: Vec<_> = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .filter(|&id| matches!(func.insn(id).kind, InsnKind::AllocArray { .. })) + .collect(); + for id in arrays { + let (vr_type, num) = match func.insn(id).kind { + InsnKind::AllocArray { vr_type, num } => (vr_type, num), + _ => unreachable!(), + }; + let size = num * vr_type.size(); + if size == 0 { + return Err(crate::invalid!("allocarray with size 0")); + } + let roundup = (size + 7) & !7; + let off = cg.new_spill(roundup); + let e = cg.extra_mut(id); + e.vr_pos.allocated = true; + e.vr_pos.spilled = off; + e.vr_pos.spilled_size = size; + e.finalized = true; + } + Ok(()) +} + +/// Materialize constants that the BPF ISA cannot encode directly. +pub fn spill_const(_env: &mut Env, func: &mut Function, cg: &mut CgState) -> Result<()> { + let insns: Vec<_> = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .collect(); + + for id in insns { + // Non-commutative binary ALU: first operand cannot be a constant. + if func.insn(id).is_bin_alu() && !func.insn(id).is_commutative_alu() { + let v0 = func.insn(id).values[0]; + if v0.is_const() { + let assign = func.build_assign_at(id, v0, InsertPos::Front); + ensure_extra(cg, func, assign); + func.change_value(id, v0, Value::Insn(assign)); + } + } + // Conditional jump: both operands cannot be constants. + if func.insn(id).is_cond_jmp() && func.insn(id).values.len() == 2 { + let v0 = func.insn(id).values[0]; + let v1 = func.insn(id).values[1]; + if v0.is_const() && v1.is_const() { + // Promote values[1] into a register. + let assign = func.build_assign_at(id, v1, InsertPos::Front); + ensure_extra(cg, func, assign); + func.change_value(id, v1, Value::Insn(assign)); + } + } + } + Ok(()) +} + +/// Fold the final stack offset into stack-relative raw constants. +pub fn add_stack_offset(func: &mut Function, offset: i32) { + let insns: Vec<_> = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .collect(); + for id in insns { + // Raw load/store address offsets. + match &mut func.insn_mut(id).kind { + InsnKind::LoadRaw { addr, .. } | InsnKind::StoreRaw { addr, .. } => match addr.offset_kind + { + ConstKind::RawOff => { + addr.offset += offset as i16; + addr.offset_kind = ConstKind::Plain; + } + ConstKind::RawOffRev => { + addr.offset -= offset as i16; + addr.offset_kind = ConstKind::Plain; + } + ConstKind::Plain => {} + }, + _ => {} + } + // Operand constants that are stack-relative. + let n = func.insn(id).values.len(); + for i in 0..n { + if let Value::Const { v, kind, .. } = &mut func.insn_mut(id).values[i] { + match *kind { + ConstKind::RawOff => { + *v += offset as i64; + *kind = ConstKind::Plain; + } + ConstKind::RawOffRev => { + *v -= offset as i64; + *kind = ConstKind::Plain; + } + ConstKind::Plain => {} + } + } + } + } +} + +// ---- helpers ---- + +fn pre_color(cg: &mut CgState, func: &Function, insn: crate::ir::InsnId, reg: u8) { + ensure_extra(cg, func, insn); + let e = cg.extra_mut(insn); + e.finalized = true; + e.vr_pos.allocated = true; + e.vr_pos.alloc_reg = reg; + e.vr_pos.spilled = 0; +} + +/// Make sure a freshly-created instruction has a CgExtra entry. +pub(super) fn ensure_extra(cg: &mut CgState, func: &Function, insn: crate::ir::InsnId) { + cg.extra.entry(insn).or_insert_with(|| { + let dst = if func.insn(insn).is_void() { + None + } else { + Some(insn) + }; + CgExtra::new(dst) + }); +} + +/// Add a fresh alloc instruction (used by the spiller) with CG metadata. +pub(super) fn create_alloc( + cg: &mut CgState, + func: &mut Function, + bb: crate::ir::BbId, + ty: VrType, + pos: InsertPos, +) -> crate::ir::InsnId { + let id = func.build_alloc_bb(bb, ty, pos); + ensure_extra(cg, func, id); + id +} diff --git a/core-rs/epass-ir/src/check.rs b/core-rs/epass-ir/src/check.rs new file mode 100644 index 00000000..d690cdad --- /dev/null +++ b/core-rs/epass-ir/src/check.rs @@ -0,0 +1,142 @@ +//! IR validity checker, run after lifting and after each pass (unless disabled). +//! +//! Verifies structural invariants the rest of the pipeline relies on: +//! - conditional jumps have exactly two successors with `bb1` being the +//! physically-next block in the layout; +//! - `ja` has exactly one successor; +//! - every operand referencing an instruction points to a live instruction; +//! - def-use chains are consistent. + +use std::collections::HashSet; + +use crate::env::Env; +use crate::error::Result; +use crate::internal; +use crate::ir::{Function, InsnKind, Value}; + +/// Run the structural checker over a function. +pub fn prog_check(env: &Env, func: &Function) -> Result<()> { + if env.opts.disable_prog_check { + return Ok(()); + } + + // Build the set of live instruction ids for operand validation. + let mut live: HashSet = HashSet::new(); + for &bb in &func.reachable_bbs { + for &id in &func.bb(bb).insns { + live.insert(id.0); + } + } + live.insert(func.sp.0); + for a in func.args { + live.insert(a.0); + } + + for (layout_idx, &bb) in func.reachable_bbs.iter().enumerate() { + let block = func.bb(bb); + + // Terminator checks. + if let Some(last) = block.last() { + let insn = func.insn(last); + match &insn.kind { + InsnKind::CondJmp { .. } => { + if block.succs.len() != 2 { + return Err(internal!( + "conditional jump in bb{} has {} successors (expected 2)", + bb.0, + block.succs.len() + )); + } + // bb1 (fallthrough) must be the physically next block. + let next = func.reachable_bbs.get(layout_idx + 1).copied(); + if insn.bb1 != next { + return Err(internal!( + "conditional jump fallthrough bb1 is not the next block in bb{}", + bb.0 + )); + } + } + InsnKind::Ja => { + if block.succs.len() != 1 { + return Err(internal!( + "ja in bb{} has {} successors (expected 1)", + bb.0, + block.succs.len() + )); + } + } + _ => {} + } + } + + // Phi placement and predecessor consistency. + let mut seen_non_phi = false; + for &id in &block.insns { + let insn = func.insn(id); + if matches!(insn.kind, InsnKind::Phi) { + if seen_non_phi { + return Err(internal!( + "phi instruction %{} in bb{} appears after a non-phi instruction", + id.0, + bb.0 + )); + } + if insn.phi.len() != block.preds.len() { + return Err(internal!( + "phi instruction %{} in bb{} has {} inputs but block has {} predecessors", + id.0, + bb.0, + insn.phi.len(), + block.preds.len() + )); + } + let mut seen = HashSet::new(); + for entry in &insn.phi { + if !block.preds.contains(&entry.bb) { + return Err(internal!( + "phi instruction %{} in bb{} has input from non-predecessor bb{}", + id.0, + bb.0, + entry.bb.0 + )); + } + if !seen.insert(entry.bb.0) { + return Err(internal!( + "phi instruction %{} in bb{} has duplicate input from bb{}", + id.0, + bb.0, + entry.bb.0 + )); + } + } + } else { + seen_non_phi = true; + } + } + + // Operand liveness + def-use consistency. + for &id in &block.insns { + let insn = func.insn(id); + for v in insn.operand_values() { + if let Value::Insn(def) = v { + if !live.contains(&def.0) { + return Err(internal!( + "instruction %{} uses a dead/unknown def %{}", + id.0, + def.0 + )); + } + // The def must list `id` as a user. + if !func.insn(def).users.contains(&id) { + return Err(internal!( + "broken def-use: %{} uses %{} but is not in its user list", + id.0, + def.0 + )); + } + } + } + } + } + Ok(()) +} diff --git a/core-rs/epass-ir/src/env.rs b/core-rs/epass-ir/src/env.rs new file mode 100644 index 00000000..024a7a38 --- /dev/null +++ b/core-rs/epass-ir/src/env.rs @@ -0,0 +1,107 @@ +//! The pipeline environment: carries options, the instruction buffer, a log +//! sink, and timing statistics through every stage. + +use std::fmt::Write as _; +use std::time::Instant; + +use crate::bytecode::BpfInsn; +use crate::opts::Opts; + +/// Log severity levels, matching the C `PRINT_LOG_*` macros. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum LogLevel { + Error = 0, + Warning = 1, + Debug = 2, + Info = 3, +} + +/// Mutable state threaded through lifting, passes, and code generation. +pub struct Env { + pub opts: Opts, + /// The working instruction buffer (input on entry, rewritten on output). + pub insns: Vec, + /// Accumulated, human-readable log text. + log: String, + + pub lift_time_ns: u128, + pub run_time_ns: u128, + pub cg_time_ns: u128, +} + +impl Env { + pub fn new(opts: Opts, insns: Vec) -> Self { + Self { + opts, + insns, + log: String::new(), + lift_time_ns: 0, + run_time_ns: 0, + cg_time_ns: 0, + } + } + + /// Append a log message if it is at or below the configured verbosity. + /// + /// `verbose` is a "level" where higher prints more; a message is kept when + /// its severity is important enough (`Error`=0) or verbosity is high enough. + pub fn log(&mut self, level: LogLevel, args: std::fmt::Arguments<'_>) { + // Errors and warnings are always recorded; debug/info gated by verbosity. + let keep = match level { + LogLevel::Error | LogLevel::Warning => true, + LogLevel::Info => self.opts.verbose >= 1, + LogLevel::Debug => self.opts.verbose >= 2, + }; + if keep { + let _ = self.log.write_fmt(args); + } + } + + pub fn log_str(&mut self, level: LogLevel, msg: &str) { + self.log(level, format_args!("{msg}")); + } + + /// Borrow the accumulated log text. + pub fn log_buffer(&self) -> &str { + &self.log + } + + /// Take ownership of the log, clearing the internal buffer. + pub fn take_log(&mut self) -> String { + std::mem::take(&mut self.log) + } + + pub fn total_time_ns(&self) -> u128 { + self.lift_time_ns + self.run_time_ns + self.cg_time_ns + } +} + +/// Convenience macros for logging into an [`Env`]. +#[macro_export] +macro_rules! log_error { + ($env:expr, $($arg:tt)*) => { $env.log($crate::env::LogLevel::Error, format_args!($($arg)*)) }; +} +#[macro_export] +macro_rules! log_warn { + ($env:expr, $($arg:tt)*) => { $env.log($crate::env::LogLevel::Warning, format_args!($($arg)*)) }; +} +#[macro_export] +macro_rules! log_info { + ($env:expr, $($arg:tt)*) => { $env.log($crate::env::LogLevel::Info, format_args!($($arg)*)) }; +} +#[macro_export] +macro_rules! log_debug { + ($env:expr, $($arg:tt)*) => { $env.log($crate::env::LogLevel::Debug, format_args!($($arg)*)) }; +} + +/// A simple scoped timer; call [`Timer::elapsed_ns`] to read the duration. +pub struct Timer(Instant); + +impl Timer { + pub fn start() -> Self { + Timer(Instant::now()) + } + pub fn elapsed_ns(&self) -> u128 { + self.0.elapsed().as_nanos() + } +} diff --git a/core-rs/epass-ir/src/error.rs b/core-rs/epass-ir/src/error.rs new file mode 100644 index 00000000..83833938 --- /dev/null +++ b/core-rs/epass-ir/src/error.rs @@ -0,0 +1,58 @@ +//! Error types for the ePass IR pipeline. + +use thiserror::Error; + +/// Errors produced while lifting, transforming, or compiling an eBPF program. +/// +/// The C implementation collapses everything into a single `env->err` integer +/// (mostly `-ENOSYS` / `-ENOMEM` / `-EINVAL`) plus a free-form log string. Here +/// we keep a richer, typed enum while still being able to surface a numeric +/// errno-like code for the CLI return path via [`Error::errno`]. +#[derive(Debug, Error, Clone, PartialEq, Eq)] +pub enum Error { + /// The program uses a feature ePass does not support yet. + #[error("unsupported: {0}")] + Unsupported(String), + + /// The input program is malformed / not a valid eBPF program. + #[error("invalid program: {0}")] + InvalidProgram(String), + + /// An internal invariant was violated (a bug in ePass). + #[error("internal error: {0}")] + Internal(String), + + /// Register allocation could not find a valid assignment. + #[error("register allocation failed: {0}")] + RegAlloc(String), +} + +impl Error { + /// Map to a negative errno, matching the conventions used by the C tool. + pub fn errno(&self) -> i32 { + match self { + Error::Unsupported(_) | Error::Internal(_) | Error::RegAlloc(_) => -38, // -ENOSYS + Error::InvalidProgram(_) => -22, // -EINVAL + } + } +} + +pub type Result = std::result::Result; + +/// Build an [`Error::Unsupported`]. +#[macro_export] +macro_rules! unsupported { + ($($arg:tt)*) => { $crate::error::Error::Unsupported(format!($($arg)*)) }; +} + +/// Build an [`Error::InvalidProgram`]. +#[macro_export] +macro_rules! invalid { + ($($arg:tt)*) => { $crate::error::Error::InvalidProgram(format!($($arg)*)) }; +} + +/// Build an [`Error::Internal`]. +#[macro_export] +macro_rules! internal { + ($($arg:tt)*) => { $crate::error::Error::Internal(format!($($arg)*)) }; +} diff --git a/core-rs/epass-ir/src/ffi.rs b/core-rs/epass-ir/src/ffi.rs new file mode 100644 index 00000000..8615ea99 --- /dev/null +++ b/core-rs/epass-ir/src/ffi.rs @@ -0,0 +1,171 @@ +//! C ABI for embedding ePass into C projects (notably the patched libbpf). +//! +//! The Rust API uses non-FFI-safe types (`Vec`, enums, `Result`), so this +//! module provides a small, stable C surface. A run takes a `bpf_insn` array +//! plus an option string, produces an opaque result holding the rewritten +//! program and a log, and exposes accessors plus a free function. +//! +//! ```c +//! struct bpf_insn; // the standard kernel/libbpf type +//! typedef struct epass_result epass_result; +//! +//! epass_result *epass_run(const struct bpf_insn *insns, size_t insn_cnt, +//! const char *gopt, int *out_err); +//! const struct bpf_insn *epass_result_insns(const epass_result *r); +//! size_t epass_result_insn_cnt(const epass_result *r); +//! const char *epass_result_log(const epass_result *r); // NUL-terminated +//! void epass_result_free(epass_result *r); +//! ``` +#![allow(non_camel_case_types)] + +use std::ffi::{c_char, c_int, CStr, CString}; + +use crate::bytecode::BpfInsn; +use crate::{autorun, default_passes, Env, Opts}; + +/// C-ABI mirror of the kernel/libbpf `struct bpf_insn` (8 bytes, packed bitfields +/// for the registers). Field order/sizes match exactly so the two are +/// bit-compatible. +#[repr(C)] +#[derive(Clone, Copy)] +pub struct epass_insn { + pub code: u8, + /// `dst_reg:4 | src_reg:4` packed (low nibble = dst). + pub regs: u8, + pub off: i16, + pub imm: i32, +} + +impl From for BpfInsn { + fn from(c: epass_insn) -> Self { + BpfInsn { + code: c.code, + dst_reg: c.regs & 0x0f, + src_reg: (c.regs >> 4) & 0x0f, + off: c.off, + imm: c.imm, + } + } +} + +impl From for epass_insn { + fn from(b: BpfInsn) -> Self { + epass_insn { + code: b.code, + regs: (b.dst_reg & 0x0f) | ((b.src_reg & 0x0f) << 4), + off: b.off, + imm: b.imm, + } + } +} + +/// Opaque result handle returned to C. +pub struct epass_result { + insns: Vec, + log: CString, +} + +/// Run the full ePass pipeline on a program. +/// +/// # Safety +/// `insns` must point to `insn_cnt` valid `epass_insn` values. `gopt` may be +/// NULL or a NUL-terminated UTF-8 string. On success returns a non-null handle +/// and writes `0` to `out_err` (if non-null); on failure returns NULL and +/// writes a negative errno-style code. +#[no_mangle] +pub unsafe extern "C" fn epass_run( + insns: *const epass_insn, + insn_cnt: usize, + gopt: *const c_char, + out_err: *mut c_int, +) -> *mut epass_result { + let set_err = |v: c_int| { + if !out_err.is_null() { + unsafe { *out_err = v }; + } + }; + + if insns.is_null() && insn_cnt != 0 { + set_err(-22); // -EINVAL + return std::ptr::null_mut(); + } + + // Decode the input program. + let input: Vec = (0..insn_cnt) + .map(|i| unsafe { *insns.add(i) }.into()) + .collect(); + + // Parse options. + let mut opts = Opts::default(); + if !gopt.is_null() { + if let Ok(s) = unsafe { CStr::from_ptr(gopt) }.to_str() { + if let Err(_e) = opts.apply_gopt(s) { + set_err(-22); + return std::ptr::null_mut(); + } + } + } + + let mut env = Env::new(opts, input); + let passes = default_passes(); + match autorun(&mut env, &passes) { + Ok(()) => { + let insns: Vec = env.insns.iter().map(|&b| b.into()).collect(); + let log = CString::new(env.take_log()).unwrap_or_default(); + set_err(0); + Box::into_raw(Box::new(epass_result { insns, log })) + } + Err(e) => { + set_err(e.errno() as c_int); + std::ptr::null_mut() + } + } +} + +/// Pointer to the rewritten instruction array (valid until `epass_result_free`). +/// +/// # Safety +/// `r` must be a handle returned by [`epass_run`]. +#[no_mangle] +pub unsafe extern "C" fn epass_result_insns(r: *const epass_result) -> *const epass_insn { + if r.is_null() { + return std::ptr::null(); + } + unsafe { (*r).insns.as_ptr() } +} + +/// Number of rewritten instructions. +/// +/// # Safety +/// `r` must be a handle returned by [`epass_run`]. +#[no_mangle] +pub unsafe extern "C" fn epass_result_insn_cnt(r: *const epass_result) -> usize { + if r.is_null() { + return 0; + } + unsafe { (*r).insns.len() } +} + +/// NUL-terminated log text accumulated during the run. +/// +/// # Safety +/// `r` must be a handle returned by [`epass_run`]. +#[no_mangle] +pub unsafe extern "C" fn epass_result_log(r: *const epass_result) -> *const c_char { + if r.is_null() { + return std::ptr::null(); + } + unsafe { (*r).log.as_ptr() } +} + +/// Free a result handle. +/// +/// # Safety +/// `r` must be a handle returned by [`epass_run`] (or NULL), and must not be +/// used afterwards. +#[no_mangle] +pub unsafe extern "C" fn epass_result_free(r: *mut epass_result) { + if !r.is_null() { + drop(unsafe { Box::from_raw(r) }); + } +} diff --git a/core-rs/epass-ir/src/helpers.rs b/core-rs/epass-ir/src/helpers.rs new file mode 100644 index 00000000..6f9e7896 --- /dev/null +++ b/core-rs/epass-ir/src/helpers.rs @@ -0,0 +1,49 @@ +//! Static eBPF helper-function argument-count table. +//! +//! Indexed by helper id. A value of `-1` means "variable / special-cased" +//! (currently only `trace_printk`, id 6). This is data ported verbatim from the +//! C implementation and should track the kernel's helper list. + +/// Number of arguments each helper takes, indexed by helper id. +/// `i8::MIN` is used for "unknown / unsupported" gaps. +pub const HELPER_ARG_NUM: &[i8] = &[ + /* 0 unused */ i8::MIN, + 2, 4, 2, 3, 0, -1, 0, 0, 5, 5, // 1-10 + 5, 3, 3, 0, 0, 2, 1, 3, 1, 4, // 11-20 + 4, 2, 2, 1, 5, 4, 3, 5, 3, 3, // 21-30 + 3, 2, 3, 1, 0, 3, 2, 3, 2, 2, // 31-40 + 1, 0, 3, 2, 3, 1, 1, 2, 5, 4, // 41-50 + 3, 4, 4, 2, 4, 3, 5, 2, 2, 4, // 51-60 + 2, 2, 4, 3, 2, 5, 4, 5, 4, 4, // 61-70 + 4, 4, 4, 4, 3, 4, 1, 4, 1, 0, // 71-80 + 2, 4, 2, 5, 5, 1, 3, 2, 2, 4, // 81-90 + 4, 3, 1, 1, 1, 1, 1, 1, 5, 5, // 91-100 + 4, 3, 3, 3, 4, 4, 5, 2, 1, 5, // 101-110 + 5, 3, 3, 3, 3, 2, 1, 0, 4, 4, // 111-120 + 5, 1, 1, 3, 0, 5, 3, 1, 2, 4, // 121-130 + 3, 2, 2, 2, 2, 1, 1, 1, 1, 1, // 131-140 + 4, 4, 4, 3, 5, 2, 3, 3, 5, 4, // 141-150 + 1, 4, 2, 1, 2, 5, 2, 0, 2, 0, // 151-160 + 3, 1, 5, 4, 5, 3, 4, 1, 3, 3, // 161-170 + 3, 1, 1, 1, 1, 3, 4, 1, 4, 5, // 171-180 + 4, 3, 3, 2, 1, 0, 1, 1, 4, 4, // 181-190 + 5, 3, 3, 2, 3, 1, 4, 4, 2, 2, // 191-200 + 5, 5, 3, 3, 3, 2, 2, 0, 4, 5, // 201-210 + 2, // 211 +]; + +/// Look up a helper's argument count. +/// +/// Returns `Some(n)` for fixed-arity helpers, `Some(-1)` for the variadic +/// `trace_printk`, and `None` for unknown / unsupported ids. +pub fn helper_arg_num(id: i32) -> Option { + if id < 0 || id as usize >= HELPER_ARG_NUM.len() { + return None; + } + let n = HELPER_ARG_NUM[id as usize]; + if n == i8::MIN { + None + } else { + Some(n) + } +} diff --git a/core-rs/epass-ir/src/ir/builder.rs b/core-rs/epass-ir/src/ir/builder.rs new file mode 100644 index 00000000..4019929f --- /dev/null +++ b/core-rs/epass-ir/src/ir/builder.rs @@ -0,0 +1,324 @@ +//! Ergonomic IR instruction builder. +//! +//! This mirrors the constructor-style API of the original C core while keeping +//! Rust's arena-based ownership model. Builder methods create an instruction at +//! the configured insertion point, fill opcode-specific fields, and maintain +//! def-use chains for all operands. + +use crate::bytecode::MAX_FUNC_ARG; +use crate::error::Result; +use crate::invalid; + +use super::insn::{BinOp, Cond, EndKind, InsnKind}; +use super::value::{AddrValue, AluOp, LoadImmExtra, PhiValue, RawPos, Value, VrType}; +use super::{BbId, Function, InsnId, InsertPos}; + +/// Where newly-built instructions are inserted. +#[derive(Debug, Clone, Copy)] +pub enum InsertPoint { + /// Insert in a basic block according to `pos`. + Bb { bb: BbId, pos: InsertPos }, + /// Insert relative to an existing instruction. + Insn { anchor: InsnId, pos: InsertPos }, +} + +/// Builder for constructing IR instructions safely and ergonomically. +pub struct IrBuilder<'f> { + func: &'f mut Function, + insert: InsertPoint, + raw_pos: RawPos, +} + +impl<'f> IrBuilder<'f> { + /// Build at a basic block insertion point. + pub fn at_bb(func: &'f mut Function, bb: BbId, pos: InsertPos) -> Self { + Self { func, insert: InsertPoint::Bb { bb, pos }, raw_pos: RawPos::default() } + } + + /// Build at the end of a block. + pub fn at_end(func: &'f mut Function, bb: BbId) -> Self { + Self::at_bb(func, bb, InsertPos::Back) + } + + /// Build at the start of a block. + pub fn at_start(func: &'f mut Function, bb: BbId) -> Self { + Self::at_bb(func, bb, InsertPos::Front) + } + + /// Build at the end of a block, before an existing terminator if present. + pub fn before_terminator(func: &'f mut Function, bb: BbId) -> Self { + Self::at_bb(func, bb, InsertPos::BackBeforeJmp) + } + + /// Build after any leading phi nodes in a block. + pub fn after_phi(func: &'f mut Function, bb: BbId) -> Self { + Self::at_bb(func, bb, InsertPos::FrontAfterPhi) + } + + /// Build relative to an existing instruction. + pub fn at_insn(func: &'f mut Function, anchor: InsnId, pos: InsertPos) -> Self { + Self { func, insert: InsertPoint::Insn { anchor, pos }, raw_pos: RawPos::default() } + } + + /// Build immediately before an instruction. + pub fn before(func: &'f mut Function, anchor: InsnId) -> Self { + Self::at_insn(func, anchor, InsertPos::Front) + } + + /// Build immediately after an instruction. + pub fn after(func: &'f mut Function, anchor: InsnId) -> Self { + Self::at_insn(func, anchor, InsertPos::Back) + } + + /// Attach source-bytecode provenance to subsequently-created instructions. + pub fn with_raw_pos(mut self, raw_pos: RawPos) -> Self { + self.raw_pos = raw_pos; + self + } + + /// Change insertion point for this builder. + pub fn set_insert(&mut self, insert: InsertPoint) { + self.insert = insert; + } + + /// Borrow the underlying function. + pub fn func(&self) -> &Function { + self.func + } + + /// Mutably borrow the underlying function. + pub fn func_mut(&mut self) -> &mut Function { + self.func + } + + fn create(&mut self, kind: InsnKind) -> InsnId { + let id = match self.insert { + InsertPoint::Bb { bb, pos } => self.func.create_insn(bb, kind, pos), + InsertPoint::Insn { anchor, pos } => self.func.create_insn_at(anchor, kind, pos), + }; + self.func.insn_mut(id).raw_pos = self.raw_pos; + id + } + + fn push_value(&mut self, id: InsnId, value: Value) { + self.func.add_value_operand(id, value); + } + + fn push_values(&mut self, id: InsnId, values: impl IntoIterator) { + for v in values { + self.push_value(id, v); + } + } + + /// `%x = alloc `. + pub fn alloc(&mut self, ty: VrType) -> InsnId { + self.create(InsnKind::Alloc { vr_type: ty }) + } + + /// `%x = allocarray x `. + pub fn alloc_array(&mut self, ty: VrType, num: u32) -> InsnId { + self.create(InsnKind::AllocArray { vr_type: ty, num }) + } + + /// `%x = getelemptr , `. + pub fn get_elem_ptr(&mut self, index: Value, array: Value) -> InsnId { + let id = self.create(InsnKind::GetElemPtr); + self.push_value(id, index); + self.push_value(id, array); + id + } + + /// `store , `. + pub fn store(&mut self, alloc: Value, value: Value) -> InsnId { + let id = self.create(InsnKind::Store); + self.push_value(id, alloc); + self.push_value(id, value); + id + } + + /// `%x = load `. + pub fn load(&mut self, alloc: Value) -> InsnId { + let id = self.create(InsnKind::Load); + self.push_value(id, alloc); + id + } + + /// `%x = loadimm. `. + pub fn load_imm_extra(&mut self, extra: LoadImmExtra, imm64: i64) -> InsnId { + self.create(InsnKind::LoadImmExtra { extra, imm64 }) + } + + /// `storeraw. [base+off], value`. + pub fn store_raw(&mut self, ty: VrType, base: Value, offset: i16, value: Value) -> InsnId { + self.store_raw_addr(ty, AddrValue::new(base, offset), value) + } + + /// `storeraw. addr, value` with a fully-specified address descriptor. + pub fn store_raw_addr(&mut self, ty: VrType, addr: AddrValue, value: Value) -> InsnId { + let base = addr.value; + let id = self.create(InsnKind::StoreRaw { vr_type: ty, addr }); + self.func.add_use(base, id); + self.push_value(id, value); + id + } + + /// `%x = loadraw. [base+off]`. + pub fn load_raw(&mut self, ty: VrType, base: Value, offset: i16) -> InsnId { + self.load_raw_addr(ty, AddrValue::new(base, offset)) + } + + /// `%x = loadraw. addr` with a fully-specified address descriptor. + pub fn load_raw_addr(&mut self, ty: VrType, addr: AddrValue) -> InsnId { + let base = addr.value; + let id = self.create(InsnKind::LoadRaw { vr_type: ty, addr }); + self.func.add_use(base, id); + id + } + + /// `%x = neg value`. + pub fn neg(&mut self, alu: AluOp, value: Value) -> InsnId { + let id = self.create(InsnKind::Neg); + self.func.insn_mut(id).alu_op = alu; + self.push_value(id, value); + id + } + + /// `%x = end. value`. + pub fn end(&mut self, kind: EndKind, swap_width: u32, value: Value) -> InsnId { + let id = self.create(InsnKind::End { kind, swap_width }); + self.func.insn_mut(id).alu_op = AluOp::Alu32; + self.push_value(id, value); + id + } + + /// `%x = lhs, rhs`. + pub fn bin(&mut self, op: BinOp, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { + let id = self.create(InsnKind::Bin { op }); + self.func.insn_mut(id).alu_op = alu; + self.push_value(id, lhs); + self.push_value(id, rhs); + id + } + + pub fn add(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Add, alu, lhs, rhs) } + pub fn sub(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Sub, alu, lhs, rhs) } + pub fn mul(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Mul, alu, lhs, rhs) } + pub fn div(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Div, alu, lhs, rhs) } + pub fn or(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Or, alu, lhs, rhs) } + pub fn and(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::And, alu, lhs, rhs) } + pub fn lsh(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Lsh, alu, lhs, rhs) } + pub fn arsh(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Arsh, alu, lhs, rhs) } + pub fn rsh(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Rsh, alu, lhs, rhs) } + pub fn modulo(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Mod, alu, lhs, rhs) } + pub fn xor(&mut self, alu: AluOp, lhs: Value, rhs: Value) -> InsnId { self.bin(BinOp::Xor, alu, lhs, rhs) } + + /// `%x = call #fid(args...)`. + pub fn call(&mut self, fid: i32, args: impl IntoIterator) -> Result { + let args: Vec = args.into_iter().collect(); + if args.len() > MAX_FUNC_ARG { + return Err(invalid!("call has {} args, max is {}", args.len(), MAX_FUNC_ARG)); + } + let id = self.create(InsnKind::Call { fid }); + self.push_values(id, args); + Ok(id) + } + + /// `%x = ecall(args...)`. + pub fn ecall(&mut self, args: impl IntoIterator) -> Result { + let args: Vec = args.into_iter().collect(); + if args.len() > MAX_FUNC_ARG { + return Err(invalid!("ecall has {} args, max is {}", args.len(), MAX_FUNC_ARG)); + } + let id = self.create(InsnKind::Ecall); + self.push_values(id, args); + Ok(id) + } + + /// `ret value`. + pub fn ret(&mut self, value: Value) -> InsnId { + let id = self.create(InsnKind::Ret); + self.push_value(id, value); + id + } + + /// `throw`. + pub fn throw(&mut self) -> InsnId { + self.create(InsnKind::Throw) + } + + /// `ja target`. + pub fn ja(&mut self, target: BbId) -> InsnId { + let id = self.create(InsnKind::Ja); + self.func.insn_mut(id).bb1 = Some(target); + id + } + + /// Conditional branch. `fallthrough` is stored in `bb1`; `taken` in `bb2`. + pub fn cond_jmp( + &mut self, + cond: Cond, + alu: AluOp, + lhs: Value, + rhs: Value, + fallthrough: BbId, + taken: BbId, + ) -> InsnId { + let id = self.create(InsnKind::CondJmp { cond }); + { + let insn = self.func.insn_mut(id); + insn.alu_op = alu; + insn.bb1 = Some(fallthrough); + insn.bb2 = Some(taken); + } + self.push_value(id, lhs); + self.push_value(id, rhs); + id + } + + pub fn jeq(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Eq, alu, lhs, rhs, fallthrough, taken) } + pub fn jne(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Ne, alu, lhs, rhs, fallthrough, taken) } + pub fn jgt(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Gt, alu, lhs, rhs, fallthrough, taken) } + pub fn jge(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Ge, alu, lhs, rhs, fallthrough, taken) } + pub fn jlt(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Lt, alu, lhs, rhs, fallthrough, taken) } + pub fn jle(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Le, alu, lhs, rhs, fallthrough, taken) } + pub fn jsgt(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Sgt, alu, lhs, rhs, fallthrough, taken) } + pub fn jsge(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Sge, alu, lhs, rhs, fallthrough, taken) } + pub fn jslt(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Slt, alu, lhs, rhs, fallthrough, taken) } + pub fn jsle(&mut self, alu: AluOp, lhs: Value, rhs: Value, fallthrough: BbId, taken: BbId) -> InsnId { self.cond_jmp(Cond::Sle, alu, lhs, rhs, fallthrough, taken) } + + /// `%x = phi [value, pred]...`. + pub fn phi(&mut self, entries: impl IntoIterator) -> InsnId { + let id = self.create(InsnKind::Phi); + for entry in entries { + self.func.add_phi_operand(id, entry.value, entry.bb); + } + id + } + + /// `%x = phi [value, pred]...` from tuple entries. + pub fn phi_entries(&mut self, entries: impl IntoIterator) -> InsnId { + let id = self.create(InsnKind::Phi); + for (value, bb) in entries { + self.func.add_phi_operand(id, value, bb); + } + id + } + + /// `%x = assign value`. + pub fn assign(&mut self, value: Value) -> InsnId { + let id = self.create(InsnKind::Assign); + self.push_value(id, value); + id + } + + /// Detached/physical register pseudo. This is usually only for codegen. + pub fn reg(&mut self, reg_id: u8) -> InsnId { + self.create(InsnKind::Reg { reg_id }) + } + + /// Function-argument pseudo. Normal functions already provide `func.args`; + /// this constructor is for specialized transformations/tests only. + pub fn function_arg(&mut self, arg_id: u8) -> InsnId { + self.create(InsnKind::FunctionArg { arg_id }) + } +} diff --git a/core-rs/epass-ir/src/ir/insn.rs b/core-rs/epass-ir/src/ir/insn.rs new file mode 100644 index 00000000..bdb03976 --- /dev/null +++ b/core-rs/epass-ir/src/ir/insn.rs @@ -0,0 +1,196 @@ +//! IR instructions: the `InsnKind` opcode enum and the `Insn` node stored in +//! the function's instruction arena. + +use smallvec::SmallVec; + +use super::value::{AddrValue, AluOp, BuiltinConst, LoadImmExtra, PhiValue, RawPos, Value, VrType}; +use super::{BbId, InsnId}; +use crate::bytecode::MAX_FUNC_ARG; + +/// Comparison/condition for conditional jumps. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Cond { + Eq, + Ne, + Gt, + Ge, + Lt, + Le, + Sgt, + Sge, + Slt, + Sle, +} + +/// Binary ALU operations. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BinOp { + Add, + Sub, + Mul, + Div, + Or, + And, + Lsh, + Arsh, + Rsh, + Mod, + Xor, +} + +impl BinOp { + /// Operations whose operands may be swapped without changing the result. + pub fn is_commutative(self) -> bool { + matches!(self, BinOp::Add | BinOp::Mul | BinOp::Or | BinOp::And | BinOp::Xor) + } +} + +/// Endianness conversion direction for `BPF_END`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum EndKind { + ToLe, + ToBe, +} + +/// The opcode and opcode-specific payload of an IR instruction. +/// +/// Operands live separately in [`Insn::values`]; this enum carries the +/// structural / non-value data (jump targets, phi entries, immediates, etc.). +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum InsnKind { + /// Allocate a stack/register slot of `vr_type`. + Alloc { vr_type: VrType }, + /// Allocate a contiguous stack array of `num * sizeof(vr_type)` bytes. + AllocArray { vr_type: VrType, num: u32 }, + /// Pointer to `values[1][values[0]]` (array element). + GetElemPtr, + /// `store values[0], values[1]` (typed; `values[0]` is an `Alloc`). + Store, + /// `values[0] = load ` (typed). + Load, + /// Load a 64-bit immediate / map fd / address. + LoadImmExtra { extra: LoadImmExtra, imm64: i64 }, + /// `storeraw addr, values[0]`. + StoreRaw { vr_type: VrType, addr: AddrValue }, + /// `loadraw addr`. + LoadRaw { vr_type: VrType, addr: AddrValue }, + /// `-values[0]`. + Neg, + /// Byte-swap of `values[0]` to the given endianness and width. + End { kind: EndKind, swap_width: u32 }, + /// Binary ALU: `values[0] values[1]`. + Bin { op: BinOp }, + /// Call helper `fid` with `values[..]` as arguments. + Call { fid: i32 }, + /// Exit with `values[0]`. + Ret, + /// Abort the program (lowered later). + Throw, + /// Unconditional jump to `bb1`. + Ja, + /// Conditional jump: if `values[0] values[1]` jump `bb2` else fall to `bb1`. + CondJmp { cond: Cond }, + /// SSA phi node; entries live in [`Insn::phi`]. + Phi, + /// Copy `values[0]` into the destination (code-gen only). + Assign, + /// A physical register pseudo-instruction (code-gen only). + Reg { reg_id: u8 }, + /// A function-argument pseudo-instruction (R1..R5). + FunctionArg { arg_id: u8 }, + /// An ePass extension call. + Ecall, +} + +impl InsnKind { + /// `true` for control-transfer instructions (jumps, ret, throw). + pub fn is_jmp(&self) -> bool { + matches!( + self, + InsnKind::Ja | InsnKind::CondJmp { .. } | InsnKind::Ret | InsnKind::Throw + ) + } + + pub fn is_cond_jmp(&self) -> bool { + matches!(self, InsnKind::CondJmp { .. }) + } + + pub fn is_bin_alu(&self) -> bool { + matches!(self, InsnKind::Bin { .. }) + } + + /// `true` if the instruction produces no value (no SSA destination). + pub fn is_void(&self) -> bool { + self.is_jmp() || matches!(self, InsnKind::Store | InsnKind::StoreRaw { .. }) + } +} + +/// An SSA instruction node. +#[derive(Debug, Clone)] +pub struct Insn { + pub kind: InsnKind, + /// Operand values (up to [`MAX_FUNC_ARG`] for calls). + pub values: SmallVec<[Value; MAX_FUNC_ARG]>, + /// Result width / ALU class (for ALU / typed memory ops). + pub alu_op: AluOp, + /// Phi entries (only for [`InsnKind::Phi`]). + pub phi: Vec, + /// Successor blocks for jump instructions. + pub bb1: Option, + pub bb2: Option, + /// Instructions that use this value (def-use chain). + pub users: Vec, + /// The basic block that owns this instruction. + pub parent_bb: BbId, + /// Source-bytecode provenance. + pub raw_pos: RawPos, + /// Builtin-constant tag carried on the *instruction* (rarely used directly). + pub builtin: BuiltinConst, +} + +impl Insn { + pub(crate) fn new(kind: InsnKind, parent_bb: BbId) -> Self { + Insn { + kind, + values: SmallVec::new(), + alu_op: AluOp::Unknown, + phi: Vec::new(), + bb1: None, + bb2: None, + users: Vec::new(), + parent_bb, + raw_pos: RawPos::default(), + builtin: BuiltinConst::None, + } + } + + pub fn is_jmp(&self) -> bool { + self.kind.is_jmp() + } + pub fn is_cond_jmp(&self) -> bool { + self.kind.is_cond_jmp() + } + pub fn is_bin_alu(&self) -> bool { + self.kind.is_bin_alu() + } + pub fn is_void(&self) -> bool { + self.kind.is_void() + } + + pub fn is_commutative_alu(&self) -> bool { + matches!(&self.kind, InsnKind::Bin { op } if op.is_commutative()) + } + + /// Iterate the operand values that participate in def-use (includes phi). + pub fn operand_values(&self) -> Vec { + if matches!(self.kind, InsnKind::Phi) { + self.phi.iter().map(|p| p.value).collect() + } else { + let mut out: Vec = self.values.to_vec(); + if let InsnKind::LoadRaw { addr, .. } | InsnKind::StoreRaw { addr, .. } = &self.kind { + out.push(addr.value); + } + out + } + } +} diff --git a/core-rs/epass-ir/src/ir/mod.rs b/core-rs/epass-ir/src/ir/mod.rs new file mode 100644 index 00000000..a58c6de8 --- /dev/null +++ b/core-rs/epass-ir/src/ir/mod.rs @@ -0,0 +1,686 @@ +//! The IR data model: arena-allocated instructions and basic blocks referenced +//! by typed index handles ([`InsnId`], [`BbId`]). This avoids the cyclic raw +//! pointers used by the C implementation while keeping cheap, `Copy` references. +//! +//! Mutation discipline: because a `&mut Function` cannot be aliased, passes that +//! iterate and mutate must first collect the handles they care about, then +//! apply changes. Helper methods on [`Function`] encapsulate the common cases +//! (insertion, def-use maintenance, value replacement). + +pub mod builder; +pub mod insn; +pub mod print; +pub mod text; +pub mod value; + +pub use builder::{InsertPoint, IrBuilder}; +pub use insn::{BinOp, Cond, EndKind, Insn, InsnKind}; +pub use value::{ + AddrValue, AluOp, BuiltinConst, ConstKind, LoadImmExtra, PhiValue, RawPos, RawPosKind, Value, + VrPos, VrType, +}; + +use crate::bytecode::MAX_FUNC_ARG; +use crate::error::Result; +use crate::invalid; + +/// Handle to an instruction in [`Function::insns`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct InsnId(pub u32); + +/// Handle to a basic block in [`Function::bbs`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct BbId(pub u32); + +impl InsnId { + pub fn index(self) -> usize { + self.0 as usize + } +} +impl BbId { + pub fn index(self) -> usize { + self.0 as usize + } +} + +/// Where to insert a new instruction relative to an anchor. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum InsertPos { + /// Immediately before the anchor (or at the start of a block). + Front, + /// Immediately after the anchor (or at the end of a block). + Back, + /// At the end of a block but before a trailing jump. + BackBeforeJmp, + /// At the start of a block but after any leading phi nodes. + FrontAfterPhi, +} + +/// A basic block: an ordered list of instruction handles plus CFG edges. +#[derive(Debug, Clone, Default)] +pub struct BasicBlock { + /// Instruction handles in program order. + pub insns: Vec, + pub preds: Vec, + pub succs: Vec, + pub flag: u32, +} + +impl BasicBlock { + pub fn first(&self) -> Option { + self.insns.first().copied() + } + pub fn last(&self) -> Option { + self.insns.last().copied() + } + pub fn is_empty(&self) -> bool { + self.insns.is_empty() + } + pub fn len(&self) -> usize { + self.insns.len() + } +} + +/// A lifted eBPF function in SSA form. +pub struct Function { + /// Instruction arena. Dead instructions are tombstoned (`None`), never moved. + insns: Vec>, + /// Basic-block arena. + bbs: Vec, + + pub entry: BbId, + /// All blocks, in creation order. + pub all_bbs: Vec, + /// Blocks reachable from `entry`, in fallthrough-chain layout order. + pub reachable_bbs: Vec, + /// Blocks with no successors. + pub end_bbs: Vec, + + /// Stack-pointer pseudo-instruction (R10). + pub sp: InsnId, + /// Function-argument pseudo-instructions (R1..R5). + pub args: [InsnId; MAX_FUNC_ARG], +} + +impl Function { + /// Create an empty function with a single entry block and the SP/arg pseudos. + pub(crate) fn new() -> Self { + let mut f = Function { + insns: Vec::new(), + bbs: Vec::new(), + entry: BbId(0), + all_bbs: Vec::new(), + reachable_bbs: Vec::new(), + end_bbs: Vec::new(), + sp: InsnId(0), + args: [InsnId(0); MAX_FUNC_ARG], + }; + let entry = f.create_bb(); + f.entry = entry; + // SP and arg pseudo-instructions are not attached to any block. + f.sp = f.alloc_detached(InsnKind::Reg { + reg_id: crate::bytecode::BPF_REG_10, + }); + for i in 0..MAX_FUNC_ARG { + f.args[i] = f.alloc_detached(InsnKind::FunctionArg { arg_id: i as u8 }); + } + f + } + + // ---- arena access ---- + + pub fn insn(&self, id: InsnId) -> &Insn { + self.insns[id.index()] + .as_ref() + .expect("dereferenced a tombstoned instruction") + } + + pub fn insn_mut(&mut self, id: InsnId) -> &mut Insn { + self.insns[id.index()] + .as_mut() + .expect("dereferenced a tombstoned instruction") + } + + pub fn try_insn(&self, id: InsnId) -> Option<&Insn> { + self.insns.get(id.index()).and_then(|x| x.as_ref()) + } + + pub fn is_alive(&self, id: InsnId) -> bool { + self.insns + .get(id.index()) + .map(|x| x.is_some()) + .unwrap_or(false) + } + + pub fn bb(&self, id: BbId) -> &BasicBlock { + &self.bbs[id.index()] + } + + pub fn bb_mut(&mut self, id: BbId) -> &mut BasicBlock { + &mut self.bbs[id.index()] + } + + pub fn insn_count(&self) -> usize { + self.insns.iter().filter(|x| x.is_some()).count() + } + + /// Number of slots in the BB arena (including any unreachable blocks). + pub fn bb_arena_len(&self) -> usize { + self.bbs.len() + } + + // ---- block creation / CFG ---- + + fn alloc_detached(&mut self, kind: InsnKind) -> InsnId { + let id = InsnId(self.insns.len() as u32); + // Detached pseudo-insns use the entry block as a nominal parent. + let insn = Insn::new(kind, self.entry); + self.insns.push(Some(insn)); + id + } + + pub fn create_bb(&mut self) -> BbId { + let id = BbId(self.bbs.len() as u32); + self.bbs.push(BasicBlock::default()); + self.all_bbs.push(id); + id + } + + /// Create a detached physical-register pseudo-instruction (used by code-gen). + pub fn create_reg_pseudo(&mut self, reg_id: u8) -> InsnId { + self.alloc_detached(InsnKind::Reg { reg_id }) + } + + pub fn connect(&mut self, from: BbId, to: BbId) { + if !self.bbs[from.index()].succs.contains(&to) { + self.bbs[from.index()].succs.push(to); + } + if !self.bbs[to.index()].preds.contains(&from) { + self.bbs[to.index()].preds.push(from); + } + } + + pub fn disconnect(&mut self, from: BbId, to: BbId) { + self.bbs[from.index()].succs.retain(|&s| s != to); + self.bbs[to.index()].preds.retain(|&p| p != from); + } + + /// The terminator instruction of `bb`, if its last instruction transfers control. + pub fn terminator(&self, bb: BbId) -> Option { + self.bb(bb).last().filter(|&id| self.insn(id).is_jmp()) + } + + pub fn is_terminated(&self, bb: BbId) -> bool { + self.terminator(bb).is_some() + } + + pub fn successor_targets(&self, bb: BbId) -> Vec { + let Some(term) = self.terminator(bb) else { return Vec::new(); }; + let insn = self.insn(term); + match insn.kind { + InsnKind::Ja => insn.bb1.into_iter().collect(), + InsnKind::CondJmp { .. } => [insn.bb1, insn.bb2].into_iter().flatten().collect(), + _ => Vec::new(), + } + } + + pub fn set_ja_target(&mut self, ja: InsnId, target: BbId) -> Result<()> { + if !matches!(self.insn(ja).kind, InsnKind::Ja) { + return Err(invalid!("instruction %{} is not a ja", ja.0)); + } + let bb = self.insn(ja).parent_bb; + if let Some(old) = self.insn(ja).bb1 { + self.disconnect(bb, old); + } + self.insn_mut(ja).bb1 = Some(target); + self.insn_mut(ja).bb2 = None; + self.connect(bb, target); + Ok(()) + } + + pub fn set_cond_targets(&mut self, cond: InsnId, fallthrough: BbId, taken: BbId) -> Result<()> { + if !matches!(self.insn(cond).kind, InsnKind::CondJmp { .. }) { + return Err(invalid!("instruction %{} is not a conditional jump", cond.0)); + } + let bb = self.insn(cond).parent_bb; + for old in [self.insn(cond).bb1, self.insn(cond).bb2].into_iter().flatten() { + self.disconnect(bb, old); + } + { + let insn = self.insn_mut(cond); + insn.bb1 = Some(fallthrough); + insn.bb2 = Some(taken); + } + self.connect(bb, fallthrough); + self.connect(bb, taken); + Ok(()) + } + + /// Redirect a terminator edge `from -> old_to` to `from -> new_to`. + /// + /// This updates the terminator and CFG edge lists. Phi inputs in the old/new + /// successor are not invented; callers that retarget semantic edges should + /// update phis as appropriate. Use [`Function::split_edge`] when inserting a + /// block on an existing edge, as it updates phi predecessor labels safely. + pub fn replace_successor(&mut self, from: BbId, old_to: BbId, new_to: BbId) -> Result<()> { + let term = self.terminator(from).ok_or_else(|| invalid!("bb{} has no terminator", from.0))?; + match self.insn(term).kind { + InsnKind::Ja => { + if self.insn(term).bb1 != Some(old_to) { + return Err(invalid!("bb{} ja does not target bb{}", from.0, old_to.0)); + } + self.set_ja_target(term, new_to)?; + } + InsnKind::CondJmp { .. } => { + let mut b1 = self.insn(term).bb1; + let mut b2 = self.insn(term).bb2; + let mut found = false; + if b1 == Some(old_to) { b1 = Some(new_to); found = true; } + if b2 == Some(old_to) { b2 = Some(new_to); found = true; } + if !found { + return Err(invalid!("bb{} conditional does not target bb{}", from.0, old_to.0)); + } + self.set_cond_targets(term, b1.unwrap(), b2.unwrap())?; + } + _ => unreachable!(), + } + Ok(()) + } + + /// Insert a new block on edge `from -> to`, returning the new block. + /// Phi inputs in `to` that came from `from` are relabeled to the new block. + pub fn split_edge(&mut self, from: BbId, to: BbId) -> Result { + if !self.bb(from).succs.contains(&to) { + return Err(invalid!("edge bb{} -> bb{} does not exist", from.0, to.0)); + } + let new_bb = self.create_bb(); + let ja = self.create_insn(new_bb, InsnKind::Ja, InsertPos::Back); + self.insn_mut(ja).bb1 = Some(to); + self.replace_successor(from, to, new_bb)?; + self.connect(new_bb, to); + + let phis: Vec<_> = self + .bb(to) + .insns + .iter() + .copied() + .take_while(|&id| matches!(self.insn(id).kind, InsnKind::Phi)) + .collect(); + for phi in phis { + for entry in &mut self.insn_mut(phi).phi { + if entry.bb == from { + entry.bb = new_bb; + } + } + } + Ok(new_bb) + } + + /// Split the parent block so that `insn` and following instructions move to a + /// fresh successor block. Returns the fresh block. + pub fn split_block_before(&mut self, insn: InsnId) -> Result { + if matches!(self.insn(insn).kind, InsnKind::Phi) { + return Err(invalid!("cannot split block before phi instruction %{}", insn.0)); + } + let old_bb = self.insn(insn).parent_bb; + let pos = self + .bb(old_bb) + .insns + .iter() + .position(|&id| id == insn) + .ok_or_else(|| invalid!("instruction %{} is not in its parent block", insn.0))?; + + let new_bb = self.create_bb(); + let moved: Vec<_> = self.bbs[old_bb.index()].insns.split_off(pos); + for id in &moved { + self.insn_mut(*id).parent_bb = new_bb; + } + self.bbs[new_bb.index()].insns = moved; + + let old_succs = self.bbs[old_bb.index()].succs.clone(); + for succ in old_succs.clone() { + self.disconnect(old_bb, succ); + self.connect(new_bb, succ); + // Edges to old successors now come from the new block; relabel phis. + let phis: Vec<_> = self + .bb(succ) + .insns + .iter() + .copied() + .take_while(|&id| matches!(self.insn(id).kind, InsnKind::Phi)) + .collect(); + for phi in phis { + for entry in &mut self.insn_mut(phi).phi { + if entry.bb == old_bb { + entry.bb = new_bb; + } + } + } + } + + let ja = self.create_insn(old_bb, InsnKind::Ja, InsertPos::Back); + self.insn_mut(ja).bb1 = Some(new_bb); + self.connect(old_bb, new_bb); + Ok(new_bb) + } + + /// Split the parent block after `insn`, moving following instructions to a + /// fresh successor block. + pub fn split_block_after(&mut self, insn: InsnId) -> Result { + if self.insn(insn).is_jmp() { + return Err(invalid!("cannot split block after terminator %{}", insn.0)); + } + let next = self.next_insn(insn).ok_or_else(|| invalid!("instruction %{} has no following instruction to split", insn.0))?; + self.split_block_before(next) + } + + pub fn create_ret_block(&mut self, value: Value) -> BbId { + let bb = self.create_bb(); + let ret = self.create_insn(bb, InsnKind::Ret, InsertPos::Back); + self.add_value_operand(ret, value); + bb + } + + pub fn create_throw_block(&mut self) -> BbId { + let bb = self.create_bb(); + self.create_insn(bb, InsnKind::Throw, InsertPos::Back); + bb + } + + // ---- instruction creation / placement ---- + + /// Resolve an insertion position into an index in `bb.insns`. + fn resolve_index(&self, bb: BbId, anchor: Option, pos: InsertPos) -> usize { + let block = self.bb(bb); + match pos { + InsertPos::Front => match anchor { + Some(a) => block.insns.iter().position(|&i| i == a).unwrap_or(0), + None => 0, + }, + InsertPos::Back => match anchor { + Some(a) => { + block.insns.iter().position(|&i| i == a).map_or(block.insns.len(), |p| p + 1) + } + None => block.insns.len(), + }, + InsertPos::BackBeforeJmp => { + if let Some(last) = block.last() { + if self.insn(last).is_jmp() { + return block.insns.len() - 1; + } + } + block.insns.len() + } + InsertPos::FrontAfterPhi => { + let mut idx = 0; + for &i in &block.insns { + if matches!(self.insn(i).kind, InsnKind::Phi) { + idx += 1; + } else { + break; + } + } + idx + } + } + } + + /// Create a new instruction and insert it into `bb` at `pos`. Returns its id. + /// + /// The caller is responsible for adding operands via [`Function::add_use`] + /// (the typed `build_*` helpers do this). + pub fn create_insn(&mut self, bb: BbId, kind: InsnKind, pos: InsertPos) -> InsnId { + let id = InsnId(self.insns.len() as u32); + self.insns.push(Some(Insn::new(kind, bb))); + let idx = self.resolve_index(bb, None, pos); + self.bbs[bb.index()].insns.insert(idx, id); + id + } + + /// Create a new instruction positioned relative to `anchor`. + pub fn create_insn_at(&mut self, anchor: InsnId, kind: InsnKind, pos: InsertPos) -> InsnId { + let bb = self.insn(anchor).parent_bb; + let id = InsnId(self.insns.len() as u32); + self.insns.push(Some(Insn::new(kind, bb))); + let idx = self.resolve_index(bb, Some(anchor), pos); + self.bbs[bb.index()].insns.insert(idx, id); + id + } + + // ---- def-use maintenance ---- + + /// Record that `user` uses `val` (adds to the def's user list if it's an insn). + pub fn add_use(&mut self, val: Value, user: InsnId) { + if let Value::Insn(def) = val { + let users = &mut self.insn_mut(def).users; + if !users.contains(&user) { + users.push(user); + } + } + } + + /// Remove `user` from the user list of `val`'s definition. + pub fn remove_use(&mut self, val: Value, user: InsnId) { + if let Value::Insn(def) = val { + if self.is_alive(def) { + self.insn_mut(def).users.retain(|&u| u != user); + } + } + } + + /// Replace every use of `old`'s value across the function with `rep`. + pub fn replace_all_uses(&mut self, old: InsnId, rep: Value) { + self.replace_all_uses_except(old, rep, None); + } + + /// Like [`Function::replace_all_uses`] but skips `except`. + pub fn replace_all_uses_except(&mut self, old: InsnId, rep: Value, except: Option) { + let users: Vec = self.insn(old).users.clone(); + let old_val = Value::Insn(old); + for user in users { + if Some(user) == except || !self.is_alive(user) { + continue; + } + self.replace_value_in(user, old_val, rep); + } + } + + /// Replace occurrences of `from` with `to` in a single instruction's operands, + /// updating def-use bookkeeping. + pub fn replace_value_in(&mut self, user: InsnId, from: Value, to: Value) { + let mut changed = false; + { + let insn = self.insn_mut(user); + match &mut insn.kind { + InsnKind::Phi => { + for p in &mut insn.phi { + if p.value == from { + p.value = to; + changed = true; + } + } + } + InsnKind::LoadRaw { addr, .. } | InsnKind::StoreRaw { addr, .. } => { + if addr.value == from { + addr.value = to; + changed = true; + } + for v in insn.values.iter_mut() { + if *v == from { + *v = to; + changed = true; + } + } + } + _ => { + for v in insn.values.iter_mut() { + if *v == from { + *v = to; + changed = true; + } + } + } + } + } + if changed { + self.remove_use(from, user); + self.add_use(to, user); + } + } + + /// Change `*slot_value` of `user` from its current value to `new`, where the + /// slot is identified by equality with `old`. + pub fn change_value(&mut self, user: InsnId, old: Value, new: Value) { + self.replace_value_in(user, old, new); + } + + /// Add a normal operand value to `user`, updating def-use chains. + pub fn add_value_operand(&mut self, user: InsnId, value: Value) { + self.insn_mut(user).values.push(value); + self.add_use(value, user); + } + + /// Add a phi operand `(value, predecessor block)`, updating def-use chains. + pub fn add_phi_operand(&mut self, phi: InsnId, value: Value, bb: BbId) { + self.insn_mut(phi).phi.push(PhiValue { value, bb }); + self.add_use(value, phi); + } + + /// Remove all non-phi operand values from `user`, updating def-use chains. + /// + /// Phi operands live in [`Insn::phi`] and are intentionally not touched by + /// this helper. + pub fn clear_values(&mut self, user: InsnId) { + let values: Vec = self.insn(user).values.to_vec(); + for v in values { + self.remove_use(v, user); + } + self.insn_mut(user).values.clear(); + } + + /// Rewrite a conditional terminator into an unconditional jump to `target`. + pub fn rewrite_cond_to_ja(&mut self, id: InsnId, target: BbId) { + self.clear_values(id); + let insn = self.insn_mut(id); + insn.kind = InsnKind::Ja; + insn.bb1 = Some(target); + insn.bb2 = None; + insn.alu_op = AluOp::Unknown; + } + + // ---- erasure ---- + + /// Tombstone an instruction, detaching it from its block and dropping its + /// outgoing uses. Panics if it still has live users. + pub fn erase_insn(&mut self, id: InsnId) { + let users: Vec = self + .insn(id) + .users + .iter() + .copied() + .filter(|&u| u != id) + .collect(); + assert!( + users.is_empty(), + "cannot erase instruction {id:?} that still has users {users:?}" + ); + // Drop our uses of operands. + for v in self.insn(id).operand_values() { + self.remove_use(v, id); + } + let bb = self.insn(id).parent_bb; + self.bbs[bb.index()].insns.retain(|&i| i != id); + self.insns[id.index()] = None; + } + + // ---- queries ---- + + pub fn prev_insn(&self, id: InsnId) -> Option { + let bb = self.insn(id).parent_bb; + let block = self.bb(bb); + let pos = block.insns.iter().position(|&i| i == id)?; + if pos == 0 { + None + } else { + Some(block.insns[pos - 1]) + } + } + + pub fn next_insn(&self, id: InsnId) -> Option { + let bb = self.insn(id).parent_bb; + let block = self.bb(bb); + let pos = block.insns.iter().position(|&i| i == id)?; + block.insns.get(pos + 1).copied() + } + + /// Find the IR instruction lifted from a given raw bytecode position. + pub fn find_by_raw_pos(&self, raw_pos: usize) -> Option { + for &bb in &self.reachable_bbs { + for &id in &self.bb(bb).insns { + let rp = self.insn(id).raw_pos; + if rp.valid && rp.pos == raw_pos { + return Some(id); + } + } + } + None + } + + // ---- typed instruction builders (maintain def-use) ---- + + /// `%x = alloc ` placed in `bb` at `pos`. + pub fn build_alloc_bb(&mut self, bb: BbId, ty: VrType, pos: InsertPos) -> InsnId { + self.create_insn(bb, InsnKind::Alloc { vr_type: ty }, pos) + } + + /// `%x = assign ` placed before/after `anchor`. + pub fn build_assign_at(&mut self, anchor: InsnId, val: Value, pos: InsertPos) -> InsnId { + let id = self.create_insn_at(anchor, InsnKind::Assign, pos); + self.insn_mut(id).values.push(val); + self.add_use(val, id); + id + } + + /// `%x = assign ` placed in `bb`. + pub fn build_assign_bb(&mut self, bb: BbId, val: Value, pos: InsertPos) -> InsnId { + let id = self.create_insn(bb, InsnKind::Assign, pos); + self.insn_mut(id).values.push(val); + self.add_use(val, id); + id + } + + /// `store , ` placed before/after `anchor`. + pub fn build_store_at( + &mut self, + anchor: InsnId, + alloc: InsnId, + val: Value, + pos: InsertPos, + ) -> InsnId { + let id = self.create_insn_at(anchor, InsnKind::Store, pos); + let aval = Value::Insn(alloc); + self.insn_mut(id).values.push(aval); + self.insn_mut(id).values.push(val); + self.add_use(aval, id); + self.add_use(val, id); + id + } + + /// `%x = load ` placed before/after `anchor`. + pub fn build_load_at(&mut self, anchor: InsnId, alloc: InsnId, pos: InsertPos) -> InsnId { + let id = self.create_insn_at(anchor, InsnKind::Load, pos); + let aval = Value::Insn(alloc); + self.insn_mut(id).values.push(aval); + self.add_use(aval, id); + id + } + + /// `%x = load ` placed in `bb`. + pub fn build_load_bb(&mut self, bb: BbId, alloc: InsnId, pos: InsertPos) -> InsnId { + let id = self.create_insn(bb, InsnKind::Load, pos); + let aval = Value::Insn(alloc); + self.insn_mut(id).values.push(aval); + self.add_use(aval, id); + id + } +} diff --git a/core-rs/epass-ir/src/ir/print.rs b/core-rs/epass-ir/src/ir/print.rs new file mode 100644 index 00000000..3852a5a7 --- /dev/null +++ b/core-rs/epass-ir/src/ir/print.rs @@ -0,0 +1,278 @@ +//! Human-friendly IR pretty-printer for debugging. +//! +//! Produces an LLVM-like textual form with stable virtual-register numbering, +//! labeled basic blocks, and explicit predecessor/successor annotations. This +//! is intentionally richer than the C printer to make debugging passes easier. + +use std::collections::HashMap; +use std::fmt::Write; + +use super::insn::{BinOp, Cond, EndKind, InsnKind}; +use super::value::{AddrValue, AluOp, BuiltinConst, ConstKind, LoadImmExtra, Value, VrPos, VrType}; +use super::{BbId, Function, InsnId}; + +/// Assigns stable, dense display ids to instructions and blocks. +struct Tags { + insn_ids: HashMap, + bb_ids: HashMap, +} + +impl Tags { + fn build(func: &Function) -> Self { + let mut insn_ids = HashMap::new(); + let mut bb_ids = HashMap::new(); + let mut vr = 0usize; + let order: &[BbId] = if func.reachable_bbs.is_empty() { + &func.all_bbs + } else { + &func.reachable_bbs + }; + for (bidx, &bb) in order.iter().enumerate() { + bb_ids.insert(bb, bidx); + for &id in &func.bb(bb).insns { + if !func.insn(id).is_void() { + insn_ids.insert(id, vr); + vr += 1; + } + } + } + Tags { insn_ids, bb_ids } + } + + fn vr(&self, id: InsnId) -> String { + match self.insn_ids.get(&id) { + Some(n) => format!("%{n}"), + None => format!("%?{}", id.0), + } + } + + fn bb(&self, id: BbId) -> String { + match self.bb_ids.get(&id) { + Some(n) => format!("bb{n}"), + None => format!("bb?{}", id.0), + } + } +} + +fn alu_suffix(op: AluOp) -> &'static str { + match op { + AluOp::Alu32 => "32", + AluOp::Alu64 => "64", + AluOp::Unknown => "?", + } +} + +fn binop_name(op: BinOp) -> &'static str { + match op { + BinOp::Add => "add", + BinOp::Sub => "sub", + BinOp::Mul => "mul", + BinOp::Div => "div", + BinOp::Or => "or", + BinOp::And => "and", + BinOp::Lsh => "lsh", + BinOp::Arsh => "arsh", + BinOp::Rsh => "rsh", + BinOp::Mod => "mod", + BinOp::Xor => "xor", + } +} + +fn cond_name(c: Cond) -> &'static str { + match c { + Cond::Eq => "jeq", + Cond::Ne => "jne", + Cond::Gt => "jgt", + Cond::Ge => "jge", + Cond::Lt => "jlt", + Cond::Le => "jle", + Cond::Sgt => "jsgt", + Cond::Sge => "jsge", + Cond::Slt => "jslt", + Cond::Sle => "jsle", + } +} + +fn vr_type_name(t: VrType) -> &'static str { + match t { + VrType::B8 => "u8", + VrType::B16 => "u16", + VrType::B32 => "u32", + VrType::B64 => "u64", + VrType::Unknown => "u?", + } +} + +fn fmt_vrpos(pos: VrPos) -> String { + if !pos.allocated { + "".to_string() + } else if pos.spilled != 0 { + format!("sp{:+}", pos.spilled) + } else { + format!("R{}", pos.alloc_reg) + } +} + +fn fmt_value(func: &Function, tags: &Tags, v: Value) -> String { + match v { + Value::Const { + v, + ty, + kind, + builtin, + } => { + if builtin != BuiltinConst::None { + return match builtin { + BuiltinConst::BbInsnCnt => "".to_string(), + BuiltinConst::BbInsnCriticalCnt => "".to_string(), + BuiltinConst::None => unreachable!(), + }; + } + let suffix = match kind { + ConstKind::Plain => "", + ConstKind::RawOff => "+sp", + ConstKind::RawOffRev => "-sp", + }; + let _ = ty; + format!("{v}{suffix}") + } + Value::Insn(id) => { + // Pseudo-instructions render as their register/arg name. + match func.try_insn(id).map(|i| &i.kind) { + Some(InsnKind::Reg { reg_id }) => format!("R{reg_id}"), + Some(InsnKind::FunctionArg { arg_id }) => format!("arg{arg_id}"), + _ => tags.vr(id), + } + } + Value::VrPos(p) | Value::FlattenDst(p) => fmt_vrpos(p), + Value::Undef => "undef".to_string(), + } +} + +fn fmt_addr(func: &Function, tags: &Tags, a: &AddrValue) -> String { + let base = fmt_value(func, tags, a.value); + let off_sp = match a.offset_kind { + ConstKind::Plain => "", + ConstKind::RawOff => "+sp", + ConstKind::RawOffRev => "-sp", + }; + format!("[{base}{:+}{off_sp}]", a.offset) +} + +fn extra_name(e: LoadImmExtra) -> &'static str { + match e { + LoadImmExtra::Imm64 => "imm64", + LoadImmExtra::MapByFd => "map_by_fd", + LoadImmExtra::MapValFd => "map_val_fd", + LoadImmExtra::VarAddr => "var_addr", + LoadImmExtra::CodeAddr => "code_addr", + LoadImmExtra::MapByIdx => "map_by_idx", + LoadImmExtra::MapValIdx => "map_val_idx", + } +} + +fn fmt_insn(func: &Function, tags: &Tags, id: InsnId) -> String { + let insn = func.insn(id); + let dst = if insn.is_void() { + String::new() + } else { + format!("{} = ", tags.vr(id)) + }; + let vals: Vec = insn + .values + .iter() + .map(|&v| fmt_value(func, tags, v)) + .collect(); + + let body = match &insn.kind { + InsnKind::Alloc { vr_type } => format!("alloc {}", vr_type_name(*vr_type)), + InsnKind::AllocArray { vr_type, num } => { + format!("allocarray {} x {}", vr_type_name(*vr_type), num) + } + InsnKind::GetElemPtr => format!("getelemptr {}, {}", vals[0], vals[1]), + InsnKind::Store => format!("store {}, {}", vals[0], vals[1]), + InsnKind::Load => format!("load {}", vals[0]), + InsnKind::LoadImmExtra { extra, imm64 } => { + format!("loadimm.{} {}", extra_name(*extra), imm64) + } + InsnKind::StoreRaw { vr_type, addr } => { + format!( + "storeraw.{} {}, {}", + vr_type_name(*vr_type), + fmt_addr(func, tags, addr), + vals.first().cloned().unwrap_or_default() + ) + } + InsnKind::LoadRaw { vr_type, addr } => { + format!("loadraw.{} {}", vr_type_name(*vr_type), fmt_addr(func, tags, addr)) + } + InsnKind::Neg => format!("neg{} {}", alu_suffix(insn.alu_op), vals[0]), + InsnKind::End { kind, swap_width } => { + let dir = match kind { + EndKind::ToLe => "le", + EndKind::ToBe => "be", + }; + format!("end.{dir}{swap_width} {}", vals[0]) + } + InsnKind::Bin { op } => { + format!("{}{} {}, {}", binop_name(*op), alu_suffix(insn.alu_op), vals[0], vals[1]) + } + InsnKind::Call { fid } => { + format!("call #{} ({})", fid, vals.join(", ")) + } + InsnKind::Ret => format!("ret {}", vals.first().cloned().unwrap_or_default()), + InsnKind::Throw => "throw".to_string(), + InsnKind::Ja => format!("ja {}", tags.bb(insn.bb1.unwrap())), + InsnKind::CondJmp { cond } => format!( + "{}{} {}, {} -> {} else {}", + cond_name(*cond), + alu_suffix(insn.alu_op), + vals[0], + vals[1], + tags.bb(insn.bb2.unwrap()), + tags.bb(insn.bb1.unwrap()), + ), + InsnKind::Phi => { + let entries: Vec = insn + .phi + .iter() + .map(|p| format!("[{}, {}]", fmt_value(func, tags, p.value), tags.bb(p.bb))) + .collect(); + format!("phi {}", entries.join(", ")) + } + InsnKind::Assign => format!("assign {}", vals[0]), + InsnKind::Reg { reg_id } => format!("reg R{reg_id}"), + InsnKind::FunctionArg { arg_id } => format!("funcarg {arg_id}"), + InsnKind::Ecall => format!("ecall ({})", vals.join(", ")), + }; + + format!("{dst}{body}") +} + +/// Render a whole function to a debug string. +pub fn print_function(func: &Function) -> String { + let tags = Tags::build(func); + let mut out = String::new(); + let order: &[BbId] = if func.reachable_bbs.is_empty() { + &func.all_bbs + } else { + &func.reachable_bbs + }; + for &bb in order { + let block = func.bb(bb); + let preds: Vec = block.preds.iter().map(|&p| tags.bb(p)).collect(); + let succs: Vec = block.succs.iter().map(|&s| tags.bb(s)).collect(); + let _ = writeln!( + out, + "{}: ; preds = [{}] succs = [{}]", + tags.bb(bb), + preds.join(", "), + succs.join(", ") + ); + for &id in &block.insns { + let _ = writeln!(out, " {}", fmt_insn(func, &tags, id)); + } + out.push('\n'); + } + out +} diff --git a/core-rs/epass-ir/src/ir/text.rs b/core-rs/epass-ir/src/ir/text.rs new file mode 100644 index 00000000..042e7f1a --- /dev/null +++ b/core-rs/epass-ir/src/ir/text.rs @@ -0,0 +1,436 @@ +//! Parseable text serialization for ePass IR. +//! +//! This is a debugging format, not a stable external ABI. It intentionally +//! serializes only semantic IR state; arena ids, users, reachable lists, and CG +//! metadata are reconstructed after parsing. + +use std::collections::HashMap; +use std::fs; +use std::path::Path; + +use crate::bytecode::MAX_FUNC_ARG; +use crate::error::{Error, Result}; +use crate::ir::insn::{BinOp, Cond, EndKind, InsnKind}; +use crate::ir::value::{AddrValue, AluOp, ConstKind, LoadImmExtra, Value, VrType}; +use crate::ir::{BbId, Function, InsnId, InsertPos}; +use crate::{cfg, invalid, unsupported}; + +#[derive(Debug, Clone, Copy)] +pub struct DumpOptions { + pub include_preds_succs: bool, +} + +impl Default for DumpOptions { + fn default() -> Self { + Self { include_preds_succs: true } + } +} + +pub fn dump_function(func: &Function) -> String { + dump_function_with_options(func, DumpOptions::default()) +} + +pub fn dump_function_with_options(func: &Function, opts: DumpOptions) -> String { + let mut names = Names::new(func); + let mut out = String::new(); + out.push_str("; ePass IR v1\n\n"); + let order: &[BbId] = if func.reachable_bbs.is_empty() { &func.all_bbs } else { &func.reachable_bbs }; + for &bb in order { + out.push_str(&format!("{}:", names.bb(bb))); + if opts.include_preds_succs { + let preds: Vec<_> = func.bb(bb).preds.iter().map(|&p| names.bb(p)).collect(); + let succs: Vec<_> = func.bb(bb).succs.iter().map(|&s| names.bb(s)).collect(); + out.push_str(&format!(" ; preds=[{}] succs=[{}]", preds.join(","), succs.join(","))); + } + out.push('\n'); + for &id in &func.bb(bb).insns { + out.push_str(" "); + out.push_str(&dump_insn(func, &mut names, id)); + out.push('\n'); + } + out.push('\n'); + } + out +} + +pub fn dump_function_to_file>(func: &Function, path: P) -> Result<()> { + fs::write(path.as_ref(), dump_function(func)) + .map_err(|e| Error::Internal(format!("failed to write IR dump '{}': {e}", path.as_ref().display()))) +} + +struct Names { + insns: HashMap, + bbs: HashMap, + next_insn: usize, +} + +impl Names { + fn new(func: &Function) -> Self { + let mut bbs = HashMap::new(); + let order: &[BbId] = if func.reachable_bbs.is_empty() { &func.all_bbs } else { &func.reachable_bbs }; + for (i, &bb) in order.iter().enumerate() { + bbs.insert(bb, format!("bb{i}")); + } + Self { insns: HashMap::new(), bbs, next_insn: 0 } + } + + fn bb(&self, bb: BbId) -> String { + self.bbs.get(&bb).cloned().unwrap_or_else(|| format!("bb?{}", bb.0)) + } + + fn insn(&mut self, func: &Function, id: InsnId) -> String { + if id == func.sp { return "%sp".to_string(); } + for i in 0..MAX_FUNC_ARG { + if id == func.args[i] { return format!("%arg{i}"); } + } + if let Some(n) = self.insns.get(&id) { return n.clone(); } + let name = format!("%{}", self.next_insn); + self.next_insn += 1; + self.insns.insert(id, name.clone()); + name + } +} + +fn dump_insn(func: &Function, names: &mut Names, id: InsnId) -> String { + let insn = func.insn(id); + let dst = if insn.is_void() { String::new() } else { format!("{} = ", names.insn(func, id)) }; + let val = |names: &mut Names, v: Value| dump_value(func, names, v); + match &insn.kind { + InsnKind::Alloc { vr_type } => format!("{dst}alloc {}", vr_name(*vr_type)), + InsnKind::AllocArray { vr_type, num } => format!("{dst}allocarray {} x {}", vr_name(*vr_type), num), + InsnKind::GetElemPtr => format!("{dst}getelemptr {}, {}", val(names, insn.values[0]), val(names, insn.values[1])), + InsnKind::Store => format!("store {}, {}", val(names, insn.values[0]), val(names, insn.values[1])), + InsnKind::Load => format!("{dst}load {}", val(names, insn.values[0])), + InsnKind::LoadImmExtra { extra, imm64 } => format!("{dst}loadimm.{} {}", extra_name(*extra), imm64), + InsnKind::StoreRaw { vr_type, addr } => format!("storeraw.{} {}, {}", vr_name(*vr_type), dump_addr(func, names, addr), val(names, insn.values[0])), + InsnKind::LoadRaw { vr_type, addr } => format!("{dst}loadraw.{} {}", vr_name(*vr_type), dump_addr(func, names, addr)), + InsnKind::Neg => format!("{dst}neg{} {}", alu_suffix(insn.alu_op), val(names, insn.values[0])), + InsnKind::End { kind, swap_width } => format!("{dst}end.{}{} {}", match kind { EndKind::ToBe => "be", EndKind::ToLe => "le" }, swap_width, val(names, insn.values[0])), + InsnKind::Bin { op } => format!("{dst}{}{} {}, {}", bin_name(*op), alu_suffix(insn.alu_op), val(names, insn.values[0]), val(names, insn.values[1])), + InsnKind::Call { fid } => { + let args: Vec<_> = insn.values.iter().map(|&v| val(names, v)).collect(); + format!("{dst}call #{}({})", fid, args.join(", ")) + } + InsnKind::Ret => format!("ret {}", val(names, insn.values[0])), + InsnKind::Throw => "throw".to_string(), + InsnKind::Ja => format!("ja {}", names.bb(insn.bb1.unwrap())), + InsnKind::CondJmp { cond } => format!("{}{} {}, {} -> {} else {}", cond_name(*cond), alu_suffix(insn.alu_op), val(names, insn.values[0]), val(names, insn.values[1]), names.bb(insn.bb2.unwrap()), names.bb(insn.bb1.unwrap())), + InsnKind::Phi => { + let entries: Vec<_> = insn.phi.iter().map(|p| format!("[{}, {}]", val(names, p.value), names.bb(p.bb))).collect(); + format!("{dst}phi {}", entries.join(", ")) + } + InsnKind::Assign => format!("{dst}assign {}", val(names, insn.values[0])), + InsnKind::Reg { reg_id } => format!("{dst}reg R{reg_id}"), + InsnKind::FunctionArg { arg_id } => format!("{dst}funcarg {arg_id}"), + InsnKind::Ecall => { + let args: Vec<_> = insn.values.iter().map(|&v| val(names, v)).collect(); + format!("{dst}ecall({})", args.join(", ")) + } + } +} + +fn dump_value(func: &Function, names: &mut Names, v: Value) -> String { + match v { + Value::Insn(id) => names.insn(func, id), + Value::Const { v, ty, kind: ConstKind::Plain, builtin } if builtin == crate::ir::BuiltinConst::None => { + format!("{}:{}", v, match ty { AluOp::Alu32 => "i32", AluOp::Alu64 => "i64", AluOp::Unknown => "i?" }) + } + Value::Const { v, ty, kind, builtin } => format!("const({},{:?},{:?},{:?})", v, ty, kind, builtin), + Value::Undef => "undef".to_string(), + Value::VrPos(_) | Value::FlattenDst(_) => "".to_string(), + } +} + +fn dump_addr(func: &Function, names: &mut Names, addr: &AddrValue) -> String { + let suffix = match addr.offset_kind { ConstKind::Plain => "", ConstKind::RawOff => "+sp", ConstKind::RawOffRev => "-sp" }; + format!("[{}{:+}{}]", dump_value(func, names, addr.value), addr.offset, suffix) +} + +pub fn load_function_from_file>(path: P) -> Result { + let text = fs::read_to_string(path.as_ref()) + .map_err(|e| Error::InvalidProgram(format!("failed to read IR file '{}': {e}", path.as_ref().display())))?; + load_function_from_str(&text) +} + +pub fn load_function_from_str(text: &str) -> Result { + Parser::new(text).parse() +} + +struct Parser<'a> { + lines: Vec<(usize, &'a str)>, + bb_names: HashMap, + val_names: HashMap, + pending_vals: Vec<(InsnId, Vec)>, + pending_addr: Vec<(InsnId, PValue)>, + pending_phi: Vec<(InsnId, Vec<(PValue, String)>)>, + pending_jumps: Vec<(InsnId, Option, Option)>, +} + +#[derive(Clone, Debug)] +enum PValue { Const(Value), Name(String), Sp, Arg(usize), Undef } + +impl<'a> Parser<'a> { + fn new(text: &'a str) -> Self { + let lines = text.lines().enumerate().filter_map(|(i, raw)| { + let line = raw.split(';').next().unwrap_or("").trim(); + if line.is_empty() { None } else { Some((i + 1, line)) } + }).collect(); + Self { lines, bb_names: HashMap::new(), val_names: HashMap::new(), pending_vals: Vec::new(), pending_addr: Vec::new(), pending_phi: Vec::new(), pending_jumps: Vec::new() } + } + + fn parse(mut self) -> Result { + let mut func = Function::new(); + let mut first = true; + for &(line_no, line) in &self.lines { + if let Some(name) = line.strip_suffix(':') { + if self.bb_names.contains_key(name) { return Err(invalid!("IR parse line {line_no}: duplicate block {name}")); } + let bb = if first { first = false; func.entry } else { func.create_bb() }; + self.bb_names.insert(name.to_string(), bb); + } + } + if self.bb_names.is_empty() { return Err(invalid!("IR parse error: no blocks")); } + + let mut cur_bb: Option = None; + let lines = self.lines.clone(); + for (line_no, line) in lines { + if let Some(name) = line.strip_suffix(':') { + cur_bb = Some(*self.bb_names.get(name).unwrap()); + continue; + } + let bb = cur_bb.ok_or_else(|| invalid!("IR parse line {line_no}: instruction before first block"))?; + self.parse_insn(&mut func, bb, line_no, line)?; + } + + self.resolve(&mut func)?; + cfg::finalize(&mut crate::Env::new(Default::default(), Vec::new()), &mut func)?; + Ok(func) + } + + fn parse_insn(&mut self, func: &mut Function, bb: BbId, line_no: usize, line: &str) -> Result<()> { + let (dst, body) = if let Some((d, b)) = line.split_once(" = ") { (Some(d.trim()), b.trim()) } else { (None, line.trim()) }; + let mut words = body.split_whitespace(); + let op = words.next().ok_or_else(|| invalid!("IR parse line {line_no}: empty instruction"))?; + + let mk = |func: &mut Function, kind| func.create_insn(bb, kind, InsertPos::Back); + let id = match op { + "alloc" => mk(func, InsnKind::Alloc { vr_type: parse_vr(words.next(), line_no)? }), + "allocarray" => { + let ty = parse_vr(words.next(), line_no)?; + if words.next() != Some("x") { return Err(invalid!("IR parse line {line_no}: expected 'x' in allocarray")); } + let n = words.next().ok_or_else(|| invalid!("IR parse line {line_no}: missing allocarray size"))?.parse().map_err(|_| invalid!("IR parse line {line_no}: bad allocarray size"))?; + mk(func, InsnKind::AllocArray { vr_type: ty, num: n }) + } + "store" => { let id = mk(func, InsnKind::Store); self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); id } + "load" => { let id = mk(func, InsnKind::Load); self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); id } + x if x.starts_with("loadimm.") => { + let extra = parse_extra(&x[8..], line_no)?; + let imm64: i64 = words.next().ok_or_else(|| invalid!("IR parse line {line_no}: missing imm64"))?.parse().map_err(|_| invalid!("IR parse line {line_no}: bad imm64"))?; + mk(func, InsnKind::LoadImmExtra { extra, imm64 }) + } + x if x.starts_with("loadraw.") => { + let ty = parse_vr_name(&x[8..], line_no)?; + let addr_s = rest_after(op, body).trim(); + let (addr, base) = parse_addr(addr_s, line_no)?; + let id = mk(func, InsnKind::LoadRaw { vr_type: ty, addr }); + self.pending_addr.push((id, base)); + id + } + x if x.starts_with("storeraw.") => { + let ty = parse_vr_name(&x[9..], line_no)?; + let rest = rest_after(op, body); + let (a, v) = split_top_comma(rest, line_no)?; + let (addr, base) = parse_addr(a.trim(), line_no)?; + let id = mk(func, InsnKind::StoreRaw { vr_type: ty, addr }); + self.pending_addr.push((id, base)); + self.pending_vals.push((id, vec![parse_pvalue(v.trim(), line_no)?])); + id + } + x if parse_bin_op(x).is_some() => { + let (op2, alu) = parse_bin_op(x).unwrap(); + let id = mk(func, InsnKind::Bin { op: op2 }); + func.insn_mut(id).alu_op = alu; + self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); + id + } + x if x.starts_with("neg") => { let id = mk(func, InsnKind::Neg); func.insn_mut(id).alu_op = parse_alu_suffix(&x[3..], line_no)?; self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); id } + x if x.starts_with("end.") => { + let spec = &x[4..]; + let (kind, width_s) = if let Some(w) = spec.strip_prefix("be") { + (EndKind::ToBe, w) + } else if let Some(w) = spec.strip_prefix("le") { + (EndKind::ToLe, w) + } else { + return Err(invalid!("IR parse line {line_no}: bad end op {x}")); + }; + let width = width_s.parse().map_err(|_| invalid!("IR parse line {line_no}: bad end width"))?; + let id = mk(func, InsnKind::End { kind, swap_width: width }); + func.insn_mut(id).alu_op = AluOp::Alu32; + self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); + id + } + "getelemptr" => { + let id = mk(func, InsnKind::GetElemPtr); + self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); + id + } + "call" => { + let rest = rest_after(op, body).trim(); + let hash = rest.strip_prefix('#').ok_or_else(|| invalid!("IR parse line {line_no}: call missing #fid"))?; + let (fid_s, args_s) = hash.split_once('(').ok_or_else(|| invalid!("IR parse line {line_no}: call missing args"))?; + let fid = fid_s.parse().map_err(|_| invalid!("IR parse line {line_no}: bad fid"))?; + let args_s = args_s.strip_suffix(')').ok_or_else(|| invalid!("IR parse line {line_no}: call missing ')'"))?; + let id = mk(func, InsnKind::Call { fid }); + self.pending_vals.push((id, parse_value_list(args_s)?)); + id + } + "ret" => { let id = mk(func, InsnKind::Ret); self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); id } + "throw" => mk(func, InsnKind::Throw), + "ja" => { let id = mk(func, InsnKind::Ja); self.pending_jumps.push((id, Some(words.next().ok_or_else(|| invalid!("IR parse line {line_no}: ja missing target"))?.to_string()), None)); id } + "phi" => { + let id = mk(func, InsnKind::Phi); + self.pending_phi.push((id, parse_phi_entries(rest_after(op, body), line_no)?)); + id + } + "assign" => { let id = mk(func, InsnKind::Assign); self.pending_vals.push((id, parse_value_list(rest_after(op, body))?)); id } + x if x.starts_with("ecall(") => { + let args_s = x.strip_prefix("ecall(").and_then(|s| s.strip_suffix(')')) + .ok_or_else(|| invalid!("IR parse line {line_no}: ecall missing args"))?; + let id = mk(func, InsnKind::Ecall); + self.pending_vals.push((id, parse_value_list(args_s)?)); + id + } + "reg" => { + let r = words.next().ok_or_else(|| invalid!("IR parse line {line_no}: reg missing id"))?; + let reg_id = r.strip_prefix('R').ok_or_else(|| invalid!("IR parse line {line_no}: bad reg id"))? + .parse().map_err(|_| invalid!("IR parse line {line_no}: bad reg id"))?; + mk(func, InsnKind::Reg { reg_id }) + } + "funcarg" => { + let arg_id = words.next().ok_or_else(|| invalid!("IR parse line {line_no}: funcarg missing id"))? + .parse().map_err(|_| invalid!("IR parse line {line_no}: bad funcarg id"))?; + mk(func, InsnKind::FunctionArg { arg_id }) + } + x if parse_cond_op(x).is_some() => { + let (cond, alu) = parse_cond_op(x).unwrap(); + let rest = rest_after(op, body); + let (lhs_rhs, targets) = rest.split_once(" -> ").ok_or_else(|| invalid!("IR parse line {line_no}: conditional missing ->"))?; + let (taken, fall) = targets.split_once(" else ").ok_or_else(|| invalid!("IR parse line {line_no}: conditional missing else"))?; + let id = mk(func, InsnKind::CondJmp { cond }); + func.insn_mut(id).alu_op = alu; + self.pending_vals.push((id, parse_value_list(lhs_rhs)?)); + self.pending_jumps.push((id, Some(fall.trim().to_string()), Some(taken.trim().to_string()))); + id + } + other => return Err(unsupported!("IR parser does not support opcode '{other}' at line {line_no}")), + }; + if let Some(dst) = dst { self.val_names.insert(dst.to_string(), id); } + Ok(()) + } + + fn resolve(self, func: &mut Function) -> Result<()> { + let bb_names = self.bb_names; + let val_names = self.val_names; + for (id, vals) in self.pending_vals { + for pv in vals { + let v = resolve_pvalue(func, &val_names, pv)?; + func.add_value_operand(id, v); + } + } + for (id, pv) in self.pending_addr { + let v = resolve_pvalue(func, &val_names, pv)?; + match &mut func.insn_mut(id).kind { InsnKind::LoadRaw { addr, .. } | InsnKind::StoreRaw { addr, .. } => addr.value = v, _ => unreachable!() } + func.add_use(v, id); + } + for (id, entries) in self.pending_phi { + for (pv, bb_name) in entries { + let v = resolve_pvalue(func, &val_names, pv)?; + let bb = *bb_names.get(&bb_name).ok_or_else(|| invalid!("IR parse error: unknown block {bb_name}"))?; + func.add_phi_operand(id, v, bb); + } + } + for (id, bb1, bb2) in self.pending_jumps { + if let Some(b) = bb1 { let bb = *bb_names.get(&b).ok_or_else(|| invalid!("IR parse error: unknown block {b}"))?; func.insn_mut(id).bb1 = Some(bb); } + if let Some(b) = bb2 { let bb = *bb_names.get(&b).ok_or_else(|| invalid!("IR parse error: unknown block {b}"))?; func.insn_mut(id).bb2 = Some(bb); } + } + // Establish CFG edges from terminators. + for bb in func.all_bbs.clone() { + if let Some(last) = func.bb(bb).last() { + let (b1, b2) = { let insn = func.insn(last); (insn.bb1, insn.bb2) }; + if let Some(t) = b1 { func.connect(bb, t); } + if let Some(t) = b2 { func.connect(bb, t); } + } + } + Ok(()) + } +} + +fn resolve_pvalue(func: &Function, val_names: &HashMap, pv: PValue) -> Result { + Ok(match pv { + PValue::Const(v) => v, + PValue::Undef => Value::Undef, + PValue::Sp => Value::Insn(func.sp), + PValue::Arg(i) if i < MAX_FUNC_ARG => Value::Insn(func.args[i]), + PValue::Arg(i) => return Err(invalid!("IR parse error: arg{i} out of range")), + PValue::Name(n) => Value::Insn(*val_names.get(&n).ok_or_else(|| invalid!("IR parse error: unknown value {n}"))?), + }) +} + +fn rest_after<'a>(op: &str, body: &'a str) -> &'a str { body[op.len()..].trim() } + +fn split_top_comma(s: &str, line_no: usize) -> Result<(&str, &str)> { + s.split_once(',').ok_or_else(|| invalid!("IR parse line {line_no}: expected comma")) +} + +fn parse_value_list(s: &str) -> Result> { + let s = s.trim(); + if s.is_empty() { return Ok(Vec::new()); } + s.split(',').map(|x| parse_pvalue(x.trim(), 0)).collect() +} + +fn parse_pvalue(s: &str, line_no: usize) -> Result { + if s == "%sp" { return Ok(PValue::Sp); } + if s == "undef" { return Ok(PValue::Undef); } + if let Some(n) = s.strip_prefix("%arg") { return Ok(PValue::Arg(n.parse().map_err(|_| invalid!("IR parse line {line_no}: bad arg"))?)); } + if s.starts_with('%') { return Ok(PValue::Name(s.to_string())); } + if let Some((num, ty)) = s.rsplit_once(':') { + let v: i64 = num.parse().map_err(|_| invalid!("IR parse line {line_no}: bad const"))?; + let alu = match ty { "i32" => AluOp::Alu32, "i64" => AluOp::Alu64, _ => return Err(unsupported!("IR parse line {line_no}: unsupported const type {ty}")) }; + return Ok(PValue::Const(Value::Const { v, ty: alu, kind: ConstKind::Plain, builtin: crate::ir::BuiltinConst::None })); + } + Err(invalid!("IR parse line {line_no}: bad value '{s}'")) +} + +fn parse_addr(s: &str, line_no: usize) -> Result<(AddrValue, PValue)> { + let inner = s.strip_prefix('[').and_then(|x| x.strip_suffix(']')).ok_or_else(|| invalid!("IR parse line {line_no}: bad address"))?; + let (base_s, off_owned): (&str, String) = if let Some((b, o)) = inner.rsplit_once('+') { + (b, o.to_string()) + } else if let Some((b, o)) = inner.rsplit_once('-') { + (b, format!("-{o}")) + } else { + (inner, "0".to_string()) + }; + let offset: i16 = off_owned.parse().map_err(|_| invalid!("IR parse line {line_no}: bad address offset"))?; + Ok((AddrValue { value: Value::Undef, offset, offset_kind: ConstKind::Plain }, parse_pvalue(base_s.trim(), line_no)?)) +} + +fn parse_phi_entries(s: &str, line_no: usize) -> Result> { + let mut out = Vec::new(); + for part in s.split("],") { + let p = part.trim().trim_start_matches('[').trim_end_matches(']').trim(); + let (v, b) = split_top_comma(p, line_no)?; + out.push((parse_pvalue(v.trim(), line_no)?, b.trim().to_string())); + } + Ok(out) +} + +fn parse_vr(v: Option<&str>, line_no: usize) -> Result { parse_vr_name(v.ok_or_else(|| invalid!("IR parse line {line_no}: missing type"))?, line_no) } +fn parse_vr_name(s: &str, line_no: usize) -> Result { Ok(match s { "u8" => VrType::B8, "u16" => VrType::B16, "u32" => VrType::B32, "u64" => VrType::B64, _ => return Err(invalid!("IR parse line {line_no}: bad vr type {s}")) }) } +fn vr_name(t: VrType) -> &'static str { match t { VrType::B8 => "u8", VrType::B16 => "u16", VrType::B32 => "u32", VrType::B64 => "u64", VrType::Unknown => "u?" } } +fn alu_suffix(a: AluOp) -> &'static str { match a { AluOp::Alu32 => "32", AluOp::Alu64 => "64", AluOp::Unknown => "?" } } +fn parse_alu_suffix(s: &str, line_no: usize) -> Result { Ok(match s { "32" => AluOp::Alu32, "64" => AluOp::Alu64, _ => return Err(invalid!("IR parse line {line_no}: bad alu suffix {s}")) }) } +fn bin_name(op: BinOp) -> &'static str { match op { BinOp::Add => "add", BinOp::Sub => "sub", BinOp::Mul => "mul", BinOp::Div => "div", BinOp::Or => "or", BinOp::And => "and", BinOp::Lsh => "lsh", BinOp::Arsh => "arsh", BinOp::Rsh => "rsh", BinOp::Mod => "mod", BinOp::Xor => "xor" } } +fn parse_bin_op(s: &str) -> Option<(BinOp, AluOp)> { let (name, suf) = s.split_at(s.len().saturating_sub(2)); let alu = match suf { "32" => AluOp::Alu32, "64" => AluOp::Alu64, _ => return None }; Some((match name { "add" => BinOp::Add, "sub" => BinOp::Sub, "mul" => BinOp::Mul, "div" => BinOp::Div, "or" => BinOp::Or, "and" => BinOp::And, "lsh" => BinOp::Lsh, "arsh" => BinOp::Arsh, "rsh" => BinOp::Rsh, "mod" => BinOp::Mod, "xor" => BinOp::Xor, _ => return None }, alu)) } +fn cond_name(c: Cond) -> &'static str { match c { Cond::Eq => "jeq", Cond::Ne => "jne", Cond::Gt => "jgt", Cond::Ge => "jge", Cond::Lt => "jlt", Cond::Le => "jle", Cond::Sgt => "jsgt", Cond::Sge => "jsge", Cond::Slt => "jslt", Cond::Sle => "jsle" } } +fn parse_cond_op(s: &str) -> Option<(Cond, AluOp)> { let (name, suf) = s.split_at(s.len().saturating_sub(2)); let alu = match suf { "32" => AluOp::Alu32, "64" => AluOp::Alu64, _ => return None }; Some((match name { "jeq" => Cond::Eq, "jne" => Cond::Ne, "jgt" => Cond::Gt, "jge" => Cond::Ge, "jlt" => Cond::Lt, "jle" => Cond::Le, "jsgt" => Cond::Sgt, "jsge" => Cond::Sge, "jslt" => Cond::Slt, "jsle" => Cond::Sle, _ => return None }, alu)) } +fn extra_name(e: LoadImmExtra) -> &'static str { match e { LoadImmExtra::Imm64 => "imm64", LoadImmExtra::MapByFd => "map_by_fd", LoadImmExtra::MapValFd => "map_val_fd", LoadImmExtra::VarAddr => "var_addr", LoadImmExtra::CodeAddr => "code_addr", LoadImmExtra::MapByIdx => "map_by_idx", LoadImmExtra::MapValIdx => "map_val_idx" } } +fn parse_extra(s: &str, line_no: usize) -> Result { Ok(match s { "imm64" => LoadImmExtra::Imm64, "map_by_fd" => LoadImmExtra::MapByFd, "map_val_fd" => LoadImmExtra::MapValFd, "var_addr" => LoadImmExtra::VarAddr, "code_addr" => LoadImmExtra::CodeAddr, "map_by_idx" => LoadImmExtra::MapByIdx, "map_val_idx" => LoadImmExtra::MapValIdx, _ => return Err(invalid!("IR parse line {line_no}: bad loadimm kind {s}")) }) } diff --git a/core-rs/epass-ir/src/ir/value.rs b/core-rs/epass-ir/src/ir/value.rs new file mode 100644 index 00000000..9ae0bf8a --- /dev/null +++ b/core-rs/epass-ir/src/ir/value.rs @@ -0,0 +1,289 @@ +//! IR values: a value is either a constant, a reference to a defining +//! instruction (SSA), a physical-register position (after RA), or undefined. + +use crate::bytecode::BpfInsn; + +use super::InsnId; + +/// Operand width / signedness class for ALU and constants. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum AluOp { + /// Unset — guards against forgetting to specify a width (matches C `IR_ALU_UNKNOWN`). + #[default] + Unknown, + /// 32-bit operation. + Alu32, + /// 64-bit operation. + Alu64, +} + +/// Virtual-register width for typed memory operations. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum VrType { + #[default] + Unknown, + B8, + B16, + B32, + B64, +} + +impl VrType { + /// Size in bytes. + pub fn size(self) -> u32 { + match self { + VrType::B8 => 1, + VrType::B16 => 2, + VrType::B32 => 4, + VrType::B64 => 8, + VrType::Unknown => 0, + } + } + + /// Map a BPF load/store size field to a [`VrType`]. + pub fn from_bpf_size(size: u8) -> Option { + use crate::bytecode::size; + Some(match size { + size::B => VrType::B8, + size::H => VrType::B16, + size::W => VrType::B32, + size::DW => VrType::B64, + _ => return None, + }) + } +} + +/// Late-bound builtin constants, resolved during code generation. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum BuiltinConst { + #[default] + None, + /// Number of instructions in the current basic block. + BbInsnCnt, + /// Number of instructions since the nearest critical block. + BbInsnCriticalCnt, +} + +/// How a constant operand relates to the stack pointer. +/// +/// `RawOff`/`RawOffRev` values get the final stack offset folded in late +/// (see the `add_stack_offset` CG step), so they must be kept distinct from +/// plain constants to avoid double-patching. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum ConstKind { + #[default] + Plain, + /// `value + stack_offset` after CG. + RawOff, + /// `value - stack_offset` after CG. + RawOffRev, +} + +/// The position of a virtual register after register allocation. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct VrPos { + /// Whether this value occupies a concrete location (regs/stack) yet. + pub allocated: bool, + /// If spilled, the size in bytes occupied on the stack. + pub spilled_size: u32, + /// Allocated physical register (valid when `spilled == 0`). + pub alloc_reg: u8, + /// Stack offset if spilled; `0` means "in a register". + pub spilled: i32, +} + +impl VrPos { + /// The fixed position of the stack pointer (R10). + pub fn stack_ptr() -> Self { + VrPos { + allocated: true, + spilled_size: 0, + alloc_reg: crate::bytecode::BPF_REG_10, + spilled: 0, + } + } +} + +/// An SSA value: an operand of an instruction. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Value { + /// A literal constant. + Const { + v: i64, + ty: AluOp, + kind: ConstKind, + builtin: BuiltinConst, + }, + /// A reference to a defining instruction (the SSA def). + Insn(InsnId), + /// A finalized register/stack position (only used during/after RA). + VrPos(VrPos), + /// A flattened destination position (code-gen only). + FlattenDst(VrPos), + /// Undefined. + Undef, +} + +impl Value { + pub fn const32(v: i32) -> Value { + Value::Const { + v: v as i64, + ty: AluOp::Alu32, + kind: ConstKind::Plain, + builtin: BuiltinConst::None, + } + } + + pub fn const64(v: i64) -> Value { + Value::Const { + v, + ty: AluOp::Alu64, + kind: ConstKind::Plain, + builtin: BuiltinConst::None, + } + } + + pub fn const32_rawoff(v: i32) -> Value { + Value::Const { + v: v as i64, + ty: AluOp::Alu32, + kind: ConstKind::RawOff, + builtin: BuiltinConst::None, + } + } + + pub fn insn(id: InsnId) -> Value { + Value::Insn(id) + } + + pub fn vrpos(pos: VrPos) -> Value { + Value::VrPos(pos) + } + + pub fn undef() -> Value { + Value::Undef + } + + /// If this value references a defining instruction, return its id. + pub fn as_insn(self) -> Option { + match self { + Value::Insn(id) => Some(id), + _ => None, + } + } + + pub fn is_const(self) -> bool { + matches!(self, Value::Const { .. }) + } +} + +/// A base value plus a signed byte offset (used by raw load/store). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct AddrValue { + pub value: Value, + pub offset: i16, + /// The kind of the offset constant (plain vs. stack-relative). + pub offset_kind: ConstKind, +} + +impl AddrValue { + pub fn new(value: Value, offset: i16) -> Self { + AddrValue { + value, + offset, + offset_kind: ConstKind::Plain, + } + } +} + +/// A single `(value, predecessor block)` entry of a phi node. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PhiValue { + pub value: Value, + pub bb: super::BbId, +} + +/// Extra immediate-load kinds for 64-bit immediates (map fds, addresses). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LoadImmExtra { + Imm64, + MapByFd, + MapValFd, + VarAddr, + CodeAddr, + MapByIdx, + MapValIdx, +} + +impl LoadImmExtra { + /// Map the `src_reg` of a `BPF_LD_IMM64` to its extra-load kind. + pub fn from_src_reg(src: u8) -> Option { + Some(match src { + 0 => LoadImmExtra::Imm64, + 1 => LoadImmExtra::MapByFd, + 2 => LoadImmExtra::MapValFd, + 3 => LoadImmExtra::VarAddr, + 4 => LoadImmExtra::CodeAddr, + 5 => LoadImmExtra::MapByIdx, + 6 => LoadImmExtra::MapValIdx, + _ => return None, + }) + } + + pub fn to_src_reg(self) -> u8 { + match self { + LoadImmExtra::Imm64 => 0, + LoadImmExtra::MapByFd => 1, + LoadImmExtra::MapValFd => 2, + LoadImmExtra::VarAddr => 3, + LoadImmExtra::CodeAddr => 4, + LoadImmExtra::MapByIdx => 5, + LoadImmExtra::MapValIdx => 6, + } + } +} + +/// Original position of an instruction/value in the source bytecode, used to +/// correlate IR back to verifier diagnostics. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct RawPos { + pub valid: bool, + pub pos: usize, + pub kind: RawPosKind, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum RawPosKind { + #[default] + Insn, + Imm, + Dst, + Src, +} + +impl RawPos { + pub fn at(pos: usize, kind: RawPosKind) -> Self { + RawPos { + valid: true, + pos, + kind, + } + } +} + +/// Helper to build the second slot of a 64-bit immediate load instruction. +#[allow(dead_code)] +pub(crate) fn imm64_low_high(imm64: i64) -> (BpfInsn, BpfInsn) { + let low = (imm64 & 0xffff_ffff) as i32; + let high = (imm64 >> 32) as i32; + ( + BpfInsn { + imm: low, + ..Default::default() + }, + BpfInsn { + imm: high, + ..Default::default() + }, + ) +} diff --git a/core-rs/epass-ir/src/lib.rs b/core-rs/epass-ir/src/lib.rs new file mode 100644 index 00000000..79f943f5 --- /dev/null +++ b/core-rs/epass-ir/src/lib.rs @@ -0,0 +1,70 @@ +//! ePass: an SSA-based intermediate representation and compiler framework for +//! eBPF programs. +//! +//! The pipeline mirrors the original C implementation: +//! +//! ```text +//! eBPF bytecode --lift--> SSA IR --run passes--> IR --compile--> eBPF bytecode +//! ``` +//! +//! This crate is userspace-only and has no dependency on libbpf or the kernel. + +pub mod bytecode; +pub mod cfg; +pub mod cg; +pub mod check; +pub mod env; +pub mod error; +pub mod ffi; +pub mod helpers; +pub mod ir; +pub mod lift; +pub mod logfmt; +pub mod opts; +pub mod pass; +pub mod passes; +pub mod pipeline; + +pub use bytecode::BpfInsn; +pub use env::{Env, LogLevel}; +pub use error::{Error, Result}; +pub use ir::{BbId, BasicBlock, Function, Insn, InsnId, InsnKind, IrBuilder, InsertPoint, Value}; +pub use ir::text::{dump_function as dump_ir, load_function_from_file as load_ir_file, load_function_from_str as load_ir_str, DumpOptions}; +pub use lift::lift; +pub use opts::{Opts, PrintMode}; +pub use pass::{Pass, PassManager}; +pub use pipeline::{autorun, run_passes_only}; + +/// Render a function to its debug textual form. +pub fn print_ir(func: &Function) -> String { + ir::print::print_function(func) +} + +/// Build the default pass pipeline with no pass options. +pub fn default_passes() -> PassManager { + passes_from_popt("").expect("builtin default pass pipeline is valid") +} + +/// Build the builtin pass pipeline from a pass-option string. +/// +/// `popt` enables/disables/configures passes, but pass order is decided by each +/// pass's `register_pass` implementation. Examples: +/// - `dump_ir(/tmp/a.epir)` enables the optional IR dump pass; +/// - `!const_prop` disables constant propagation; +/// - `!phi` is rejected because `phi` is not disableable. +pub fn passes_from_popt(popt: &str) -> Result { + PassManager::from_passes( + vec![ + Box::new(passes::dump_ir::pass()), + Box::new(passes::const_prop::pass()), + Box::new(passes::phi::pass()), + Box::new(passes::optimization::pass()), + ], + popt, + ) +} + +/// Backward-compatible alias for callers using the old name. +pub fn default_passes_with_popt(popt: &str) -> PassManager { + passes_from_popt(popt).expect("pass options produced an invalid builtin pipeline") +} diff --git a/core-rs/epass-ir/src/lift.rs b/core-rs/epass-ir/src/lift.rs new file mode 100644 index 00000000..669fb2fd --- /dev/null +++ b/core-rs/epass-ir/src/lift.rs @@ -0,0 +1,854 @@ +//! The lifter: translates raw eBPF bytecode into SSA-form IR. +//! +//! Pipeline (mirrors the C `bpf_ir.c`): +//! 1. Discover basic blocks by scanning for jump targets and fallthroughs. +//! 2. Build the SSA form on the fly using Braun et al.'s algorithm +//! ("Simple and Efficient Construction of SSA Form"). +//! 3. Translate each eBPF opcode into one or more IR instructions. +//! 4. Compute CFG successors, the reachable-block chain layout, and end blocks. + +use std::collections::HashMap; + +use crate::bytecode::{self as bc, BpfInsn}; +use crate::env::{Env, Timer}; +use crate::error::Result; +use crate::helpers::helper_arg_num; +use crate::ir::insn::{BinOp, Cond, EndKind, InsnKind}; +use crate::ir::value::{AddrValue, AluOp, ConstKind, LoadImmExtra, PhiValue, RawPos, RawPosKind, VrType}; +use crate::ir::{BbId, Function, InsnId, InsertPos, Value}; +use crate::{internal, invalid, unsupported}; + +/// A pre-IR (raw) instruction, with 64-bit immediates already joined. +#[derive(Debug, Clone, Copy)] +struct PreInsn { + insn: BpfInsn, + imm64: i64, + /// Position in the original bytecode. + pos: usize, +} + +/// A pre-IR basic block, identified by the bytecode position of its entrance. +struct PreBlock { + start: usize, + end: usize, // exclusive + insns: Vec, + pred_positions: Vec, + /// The corresponding real IR block. + ir_bb: BbId, + sealed: bool, + filled: bool, + /// Incomplete phi nodes awaiting sealing, keyed by register. + incomplete_phis: [Option; bc::MAX_BPF_REG], +} + +/// SSA construction state. +struct Ssa<'e> { + #[allow(dead_code)] + env: &'e mut Env, + func: Function, + blocks: Vec, + /// `entrance position -> pre-block index`. + by_entrance: HashMap, + entry_idx: usize, + /// `current_def[reg][block_idx] = value`. + current_def: Vec>, + /// Total bytecode length (for fallthrough successor detection). + total_len: usize, +} + +/// Returns true if a raw instruction ends a basic block (any jump except call). +fn is_breakpoint(insn: &BpfInsn) -> bool { + let code = insn.code; + let class = bc::class_of(code); + if class == bc::class::JMP || class == bc::class::JMP32 { + bc::op_of(code) != bc::op::CALL + } else { + false + } +} + +fn is_cond_jump_op(op: u8) -> bool { + matches!( + op, + bc::op::JEQ + | bc::op::JGT + | bc::op::JGE + | bc::op::JSET + | bc::op::JNE + | bc::op::JSGT + | bc::op::JSGE + | bc::op::JLT + | bc::op::JLE + | bc::op::JSLT + | bc::op::JSLE + ) +} + +/// Discover basic-block entrances and build pre-blocks with predecessor links. +fn discover_blocks(env: &mut Env, insns: &mut [BpfInsn]) -> Result<(Vec<(usize, Vec)>,)> { + // Normalize `pc+0` conditional jumps into NOPs (ja +0), as the C lifter does. + for i in 0..insns.len() { + let code = insns[i].code; + let class = bc::class_of(code); + if (class == bc::class::JMP || class == bc::class::JMP32) + && is_cond_jump_op(bc::op_of(code)) + && insns[i].off == 0 + { + insns[i] = BpfInsn::new(bc::class::JMP | bc::op::JA, 0, 0, 0, 0); + } + } + + // entrance position -> list of predecessor (source) positions + let mut entrances: HashMap> = HashMap::new(); + let mut order: Vec = Vec::new(); + let ensure = |entrances: &mut HashMap>, + order: &mut Vec, + pos: usize| { + if !entrances.contains_key(&pos) { + entrances.insert(pos, Vec::new()); + order.push(pos); + } + }; + + let len = insns.len(); + for i in 0..len { + let code = insns[i].code; + let class = bc::class_of(code); + if class != bc::class::JMP && class != bc::class::JMP32 { + continue; + } + let op = bc::op_of(code); + if op == bc::op::JA { + if class != bc::class::JMP { + return Err(unsupported!("BPF_JA only allows JMP class (no JMP32 JA)")); + } + let target = (i as i64 + insns[i].off as i64 + 1) as usize; + ensure(&mut entrances, &mut order, target); + entrances.get_mut(&target).unwrap().push(i); + } else if is_cond_jump_op(op) { + let target = (i as i64 + insns[i].off as i64 + 1) as usize; + let fallthrough = i + 1; + ensure(&mut entrances, &mut order, target); + entrances.get_mut(&target).unwrap().push(i); + ensure(&mut entrances, &mut order, fallthrough); + entrances.get_mut(&fallthrough).unwrap().push(i); + } else if op == bc::op::EXIT { + if i + 1 < len { + ensure(&mut entrances, &mut order, i + 1); + } + } + } + // Entry block at 0 (no predecessors). + ensure(&mut entrances, &mut order, 0); + + let _ = env; + order.sort_unstable(); + + // Add fallthrough predecessors: for every entrance `e > 0`, if the previous + // instruction `e-1` does not itself terminate a block (not a breakpoint), + // then control falls through from `e-1` into `e`. We record `e-1` as a + // predecessor source position; `block_of_pos` later maps it to its block. + for &e in &order { + if e == 0 { + continue; + } + let prev = e - 1; + if !is_breakpoint(&insns[prev]) { + let preds = entrances.get_mut(&e).unwrap(); + if !preds.contains(&prev) { + preds.push(prev); + } + } + } + + let result: Vec<(usize, Vec)> = order + .into_iter() + .map(|pos| (pos, entrances.remove(&pos).unwrap())) + .collect(); + Ok((result,)) +} + +impl<'e> Ssa<'e> { + /// Given a bytecode position, find the pre-block that contains it. + fn block_of_pos(&self, pos: usize) -> Option { + // blocks are sorted by start; find the last block whose start <= pos + // and whose range contains pos. + let mut found = None; + for (idx, b) in self.blocks.iter().enumerate() { + if b.start <= pos { + found = Some(idx); + } else { + break; + } + } + let idx = found?; + if pos < self.blocks[idx].end { + Some(idx) + } else { + None + } + } + + fn write_var(&mut self, reg: u8, block: usize, val: Value) { + self.current_def[reg as usize].insert(block, val); + } + + fn is_var_defined(&self, reg: u8, block: usize) -> bool { + self.current_def[reg as usize].contains_key(&block) + } + + fn read_var(&mut self, reg: u8, block: usize) -> Result { + if reg == bc::BPF_REG_10 { + return Ok(Value::Insn(self.func.sp)); + } + if let Some(&v) = self.current_def[reg as usize].get(&block) { + return Ok(v); + } + if block == self.entry_idx { + // Entry block defines r1..r5 from function arguments. + if reg >= bc::BPF_REG_1 && (reg as usize) <= bc::MAX_FUNC_ARG { + return Ok(Value::Insn(self.func.args[(reg - 1) as usize])); + } + return Err(invalid!( + "read of undefined r{reg} in entry block (invalid program)" + )); + } + self.read_var_recursive(reg, block) + } + + fn read_var_recursive(&mut self, reg: u8, block: usize) -> Result { + let (sealed, single_pred) = { + let b = &self.blocks[block]; + (b.sealed, if b.pred_positions.len() == 1 { Some(b.pred_positions[0]) } else { None }) + }; + let val; + if !sealed { + // Incomplete CFG: place an empty phi to be completed at seal time. + let ir_bb = self.blocks[block].ir_bb; + let phi = self.func.create_insn(ir_bb, InsnKind::Phi, InsertPos::Front); + self.blocks[block].incomplete_phis[reg as usize] = Some(phi); + val = Value::Insn(phi); + } else if let Some(pred_pos) = single_pred { + let pred_block = self + .block_of_pos(pred_pos) + .ok_or_else(|| internal!("predecessor block not found for pos {pred_pos}"))?; + val = self.read_var(reg, pred_block)?; + } else { + let ir_bb = self.blocks[block].ir_bb; + let phi = self.func.create_insn(ir_bb, InsnKind::Phi, InsertPos::Front); + let v = Value::Insn(phi); + self.write_var(reg, block, v); + self.add_phi_operands(reg, phi, block)?; + val = v; + } + self.write_var(reg, block, val); + Ok(val) + } + + fn add_phi_operands(&mut self, reg: u8, phi: InsnId, block: usize) -> Result<()> { + // Map predecessor source positions to their owning blocks, deduplicating + // so each predecessor block contributes exactly one phi entry. + let pred_positions = self.blocks[block].pred_positions.clone(); + let mut pred_blocks: Vec = Vec::new(); + for pred_pos in pred_positions { + let pb = self + .block_of_pos(pred_pos) + .ok_or_else(|| internal!("phi predecessor block not found for pos {pred_pos}"))?; + if !pred_blocks.contains(&pb) { + pred_blocks.push(pb); + } + } + for pred_block in pred_blocks { + let value = self.read_var(reg, pred_block)?; + let pred_bb = self.blocks[pred_block].ir_bb; + self.func.insn_mut(phi).phi.push(PhiValue { value, bb: pred_bb }); + self.func.add_use(value, phi); + } + Ok(()) + } + + /// Seal a block: complete any incomplete phis now that its predecessors + /// are all filled. + fn seal_block(&mut self, block: usize) -> Result<()> { + for reg in 0..bc::MAX_BPF_REG { + if let Some(phi) = self.blocks[block].incomplete_phis[reg] { + self.blocks[block].incomplete_phis[reg] = None; + self.add_phi_operands(reg as u8, phi, block)?; + } + } + self.blocks[block].sealed = true; + Ok(()) + } + + /// Build a constant or register source value for an instruction operand. + fn src_value(&mut self, block: usize, p: &PreInsn) -> Result { + let code = p.insn.code; + if bc::src_of(code) == bc::src::K { + Ok(Value::Const { + v: p.insn.imm as i64, + ty: AluOp::Alu32, + kind: ConstKind::Plain, + builtin: crate::ir::value::BuiltinConst::None, + }) + } else { + self.read_var(p.insn.src_reg, block) + } + } + + fn alu_class(code: u8) -> AluOp { + if bc::class_of(code) == bc::class::ALU { + AluOp::Alu32 + } else { + AluOp::Alu64 + } + } + + /// Build `dst = dst src` and record the new SSA def for `dst_reg`. + fn emit_bin(&mut self, block: usize, p: &PreInsn, op: BinOp) -> Result<()> { + let alu = Self::alu_class(p.insn.code); + let lhs = self.read_var(p.insn.dst_reg, block)?; + let rhs = self.src_value(block, p)?; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, InsnKind::Bin { op }, InsertPos::Back); + { + let insn = self.func.insn_mut(id); + insn.alu_op = alu; + insn.values.push(lhs); + insn.values.push(rhs); + insn.raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + } + self.func.add_use(lhs, id); + self.func.add_use(rhs, id); + self.write_var(p.insn.dst_reg, block, Value::Insn(id)); + Ok(()) + } + + fn emit_unary(&mut self, block: usize, p: &PreInsn, kind: InsnKind) -> Result<()> { + let alu = Self::alu_class(p.insn.code); + let operand = self.read_var(p.insn.dst_reg, block)?; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, kind, InsertPos::Back); + { + let insn = self.func.insn_mut(id); + insn.alu_op = alu; + insn.values.push(operand); + insn.raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + } + self.func.add_use(operand, id); + self.write_var(p.insn.dst_reg, block, Value::Insn(id)); + Ok(()) + } + + fn emit_cond_jmp(&mut self, block: usize, p: &PreInsn, cond: Cond) -> Result<()> { + let alu = if bc::class_of(p.insn.code) == bc::class::JMP { + AluOp::Alu64 + } else { + AluOp::Alu32 + }; + let lhs = self.read_var(p.insn.dst_reg, block)?; + let rhs = self.src_value(block, p)?; + let fallthrough = self.ir_bb_at_pos(p.pos + 1)?; + let target = self.ir_bb_at_pos((p.pos as i64 + p.insn.off as i64 + 1) as usize)?; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, InsnKind::CondJmp { cond }, InsertPos::Back); + { + let insn = self.func.insn_mut(id); + insn.alu_op = alu; + insn.values.push(lhs); + insn.values.push(rhs); + insn.bb1 = Some(fallthrough); + insn.bb2 = Some(target); + insn.raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + } + self.func.add_use(lhs, id); + self.func.add_use(rhs, id); + Ok(()) + } + + fn ir_bb_at_pos(&self, pos: usize) -> Result { + let idx = *self + .by_entrance + .get(&pos) + .ok_or_else(|| internal!("no basic block starts at pos {pos}"))?; + Ok(self.blocks[idx].ir_bb) + } + + /// Translate every instruction in a pre-block into IR, recursing into succs. + fn fill_block(&mut self, block: usize) -> Result<()> { + if self.blocks[block].sealed && self.blocks[block].filled { + return Ok(()); + } + // Seal if all predecessors are filled. + let all_preds_filled = { + let pred_positions = self.blocks[block].pred_positions.clone(); + pred_positions.iter().all(|&pp| { + self.block_of_pos(pp) + .map(|pb| self.blocks[pb].filled) + .unwrap_or(true) + }) + }; + if all_preds_filled && !self.blocks[block].sealed { + self.seal_block(block)?; + } + if self.blocks[block].filled { + return Ok(()); + } + + let pre_insns = self.blocks[block].insns.clone(); + for p in &pre_insns { + if let Err(e) = self.translate(block, p) { + // libbpf CO-RE can intentionally poison instructions in branches + // that it also makes statically unreachable; the verifier prunes + // those branches, but ePass lifts before verifier pruning. Treat a + // per-instruction translation failure as a dummy no-op: emit no IR, + // do not update SSA defs, and do not affect CFG. If such an + // instruction is actually reachable, the rewritten program should + // still fail verifier validation rather than being silently accepted. + crate::log_warn!( + self.env, + "warning: replacing instruction at pos {} with dummy no-op: {}\n", + p.pos, + e + ); + } + } + self.blocks[block].filled = true; + + // Recurse into successors (by CFG positions). + let succ_positions = self.successor_positions(block); + for sp in succ_positions { + if let Some(sb) = self.by_entrance.get(&sp).copied() { + self.fill_block(sb)?; + } + } + Ok(()) + } + + /// The bytecode positions of a pre-block's CFG successors. + fn successor_positions(&self, block: usize) -> Vec { + let b = &self.blocks[block]; + let last = b.insns.last(); + let mut succs = Vec::new(); + if let Some(p) = last { + let code = p.insn.code; + let class = bc::class_of(code); + if class == bc::class::JMP || class == bc::class::JMP32 { + let op = bc::op_of(code); + if op == bc::op::JA { + succs.push((p.pos as i64 + p.insn.off as i64 + 1) as usize); + return succs; + } else if is_cond_jump_op(op) { + succs.push(p.pos + 1); + succs.push((p.pos as i64 + p.insn.off as i64 + 1) as usize); + return succs; + } else if op == bc::op::EXIT { + return succs; // no successors + } + } + } + // Fallthrough to the next block. + if b.end < self.total_len { + succs.push(b.end); + } + succs + } + + fn translate(&mut self, block: usize, p: &PreInsn) -> Result<()> { + let code = p.insn.code; + let class = bc::class_of(code); + match class { + bc::class::ALU | bc::class::ALU64 => self.translate_alu(block, p), + bc::class::LD if bc::mode_of(code) == bc::mode::IMM && bc::size_of(code) == bc::size::DW => { + self.translate_ld_imm64(block, p) + } + bc::class::LDX if bc::mode_of(code) == bc::mode::MEM || bc::mode_of(code) == bc::mode::MEMSX => { + self.translate_load(block, p) + } + bc::class::ST if bc::mode_of(code) == bc::mode::MEM => self.translate_store_imm(block, p), + bc::class::STX if bc::mode_of(code) == bc::mode::MEM => self.translate_store_reg(block, p), + bc::class::JMP | bc::class::JMP32 => self.translate_jmp(block, p), + _ => Err(unsupported!( + "instruction class 0x{:02x} at pos {} not supported", + class, + p.pos + )), + } + } + + fn translate_alu(&mut self, block: usize, p: &PreInsn) -> Result<()> { + let code = p.insn.code; + let op = bc::op_of(code); + let alu = Self::alu_class(code); + if p.insn.off != 0 && op != bc::op::END { + return Err(invalid!("ALU instruction with nonzero offset at pos {}", p.pos)); + } + match op { + bc::op::ADD => self.emit_bin(block, p, BinOp::Add), + bc::op::SUB => self.emit_bin(block, p, BinOp::Sub), + bc::op::MUL => self.emit_bin(block, p, BinOp::Mul), + bc::op::DIV => self.emit_bin(block, p, BinOp::Div), + bc::op::OR => self.emit_bin(block, p, BinOp::Or), + bc::op::AND => self.emit_bin(block, p, BinOp::And), + bc::op::LSH => self.emit_bin(block, p, BinOp::Lsh), + bc::op::RSH => self.emit_bin(block, p, BinOp::Rsh), + bc::op::ARSH => self.emit_bin(block, p, BinOp::Arsh), + bc::op::MOD => self.emit_bin(block, p, BinOp::Mod), + bc::op::XOR => self.emit_bin(block, p, BinOp::Xor), + bc::op::MOV => { + // MOV does not create an instruction; it just rebinds the SSA value. + let mut v = self.src_value(block, p)?; + if bc::src_of(code) == bc::src::K { + if let Value::Const { ty, .. } = &mut v { + *ty = AluOp::Alu32; + } + } + let _ = alu; + self.write_var(p.insn.dst_reg, block, v); + Ok(()) + } + bc::op::NEG => { + if bc::src_of(code) != bc::src::K { + return Err(invalid!("neg with src != K at pos {}", p.pos)); + } + self.emit_unary(block, p, InsnKind::Neg) + } + bc::op::END => { + if alu == AluOp::Alu64 { + return Err(unsupported!("BPF_END is not supported in 64-bit mode")); + } + let kind = match bc::src_of(code) { + bc::end::TO_BE => EndKind::ToBe, + bc::end::TO_LE => EndKind::ToLe, + _ => return Err(unsupported!("unknown BPF_END subcode at pos {}", p.pos)), + }; + self.emit_unary( + block, + p, + InsnKind::End { + kind, + swap_width: p.insn.imm as u32, + }, + ) + } + _ => Err(unsupported!("unknown ALU op 0x{:02x} at pos {}", op, p.pos)), + } + } + + fn translate_ld_imm64(&mut self, block: usize, p: &PreInsn) -> Result<()> { + let extra = LoadImmExtra::from_src_reg(p.insn.src_reg) + .ok_or_else(|| unsupported!("unsupported LD_IMM64 src_reg {}", p.insn.src_reg))?; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn( + ir_bb, + InsnKind::LoadImmExtra { + extra, + imm64: p.imm64, + }, + InsertPos::Back, + ); + self.func.insn_mut(id).raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + self.write_var(p.insn.dst_reg, block, Value::Insn(id)); + Ok(()) + } + + fn translate_load(&mut self, block: usize, p: &PreInsn) -> Result<()> { + // dst = *(size *)(src + off) + let vr_type = VrType::from_bpf_size(bc::size_of(p.insn.code)) + .ok_or_else(|| invalid!("bad load size at pos {}", p.pos))?; + let base = self.read_var(p.insn.src_reg, block)?; + let offset_kind = if p.insn.src_reg == bc::BPF_REG_10 { + ConstKind::RawOff + } else { + ConstKind::Plain + }; + let mut addr = AddrValue::new(base, p.insn.off); + addr.offset_kind = offset_kind; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, InsnKind::LoadRaw { vr_type, addr }, InsertPos::Back); + self.func.insn_mut(id).raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + self.func.add_use(base, id); + self.write_var(p.insn.dst_reg, block, Value::Insn(id)); + Ok(()) + } + + fn translate_store_imm(&mut self, block: usize, p: &PreInsn) -> Result<()> { + // *(size *)(dst + off) = imm + let vr_type = VrType::from_bpf_size(bc::size_of(p.insn.code)) + .ok_or_else(|| invalid!("bad store size at pos {}", p.pos))?; + let base = self.read_var(p.insn.dst_reg, block)?; + let offset_kind = if p.insn.dst_reg == bc::BPF_REG_10 { + ConstKind::RawOff + } else { + ConstKind::Plain + }; + let mut addr = AddrValue::new(base, p.insn.off); + addr.offset_kind = offset_kind; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, InsnKind::StoreRaw { vr_type, addr }, InsertPos::Back); + let imm = Value::const32(p.insn.imm); + { + let insn = self.func.insn_mut(id); + insn.values.push(imm); + insn.raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + } + self.func.add_use(base, id); + Ok(()) + } + + fn translate_store_reg(&mut self, block: usize, p: &PreInsn) -> Result<()> { + // *(size *)(dst + off) = src + let vr_type = VrType::from_bpf_size(bc::size_of(p.insn.code)) + .ok_or_else(|| invalid!("bad store size at pos {}", p.pos))?; + let base = self.read_var(p.insn.dst_reg, block)?; + let src = self.read_var(p.insn.src_reg, block)?; + let offset_kind = if p.insn.dst_reg == bc::BPF_REG_10 { + ConstKind::RawOff + } else { + ConstKind::Plain + }; + let mut addr = AddrValue::new(base, p.insn.off); + addr.offset_kind = offset_kind; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, InsnKind::StoreRaw { vr_type, addr }, InsertPos::Back); + { + let insn = self.func.insn_mut(id); + insn.values.push(src); + insn.raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + } + self.func.add_use(base, id); + self.func.add_use(src, id); + Ok(()) + } + + fn translate_jmp(&mut self, block: usize, p: &PreInsn) -> Result<()> { + let op = bc::op_of(p.insn.code); + match op { + bc::op::JA => { + let target = self.ir_bb_at_pos((p.pos as i64 + p.insn.off as i64 + 1) as usize)?; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, InsnKind::Ja, InsertPos::Back); + let insn = self.func.insn_mut(id); + insn.bb1 = Some(target); + insn.raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + Ok(()) + } + bc::op::EXIT => { + let r0 = self.read_var(bc::BPF_REG_0, block)?; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, InsnKind::Ret, InsertPos::Back); + { + let insn = self.func.insn_mut(id); + insn.values.push(r0); + insn.raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + } + self.func.add_use(r0, id); + Ok(()) + } + bc::op::JEQ => self.emit_cond_jmp(block, p, Cond::Eq), + bc::op::JNE => self.emit_cond_jmp(block, p, Cond::Ne), + bc::op::JGT => self.emit_cond_jmp(block, p, Cond::Gt), + bc::op::JGE => self.emit_cond_jmp(block, p, Cond::Ge), + bc::op::JLT => self.emit_cond_jmp(block, p, Cond::Lt), + bc::op::JLE => self.emit_cond_jmp(block, p, Cond::Le), + bc::op::JSGT => self.emit_cond_jmp(block, p, Cond::Sgt), + bc::op::JSGE => self.emit_cond_jmp(block, p, Cond::Sge), + bc::op::JSLT => self.emit_cond_jmp(block, p, Cond::Slt), + bc::op::JSLE => self.emit_cond_jmp(block, p, Cond::Sle), + bc::op::CALL => self.translate_call(block, p), + _ => Err(unsupported!("unknown jmp op 0x{:02x} at pos {}", op, p.pos)), + } + } + + fn translate_call(&mut self, block: usize, p: &PreInsn) -> Result<()> { + let src = p.insn.src_reg; + if src == 1 { + return Err(unsupported!("BPF-local (pc+offset) calls not supported")); + } + if src == 2 { + return Err(unsupported!("platform-specific helper calls not supported")); + } + + let is_ecall = src == bc::EPASS_CALL; + let kind = if is_ecall { InsnKind::Ecall } else { InsnKind::Call { fid: p.insn.imm } }; + let ir_bb = self.blocks[block].ir_bb; + let id = self.func.create_insn(ir_bb, kind, InsertPos::Back); + self.func.insn_mut(id).raw_pos = RawPos::at(p.pos, RawPosKind::Insn); + + let argc: usize = if is_ecall { + if p.insn.imm < 0 { + return Err(unsupported!("unknown ecall function {} at pos {}", p.insn.imm, p.pos)); + } + p.insn.dst_reg as usize + } else { + match helper_arg_num(p.insn.imm) { + Some(-1) => { + // trace_printk: variable length; infer from defined registers. + let mut n = 2usize; + let mut j = 2u8; + while (j as usize) < bc::MAX_FUNC_ARG { + if self.is_var_defined(j + bc::BPF_REG_1, block) { + n = (j + bc::BPF_REG_1) as usize; + j += 1; + } else { + break; + } + } + n + } + Some(n) => n as usize, + None => { + return Err(unsupported!( + "unsupported helper function {} at pos {}", + p.insn.imm, + p.pos + )); + } + } + }; + + if argc > bc::MAX_FUNC_ARG { + return Err(invalid!("too many call arguments ({argc}) at pos {}", p.pos)); + } + + for j in 0..argc { + let arg = self.read_var(bc::BPF_REG_1 + j as u8, block)?; + self.func.insn_mut(id).values.push(arg); + self.func.add_use(arg, id); + } + self.write_var(bc::BPF_REG_0, block, Value::Insn(id)); + Ok(()) + } +} + +/// Lift an eBPF program to SSA IR. +pub fn lift(env: &mut Env) -> Result { + let timer = Timer::start(); + let mut insns = std::mem::take(&mut env.insns); + let total_len = insns.len(); + + let (entrances,) = discover_blocks(env, &mut insns)?; + + // Build the function and one IR block per entrance. + let mut func = Function::new(); + // The entry pre-block reuses the function's entry IR block; others are new. + let mut blocks: Vec = Vec::with_capacity(entrances.len()); + let mut by_entrance: HashMap = HashMap::new(); + let entrance_starts: Vec = entrances.iter().map(|(p, _)| *p).collect(); + + for (idx, (start, pred_positions)) in entrances.iter().enumerate() { + // End is the next entrance, or end-of-program; trimmed at a breakpoint. + let mut end = entrance_starts + .get(idx + 1) + .copied() + .unwrap_or(total_len); + for pos in *start..end { + if is_breakpoint_or_exit(&insns[pos]) { + end = pos + 1; + break; + } + } + let ir_bb = if *start == 0 { func.entry } else { func.create_bb() }; + by_entrance.insert(*start, idx); + + // Collect pre-insns, joining LD_IMM64 pairs. + let mut pre_insns = Vec::new(); + let mut pos = *start; + while pos < end { + let insn = insns[pos]; + let mut imm64 = (insn.imm as u32) as i64; + let cur_pos = pos; + if pos + 1 < end && insns[pos + 1].is_imm64_continuation() { + imm64 = bc::join_imm64(insn.imm, insns[pos + 1].imm); + pos += 1; + } + pre_insns.push(PreInsn { + insn, + imm64, + pos: cur_pos, + }); + pos += 1; + } + + blocks.push(PreBlock { + start: *start, + end, + insns: pre_insns, + pred_positions: pred_positions.clone(), + ir_bb, + sealed: false, + filled: false, + incomplete_phis: [None; bc::MAX_BPF_REG], + }); + } + + // Set up IR-level pred/succ edges between blocks (positions -> blocks). + // (Filled after translation; SSA construction uses pred_positions directly.) + + let entry_idx = *by_entrance.get(&0).expect("entry block at pos 0"); + + let mut ssa = Ssa { + env, + func, + blocks, + by_entrance, + entry_idx, + current_def: vec![HashMap::new(); bc::MAX_BPF_REG], + total_len, + }; + + ssa.fill_block(entry_idx)?; + + // Seal any blocks still open (e.g. loop headers whose back-edge predecessor + // was filled after the header). All blocks are filled by now. Completing a + // phi may create new incomplete phis in still-open predecessors, so we drain + // to a fixpoint: mark every block sealed, then complete every phi that still + // has no operands. + for block in 0..ssa.blocks.len() { + ssa.blocks[block].sealed = true; + } + loop { + // Find a recorded-but-empty incomplete phi anywhere and complete it. + let mut pending: Option<(usize, u8, InsnId)> = None; + 'outer: for block in 0..ssa.blocks.len() { + for reg in 0..bc::MAX_BPF_REG { + if let Some(phi) = ssa.blocks[block].incomplete_phis[reg] { + if ssa.func.insn(phi).phi.is_empty() { + pending = Some((block, reg as u8, phi)); + break 'outer; + } + } + } + } + match pending { + Some((block, reg, phi)) => { + ssa.blocks[block].incomplete_phis[reg as usize] = None; + ssa.add_phi_operands(reg, phi, block)?; + } + None => break, + } + } + + // Establish IR CFG edges from pre-block successor positions. + for block in 0..ssa.blocks.len() { + let from = ssa.blocks[block].ir_bb; + for sp in ssa.successor_positions(block) { + if let Some(&sb) = ssa.by_entrance.get(&sp) { + let to = ssa.blocks[sb].ir_bb; + ssa.func.connect(from, to); + } + } + } + + let mut func = ssa.func; + drop(ssa.blocks); + + crate::cfg::finalize(env, &mut func)?; + + env.insns = insns; // restore (the original program is kept until compile writes back) + env.lift_time_ns += timer.elapsed_ns(); + Ok(func) +} + +fn is_breakpoint_or_exit(insn: &BpfInsn) -> bool { + is_breakpoint(insn) +} diff --git a/core-rs/epass-ir/src/logfmt.rs b/core-rs/epass-ir/src/logfmt.rs new file mode 100644 index 00000000..8b3dfa08 --- /dev/null +++ b/core-rs/epass-ir/src/logfmt.rs @@ -0,0 +1,56 @@ +//! The "dump" text format: one decimal `u64` per line, each the packed encoding +//! of a single [`BpfInsn`]. This is the machine-diffable format used by the C +//! tool's `readlog` input and `-F log` output, and is our primary validation +//! channel against the reference implementation. + +use crate::bytecode::BpfInsn; + +/// Parse a dump-format string into a list of instructions. +/// +/// Parsing stops at the first blank line (matching the C `readlog` behavior). +/// Lines that are not parseable as `u64` are skipped. +pub fn parse_dump(text: &str) -> Vec { + let mut insns = Vec::new(); + for line in text.lines() { + let trimmed = line.trim(); + if trimmed.is_empty() { + break; + } + match trimmed.parse::() { + Ok(raw) => insns.push(BpfInsn::from_u64(raw)), + Err(_) => continue, + } + } + insns +} + +/// Serialize instructions to dump format (one decimal `u64` per line). +pub fn to_dump(insns: &[BpfInsn]) -> String { + let mut out = String::new(); + for insn in insns { + out.push_str(&insn.to_u64().to_string()); + out.push('\n'); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn roundtrip_dump() { + let insns = vec![ + BpfInsn::from_u64(135), // r0 = 0 (mov) + BpfInsn::from_u64(149), // exit + ]; + let text = to_dump(&insns); + assert_eq!(parse_dump(&text), insns); + } + + #[test] + fn stops_at_blank_line() { + let text = "135\n149\n\n999\n"; + assert_eq!(parse_dump(text).len(), 2); + } +} diff --git a/core-rs/epass-ir/src/opts.rs b/core-rs/epass-ir/src/opts.rs new file mode 100644 index 00000000..a73d148f --- /dev/null +++ b/core-rs/epass-ir/src/opts.rs @@ -0,0 +1,83 @@ +//! Pipeline options for the userspace ePass library. + +/// How the BPF program should be rendered when printing. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum PrintMode { + /// Disassembled, human-readable BPF. + #[default] + Bpf, + /// Per-field detail table. + Detail, + /// Both disassembly and detail. + BpfDetail, + /// One packed `u64` per instruction (the machine-diffable "dump" format). + Dump, +} + +/// Options controlling lifting, transformation, and code generation. +#[derive(Debug, Clone)] +pub struct Opts { + /// Only print the program; perform no transformation. + pub print_only: bool, + /// Verbosity level (0 = quiet, higher = more log detail). + pub verbose: i32, + /// Emit the interference graph in Graphviz DOT format. + pub dotgraph: bool, + /// Skip the IR validity checker after each pass. + pub disable_prog_check: bool, + /// Disable register coalescing in code generation. + pub disable_coalesce: bool, + /// How to render printed programs. + pub print_mode: PrintMode, + /// Load initial IR from a text `.epir` file instead of lifting bytecode. + pub load_ir: Option, +} + +impl Default for Opts { + fn default() -> Self { + Self { + print_only: false, + verbose: 1, + dotgraph: false, + disable_prog_check: false, + disable_coalesce: false, + print_mode: PrintMode::Bpf, + load_ir: None, + } + } +} + +impl Opts { + /// Apply a comma-separated global-option string (`key` or `key=value`), + /// e.g. `"verbose=2,disable_coalesce"`. Unknown keys return an error string. + pub fn apply_gopt(&mut self, gopt: &str) -> Result<(), String> { + for tok in gopt.split(',').map(str::trim).filter(|s| !s.is_empty()) { + let (key, val) = match tok.split_once('=') { + Some((k, v)) => (k, Some(v)), + None => (tok, None), + }; + match key { + "verbose" => { + self.verbose = val + .and_then(|v| v.parse::().ok()) + .ok_or_else(|| "verbose requires an integer".to_string())?; + } + "disable_coalesce" => self.disable_coalesce = true, + "print_bpf" => self.print_mode = PrintMode::Bpf, + "print_dump" => self.print_mode = PrintMode::Dump, + "print_detail" => self.print_mode = PrintMode::Detail, + "print_bpf_detail" => self.print_mode = PrintMode::BpfDetail, + "no_prog_check" => self.disable_prog_check = true, + "printonly" => self.print_only = true, + "dotgraph" => self.dotgraph = true, + "load_ir" => { + self.load_ir = Some( + val.ok_or_else(|| "load_ir requires a path".to_string())?.to_string(), + ); + } + other => return Err(format!("unknown global option '{other}'")), + } + } + Ok(()) + } +} diff --git a/core-rs/epass-ir/src/pass.rs b/core-rs/epass-ir/src/pass.rs new file mode 100644 index 00000000..ce94334c --- /dev/null +++ b/core-rs/epass-ir/src/pass.rs @@ -0,0 +1,361 @@ +//! The pass framework: a [`Pass`] trait plus a [`PassManager`] that runs the +//! pre / custom / post pipeline, re-validating and re-laying-out the CFG after +//! each pass (matching the C `run_single_pass` / `bpf_ir_pass_postprocess`). + +use crate::env::{Env, LogLevel, Timer}; +use crate::error::Result; +use crate::ir::Function; +use crate::{cfg, check, internal, invalid, log_debug}; +use indexmap::IndexMap; + +/// A transformation or analysis over a function. +pub trait Pass { + /// A short, stable name (used for logging and pass options). + fn name(&self) -> &str; + + /// Whether this pass runs when absent from `--popt`. + fn enabled_by_default(&self) -> bool { + false + } + + /// Whether the pass can be disabled with `!pass_name`. + fn allow_disable(&self) -> bool { + true + } + + /// Initialize/configure this pass from its pass-option argument. + /// + /// Called once while building the pass manager, before ordering and before + /// any IR mutation. Most passes take no options and inherit this default. + fn init(&mut self, arg: Option<&str>) -> Result<()> { + if arg.is_some() { + return Err(invalid!("pass '{}' takes no options", self.name())); + } + Ok(()) + } + + /// Adjust this pass's position in the pass list. + /// + /// The pass manager verifies that the returned list only inserted, removed, + /// or moved entries with this pass's own name. Other passes' relative order + /// and multiplicity must remain unchanged. + fn register_pass(&self, order: Vec) -> Result> { + Ok(order) + } + + /// Run the pass. Mutating the function is allowed. + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()>; +} + +/// Recompute CFG metadata and validate the IR after a pass mutated it. +pub fn postprocess(env: &mut Env, func: &mut Function) -> Result<()> { + // Recompute successors from terminators, then chain layout + end blocks. + recompute_succs(func)?; + cfg::finalize(env, func)?; + drop_unreachable_edges(func); + prune_phi_inputs(func); + check::prog_check(env, func)?; + Ok(()) +} + +/// Re-derive each block's `succs` from its terminator's jump targets. +/// +/// Insertion/erasure during a pass can leave `succs` stale; the terminator's +/// `bb1`/`bb2` are the source of truth. +fn recompute_succs(func: &mut Function) -> Result<()> { + use crate::ir::InsnKind; + let bbs: Vec<_> = func.all_bbs.clone(); + for bb in bbs { + let Some(last) = func.bb(bb).last() else { + continue; + }; + let (b1, b2) = { + let insn = func.insn(last); + match &insn.kind { + InsnKind::CondJmp { .. } => (insn.bb1, insn.bb2), + InsnKind::Ja => (insn.bb1, None), + _ => continue, + } + }; + // Disconnect old successors, then reconnect from the terminator. + let old: Vec<_> = func.bb(bb).succs.clone(); + for s in old { + func.disconnect(bb, s); + } + if let Some(b1) = b1 { + func.connect(bb, b1); + } + if let Some(b2) = b2 { + func.connect(bb, b2); + } + } + Ok(()) +} + +/// Drop predecessor/successor edges that cross from the reachable subgraph to an +/// unreachable block after a CFG rewrite. The BB arena keeps unreachable blocks +/// around, but later analyses walk `preds`, so reachable blocks must not retain +/// stale predecessors from dead blocks. +fn drop_unreachable_edges(func: &mut Function) { + let reachable: std::collections::HashSet<_> = func.reachable_bbs.iter().copied().collect(); + let bbs = func.all_bbs.clone(); + for bb in bbs { + func.bb_mut(bb).preds.retain(|p| reachable.contains(p)); + if reachable.contains(&bb) { + func.bb_mut(bb).succs.retain(|s| reachable.contains(s)); + } + } +} + +/// Remove phi operands whose incoming block is no longer a predecessor after a +/// CFG rewrite. This keeps SSA edge uses aligned with the current CFG and lets +/// later phi-simplification see constants/trivial phis exposed by branch folding. +fn prune_phi_inputs(func: &mut Function) { + use crate::ir::InsnKind; + let bbs = func.reachable_bbs.clone(); + let reachable: std::collections::HashSet<_> = bbs.iter().copied().collect(); + for bb in bbs { + let preds: Vec<_> = func + .bb(bb) + .preds + .iter() + .copied() + .filter(|p| reachable.contains(p)) + .collect(); + let phis: Vec<_> = func + .bb(bb) + .insns + .iter() + .copied() + .take_while(|&id| matches!(func.insn(id).kind, InsnKind::Phi)) + .collect(); + for phi in phis { + let old = func.insn(phi).phi.clone(); + let mut new_phi = Vec::with_capacity(old.len()); + let mut removed_values = Vec::new(); + for entry in old { + if preds.contains(&entry.bb) { + new_phi.push(entry); + } else { + removed_values.push(entry.value); + } + } + for value in removed_values { + if !new_phi.iter().any(|entry| entry.value == value) { + func.remove_use(value, phi); + } + } + func.insn_mut(phi).phi = new_phi; + } + } +} + +/// Parsed pass-option entry. +#[derive(Debug, Clone, PartialEq, Eq)] +struct ParsedPassOpt { + name: String, + disabled: bool, + arg: Option, +} + +/// Runs an ordered list of passes. +#[derive(Default)] +pub struct PassManager { + passes: IndexMap>, + order: Vec, +} + +impl PassManager { + pub fn new() -> Self { + PassManager::default() + } + + /// Build a pass manager from available pass objects and a `--popt` string. + pub fn from_passes(mut passes: Vec>, popt: &str) -> Result { + let mut map: IndexMap> = IndexMap::new(); + for pass in passes.drain(..) { + let name = pass.name().to_string(); + if map.contains_key(&name) { + return Err(invalid!("duplicate pass registration '{name}'")); + } + map.insert(name, pass); + } + + let parsed = parse_popt(popt)?; + let mut enabled: IndexMap = map + .iter() + .map(|(name, pass)| (name.clone(), pass.enabled_by_default())) + .collect(); + let mut args: IndexMap> = map.keys().map(|name| (name.clone(), None)).collect(); + let mut explicit: IndexMap = map.keys().map(|name| (name.clone(), false)).collect(); + + for opt in parsed { + let Some(pass) = map.get(opt.name.as_str()) else { + return Err(invalid!("unknown pass '{}'", opt.name)); + }; + if opt.disabled { + if opt.arg.is_some() { + return Err(invalid!("disabled pass '{}' cannot have options", opt.name)); + } + if !pass.allow_disable() { + return Err(invalid!("pass '{}' cannot be disabled", opt.name)); + } + enabled.insert(opt.name.clone(), false); + explicit.insert(opt.name, true); + } else { + enabled.insert(opt.name.clone(), true); + args.insert(opt.name.clone(), opt.arg); + explicit.insert(opt.name, true); + } + } + + // Initialize enabled passes (default-enabled passes get None unless the + // user supplied an explicit option). + let keys: Vec = map.keys().cloned().collect(); + for name in &keys { + if *enabled.get(name).unwrap_or(&false) { + let arg = args.get(name).and_then(|x| x.as_deref()); + map.get_mut(name).unwrap().init(arg)?; + } + } + + let mut order: Vec = keys + .iter() + .filter(|name| *enabled.get(*name).unwrap_or(&false)) + .cloned() + .collect(); + + const MAX_ORDER_ITERS: usize = 32; + let enabled_names = order.clone(); + let mut stable = false; + for _ in 0..MAX_ORDER_ITERS { + let before_iter = order.clone(); + for name in &enabled_names { + if !order.iter().any(|n| n == name) { + // The pass removed itself; don't ask it to order again. + continue; + } + let pass = map.get(name).unwrap(); + let old = order.clone(); + let new = pass.register_pass(order)?; + verify_only_own_changes(pass.name(), &old, &new)?; + order = new; + } + if order == before_iter { + stable = true; + break; + } + } + if !stable { + return Err(internal!("pass registration order did not converge after {MAX_ORDER_ITERS} iterations")); + } + + Ok(PassManager { passes: map, order }) + } + + /// Add a pass directly to this manager, enabled at the end of the current order. + /// Useful for tests or custom callers that construct their own pipeline. + pub fn add_pass(&mut self, pass: Box) -> Result<()> { + let name = pass.name().to_string(); + if self.passes.contains_key(&name) { + return Err(invalid!("duplicate pass '{name}'")); + } + self.order.push(name.clone()); + self.passes.insert(name, pass); + Ok(()) + } + + fn run_one(&self, env: &mut Env, func: &mut Function, pass: &dyn Pass) -> Result<()> { + log_debug!(env, "------ Running Pass: {} ------\n", pass.name()); + pass.run(env, func)?; + postprocess(env, func)?; + if env.opts.verbose >= 2 { + let txt = crate::ir::print::print_function(func); + env.log(LogLevel::Debug, format_args!("{txt}")); + } + Ok(()) + } + + /// Run passes in the finalized order. + pub fn run(&self, env: &mut Env, func: &mut Function) -> Result<()> { + let timer = Timer::start(); + for name in &self.order { + let pass = self + .passes + .get(name.as_str()) + .ok_or_else(|| internal!("pass order references unknown pass '{name}'"))?; + self.run_one(env, func, pass.as_ref())?; + } + env.run_time_ns += timer.elapsed_ns(); + Ok(()) + } +} + +fn verify_only_own_changes(own: &str, old: &[String], new: &[String]) -> Result<()> { + let old_without: Vec<_> = old.iter().filter(|n| n.as_str() != own).cloned().collect(); + let new_without: Vec<_> = new.iter().filter(|n| n.as_str() != own).cloned().collect(); + if old_without != new_without { + return Err(internal!("pass '{own}' illegally modified other passes during registration")); + } + let own_count = new.iter().filter(|n| n.as_str() == own).count(); + if own_count > 1 { + return Err(internal!("pass '{own}' duplicated itself; duplicate pass instances are not supported yet")); + } + Ok(()) +} + +fn parse_popt(popt: &str) -> Result> { + let mut out = Vec::new(); + for item in split_top_level_commas(popt)? { + let item = item.trim(); + if item.is_empty() { + continue; + } + let (disabled, rest) = match item.strip_prefix('!') { + Some(r) => (true, r.trim()), + None => (false, item), + }; + let (name, arg) = if let Some(open) = rest.find('(') { + if !rest.ends_with(')') { + return Err(invalid!("unterminated pass option parentheses in '{item}'")); + } + let name = rest[..open].trim(); + let arg = rest[open + 1..rest.len() - 1].to_string(); + (name, Some(arg)) + } else { + (rest, None) + }; + if name.is_empty() { + return Err(invalid!("empty pass name in '{item}'")); + } + out.push(ParsedPassOpt { name: name.to_string(), disabled, arg }); + } + Ok(out) +} + +fn split_top_level_commas(s: &str) -> Result> { + let mut parts = Vec::new(); + let mut depth = 0i32; + let mut start = 0usize; + for (idx, ch) in s.char_indices() { + match ch { + '(' => depth += 1, + ')' => { + depth -= 1; + if depth < 0 { + return Err(invalid!("unmatched ')' in pass options")); + } + } + ',' if depth == 0 => { + parts.push(&s[start..idx]); + start = idx + 1; + } + _ => {} + } + } + if depth != 0 { + return Err(invalid!("unterminated pass option parentheses")); + } + parts.push(&s[start..]); + Ok(parts) +} diff --git a/core-rs/epass-ir/src/passes/const_prop.rs b/core-rs/epass-ir/src/passes/const_prop.rs new file mode 100644 index 00000000..31c7fb42 --- /dev/null +++ b/core-rs/epass-ir/src/passes/const_prop.rs @@ -0,0 +1,273 @@ +//! Conservative constant propagation and folding. +//! +//! This pass only folds pure SSA values with plain constants. It deliberately +//! avoids memory, calls, map/value immediates, stack-relative constants, and +//! builtin constants. The main CFG transform is folding conditional branches +//! whose operands are both known constants. + +use crate::env::Env; +use crate::error::Result; +use crate::ir::insn::{BinOp, Cond, InsnKind}; +use crate::ir::value::{AluOp, BuiltinConst, ConstKind, Value}; +use crate::ir::{Function, InsnId}; +use crate::pass::Pass; + +fn plain_const(v: Value) -> Option<(i64, AluOp)> { + match v { + Value::Const { + v, + ty, + kind: ConstKind::Plain, + builtin: BuiltinConst::None, + } => Some((v, ty)), + _ => None, + } +} + +fn const_value(v: i64, ty: AluOp) -> Value { + Value::Const { + v: match ty { + AluOp::Alu32 => (v as u32 as i32) as i64, + AluOp::Alu64 | AluOp::Unknown => v, + }, + ty, + kind: ConstKind::Plain, + builtin: BuiltinConst::None, + } +} + +fn fold_bin(op: BinOp, alu: AluOp, lhs: i64, rhs: i64) -> Option { + let width = if alu == AluOp::Alu32 { 32 } else { 64 }; + let sh = rhs as u64; + if matches!(op, BinOp::Lsh | BinOp::Rsh | BinOp::Arsh) && sh >= width { + return None; + } + + let out = match alu { + AluOp::Alu32 => { + let a = lhs as u32; + let b = rhs as u32; + let r = match op { + BinOp::Add => a.wrapping_add(b), + BinOp::Sub => a.wrapping_sub(b), + BinOp::Mul => a.wrapping_mul(b), + BinOp::Div => { + if b == 0 { return None; } + a.wrapping_div(b) + } + BinOp::Mod => { + if b == 0 { return None; } + a.wrapping_rem(b) + } + BinOp::Or => a | b, + BinOp::And => a & b, + BinOp::Xor => a ^ b, + BinOp::Lsh => a.wrapping_shl(sh as u32), + BinOp::Rsh => a.wrapping_shr(sh as u32), + BinOp::Arsh => ((a as i32) >> (sh as u32)) as u32, + }; + (r as i32) as i64 + } + AluOp::Alu64 | AluOp::Unknown => { + let a = lhs as u64; + let b = rhs as u64; + let r = match op { + BinOp::Add => a.wrapping_add(b), + BinOp::Sub => a.wrapping_sub(b), + BinOp::Mul => a.wrapping_mul(b), + BinOp::Div => { + if b == 0 { return None; } + a.wrapping_div(b) + } + BinOp::Mod => { + if b == 0 { return None; } + a.wrapping_rem(b) + } + BinOp::Or => a | b, + BinOp::And => a & b, + BinOp::Xor => a ^ b, + BinOp::Lsh => a.wrapping_shl(sh as u32), + BinOp::Rsh => a.wrapping_shr(sh as u32), + BinOp::Arsh => ((a as i64) >> (sh as u32)) as u64, + }; + r as i64 + } + }; + Some(const_value(out, alu)) +} + +fn fold_neg(alu: AluOp, v: i64) -> Value { + match alu { + AluOp::Alu32 => const_value((v as u32).wrapping_neg() as i32 as i64, AluOp::Alu32), + AluOp::Alu64 | AluOp::Unknown => const_value((v as u64).wrapping_neg() as i64, alu), + } +} + +fn eval_cond(cond: Cond, alu: AluOp, lhs: i64, rhs: i64) -> bool { + match cond { + Cond::Eq => lhs == rhs, + Cond::Ne => lhs != rhs, + Cond::Gt => { + if alu == AluOp::Alu32 { (lhs as u32) > (rhs as u32) } else { (lhs as u64) > (rhs as u64) } + } + Cond::Ge => { + if alu == AluOp::Alu32 { (lhs as u32) >= (rhs as u32) } else { (lhs as u64) >= (rhs as u64) } + } + Cond::Lt => { + if alu == AluOp::Alu32 { (lhs as u32) < (rhs as u32) } else { (lhs as u64) < (rhs as u64) } + } + Cond::Le => { + if alu == AluOp::Alu32 { (lhs as u32) <= (rhs as u32) } else { (lhs as u64) <= (rhs as u64) } + } + Cond::Sgt => { + if alu == AluOp::Alu32 { (lhs as i32) > (rhs as i32) } else { lhs > rhs } + } + Cond::Sge => { + if alu == AluOp::Alu32 { (lhs as i32) >= (rhs as i32) } else { lhs >= rhs } + } + Cond::Slt => { + if alu == AluOp::Alu32 { (lhs as i32) < (rhs as i32) } else { lhs < rhs } + } + Cond::Sle => { + if alu == AluOp::Alu32 { (lhs as i32) <= (rhs as i32) } else { lhs <= rhs } + } + } +} + +fn replace_with_const(func: &mut Function, id: InsnId, c: Value) { + func.replace_all_uses(id, c); + if func.is_alive(id) && func.insn(id).users.iter().all(|&u| u == id) { + // Drop any self-use first (possible for degenerate phis). + if matches!(func.insn(id).kind, InsnKind::Phi) { + let entries = func.insn(id).phi.clone(); + for entry in entries { + func.remove_use(entry.value, id); + } + func.insn_mut(id).phi.clear(); + } + func.erase_insn(id); + } +} + +pub fn const_prop(_env: &mut Env, func: &mut Function) -> Result<()> { + let mut changed = true; + while changed { + changed = false; + let insns: Vec = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .collect(); + + for id in insns { + if !func.is_alive(id) { + continue; + } + let kind = func.insn(id).kind.clone(); + match kind { + InsnKind::Assign => { + if let Some((v, ty)) = plain_const(func.insn(id).values[0]) { + replace_with_const(func, id, const_value(v, ty)); + changed = true; + } + } + InsnKind::Bin { op } => { + let vals = func.insn(id).values.clone(); + if vals.len() == 2 { + if let (Some((l, _)), Some((r, _))) = (plain_const(vals[0]), plain_const(vals[1])) { + if let Some(c) = fold_bin(op, func.insn(id).alu_op, l, r) { + replace_with_const(func, id, c); + changed = true; + } + } + } + } + InsnKind::Neg => { + if let Some((v, _)) = plain_const(func.insn(id).values[0]) { + let c = fold_neg(func.insn(id).alu_op, v); + replace_with_const(func, id, c); + changed = true; + } + } + InsnKind::Phi => { + let mut same: Option = None; + let mut foldable = true; + for entry in func.insn(id).phi.clone() { + let Some((v, ty)) = plain_const(entry.value) else { + foldable = false; + break; + }; + let c = const_value(v, ty); + if let Some(prev) = same { + if prev != c { + foldable = false; + break; + } + } else { + same = Some(c); + } + } + if foldable { + if let Some(c) = same { + replace_with_const(func, id, c); + changed = true; + } + } + } + InsnKind::CondJmp { cond } => { + let vals = func.insn(id).values.clone(); + if vals.len() == 2 { + if let (Some((l, _)), Some((r, _))) = (plain_const(vals[0]), plain_const(vals[1])) { + let taken = eval_cond(cond, func.insn(id).alu_op, l, r); + let target = if taken { func.insn(id).bb2 } else { func.insn(id).bb1 }; + if let Some(target) = target { + func.rewrite_cond_to_ja(id, target); + changed = true; + } + } + } + } + _ => {} + } + } + } + Ok(()) +} + +#[derive(Default)] +pub struct ConstPropPass; + +impl Pass for ConstPropPass { + fn name(&self) -> &str { + "const_prop" + } + + fn enabled_by_default(&self) -> bool { + true + } + + fn allow_disable(&self) -> bool { + true + } + + fn register_pass(&self, mut order: Vec) -> Result> { + let name = self.name(); + if let Some(pos) = order.iter().position(|p| p == name) { + let own = order.remove(pos); + if let Some(phi_pos) = order.iter().position(|p| p == "phi") { + order.insert(phi_pos, own); + } else { + order.push(own); + } + } + Ok(order) + } + + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()> { + const_prop(env, func) + } +} + +pub fn pass() -> impl Pass { + ConstPropPass +} diff --git a/core-rs/epass-ir/src/passes/dump_ir.rs b/core-rs/epass-ir/src/passes/dump_ir.rs new file mode 100644 index 00000000..f8cde174 --- /dev/null +++ b/core-rs/epass-ir/src/passes/dump_ir.rs @@ -0,0 +1,63 @@ +//! Optional IR dump pass. +//! +//! Enabled with pass option `dump_ir()` or `dump_ir(path=)`. +//! The pass orders itself first, so it dumps the IR immediately after lift (or +//! immediately after `load_ir`) before other passes mutate it. + +use crate::env::Env; +use crate::error::{Error, Result}; +use crate::ir::Function; +use crate::pass::Pass; +use crate::invalid; + +#[derive(Default)] +pub struct DumpIrPass { + path: Option, +} + +impl Pass for DumpIrPass { + fn name(&self) -> &str { + "dump_ir" + } + + fn enabled_by_default(&self) -> bool { + false + } + + fn allow_disable(&self) -> bool { + true + } + + fn init(&mut self, arg: Option<&str>) -> Result<()> { + let arg = arg.ok_or_else(|| invalid!("dump_ir requires a path: use dump_ir(/tmp/file.epir)"))?; + let path = arg.strip_prefix("path=").unwrap_or(arg).trim(); + if path.is_empty() { + return Err(invalid!("dump_ir path is empty")); + } + self.path = Some(path.to_string()); + Ok(()) + } + + fn register_pass(&self, mut order: Vec) -> Result> { + let name = self.name(); + order.retain(|p| p != name); + order.insert(0, name.to_string()); + Ok(order) + } + + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()> { + let Some(path) = self.path.as_ref() else { + return Ok(()); + }; + crate::ir::text::dump_function_to_file(func, path).map_err(|e| match e { + Error::Internal(msg) => Error::Internal(format!("dump_ir failed for '{path}': {msg}")), + other => other, + })?; + crate::log_info!(env, "ePass: dumped IR to {}\n", path); + Ok(()) + } +} + +pub fn pass() -> impl Pass { + DumpIrPass::default() +} diff --git a/core-rs/epass-ir/src/passes/mod.rs b/core-rs/epass-ir/src/passes/mod.rs new file mode 100644 index 00000000..552f6b87 --- /dev/null +++ b/core-rs/epass-ir/src/passes/mod.rs @@ -0,0 +1,6 @@ +//! Built-in passes. + +pub mod const_prop; +pub mod dump_ir; +pub mod optimization; +pub mod phi; diff --git a/core-rs/epass-ir/src/passes/optimization.rs b/core-rs/epass-ir/src/passes/optimization.rs new file mode 100644 index 00000000..aadf0b49 --- /dev/null +++ b/core-rs/epass-ir/src/passes/optimization.rs @@ -0,0 +1,172 @@ +//! `optimize_ir`: light-weight dead-code elimination over the IR. +//! +//! Two transforms (ported from the C `optimization.c`): +//! 1. `remove_no_user_insn` — iteratively erase value-producing instructions +//! (not void, not calls) that have no users. +//! 2. `remove_unused_alloc` — erase `alloc`s that are only ever stored to +//! (never loaded), together with those stores. +//! +//! In the Rust pipeline this is a default pass ordered after phi cleanup and +//! before code generation. + +use crate::env::Env; +use crate::error::Result; +use crate::ir::insn::InsnKind; +use crate::ir::{Function, InsnId}; +use crate::pass::Pass; + +/// Options controlling the optimizer (parsed from a pass-option string). +#[derive(Debug, Clone, Copy, Default)] +pub struct OptimizeOpts { + /// Skip dead-code (no-user) elimination. + pub no_dead_elim: bool, + /// Skip the whole pass. + pub no_opt: bool, +} + +impl OptimizeOpts { + /// Parse a pass-option string (`"no_dead_elim,noopt"` or `"no_dead_elim noopt"`). + pub fn parse(s: &str) -> Self { + let mut o = OptimizeOpts::default(); + for tok in s.split(|c: char| c == ',' || c.is_whitespace()).filter(|t| !t.is_empty()) { + match tok { + "no_dead_elim" => o.no_dead_elim = true, + "noopt" => o.no_opt = true, + _ => {} + } + } + o + } +} + +/// Iteratively erase value-producing instructions with no users. +fn remove_no_user_insn(func: &mut Function) { + loop { + let mut changed = false; + let candidates: Vec = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .collect(); + for id in candidates { + if !func.is_alive(id) { + continue; + } + let insn = func.insn(id); + if insn.is_void() || matches!(insn.kind, InsnKind::Call { .. } | InsnKind::Ecall) { + continue; + } + // A self-reference (e.g. an unsimplified phi) does not count as a use. + let live_users = insn.users.iter().any(|&u| u != id); + if !live_users { + func.erase_insn(id); + changed = true; + } + } + if !changed { + break; + } + } +} + +/// Erase `alloc`s that are never loaded (only stored), and their stores. +fn remove_unused_alloc(func: &mut Function) { + let allocs: Vec = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .filter(|&id| matches!(func.insn(id).kind, InsnKind::Alloc { .. })) + .collect(); + + for alloc in allocs { + if !func.is_alive(alloc) { + continue; + } + let users = func.insn(alloc).users.clone(); + let has_load = users + .iter() + .any(|&u| matches!(func.insn(u).kind, InsnKind::Load)); + if has_load { + continue; + } + // No loads: drop every store (the only remaining users), then the alloc. + for user in users { + if func.is_alive(user) { + // Detach the store's operand uses first so erase_insn's + // "no users" invariant holds. + func.erase_insn(user); + } + } + func.erase_insn(alloc); + } +} + +/// Run the optimizer with the given options. +pub fn optimize_ir_opts(env: &mut Env, func: &mut Function, opts: OptimizeOpts) -> Result<()> { + if opts.no_opt { + crate::log_debug!(env, "skip optimization\n"); + return Ok(()); + } + if !opts.no_dead_elim { + remove_no_user_insn(func); + } else { + crate::log_debug!(env, "skip remove_no_user_insn\n"); + } + remove_unused_alloc(func); + Ok(()) +} + +/// Run the optimizer with default options. +pub fn optimize_ir(env: &mut Env, func: &mut Function) -> Result<()> { + optimize_ir_opts(env, func, OptimizeOpts::default()) +} + +#[derive(Debug, Clone, Copy, Default)] +pub struct OptimizeIrPass { + opts: OptimizeOpts, +} + +impl Pass for OptimizeIrPass { + fn name(&self) -> &str { + "optimize_ir" + } + + fn enabled_by_default(&self) -> bool { + true + } + + fn allow_disable(&self) -> bool { + true + } + + fn init(&mut self, arg: Option<&str>) -> Result<()> { + self.opts = arg.map(OptimizeOpts::parse).unwrap_or_default(); + Ok(()) + } + + fn register_pass(&self, mut order: Vec) -> Result> { + let name = self.name(); + if let Some(pos) = order.iter().position(|p| p == name) { + let own = order.remove(pos); + // The optimizer should run after phi cleanup, and after const_prop if + // phi was disabled in a custom build. Put it behind the latest known + // cleanup pass, otherwise keep it at the end. + let insert_after = order + .iter() + .rposition(|p| p == "phi" || p == "const_prop") + .map(|i| i + 1) + .unwrap_or(order.len()); + order.insert(insert_after, own); + } + Ok(order) + } + + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()> { + optimize_ir_opts(env, func, self.opts) + } +} + +/// Construct the pass object. +pub fn pass() -> impl Pass { + OptimizeIrPass::default() +} diff --git a/core-rs/epass-ir/src/passes/phi.rs b/core-rs/epass-ir/src/passes/phi.rs new file mode 100644 index 00000000..2c136e49 --- /dev/null +++ b/core-rs/epass-ir/src/passes/phi.rs @@ -0,0 +1,103 @@ +//! `remove_trivial_phi`: iteratively delete phi nodes whose operands are all +//! identical (ignoring self-references), replacing uses with that single value. + +use crate::env::Env; +use crate::error::Result; +use crate::invalid; +use crate::ir::{Function, InsnId, InsnKind, Value}; +use crate::pass::Pass; + +/// Try to remove one trivial phi. Returns `Ok(true)` if it was removed. +fn try_remove(func: &mut Function, phi: InsnId) -> Result { + let mut same: Option = None; + for entry in func.insn(phi).phi.clone() { + let v = entry.value; + // Skip self-references and operands equal to the running unique value. + if let Value::Insn(def) = v { + if def == phi { + continue; + } + } + if Some(v) == same { + continue; + } + if same.is_some() { + // Two distinct non-self operands: not trivial. + return Ok(false); + } + same = Some(v); + } + + let Some(rep) = same else { + return Err(invalid!("phi instruction %{} has no operands", phi.0)); + }; + + // Drop our own uses of the phi operands before replacing/erasing. + for entry in func.insn(phi).phi.clone() { + func.remove_use(entry.value, phi); + } + func.insn_mut(phi).phi.clear(); + + func.replace_all_uses_except(phi, rep, Some(phi)); + func.erase_insn(phi); + Ok(true) +} + +/// Remove all trivial phis to a fixpoint. +pub fn remove_trivial_phi(_env: &mut Env, func: &mut Function) -> Result<()> { + let mut changed = true; + while changed { + changed = false; + let phis: Vec = func + .reachable_bbs + .iter() + .flat_map(|&bb| func.bb(bb).insns.clone()) + .filter(|&id| matches!(func.insn(id).kind, InsnKind::Phi)) + .collect(); + for phi in phis { + if func.is_alive(phi) { + changed |= try_remove(func, phi)?; + } + } + } + Ok(()) +} + +#[derive(Default)] +pub struct PhiPass; + +impl Pass for PhiPass { + fn name(&self) -> &str { + "phi" + } + + fn enabled_by_default(&self) -> bool { + true + } + + fn allow_disable(&self) -> bool { + false + } + + fn register_pass(&self, mut order: Vec) -> Result> { + let name = self.name(); + if let Some(pos) = order.iter().position(|p| p == name) { + let own = order.remove(pos); + if let Some(opt_pos) = order.iter().position(|p| p == "optimize_ir") { + order.insert(opt_pos, own); + } else { + order.push(own); + } + } + Ok(order) + } + + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()> { + remove_trivial_phi(env, func) + } +} + +/// Construct the pass object. +pub fn pass() -> impl Pass { + PhiPass +} diff --git a/core-rs/epass-ir/src/pipeline.rs b/core-rs/epass-ir/src/pipeline.rs new file mode 100644 index 00000000..94358c6b --- /dev/null +++ b/core-rs/epass-ir/src/pipeline.rs @@ -0,0 +1,50 @@ +//! The end-to-end driver: lift, run passes, and compile, mirroring the C +//! `bpf_ir_autorun`. + +use crate::cg; +use crate::env::Env; +use crate::error::Result; +use crate::lift; +use crate::pass::PassManager; +use crate::{log_debug, log_info}; + +fn initial_function(env: &mut Env) -> Result { + if let Some(path) = env.opts.load_ir.clone() { + let mut func = crate::ir::text::load_function_from_file(path)?; + crate::pass::postprocess(env, &mut func)?; + Ok(func) + } else { + lift::lift(env) + } +} + +/// Lift, run the given passes, and compile the program in `env.insns`, +/// writing the rewritten bytecode back into `env.insns`. +pub fn autorun(env: &mut Env, passes: &PassManager) -> Result<()> { + let len = env.insns.len(); + + if env.opts.print_only { + return Ok(()); + } + + let mut func = initial_function(env)?; + log_debug!(env, "{}", crate::ir::print::print_function(&func)); + + passes.run(env, &mut func)?; + cg::compile(env, &mut func)?; + + log_info!( + env, + "ePass: {} -> {} instructions\n", + len, + env.insns.len() + ); + Ok(()) +} + +/// Lift and run passes but skip code generation (the `--pass-only` path). +pub fn run_passes_only(env: &mut Env, passes: &PassManager) -> Result<()> { + let mut func = initial_function(env)?; + passes.run(env, &mut func)?; + Ok(()) +} diff --git a/core-rs/epass-ir/tests/builder.rs b/core-rs/epass-ir/tests/builder.rs new file mode 100644 index 00000000..3b2db7c1 --- /dev/null +++ b/core-rs/epass-ir/tests/builder.rs @@ -0,0 +1,38 @@ +use epass_ir::bytecode::{class, op, src, BpfInsn}; +use epass_ir::ir::{AluOp, BinOp, InsertPos, IrBuilder, Value}; +use epass_ir::{check, lift, Env, Opts}; + +fn env_minimal() -> Env { + Env::new( + Opts::default(), + vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ], + ) +} + +#[test] +fn builder_bin_updates_def_use() { + let mut env = env_minimal(); + let mut func = lift(&mut env).expect("lift"); + let bb = func.entry; + let ret = func.bb(bb).last().unwrap(); + let x = { + let mut b = IrBuilder::at_insn(&mut func, ret, InsertPos::Front); + b.bin(BinOp::Add, AluOp::Alu64, Value::const64(1), Value::const64(2)) + }; + assert!(func.insn(x).users.is_empty()); + check::prog_check(&env, &func).expect("check"); +} + +#[test] +fn builder_call_checks_arg_limit() { + let mut env = env_minimal(); + let mut func = lift(&mut env).expect("lift"); + let bb = func.entry; + let ret = func.bb(bb).last().unwrap(); + let mut b = IrBuilder::at_insn(&mut func, ret, InsertPos::Front); + let args = vec![Value::const64(0); 6]; + assert!(b.call(1, args).is_err()); +} diff --git a/core-rs/epass-ir/tests/cfg_utils.rs b/core-rs/epass-ir/tests/cfg_utils.rs new file mode 100644 index 00000000..25dce2a2 --- /dev/null +++ b/core-rs/epass-ir/tests/cfg_utils.rs @@ -0,0 +1,48 @@ +use epass_ir::bytecode::{class, op, src, BpfInsn}; +use epass_ir::ir::Value; +use epass_ir::{check, lift, Env, Opts}; + +fn branch_prog() -> Vec { + vec![ + BpfInsn::new(class::JMP | op::JEQ | src::K, 1, 0, 1, 0), + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 1), + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 2), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ] +} + +#[test] +fn split_block_before_moves_suffix_and_preserves_check() { + let mut env = Env::new(Opts::default(), vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ]); + let mut func = lift(&mut env).expect("lift"); + let entry = func.entry; + let ret = func.bb(entry).last().unwrap(); + let new_bb = func.split_block_before(ret).expect("split"); + assert!(func.bb(entry).succs.contains(&new_bb)); + assert!(func.bb(new_bb).preds.contains(&entry)); + check::prog_check(&env, &func).expect("check"); +} + +#[test] +fn split_edge_inserts_block_and_relabels_phi_preds() { + let mut env = Env::new(Opts::default(), branch_prog()); + let mut func = lift(&mut env).expect("lift"); + let from = func.entry; + let to = func.bb(from).succs[0]; + let mid = func.split_edge(from, to).expect("split edge"); + assert!(func.bb(from).succs.contains(&mid)); + assert!(func.bb(mid).succs.contains(&to)); + assert!(func.bb(to).preds.contains(&mid)); + assert!(!func.bb(to).preds.contains(&from)); +} + +#[test] +fn create_ret_block_builds_valid_block() { + let mut env = Env::new(Opts::default(), branch_prog()); + let mut func = lift(&mut env).expect("lift"); + let bb = func.create_ret_block(Value::const64(1)); + assert!(func.bb(bb).last().is_some()); +} diff --git a/core-rs/epass-ir/tests/compile.rs b/core-rs/epass-ir/tests/compile.rs new file mode 100644 index 00000000..4511e7d8 --- /dev/null +++ b/core-rs/epass-ir/tests/compile.rs @@ -0,0 +1,73 @@ +//! End-to-end compile tests: lift -> passes -> codegen, then sanity-check the +//! emitted bytecode (and that it round-trips through the dump format). + +use epass_ir::bytecode::{class, mode, op, size, src, BpfInsn}; +use epass_ir::{autorun, default_passes, logfmt, Env, Opts}; + +fn compile(prog: Vec) -> (Env, Vec) { + let mut opts = Opts::default(); + opts.verbose = 3; + let mut env = Env::new(opts, prog); + let passes = default_passes(); + autorun(&mut env, &passes).expect("autorun failed"); + let out = env.insns.clone(); + (env, out) +} + +#[test] +fn compile_minimal_exit() { + let prog = vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ]; + let (_env, out) = compile(prog); + assert!(!out.is_empty()); + // Must end with EXIT. + let last = *out.last().unwrap(); + assert_eq!(last.code, class::JMP | op::EXIT, "last insn should be exit"); +} + +#[test] +fn compile_add_and_return() { + // r1 = 5; r2 = 7; r1 += r2; r0 = r1; exit + let prog = vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 1, 0, 0, 5), + BpfInsn::new(class::ALU64 | op::MOV | src::K, 2, 0, 0, 7), + BpfInsn::new(class::ALU64 | op::ADD | src::X, 1, 2, 0, 0), + BpfInsn::new(class::ALU64 | op::MOV | src::X, 0, 1, 0, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ]; + let (_env, out) = compile(prog); + assert_eq!(out.last().unwrap().code, class::JMP | op::EXIT); + // Dump round-trips. + let dumped = logfmt::to_dump(&out); + assert_eq!(logfmt::parse_dump(&dumped), out); +} + +#[test] +fn compile_loop() { + // counting loop + let prog = vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 1, 0, 0, 0), // r1 = 0 + BpfInsn::new(class::ALU64 | op::ADD | src::K, 1, 0, 0, 1), // r1 += 1 + BpfInsn::new(class::JMP | op::JLT | src::K, 1, 0, -2, 10), // if r1<10 goto -2 + BpfInsn::new(class::ALU64 | op::MOV | src::X, 0, 1, 0, 0), // r0 = r1 + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ]; + let (_env, out) = compile(prog); + assert!(out.iter().any(|i| class::JMP == (i.code & 0x07) && (i.code & 0xf0) == op::JLT + || (i.code & 0x07) == class::JMP32 && (i.code & 0xf0) == op::JLT + || (i.code & 0xf0) == op::JGT)); + assert_eq!(out.last().unwrap().code, class::JMP | op::EXIT); +} + +#[test] +fn compile_stack_load_store() { + let prog = vec![ + BpfInsn::new(class::STX | mode::MEM | size::DW, 10, 1, -8, 0), + BpfInsn::new(class::LDX | mode::MEM | size::DW, 0, 10, -8, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ]; + let (_env, out) = compile(prog); + assert_eq!(out.last().unwrap().code, class::JMP | op::EXIT); +} diff --git a/core-rs/epass-ir/tests/ir_text.rs b/core-rs/epass-ir/tests/ir_text.rs new file mode 100644 index 00000000..31e9cbfd --- /dev/null +++ b/core-rs/epass-ir/tests/ir_text.rs @@ -0,0 +1,34 @@ +use epass_ir::bytecode::{class, op, src, BpfInsn}; +use epass_ir::{autorun, default_passes, dump_ir, lift, load_ir_str, Env, Opts}; + +fn minimal_prog() -> Vec { + vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ] +} + +#[test] +fn dump_load_roundtrip_compiles() { + let mut env = Env::new(Opts::default(), minimal_prog()); + let func = lift(&mut env).expect("lift"); + let text = dump_ir(&func); + let func2 = load_ir_str(&text).expect("load ir"); + let text2 = dump_ir(&func2); + assert!(text2.contains("ret"), "{text2}"); +} + +#[test] +fn load_ir_gopt_bypasses_lift_and_compiles() { + let mut env = Env::new(Opts::default(), minimal_prog()); + let func = lift(&mut env).expect("lift"); + let text = dump_ir(&func); + let path = std::env::temp_dir().join("epass_ir_text_test.epir"); + std::fs::write(&path, text).unwrap(); + + let mut opts = Opts::default(); + opts.apply_gopt(&format!("load_ir={}", path.display())).unwrap(); + let mut env2 = Env::new(opts, vec![BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0)]); + autorun(&mut env2, &default_passes()).expect("autorun load_ir"); + assert_eq!(env2.insns.last().unwrap().code, class::JMP | op::EXIT); +} diff --git a/core-rs/epass-ir/tests/lift.rs b/core-rs/epass-ir/tests/lift.rs new file mode 100644 index 00000000..a66efeda --- /dev/null +++ b/core-rs/epass-ir/tests/lift.rs @@ -0,0 +1,89 @@ +//! End-to-end lifter tests: build small eBPF programs, lift them, run the +//! trivial-phi pass, and check the resulting IR is well-formed and prints. + +use epass_ir::bytecode::{op, src, BpfInsn}; +use epass_ir::bytecode::{class, size, mode}; +use epass_ir::{check, lift, Env, Opts}; + +fn env_with(insns: Vec) -> Env { + let mut opts = Opts::default(); + opts.verbose = 3; + Env::new(opts, insns) +} + +#[test] +fn lift_minimal_exit() { + // r0 = 0; exit + let prog = vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ]; + let mut env = env_with(prog); + let func = lift(&mut env).expect("lift failed"); + check::prog_check(&env, &func).expect("prog_check failed"); + let txt = epass_ir::print_ir(&func); + assert!(txt.contains("ret"), "should contain a ret:\n{txt}"); +} + +#[test] +fn lift_branch_creates_two_blocks() { + // if r1 == 0 goto +1; r0 = 1; r0 = 2; exit + let prog = vec![ + BpfInsn::new(class::JMP | op::JEQ | src::K, 1, 0, 1, 0), // 0: jeq r1, 0 -> +1 + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 1), // 1: r0 = 1 + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 2), // 2: r0 = 2 + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), // 3: exit + ]; + let mut env = env_with(prog); + let func = lift(&mut env).expect("lift failed"); + check::prog_check(&env, &func).expect("prog_check failed"); + // Entry + fallthrough + target(merge) = at least 3 reachable blocks. + assert!(func.reachable_bbs.len() >= 2); +} + +#[test] +fn lift_loop_with_phi_then_simplify() { + // A simple counting loop that requires a phi for r1. + // r1 = 0 + // loop: + // r1 += 1 + // if r1 < 10 goto loop + // r0 = r1 + // exit + let prog = vec![ + BpfInsn::new(class::ALU64 | op::MOV | src::K, 1, 0, 0, 0), // 0: r1 = 0 + BpfInsn::new(class::ALU64 | op::ADD | src::K, 1, 0, 0, 1), // 1: r1 += 1 + BpfInsn::new(class::JMP | op::JLT | src::K, 1, 0, -2, 10), // 2: if r1 < 10 goto 1 + BpfInsn::new(class::ALU64 | op::MOV | src::X, 0, 1, 0, 0), // 3: r0 = r1 + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), // 4: exit + ]; + let mut env = env_with(prog); + let mut func = lift(&mut env).expect("lift failed"); + check::prog_check(&env, &func).expect("prog_check after lift failed"); + + let txt_before = epass_ir::print_ir(&func); + assert!(txt_before.contains("phi"), "loop should produce a phi:\n{txt_before}"); + + // Run the trivial-phi pass through the manager (also re-validates). + let mut pm = epass_ir::PassManager::new(); + pm.add_pass(Box::new(epass_ir::passes::phi::pass())).expect("add pass"); + pm.run(&mut env, &mut func).expect("phi pass failed"); + check::prog_check(&env, &func).expect("prog_check after phi failed"); +} + +#[test] +fn lift_load_store_stack() { + // *(u64 *)(r10 - 8) = r1; r2 = *(u64 *)(r10 - 8); r0 = 0; exit + let prog = vec![ + BpfInsn::new(class::STX | mode::MEM | size::DW, 10, 1, -8, 0), + BpfInsn::new(class::LDX | mode::MEM | size::DW, 2, 10, -8, 0), + BpfInsn::new(class::ALU64 | op::MOV | src::K, 0, 0, 0, 0), + BpfInsn::new(class::JMP | op::EXIT, 0, 0, 0, 0), + ]; + let mut env = env_with(prog); + let func = lift(&mut env).expect("lift failed"); + check::prog_check(&env, &func).expect("prog_check failed"); + let txt = epass_ir::print_ir(&func); + assert!(txt.contains("storeraw"), "{txt}"); + assert!(txt.contains("loadraw"), "{txt}"); +} diff --git a/core-rs/epasstool/Cargo.toml b/core-rs/epasstool/Cargo.toml new file mode 100644 index 00000000..e6e79bf9 --- /dev/null +++ b/core-rs/epasstool/Cargo.toml @@ -0,0 +1,14 @@ +[package] +name = "epasstool" +version.workspace = true +edition.workspace = true +license.workspace = true +description = "Userspace CLI for the ePass eBPF IR compiler." + +[[bin]] +name = "epasstool" +path = "src/main.rs" + +[dependencies] +epass-ir = { path = "../epass-ir" } +libbpf-sys = "1" diff --git a/core-rs/epasstool/src/cli.rs b/core-rs/epasstool/src/cli.rs new file mode 100644 index 00000000..0cf75983 --- /dev/null +++ b/core-rs/epasstool/src/cli.rs @@ -0,0 +1,113 @@ +//! Command-line parsing, mirroring the original C tool's interface. + +use epass_ir::Opts; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Mode { + Read, + Print, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum OutputFormat { + /// Raw section bytes. + Sec, + /// Dump format (one `u64` per line). + Log, +} + +#[derive(Debug, Clone)] +pub struct UserOpts { + pub mode: Mode, + pub prog: String, + pub prog_out: Option, + pub output_format: OutputFormat, + pub section: Option, + pub no_compile: bool, + pub gopt: String, + pub popt: String, + pub opts: Opts, +} + +pub enum CliError { + Usage, + Message(String), +} + +pub fn print_usage() { + eprintln!( + "Usage: epasstool [options] \n\n\ +Commands:\n\ +\x20 read Read (lift, transform and compile) the specified file\n\ +\x20 print Print the specified file\n\n\ +Options:\n\ +\x20 --pass-only, -P Skip compilation\n\ +\x20 --gopt Specify a global option (comma-separated)\n\ +\x20 --popt Specify a pass option\n\ +\x20 --sec, -s Specify the ELF section/program name\n\ +\x20 -F Output format: sec | log (default)\n\ +\x20 -o Write the modified program to a file\n\n\ +Global options (--gopt):\n\ +\x20 verbose= Set verbosity level\n\ +\x20 disable_coalesce Disable register coalescing\n\ +\x20 print_bpf Print disassembled BPF (default)\n\ +\x20 print_dump Print packed u64 dump\n\ +\x20 print_detail Print detailed per-field view\n\ +\x20 no_prog_check Disable the IR validity checker\n" + ); +} + +pub fn parse>(mut args: I) -> Result { + let cmd = args.next().ok_or(CliError::Usage)?; + let mode = match cmd.as_str() { + "read" => Mode::Read, + "print" => Mode::Print, + _ => return Err(CliError::Usage), + }; + + let mut uo = UserOpts { + mode, + prog: String::new(), + prog_out: None, + output_format: OutputFormat::Log, + section: None, + no_compile: false, + gopt: String::new(), + popt: String::new(), + opts: Opts::default(), + }; + + while let Some(arg) = args.next() { + match arg.as_str() { + "--pass-only" | "-P" if mode == Mode::Read => uo.no_compile = true, + "--gopt" => uo.gopt = args.next().ok_or(CliError::Usage)?, + "--popt" => uo.popt = args.next().ok_or(CliError::Usage)?, + "--sec" | "-s" => uo.section = Some(args.next().ok_or(CliError::Usage)?), + "-o" if mode == Mode::Read => uo.prog_out = Some(args.next().ok_or(CliError::Usage)?), + "-F" if mode == Mode::Read => { + let f = args.next().ok_or(CliError::Usage)?; + uo.output_format = match f.as_str() { + "sec" => OutputFormat::Sec, + "log" => OutputFormat::Log, + _ => return Err(CliError::Usage), + }; + } + other if !other.starts_with('-') => { + if uo.prog.is_empty() { + uo.prog = other.to_string(); + } else { + return Err(CliError::Usage); + } + } + _ => return Err(CliError::Usage), + } + } + + if uo.prog.is_empty() { + return Err(CliError::Usage); + } + + let gopt = uo.gopt.clone(); + uo.opts.apply_gopt(&gopt).map_err(CliError::Message)?; + Ok(uo) +} diff --git a/core-rs/epasstool/src/elf.rs b/core-rs/epasstool/src/elf.rs new file mode 100644 index 00000000..e40b381f --- /dev/null +++ b/core-rs/epasstool/src/elf.rs @@ -0,0 +1,111 @@ +//! ELF object input/output via libbpf (raw FFI through `libbpf-sys`). +//! +//! We read each program's instruction stream, run it through ePass, and +//! (optionally) write the rewritten instructions back into the object. + +use std::ffi::{CStr, CString}; + +use epass_ir::BpfInsn; +use libbpf_sys as bpf; + +use crate::cli::{Mode, UserOpts}; +use crate::run; + +type Error = Box; + +/// Convert a libbpf `bpf_insn` to our representation via its packed bits. +fn from_lib(insn: &bpf::bpf_insn) -> BpfInsn { + // Reconstruct the packed u64 from the C bitfields. + // libbpf-sys exposes dst_reg/src_reg as a single byte via bitfields; we + // read the raw bytes to be layout-independent. + let raw: u64 = unsafe { std::mem::transmute_copy(insn) }; + BpfInsn::from_u64(raw) +} + +fn to_lib(insn: &BpfInsn) -> bpf::bpf_insn { + let raw = insn.to_u64(); + unsafe { std::mem::transmute_copy(&raw) } +} + +pub fn run_elf(uo: &UserOpts) -> Result<(), Error> { + let path = CString::new(uo.prog.as_str())?; + let obj = unsafe { bpf::bpf_object__open(path.as_ptr()) }; + if obj.is_null() { + return Err(format!("failed to open ELF object '{}'", uo.prog).into()); + } + let result = process_object(uo, obj); + unsafe { bpf::bpf_object__close(obj) }; + result +} + +fn process_object(uo: &UserOpts, obj: *mut bpf::bpf_object) -> Result<(), Error> { + let want_section = uo.section.clone(); + let mut prog = unsafe { bpf::bpf_object__next_program(obj, std::ptr::null_mut()) }; + let mut processed_any = false; + let mut total = 0usize; + + while !prog.is_null() { + let name = unsafe { + let n = bpf::bpf_program__name(prog); + if n.is_null() { + String::new() + } else { + CStr::from_ptr(n).to_string_lossy().into_owned() + } + }; + + // If a specific section was requested, skip non-matching programs. + let selected = match &want_section { + Some(s) => &name == s, + None => true, + }; + + if selected { + let cnt = unsafe { bpf::bpf_program__insn_cnt(prog) } as usize; + let insns_ptr = unsafe { bpf::bpf_program__insns(prog) }; + let mut prog_insns = Vec::with_capacity(cnt); + for i in 0..cnt { + let insn = unsafe { &*insns_ptr.add(i) }; + prog_insns.push(from_lib(insn)); + } + total += cnt; + processed_any = true; + + let out = run::process(uo, prog_insns)?; + + // Write back into the program (read mode only, when modified). + if uo.mode == Mode::Read { + let lib_insns: Vec = out.iter().map(to_lib).collect(); + let rc = unsafe { + bpf::bpf_program__set_insns( + prog, + lib_insns.as_ptr() as *mut bpf::bpf_insn, + lib_insns.len() as bpf::size_t, + ) + }; + if rc != 0 { + return Err(format!("bpf_program__set_insns failed ({rc})").into()); + } + } + + // Write the output file if requested (last selected program wins, + // matching the per-program behavior of the C tool). + if let Some(path) = &uo.prog_out { + run::write_output(uo, path, &out)?; + } + } + + prog = unsafe { bpf::bpf_object__next_program(obj, prog) }; + // If a section was explicitly requested, stop after the first program + // when not iterating all. + if want_section.is_some() && selected { + break; + } + } + + if !processed_any { + return Err("no matching program found in object".into()); + } + eprintln!("processed {total} instructions"); + Ok(()) +} diff --git a/core-rs/epasstool/src/main.rs b/core-rs/epasstool/src/main.rs new file mode 100644 index 00000000..f5bae0ce --- /dev/null +++ b/core-rs/epasstool/src/main.rs @@ -0,0 +1,36 @@ +//! ePass userspace CLI. +//! +//! Commands: +//! read — lift, transform, and compile a program, then write it back +//! print — print a program without transforming it +//! +//! Inputs may be ELF object files (parsed via libbpf) or dump-format text files +//! (one packed `u64` per line). Options mirror the original C tool. + +mod cli; +mod elf; +mod run; + +use std::process::ExitCode; + +fn main() -> ExitCode { + let opts = match cli::parse(std::env::args().skip(1)) { + Ok(o) => o, + Err(cli::CliError::Usage) => { + cli::print_usage(); + return ExitCode::from(2); + } + Err(cli::CliError::Message(m)) => { + eprintln!("error: {m}"); + return ExitCode::from(2); + } + }; + + match run::run(opts) { + Ok(()) => ExitCode::SUCCESS, + Err(e) => { + eprintln!("error: {e}"); + ExitCode::FAILURE + } + } +} diff --git a/core-rs/epasstool/src/run.rs b/core-rs/epasstool/src/run.rs new file mode 100644 index 00000000..1476705d --- /dev/null +++ b/core-rs/epasstool/src/run.rs @@ -0,0 +1,106 @@ +//! Command dispatch: read/print on ELF or dump-format inputs. + +use std::fs; +use std::io::Read; + +use epass_ir::{autorun, logfmt, passes_from_popt, run_passes_only, BpfInsn, Env, LogLevel, PrintMode}; + +use crate::cli::{Mode, OutputFormat, UserOpts}; +use crate::elf; + +type Error = Box; + +/// Detect whether a file begins with the ELF magic. +fn is_elf(path: &str) -> bool { + let mut buf = [0u8; 4]; + if let Ok(mut f) = fs::File::open(path) { + if f.read_exact(&mut buf).is_ok() { + return buf == [0x7f, b'E', b'L', b'F']; + } + } + false +} + +pub fn run(uo: UserOpts) -> Result<(), Error> { + if is_elf(&uo.prog) { + elf::run_elf(&uo) + } else { + run_logfile(&uo) + } +} + +/// Read a dump-format text file and process it. +fn run_logfile(uo: &UserOpts) -> Result<(), Error> { + let text = fs::read_to_string(&uo.prog)?; + let prog = logfmt::parse_dump(&text); + let out = process(uo, prog)?; + if let Some(path) = &uo.prog_out { + write_output(uo, path, &out)?; + } + Ok(()) +} + +/// Run the pipeline (or just print) on one program, returning the result. +pub fn process(uo: &UserOpts, prog: Vec) -> Result, Error> { + let mut env = Env::new(uo.opts.clone(), prog.clone()); + + match uo.mode { + Mode::Print => { + print_program(&mut env, &prog); + flush_log(&env); + Ok(prog) + } + Mode::Read => { + let passes = passes_from_popt(&uo.popt)?; + let result = if uo.no_compile { + run_passes_only(&mut env, &passes).map(|_| prog.clone()) + } else { + autorun(&mut env, &passes).map(|_| env.insns.clone()) + }; + flush_log(&env); + Ok(result?) + } + } +} + +fn print_program(env: &mut Env, prog: &[BpfInsn]) { + match env.opts.print_mode { + PrintMode::Dump => { + for insn in prog { + env.log(LogLevel::Error, format_args!("{}\n", insn.to_u64())); + } + } + _ => { + for (i, insn) in prog.iter().enumerate() { + env.log( + LogLevel::Error, + format_args!( + "[{i}] code={:#04x} dst=r{} src=r{} off={} imm={}\n", + insn.code, insn.dst_reg, insn.src_reg, insn.off, insn.imm + ), + ); + } + } + } +} + +pub(crate) fn write_output(uo: &UserOpts, path: &str, out: &[BpfInsn]) -> Result<(), Error> { + match uo.output_format { + OutputFormat::Log => fs::write(path, logfmt::to_dump(out))?, + OutputFormat::Sec => { + let mut bytes = Vec::with_capacity(out.len() * 8); + for insn in out { + bytes.extend_from_slice(&insn.to_u64().to_le_bytes()); + } + fs::write(path, bytes)?; + } + } + Ok(()) +} + +fn flush_log(env: &Env) { + let log = env.log_buffer(); + if !log.is_empty() { + print!("{log}"); + } +} diff --git a/core-rs/epasstool/tests/bpftests.rs b/core-rs/epasstool/tests/bpftests.rs new file mode 100644 index 00000000..945dc375 --- /dev/null +++ b/core-rs/epasstool/tests/bpftests.rs @@ -0,0 +1,104 @@ +use std::path::{Path, PathBuf}; +use std::process::Command; + +fn have_cmd(cmd: &str) -> bool { + Command::new("sh") + .arg("-c") + .arg(format!("command -v {cmd} >/dev/null 2>&1")) + .status() + .map(|s| s.success()) + .unwrap_or(false) +} + +fn uname_m() -> Option { + let out = Command::new("uname").arg("-m").output().ok()?; + if !out.status.success() { + return None; + } + Some(String::from_utf8_lossy(&out.stdout).trim().to_string()) +} + +fn compile_bpf(src: &Path, out: &Path) -> bool { + let arch = uname_m().unwrap_or_else(|| "x86_64".to_string()); + Command::new("clang") + .arg("-O2") + .arg("-target") + .arg("bpf") + .arg("-I") + .arg(format!("/usr/include/{arch}-linux-gnu")) + .arg("-c") + .arg(src) + .arg("-o") + .arg(out) + .status() + .map(|s| s.success()) + .unwrap_or(false) +} + +#[test] +fn bpftests_compile_and_rewrite_allowlist() { + if !have_cmd("clang") { + eprintln!("skipping bpftests: clang not available"); + return; + } + + let manifest = PathBuf::from(env!("CARGO_MANIFEST_DIR")); + let bpftests = manifest.parent().unwrap().join("bpftests"); + let cases = [ + "empty.c", + "exit.c", + "simple1.c", + "simple2.c", + "alu64.c", + "mem2.c", + "loop2.c", + "loop3.c", + "spillconst.c", + "test_spill.c", + ]; + + let tmp = std::env::temp_dir().join(format!("epass-bpftests-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&tmp); + std::fs::create_dir_all(&tmp).unwrap(); + + let epasstool = env!("CARGO_BIN_EXE_epasstool"); + let mut compiled = 0usize; + let mut rewritten = 0usize; + + for case in cases { + let src = bpftests.join(case); + if !src.exists() { + eprintln!("skipping missing bpftest {case}"); + continue; + } + let obj = tmp.join(format!("{}.o", case.trim_end_matches(".c"))); + if !compile_bpf(&src, &obj) { + eprintln!("skipping bpftest {case}: clang failed"); + continue; + } + compiled += 1; + + let out = tmp.join(format!("{}.dump", case.trim_end_matches(".c"))); + let status = Command::new(epasstool) + .arg("read") + .arg("-s") + .arg("prog") + .arg("-F") + .arg("log") + .arg("-o") + .arg(&out) + .arg(&obj) + .status() + .expect("run epasstool"); + assert!(status.success(), "epasstool failed on {case}"); + assert!(out.metadata().map(|m| m.len() > 0).unwrap_or(false), "empty dump for {case}"); + rewritten += 1; + } + + let _ = std::fs::remove_dir_all(&tmp); + if compiled == 0 { + eprintln!("skipping bpftests: no allowlisted C tests compiled in this environment"); + return; + } + assert_eq!(compiled, rewritten); +} diff --git a/core/.clang-format b/deprecated/core/.clang-format similarity index 100% rename from core/.clang-format rename to deprecated/core/.clang-format diff --git a/core/.gitignore b/deprecated/core/.gitignore similarity index 100% rename from core/.gitignore rename to deprecated/core/.gitignore diff --git a/core/.vscode/launch.json b/deprecated/core/.vscode/launch.json similarity index 100% rename from core/.vscode/launch.json rename to deprecated/core/.vscode/launch.json diff --git a/core/.vscode/settings.json b/deprecated/core/.vscode/settings.json similarity index 100% rename from core/.vscode/settings.json rename to deprecated/core/.vscode/settings.json diff --git a/core/Bugs.md b/deprecated/core/Bugs.md similarity index 100% rename from core/Bugs.md rename to deprecated/core/Bugs.md diff --git a/core/CMakeLists.txt b/deprecated/core/CMakeLists.txt similarity index 100% rename from core/CMakeLists.txt rename to deprecated/core/CMakeLists.txt diff --git a/core/Makefile b/deprecated/core/Makefile similarity index 100% rename from core/Makefile rename to deprecated/core/Makefile diff --git a/core/array.c b/deprecated/core/array.c similarity index 100% rename from core/array.c rename to deprecated/core/array.c diff --git a/core/aux/cg_prog_check.c b/deprecated/core/aux/cg_prog_check.c similarity index 100% rename from core/aux/cg_prog_check.c rename to deprecated/core/aux/cg_prog_check.c diff --git a/core/aux/disasm.c b/deprecated/core/aux/disasm.c similarity index 100% rename from core/aux/disasm.c rename to deprecated/core/aux/disasm.c diff --git a/core/aux/ir_utils.c b/deprecated/core/aux/ir_utils.c similarity index 100% rename from core/aux/ir_utils.c rename to deprecated/core/aux/ir_utils.c diff --git a/core/aux/kern_utils.c b/deprecated/core/aux/kern_utils.c similarity index 100% rename from core/aux/kern_utils.c rename to deprecated/core/aux/kern_utils.c diff --git a/core/aux/prog_check.c b/deprecated/core/aux/prog_check.c similarity index 100% rename from core/aux/prog_check.c rename to deprecated/core/aux/prog_check.c diff --git a/core/bpf_ir.c b/deprecated/core/bpf_ir.c similarity index 100% rename from core/bpf_ir.c rename to deprecated/core/bpf_ir.c diff --git a/core/epasstool/CMakeLists.txt b/deprecated/core/epasstool/CMakeLists.txt similarity index 100% rename from core/epasstool/CMakeLists.txt rename to deprecated/core/epasstool/CMakeLists.txt diff --git a/core/epasstool/epasstool.c b/deprecated/core/epasstool/epasstool.c similarity index 100% rename from core/epasstool/epasstool.c rename to deprecated/core/epasstool/epasstool.c diff --git a/core/epasstool/epasstool.h b/deprecated/core/epasstool/epasstool.h similarity index 100% rename from core/epasstool/epasstool.h rename to deprecated/core/epasstool/epasstool.h diff --git a/core/epasstool/print.c b/deprecated/core/epasstool/print.c similarity index 100% rename from core/epasstool/print.c rename to deprecated/core/epasstool/print.c diff --git a/core/epasstool/printlog.c b/deprecated/core/epasstool/printlog.c similarity index 100% rename from core/epasstool/printlog.c rename to deprecated/core/epasstool/printlog.c diff --git a/core/epasstool/read.c b/deprecated/core/epasstool/read.c similarity index 100% rename from core/epasstool/read.c rename to deprecated/core/epasstool/read.c diff --git a/core/epasstool/readlog.c b/deprecated/core/epasstool/readlog.c similarity index 100% rename from core/epasstool/readlog.c rename to deprecated/core/epasstool/readlog.c diff --git a/core/epasstool/test_pass1.c b/deprecated/core/epasstool/test_pass1.c similarity index 100% rename from core/epasstool/test_pass1.c rename to deprecated/core/epasstool/test_pass1.c diff --git a/core/include/ir.h b/deprecated/core/include/ir.h similarity index 100% rename from core/include/ir.h rename to deprecated/core/include/ir.h diff --git a/core/include/ir_cg.h b/deprecated/core/include/ir_cg.h similarity index 100% rename from core/include/ir_cg.h rename to deprecated/core/include/ir_cg.h diff --git a/core/include/linux/bpf.h b/deprecated/core/include/linux/bpf.h similarity index 100% rename from core/include/linux/bpf.h rename to deprecated/core/include/linux/bpf.h diff --git a/core/include/linux/bpf_ir.h b/deprecated/core/include/linux/bpf_ir.h similarity index 95% rename from core/include/linux/bpf_ir.h rename to deprecated/core/include/linux/bpf_ir.h index e729b56f..bdb2aa0d 100644 --- a/core/include/linux/bpf_ir.h +++ b/deprecated/core/include/linux/bpf_ir.h @@ -1058,36 +1058,33 @@ struct builtin_pass_cfg { }; #define DEF_CUSTOM_PASS(pass_def, check_applyc, param_loadc, param_unloadc) \ - { \ - .pass = pass_def, .param = NULL, .param_load = param_loadc, \ - .param_unload = param_unloadc, .check_apply = check_applyc \ - } + { .pass = pass_def, \ + .param = NULL, \ + .param_load = param_loadc, \ + .param_unload = param_unloadc, \ + .check_apply = check_applyc } #define DEF_BUILTIN_PASS_CFG(namec, param_loadc, param_unloadc) \ - { \ - .name = namec, .param = NULL, .enable = false, \ - .enable_cfg = false, .param_load = param_loadc, \ - .param_unload = param_unloadc \ - } + { .name = namec, \ + .param = NULL, \ + .enable = false, \ + .enable_cfg = false, \ + .param_load = param_loadc, \ + .param_unload = param_unloadc } #define DEF_BUILTIN_PASS_ENABLE_CFG(namec, param_loadc, param_unloadc) \ - { \ - .name = namec, .param = NULL, .enable = true, \ - .enable_cfg = false, .param_load = param_loadc, \ - .param_unload = param_unloadc \ - } + { .name = namec, \ + .param = NULL, \ + .enable = true, \ + .enable_cfg = false, \ + .param_load = param_loadc, \ + .param_unload = param_unloadc } -#define DEF_FUNC_PASS(fun, msg, en_def) \ - { \ - .pass = fun, .name = msg, .enabled = en_def, \ - .force_enable = false \ - } +#define DEF_FUNC_PASS(fun, msg, en_def) \ + { .pass = fun, .name = msg, .enabled = en_def, .force_enable = false } -#define DEF_NON_OVERRIDE_FUNC_PASS(fun, msg) \ - { \ - .pass = fun, .name = msg, .enabled = true, \ - .force_enable = true \ - } +#define DEF_NON_OVERRIDE_FUNC_PASS(fun, msg) \ + { .pass = fun, .name = msg, .enabled = true, .force_enable = true } void insn_counter(struct bpf_ir_env *env, struct ir_function *fun, void *param); @@ -1131,11 +1128,10 @@ struct ir_value bpf_ir_value_stack_ptr(struct ir_function *fun); struct ir_value bpf_ir_value_r0(struct ir_function *fun); -#define VR_POS_STACK_PTR \ - (struct ir_vr_pos) \ - { \ - .allocated = true, .alloc_reg = BPF_REG_10, .spilled = 0 \ - } +#define VR_POS_STACK_PTR \ + (struct ir_vr_pos){ .allocated = true, \ + .alloc_reg = BPF_REG_10, \ + .spilled = 0 } struct ir_value bpf_ir_value_norm_stack_ptr(void); diff --git a/core/include/linux/hash.h b/deprecated/core/include/linux/hash.h similarity index 100% rename from core/include/linux/hash.h rename to deprecated/core/include/linux/hash.h diff --git a/core/include/linux/list.h b/deprecated/core/include/linux/list.h similarity index 98% rename from core/include/linux/list.h rename to deprecated/core/include/linux/list.h index dcc0db62..686abafb 100644 --- a/core/include/linux/list.h +++ b/deprecated/core/include/linux/list.h @@ -41,10 +41,7 @@ struct list_head { struct list_head *next, *prev; }; -#define LIST_HEAD_INIT(name) \ - { \ - &(name), &(name) \ - } +#define LIST_HEAD_INIT(name) { &(name), &(name) } #define LIST_HEAD(name) struct list_head name = LIST_HEAD_INIT(name) @@ -352,10 +349,7 @@ struct hlist_node { struct hlist_node *next, **pprev; }; -#define HLIST_HEAD_INIT \ - { \ - .first = NULL \ - } +#define HLIST_HEAD_INIT { .first = NULL } #define HLIST_HEAD(name) struct hlist_head name = { .first = NULL } #define INIT_HLIST_HEAD(ptr) ((ptr)->first = NULL) static inline void INIT_HLIST_NODE(struct hlist_node *h) diff --git a/core/ir_bb.c b/deprecated/core/ir_bb.c similarity index 100% rename from core/ir_bb.c rename to deprecated/core/ir_bb.c diff --git a/core/ir_cg.c b/deprecated/core/ir_cg.c similarity index 100% rename from core/ir_cg.c rename to deprecated/core/ir_cg.c diff --git a/core/ir_cg_norm.c b/deprecated/core/ir_cg_norm.c similarity index 100% rename from core/ir_cg_norm.c rename to deprecated/core/ir_cg_norm.c diff --git a/core/ir_helper.c b/deprecated/core/ir_helper.c similarity index 100% rename from core/ir_helper.c rename to deprecated/core/ir_helper.c diff --git a/core/ir_insn.c b/deprecated/core/ir_insn.c similarity index 100% rename from core/ir_insn.c rename to deprecated/core/ir_insn.c diff --git a/core/ir_value.c b/deprecated/core/ir_value.c similarity index 100% rename from core/ir_value.c rename to deprecated/core/ir_value.c diff --git a/core/lli.c b/deprecated/core/lli.c similarity index 100% rename from core/lli.c rename to deprecated/core/lli.c diff --git a/core/passes/cg_prepare.c b/deprecated/core/passes/cg_prepare.c similarity index 100% rename from core/passes/cg_prepare.c rename to deprecated/core/passes/cg_prepare.c diff --git a/core/passes/code_compaction.c b/deprecated/core/passes/code_compaction.c similarity index 100% rename from core/passes/code_compaction.c rename to deprecated/core/passes/code_compaction.c diff --git a/core/passes/constraint.c b/deprecated/core/passes/constraint.c similarity index 100% rename from core/passes/constraint.c rename to deprecated/core/passes/constraint.c diff --git a/core/passes/div_by_zero.c b/deprecated/core/passes/div_by_zero.c similarity index 100% rename from core/passes/div_by_zero.c rename to deprecated/core/passes/div_by_zero.c diff --git a/core/passes/ecall.c b/deprecated/core/passes/ecall.c similarity index 100% rename from core/passes/ecall.c rename to deprecated/core/passes/ecall.c diff --git a/core/passes/insn_counter_pass.c b/deprecated/core/passes/insn_counter_pass.c similarity index 100% rename from core/passes/insn_counter_pass.c rename to deprecated/core/passes/insn_counter_pass.c diff --git a/core/passes/jmp_complexity.c b/deprecated/core/passes/jmp_complexity.c similarity index 100% rename from core/passes/jmp_complexity.c rename to deprecated/core/passes/jmp_complexity.c diff --git a/core/passes/msan.c b/deprecated/core/passes/msan.c similarity index 100% rename from core/passes/msan.c rename to deprecated/core/passes/msan.c diff --git a/core/passes/optimization.c b/deprecated/core/passes/optimization.c similarity index 100% rename from core/passes/optimization.c rename to deprecated/core/passes/optimization.c diff --git a/core/passes/phi_pass.c b/deprecated/core/passes/phi_pass.c similarity index 100% rename from core/passes/phi_pass.c rename to deprecated/core/passes/phi_pass.c diff --git a/core/passes/translate_throw.c b/deprecated/core/passes/translate_throw.c similarity index 100% rename from core/passes/translate_throw.c rename to deprecated/core/passes/translate_throw.c diff --git a/core/ptrset.c b/deprecated/core/ptrset.c similarity index 100% rename from core/ptrset.c rename to deprecated/core/ptrset.c diff --git a/core/scripts/buildobj.sh b/deprecated/core/scripts/buildobj.sh similarity index 100% rename from core/scripts/buildobj.sh rename to deprecated/core/scripts/buildobj.sh diff --git a/core/scripts/format.sh b/deprecated/core/scripts/format.sh similarity index 89% rename from core/scripts/format.sh rename to deprecated/core/scripts/format.sh index 2f682cb0..7225cc25 100755 --- a/core/scripts/format.sh +++ b/deprecated/core/scripts/format.sh @@ -6,7 +6,7 @@ files=$(find . -iname '*.h' -o -iname '*.c' -not -path "./build/*") clang_path=$(command -v clang-format 2>/dev/null) if [ -z "$clang_path" ]; then - for ver in {18..14}; do # check clang-20 down to clang-10 + for ver in {30..14}; do # check clang-20 down to clang-10 if command -v clang-format-$ver &>/dev/null; then clang_path=$(command -v clang-format-$ver) break diff --git a/core/scripts/gen_insn_ctor.py b/deprecated/core/scripts/gen_insn_ctor.py similarity index 100% rename from core/scripts/gen_insn_ctor.py rename to deprecated/core/scripts/gen_insn_ctor.py diff --git a/core/scripts/gen_kernel.sh b/deprecated/core/scripts/gen_kernel.sh similarity index 100% rename from core/scripts/gen_kernel.sh rename to deprecated/core/scripts/gen_kernel.sh diff --git a/core/scripts/helpers.txt b/deprecated/core/scripts/helpers.txt similarity index 100% rename from core/scripts/helpers.txt rename to deprecated/core/scripts/helpers.txt diff --git a/core/tests/test_list.c b/deprecated/core/tests/test_list.c similarity index 100% rename from core/tests/test_list.c rename to deprecated/core/tests/test_list.c diff --git a/core/tests/test_ptrset.c b/deprecated/core/tests/test_ptrset.c similarity index 100% rename from core/tests/test_ptrset.c rename to deprecated/core/tests/test_ptrset.c diff --git a/docs/CONTRIBUTION_GUIDE.md b/docs/CONTRIBUTION_GUIDE.md deleted file mode 100644 index 7ecea160..00000000 --- a/docs/CONTRIBUTION_GUIDE.md +++ /dev/null @@ -1,34 +0,0 @@ -# Contribution Guide - - -## Project Structure - -``` -ePass/ -├── core/ # Main compiler implementation -│ ├── include/ # Header files -│ ├── docs/ # Technical documentation -│ ├── passes/ # Optimization passes -│ ├── aux/ # Auxiliary utilities -│ ├── epasstool/ # CLI tool -│ └── tests/ # Simple BPF tests -├── test/ # Integration tests and evaluation -├── rejected/ # Collected rejected programs -└── tools/ # Helper scripts and utilities -``` - -## Common Development Patterns - -### Iterating Through Instructions - -```c -struct ir_basic_block **pos; -array_for(pos, fun->reachable_bbs) -{ - struct ir_basic_block *bb = *pos; - struct ir_insn *insn; - list_for_each_entry(insn, &bb->ir_insn_head, list_ptr) { - // Process instruction - } -} -``` diff --git a/docs/CORE_RS.md b/docs/CORE_RS.md new file mode 100644 index 00000000..d6c6364d --- /dev/null +++ b/docs/CORE_RS.md @@ -0,0 +1,173 @@ +# Rust Core Architecture + +The Rust core (`core-rs/epass-ir`) is the active ePass compiler library. It is a +userspace-only rewrite of the old C core and is designed to be embedded by +CLIs, tests, and patched libbpf. + +## Pipeline + +```text +BpfInsn[] + -> lift + -> Function (SSA IR) + -> PassManager + -> codegen prep + -> liveness/interference + -> register allocation + spilling + -> SSA-out + -> normalize/emit + -> BpfInsn[] +``` + +## IR model + +The IR uses stable arena indices instead of raw pointers: + +- `Function` owns instruction and block arenas. +- `InsnId` identifies an instruction. +- `BbId` identifies a basic block. +- Removed instructions are tombstoned rather than moving arena entries. +- Def-use lists are maintained by helper methods and checked after passes. + +Important types: + +```rust +Function +BasicBlock +Insn +InsnKind +Value +InsnId +BbId +``` + +Pseudo values: + +- `func.sp` is the stack/frame pointer pseudo (`R10`). +- `func.args[0..5]` are function argument pseudos (`R1..R5`). + +## Building IR + +Use `IrBuilder` for pass code whenever possible: + +```rust +let mut b = IrBuilder::before_terminator(func, bb); +let x = b.add(AluOp::Alu64, lhs, rhs); +b.ret(Value::Insn(x)); +``` + +`IrBuilder` updates def-use automatically. + +Low-level construction remains available: + +```rust +let id = func.create_insn(bb, InsnKind::Assign, InsertPos::Back); +func.add_value_operand(id, value); +``` + +## CFG utilities + +`Function` exposes helpers for common CFG edits: + +```rust +terminator(bb) +is_terminated(bb) +successor_targets(bb) +set_ja_target(ja, target) +set_cond_targets(cond, fallthrough, taken) +replace_successor(from, old_to, new_to) +split_edge(from, to) +split_block_before(insn) +split_block_after(insn) +create_ret_block(value) +create_throw_block() +``` + +`split_edge` also relabels phi predecessors in the successor block. + +## Pass framework + +Passes implement one trait: + +```rust +trait Pass { + fn name(&self) -> &str; + fn enabled_by_default(&self) -> bool; + fn allow_disable(&self) -> bool; + fn init(&mut self, arg: Option<&str>) -> Result<()>; + fn register_pass(&self, order: Vec) -> Result>; + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()>; +} +``` + +The pass manager: + +1. parses `--popt`; +2. enables/disables/configures passes; +3. lets enabled passes order themselves; +4. verifies each pass only moves/inserts/removes itself; +5. runs passes with `postprocess` after each pass. + +See [PASS_MANAGER.md](PASS_MANAGER.md). + +## Postprocess after every pass + +After each pass: + +```text +recompute successors +cfg::finalize +remove unreachable edges +prune dead phi inputs +prog_check +``` + +`prog_check` validates: + +- branch successor structure; +- conditional fallthrough layout; +- phi placement; +- phi inputs matching predecessor blocks; +- def-use consistency. + +## IR text format + +`.epir` is a parseable debugging format. See [IR_TEXT.md](IR_TEXT.md). + +Library APIs: + +```rust +epass_ir::dump_ir(&func) +epass_ir::load_ir_str(&text) +epass_ir::load_ir_file(path) +``` + +Pipeline option: + +```bash +--gopt load_ir=/tmp/prog.epir +``` + +Dump pass option: + +```bash +--popt 'dump_ir(/tmp/prog.epir)' +``` + +## Code generation + +The code generator performs: + +- call/argument lowering to physical register copies; +- array/constant spill preparation; +- liveness and interference construction; +- pre-spilling of oversized cliques; +- greedy coloring; +- post-spill fallback if coloring fails on a non-chordal graph; +- copy coalescing; +- phi removal / SSA-out; +- BPF instruction emission. + +The post-spill fallback is required because ePass adds register constraints that +can make the interference graph non-chordal even though pure SSA interference +graphs are chordal. diff --git a/docs/CREATE_INSTRUCTION.md b/docs/CREATE_INSTRUCTION.md index 48138f57..c1f5a826 100644 --- a/docs/CREATE_INSTRUCTION.md +++ b/docs/CREATE_INSTRUCTION.md @@ -1 +1,168 @@ -# Create New Instruction \ No newline at end of file +# Creating IR Instructions in `core-rs` + +The old C core exposed many `create_*_insn` constructors. In the Rust core, use +`IrBuilder` for pass code and `Function` primitives for low-level cases. + +## Preferred API: `IrBuilder` + +```rust +use epass_ir::ir::{IrBuilder, InsertPos, Value, AluOp, BinOp, VrType}; + +let mut b = IrBuilder::before_terminator(func, bb); +let tmp = b.add(AluOp::Alu64, lhs, Value::const64(1)); +b.store_raw(VrType::B64, base, -8, Value::Insn(tmp)); +``` + +The builder: + +- creates the instruction; +- inserts it at the configured insertion point; +- fills opcode-specific fields; +- updates def-use chains; +- returns the new `InsnId`. + +## Insertion points + +```rust +IrBuilder::at_bb(func, bb, InsertPos::Back) +IrBuilder::at_end(func, bb) +IrBuilder::at_start(func, bb) +IrBuilder::before_terminator(func, bb) +IrBuilder::after_phi(func, bb) + +IrBuilder::at_insn(func, anchor, InsertPos::Front) +IrBuilder::before(func, anchor) +IrBuilder::after(func, anchor) +``` + +## Supported constructors + +Allocation: + +```rust +b.alloc(VrType::B64) +b.alloc_array(VrType::B8, 32) +``` + +Memory: + +```rust +b.store(slot, value) +b.load(slot) +b.store_raw(VrType::B32, base, offset, value) +b.load_raw(VrType::B64, base, offset) +``` + +Immediates: + +```rust +b.load_imm_extra(LoadImmExtra::Imm64, 42) +``` + +ALU: + +```rust +b.bin(BinOp::Add, AluOp::Alu64, lhs, rhs) +b.add(...) +b.sub(...) +b.mul(...) +b.div(...) +b.and(...) +b.or(...) +b.xor(...) +b.lsh(...) +b.rsh(...) +b.arsh(...) +b.modulo(...) +b.neg(AluOp::Alu64, value) +b.end(EndKind::ToBe, 32, value) +``` + +Calls and returns: + +```rust +b.call(6, [arg0, arg1])? +b.ecall([arg0])? +b.ret(value) +b.throw() +``` + +Branches: + +```rust +b.ja(target) +b.cond_jmp(Cond::Eq, AluOp::Alu64, lhs, rhs, fallthrough, taken) +b.jeq(...) +b.jne(...) +b.jgt(...) +b.jge(...) +b.jlt(...) +b.jle(...) +b.jsgt(...) +b.jsge(...) +b.jslt(...) +b.jsle(...) +``` + +Phi and copies: + +```rust +b.phi_entries([(value1, pred1), (value2, pred2)]) +b.assign(value) +``` + +Misc: + +```rust +b.get_elem_ptr(index, array) +b.reg(reg_id) // mostly for codegen/tests +b.function_arg(arg_id) // mostly for specialized tests +``` + +## Low-level API + +Use this only when a builder method is not appropriate: + +```rust +let id = func.create_insn(bb, InsnKind::Assign, InsertPos::Back); +func.add_value_operand(id, value); +``` + +Do **not** push operands manually unless you also maintain def-use: + +```rust +// Avoid this in passes: +func.insn_mut(id).values.push(value); // missing add_use! +``` + +If you must manipulate operands manually, use: + +```rust +func.add_value_operand(id, value) +func.add_phi_operand(phi, value, pred_bb) +func.clear_values(id) +func.replace_value_in(user, old, new) +func.replace_all_uses(def, replacement) +``` + +`prog_check` validates def-use after each pass and will catch many mistakes. + +## CFG helpers + +For block and branch manipulation, use `Function` CFG utilities: + +```rust +func.terminator(bb) +func.successor_targets(bb) +func.set_ja_target(ja, target)? +func.set_cond_targets(cond, fallthrough, taken)? +func.replace_successor(from, old_to, new_to)? +func.split_edge(from, to)? +func.split_block_before(insn)? +func.split_block_after(insn)? +func.create_ret_block(value) +func.create_throw_block() +``` + +`split_edge` relabels phi inputs in the successor block and is the safest way to +insert instrumentation on an existing CFG edge. diff --git a/docs/EPASS_OPTIONS.md b/docs/EPASS_OPTIONS.md index 1dadda98..b87d5714 100644 --- a/docs/EPASS_OPTIONS.md +++ b/docs/EPASS_OPTIONS.md @@ -1,7 +1,109 @@ -# Epass Options +# ePass Options -## Global Options +Options are split into global options (`--gopt`) and pass options (`--popt`). +## Global options (`--gopt`) +`--gopt` is a comma-separated list: -## Pass Options \ No newline at end of file +```bash +--gopt 'verbose=2,disable_coalesce' +``` + +Supported options: + +| Option | Meaning | +|--------|---------| +| `verbose=` | Set log verbosity. `0` quiet, `1` info, `2+` debug IR logs. | +| `disable_coalesce` | Disable register coalescing in codegen. | +| `print_bpf` | Print BPF disassembly format. | +| `print_dump` | Print dump format: one packed `u64` per instruction. | +| `print_detail` | Print detailed instruction view. | +| `print_bpf_detail` | Print both BPF and detailed view. | +| `no_prog_check` | Disable IR verifier checks after passes. Mostly for debugging broken IR. | +| `printonly` | Skip transformation. | +| `dotgraph` | Reserved/debug option for graph output. | +| `load_ir=` | Load initial IR from `.epir` instead of lifting bytecode. Passes/codegen still run. | + +## Pass options (`--popt`) + +`--popt` configures pass enable/disable/options. It does **not** directly define +pass order. Each pass decides its own order via `register_pass`. + +Syntax: + +```text +pass +pass(arg) +!pass +``` + +Examples: + +```bash +--popt 'dump_ir(/tmp/lift.epir)' +--popt 'dump_ir(path=/tmp/lift.epir),!const_prop' +--popt 'optimize_ir(no_dead_elim)' +``` + +Commas split options only at the top level, so arguments can contain commas +inside parentheses. + +## Builtin passes + +### `dump_ir` + +Optional. Disabled by default. + +```text +dump_ir(/tmp/a.epir) +dump_ir(path=/tmp/a.epir) +``` + +Orders itself first and dumps the IR immediately after lift/load, before other +passes mutate it. + +### `const_prop` + +Default enabled. Disableable. + +```text +!const_prop +``` + +Takes no options. Conservatively folds constants and constant conditional +branches. + +### `phi` + +Default enabled. **Not disableable**. + +Removes trivial phi nodes. + +### `optimize_ir` + +Default enabled. Disableable. + +Options: + +```text +optimize_ir(no_dead_elim) +optimize_ir(noopt) +optimize_ir(no_dead_elim,noopt) +``` + +Performs dead-code elimination and unused-alloc removal. + +## Default order + +Without `--popt`: + +```text +const_prop -> phi -> optimize_ir +``` + +With dump enabled: + +```text +dump_ir -> const_prop -> phi -> optimize_ir +``` diff --git a/docs/IR.md b/docs/IR.md deleted file mode 100644 index ae95dab6..00000000 --- a/docs/IR.md +++ /dev/null @@ -1,231 +0,0 @@ -# bpf IR Specification (v0.1) - -## `bpf_insn` Structure - -```c -struct ir_insn { - struct ir_value values[MAX_FUNC_ARG]; - u8 value_num; - - // Used in ALLOC instructions - enum ir_vr_type vr_type; - - // Used in RAW instructions - struct ir_address_value addr_val; - - // Used in JMP instructions - struct ir_basic_block *bb1; - struct ir_basic_block *bb2; - - // Array of phi_value - struct array phi; - - s32 fid; - u32 f_arg_num; - enum ir_insn_type op; - - // Linked list - struct list_head list_ptr; - - // Parent BB - struct ir_basic_block *parent_bb; - - // Array of struct ir_insn * - // Users - struct array users; - - // Might be useful? - // Too difficult, need BTF - // enum ir_vr_type type; - - // Used when generating the real code - size_t _insn_id; - void *user_data; - u8 _visited; -}; -``` - -There are currently 20 instructions supported. - -## IR Instructions - -General syntax notation for documenting the instructions: - -`INSN ...` - -`FIELD_1` is a field name in the `bpf_insn` struct. - -For example, the following notation is valid syntax notation: - -`alloc ` - -`abort` - -`ja ` - -### `alloc` - -Syntax: `alloc `. - -Allocate a space on stack or on a register (decided by the code gen). - -Example: - -``` -%1 = alloc IR_VR_TYPE_U32 -store %1 200 -``` - -(Currently other vr type is not working. All alloc uses 64 bits reg/stack) - -### `allocarray` - -Syntax: `allocarray ` - -Allocate a continuous array on stack with size `sizeof(vr_type)*array_num`. - -Users could load or store values to it by using `storeraw` and `loadraw`. - -### `getelemptr` - -Syntax: `getelemptr ` - -Get a pointer to the element in an array. - -`values[1]` should point to a `allocarray` instruction. - -### `store` - -Syntax: `store ` - -Requirement: `values[0]` is an `alloc` instruction. - -Store a value `values[1]` in an address `values[0]`. - -### `load` - -Syntax: `load ` - -Requirement: `values[0]` is an `alloc` instruction. - -Load a value `values[0]` with size `vr_type`. - -### `storeraw` - -Syntax: `storeraw ` - -Store a value `values[0]` in manually set `addr_val` with size `vr_type`. - -### `loadraw` - -Syntax: `loadraw ` - -Load a value `addr_val` with size `vr_type`. - -### `loadrawextra` - -Syntax: `loadrawextra ` - -Extra instruction to load imm64. - -### ALU Binary Instructions - -This includes `add`, `sub`, etc. - -Syntax: `INSN ` - -Do ALU binary computation. - -Example: - -``` -%3 = add %1 %2 -``` - -### `call` - -Syntax: `call ...` - -Call a eBPF helper function with arguments `values[0]`... - -### `ret` - -Syntax: `ret ` - -Exit the program with exit code `values[0]`. - -### `ja` - -Syntax: `ja ` - -Jump to basic block `bb1`. - -### Conditional Jump Instructions - -Syntax: `INSN ` - -Do condition jump based on testing `values[0]` and `values[1]`. - -`bb1` is the basic block next to this basic block if not jumping, `bb2` is the basic block to jump. - -Requirement: `bb1` must be next to this basic block. - -### `phi` - -Syntax: `phi ...` - -Phi instruction. `phi` is an array of `phi_value`. Each `phi_value` is a `(ir_value, ir_basic_block*)` pair. - -## BasicBlock - -The basic block structure is `struct ir_basic_block*`. - -The instructions in the basic block is stored in `ir_insn_head`. It is a doubly linked list. - -The predecessors and successors are stored in `preds` and `succs`. They are arrays of `struct ir_basic_block *`. - -Users could add custom data in the `user_data` field. Make sure to free the user data after using it. - -## How to build IR - -### Create a new instruction - -Use functions in `ir_insn`. - -It's possible to create an instruction after/before one existing instruction or at the back/front of a basic block. - -For example, to create a `alloc` instruction, there are two functions: - -```c - -struct ir_insn *create_alloc_insn(struct ir_insn *insn, enum ir_vr_type type, - enum insert_position pos); - -struct ir_insn *create_alloc_insn_bb(struct ir_basic_block *bb, enum ir_vr_type type, - enum insert_position pos); -``` - -`insn` is the instruction that you want to insert after/before. `type` is the specific data needed for this instruction. `pos` is the relative position to insert. There are two options: - -```c -enum insert_position { - INSERT_BACK, - INSERT_FRONT, -}; -``` - -# BPF ISA - -BPF has 10 general purpose registers and a read-only frame pointer register, all of which are 64-bits wide. - -The BPF calling convention is defined as: - -- R0: return value from function calls, and exit value for BPF programs -- R1 - R5: arguments for function calls -- R6 - R9: callee saved registers that function calls will preserve -- R10: read-only frame pointer to access stack - -R0 - R5 are scratch registers and BPF programs needs to spill/fill them if necessary across calls. - -The BPF program needs to store the return value into register R0 before doing an EXIT. - diff --git a/docs/IR_TEXT.md b/docs/IR_TEXT.md new file mode 100644 index 00000000..3c4a4514 --- /dev/null +++ b/docs/IR_TEXT.md @@ -0,0 +1,195 @@ +# ePass IR Text Format (`.epir`) + +`epass-ir` provides a parseable text format for debugging, pass development, and +round-tripping IR through files. The format is intentionally human-readable and +is **not** a stable external ABI yet. + +## Library API + +```rust +use epass_ir::{dump_ir, load_ir_file, load_ir_str, DumpOptions}; + +let text = dump_ir(&func); +let func = load_ir_str(&text)?; +let func = load_ir_file("/tmp/prog.epir")?; +``` + +Passes can call the dumper directly: + +```rust +let text = epass_ir::ir::text::dump_function(func); +std::fs::write("/tmp/after-pass.epir", text)?; +``` + +The loader rebuilds semantic IR state only. Arena indices, def-use lists, +reachable block lists, and CFG metadata are reconstructed after parsing. + +## Loading IR through the pipeline + +`load_ir=` is available as a global option: + +```bash +epasstool read --gopt load_ir=/tmp/prog.epir dummy.txt +``` + +When set, ePass skips bytecode lifting, loads the IR file as the initial +`Function`, then still runs the normal pass pipeline and code generator: + +```text +load_ir -> postprocess/check -> const_prop -> phi -> custom passes -> codegen +``` + +There is no global dump option. To dump the IR immediately after lift/load, use +the optional `dump_ir` pass option: + +```bash +epasstool read -P --popt 'dump_ir(/tmp/prog.epir)' prog.o +``` + +The dump pass is optional and orders itself before all other passes. If enabled, +the effective default order is: + +```text +dump_ir -> const_prop -> phi -> optimize_ir +``` + +Without `dump_ir`, the default order is: + +```text +const_prop -> phi -> optimize_ir +``` + +Pass options use the general pass-option format `pass(arg)` / `!pass`; they do +not directly define ordering. Each pass defines its own order. + +## Basic syntax + +Comments begin with `;`. + +```text +; ePass IR v1 + +bb0: ; preds=[] succs=[bb1,bb2] + %0 = loadraw.u64 [%arg0+0] + %1 = add64 %0, 1:i64 + jeq64 %1, 10:i64 -> bb2 else bb1 + +bb1: + ret %1 + +bb2: + ret 0:i64 +``` + +`preds`/`succs` comments are emitted for readability and ignored by the parser; +CFG edges are reconstructed from terminators. + +## Values + +Special values: + +```text +%sp ; stack/frame pointer pseudo (R10) +%arg0 ; function argument pseudo (R1) +%arg1 +... +%arg4 +undef +``` + +Plain constants: + +```text +0:i32 +0:i64 +-1:i32 +42:i64 +``` + +Only plain constants are accepted by the parser in the MVP. Non-plain constants +such as stack-relative offsets and builtin constants may be dumped as diagnostic +`const(...)` forms, but are not intended as stable parse input yet. + +## Supported instructions + +The parser supports the main pre-codegen IR instructions: + +```text +%x = alloc u64 +%x = allocarray u8 x 32 +%x = getelemptr %idx, %arr + +store %slot, %value +%x = load %slot + +%x = loadimm.imm64 123 +%x = loadimm.map_by_fd 4 +%x = loadimm.map_val_fd 17179869188 + +%x = loadraw.u64 [%base+8] +storeraw.u32 [%base-16], %value + +%x = add64 %a, %b +%x = sub32 %a, 1:i32 +%x = mul64 %a, %b +%x = div64 %a, %b +%x = mod64 %a, %b +%x = and32 %a, 255:i32 +%x = or64 %a, %b +%x = xor64 %a, %b +%x = lsh64 %a, 1:i64 +%x = rsh64 %a, 1:i64 +%x = arsh64 %a, 1:i64 +%x = neg64 %a +%x = end.be32 %a +%x = end.le16 %a + +%x = call #6(%fmt, 16:i32) +%x = ecall(%a, %b) + +ret %x +throw +ja bb1 +jeq64 %a, 0:i64 -> bb2 else bb1 +jsgt32 %a, -1:i32 -> bb3 else bb4 + +%x = phi [%a, bb1], [0:i64, bb2] +%x = assign %a +``` + +Pseudo instructions (`reg Rn`, `funcarg n`) can be dumped and parsed, but normal +pre-codegen IR should reference `%sp` and `%argN` instead of creating new pseudos. + +## Branch target convention + +Conditional branches use explicit labels: + +```text +jeq64 lhs, rhs -> taken_bb else fallthrough_bb +``` + +Internally this maps to: + +- `bb2 = taken_bb` +- `bb1 = fallthrough_bb` + +## Validation after load + +The pipeline runs `postprocess` after loading: + +1. recompute CFG successors from terminators; +2. recompute reachable block layout; +3. drop edges from unreachable blocks; +4. prune phi inputs from dead predecessors; +5. run `prog_check`. + +`prog_check` validates def-use consistency, branch structure, and phi predecessor +consistency. + +## Current limitations + +- The text format is versioned as `v1` but not yet a stable ABI. +- Parser support is focused on pre-codegen IR. Post-codegen/CG-specific values are + diagnostic only. +- Parser currently accepts plain constants only as stable input. +- The loader reconstructs fresh arena IDs; do not rely on old `InsnId` values. diff --git a/docs/KERNEL_TESTING.md b/docs/KERNEL_TESTING.md deleted file mode 100644 index b6692761..00000000 --- a/docs/KERNEL_TESTING.md +++ /dev/null @@ -1 +0,0 @@ -# Kernel Testing \ No newline at end of file diff --git a/docs/PASS_MANAGER.md b/docs/PASS_MANAGER.md new file mode 100644 index 00000000..4e451de1 --- /dev/null +++ b/docs/PASS_MANAGER.md @@ -0,0 +1,140 @@ +# Pass Manager + +The Rust pass manager uses pass-owned metadata and ordering. Users do not specify +pass order directly. Instead, `--popt` enables, disables, or configures passes; +each pass decides where it belongs. + +## Pass trait + +```rust +pub trait Pass { + fn name(&self) -> &str; + + fn enabled_by_default(&self) -> bool { false } + fn allow_disable(&self) -> bool { true } + + fn init(&mut self, arg: Option<&str>) -> Result<()> { ... } + + fn register_pass(&self, order: Vec) -> Result> { + Ok(order) + } + + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()>; +} +``` + +`init` is called before any IR mutation. Pass options are parsed once and stored +inside the pass object. + +`register_pass` receives the current pass name list and returns a modified list. +The pass manager verifies that a pass only changes entries with its own name. +This permits a pass to move, insert, or remove itself but prevents it from +manipulating other passes. + +## Pass option syntax + +```text +pass +pass(arg) +!pass +``` + +Examples: + +```bash +--popt 'dump_ir(/tmp/lift.epir)' +--popt '!const_prop' +--popt 'optimize_ir(no_dead_elim)' +``` + +Commas split options at top level, so pass arguments may contain commas inside +parentheses. + +## Default passes + +Default enabled passes: + +```text +const_prop -> phi -> optimize_ir +``` + +Optional pass: + +```text +dump_ir(path) +``` + +When enabled, `dump_ir` orders itself first: + +```text +dump_ir -> const_prop -> phi -> optimize_ir +``` + +## Builtin passes + +### `dump_ir` + +- Default: disabled +- Disableable: yes +- Args: required path +- Forms: + ```text + dump_ir(/tmp/a.epir) + dump_ir(path=/tmp/a.epir) + ``` +- Order: first + +### `const_prop` + +- Default: enabled +- Disableable: yes +- Args: none +- Order: before `phi` + +### `phi` + +- Default: enabled +- Disableable: no +- Args: none +- Order: before `optimize_ir` if optimizer is present, otherwise last + +### `optimize_ir` + +- Default: enabled +- Disableable: yes +- Args: + ```text + no_dead_elim + noopt + ``` +- Order: after `phi` / after cleanup passes + +## Ordering stabilization + +Pass ordering is computed by repeatedly asking enabled passes to register their +preferred position until the list is stable. + +Limits: + +- maximum iterations: 32 +- duplicate pass instances: not supported yet +- if a pass changes any pass except itself, registration fails +- if ordering does not converge, registration fails + +This catches ordering cycles such as two passes repeatedly moving themselves +before each other. + +## Postprocess + +After every pass, the manager runs: + +```text +recompute_succs +cfg::finalize +drop_unreachable_edges +prune_phi_inputs +prog_check +``` + +This means passes may rewrite branches and CFG targets, then rely on postprocess +to clean reachable layout and validate IR. diff --git a/docs/TESTING.md b/docs/TESTING.md index 8c180903..29700c8d 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -1 +1,93 @@ -# ePass Testing \ No newline at end of file +# Testing + +## Rust unit/integration tests + +```bash +cd core-rs +cargo test --release +``` + +The current tests cover: + +- bytecode packing/unpacking; +- lifter basics; +- compile/codegen basics; +- IR text dump/load; +- `IrBuilder`; +- CFG utility helpers. + +## Dump-format roundtrip + +The dump format is one packed `u64` per BPF instruction. + +```bash +core-rs/target/release/epasstool read -F log -o out.txt input.txt +``` + +## EPIR corpus + +Parseable IR dumps for sample programs live under: + +```text +core-rs/epass-ir/tests/epir +``` + +Regenerate them from `test/output/*.o` with the `dump_ir` pass: + +```bash +cd test/output +EP=../../core-rs/target/release/epasstool +OUT=../../core-rs/epass-ir/tests/epir +mkdir -p "$OUT" +for obj in *.o; do + for fn in $(readelf -sW "$obj" | awk '$4=="FUNC"{print $8}' | sort -u); do + safe=$(printf '%s__%s' "${obj%.o}" "$fn" | tr '/:' '__') + "$EP" read -P --popt "dump_ir($OUT/$safe.epir)" --gopt verbose=0 -s "$fn" "$obj" || true + done +done +``` + +Some ELF `FUNC` symbols are not libbpf programs and will be skipped. + +## Verifier testing through patched libbpf + +Build patched bpftool: + +```bash +cd third-party/ePass-bpftool/src +make -j +``` + +Load through ePass: + +```bash +sudo LIBBPF_ENABLE_EPASS=1 \ + third-party/ePass-bpftool/src/bpftool prog load test/output/progs_simple1.o /sys/fs/bpf/test +sudo rm -f /sys/fs/bpf/test +``` + +Use a timeout for batch testing because some sample programs or attach types can +hang/fail independently of ePass: + +```bash +timeout -k 2 15 sudo LIBBPF_ENABLE_EPASS=1 ... +``` + +For meaningful comparisons, first check whether the original program loads +without ePass; only baseline-loadable programs should be counted as verifier +regressions. + +## Old C core comparison + +The old C core under `core/` can still be built and used as a reference, but it +is no longer the active implementation. + +```bash +cd core +make configure +make build +``` + +The Rust implementation is expected to be verifier-correct, but byte-for-byte +identity with the old C code is not guaranteed for all programs because register +allocation and cleanup passes can choose different but equivalent bytecode. diff --git a/docs/TODO.md b/docs/TODO.md deleted file mode 100644 index aebb85a2..00000000 --- a/docs/TODO.md +++ /dev/null @@ -1,5 +0,0 @@ -# TODOs - -- [ ] Huge tests in falco -- [ ] bpf-to-bpf calls -- [ ] Full test suite diff --git a/docs/USERSPACE_TESTING.md b/docs/USERSPACE_TESTING.md index 16b61818..3a22c147 100644 --- a/docs/USERSPACE_TESTING.md +++ b/docs/USERSPACE_TESTING.md @@ -1,65 +1,97 @@ # Userspace Testing -The main development happens in `core` directory. To start, `cd` into `core`. +The active userspace implementation is `core-rs`. The old C `core/` is kept as +reference. -### Build +## Build Rust core and CLI ```bash -make configure # Do it once - -make build +cd core-rs +cargo build --release +cargo test --release ``` -### Install +The CLI binary is: -```bash -make install +```text +core-rs/target/release/epasstool ``` -### Basic Usage +## Basic CLI usage ```bash -# Run ePass on the program -epass read prog.o +# Rewrite an ELF program and write dump-format output. +core-rs/target/release/epasstool read -s prog -F log -o out.txt test/output/progs_simple1.o -# Run ePass on the program with gopt and popt -epass read --popt popts --gopt gopts prog.o +# Print original BPF in dump format. +core-rs/target/release/epasstool print --gopt print_dump test/output/progs_simple1.o -# Print the BPF program -epass print prog.o +# Dump lifted IR before passes. +core-rs/target/release/epasstool read -P --popt 'dump_ir(/tmp/prog.epir)' -s prog test/output/progs_simple1.o + +# Load IR instead of lifting bytecode. +core-rs/target/release/epasstool read --gopt load_ir=/tmp/prog.epir -F log -o out.txt dummy.txt ``` -For `gopt` and `popt`, see [ePass Options](./EPASS_OPTIONS.md). +See [EPASS_OPTIONS.md](EPASS_OPTIONS.md) for `--gopt` and `--popt`. -### Use ePass with `libbpf` +## Patched libbpf -We may want to load a ePass-modified program to the kernel to see its effect. ePass provides a modified libbpf that allows users to run ePass before loading programs to the kernel. The advantage is that you do not need to change the kernel. However, running ePass in userspace cannot leverage the verifier, so it cannot use verifier information, cannot run verifier dependent passes, and cannot run kernel passes. +`third-party/ePass-libbpf` is patched to call ePass before program load when +`LIBBPF_ENABLE_EPASS=1` is set. -First, initializing all submodules. +Build: ```bash -git submodule update --init --recursive +cd third-party/ePass-libbpf/src +make -j ``` -Now open the `libbpf` source code directory and build: +This builds both `libbpf.a` and `libbpf.so`. The static archive includes the +Rust C ABI library `libepass_ir.a`. + +## Patched bpftool + +Many system bpftool builds are statically linked, so build the repository copy: ```bash -cd third-party/ePass-libbpf/src +cd third-party/ePass-bpftool/src make -j ``` -To install `ePass libbpf`, install: +Load a program through ePass and the kernel verifier: ```bash -sudo make install +sudo LIBBPF_ENABLE_EPASS=1 \ + LIBBPF_EPASS_GOPT='verbose=1' \ + third-party/ePass-bpftool/src/bpftool prog load test/output/progs_simple1.o /sys/fs/bpf/test ``` -After installing ePass libbpf, you could run any programs that depends on the `libbpf` shared library with `ePass` commands. +Clean up: -For `bpftool`, you need to build `bpftool` because by default it statically link `libbpf`. +```bash +sudo rm -f /sys/fs/bpf/test +``` -An example of using ePass to load `test.o` eBPF program using `bpftool`: +To prove the rewritten program was loaded, compare kernel `xlated` size: ```bash -sudo LIBBPF_ENABLE_EPASS=1 LIBBPF_EPASS_GOPT="verbose=3" LIBBPF_EPASS_POPT="msan" bpftool prog load test.o /sys/fs/bpf/test -``` \ No newline at end of file +sudo third-party/ePass-bpftool/src/bpftool prog show pinned /sys/fs/bpf/test +``` + +## Dynamic libbpf consumers + +Any application dynamically linked against libbpf can use the patched shared +library by setting `LD_LIBRARY_PATH` to `third-party/ePass-libbpf/src`, then +setting `LIBBPF_ENABLE_EPASS=1`. + +## Useful environment variables + +| Variable | Meaning | +|----------|---------| +| `LIBBPF_ENABLE_EPASS=1` | Enable ePass rewrite before load. | +| `LIBBPF_EPASS_GOPT='...'` | Global options passed to ePass. | +| `LIBBPF_ENABLE_AUTORELOAD=1` | If rewritten load fails, retry original instructions. | + +Pass options are currently CLI-side (`--popt`) for `epasstool`. The libbpf C ABI +uses the default pass pipeline unless extended to pass popt separately. diff --git a/docs/WRITING_PASSES.md b/docs/WRITING_PASSES.md new file mode 100644 index 00000000..551345cc --- /dev/null +++ b/docs/WRITING_PASSES.md @@ -0,0 +1,171 @@ +# Writing Passes for `core-rs` + +Passes transform or analyze `Function` IR. This guide describes the Rust pass API, +pass options, ordering, and safe IR construction helpers. + +## Pass trait + +Implement `epass_ir::Pass`: + +```rust +use epass_ir::{Env, Result, Pass}; +use epass_ir::ir::Function; + +#[derive(Default)] +struct MyPass { + enabled_feature: bool, +} + +impl Pass for MyPass { + fn name(&self) -> &str { "my_pass" } + + fn enabled_by_default(&self) -> bool { false } + fn allow_disable(&self) -> bool { true } + + fn init(&mut self, arg: Option<&str>) -> Result<()> { + if let Some(arg) = arg { + self.enabled_feature = arg.contains("feature=1"); + } + Ok(()) + } + + fn register_pass(&self, mut order: Vec) -> Result> { + // Move this pass after const_prop if both are enabled. + let name = self.name(); + if let Some(pos) = order.iter().position(|p| p == name) { + let own = order.remove(pos); + if let Some(cp) = order.iter().position(|p| p == "const_prop") { + order.insert(cp + 1, own); + } else { + order.push(own); + } + } + Ok(order) + } + + fn run(&self, env: &mut Env, func: &mut Function) -> Result<()> { + // mutate/analyze func here + Ok(()) + } +} +``` + +Options are parsed in `init` before any IR is mutated. `run` receives a configured +pass object. + +## Pass options (`--popt`) + +Syntax: + +```text +pass +pass(arg) +!pass +``` + +Examples: + +```bash +--popt 'dump_ir(/tmp/lift.epir)' +--popt '!const_prop' +--popt 'optimize_ir(no_dead_elim)' +``` + +Pass options do not specify order. Each pass decides its own position in +`register_pass`. + +## Ordering rules + +`register_pass` receives the current ordered pass-name list and returns a new +list. The pass manager verifies that the pass only changes entries with its own +name. It may move, insert, or remove itself; it may not reorder or edit other +passes. + +The manager iterates registration until stable, with a limit to detect cycles. + +## Postprocess after every pass + +After every pass run, the manager performs: + +```text +recompute CFG successors +cfg::finalize +remove unreachable edges +prune dead phi inputs +prog_check +``` + +This means a pass can rewrite branch targets and rely on postprocess to refresh +CFG metadata and validate def-use/phis. + +## Safe IR construction + +Prefer `IrBuilder`: + +```rust +use epass_ir::ir::{IrBuilder, AluOp, Value}; + +let mut b = IrBuilder::before_terminator(func, bb); +let x = b.add(AluOp::Alu64, lhs, Value::const64(1)); +b.ret(Value::Insn(x)); +``` + +`IrBuilder` updates def-use automatically. + +If using low-level APIs, maintain def-use with helpers: + +```rust +let id = func.create_insn(bb, InsnKind::Assign, InsertPos::Back); +func.add_value_operand(id, value); +``` + +Do not push to `insn.values` directly unless you also update users. + +## CFG helpers + +Use `Function` helpers for common CFG edits: + +```rust +func.split_edge(from, to)?; +func.split_block_before(insn)?; +func.replace_successor(from, old_to, new_to)?; +func.set_ja_target(ja, target)?; +func.set_cond_targets(cond, fallthrough, taken)?; +func.create_ret_block(Value::const64(1)); +``` + +`split_edge` updates phi predecessor labels and is the safest way to insert a +block on an existing edge. + +## Debugging passes + +Enable IR dumps: + +```bash +epasstool read -P --popt 'dump_ir(/tmp/lift.epir)' prog.o +``` + +Or call from pass code: + +```rust +let text = epass_ir::ir::text::dump_function(func); +std::fs::write("/tmp/after-my-pass.epir", text)?; +``` + +Load IR directly: + +```bash +epasstool read --gopt load_ir=/tmp/lift.epir dummy.txt +``` + +## Checklist + +Before adding a new pass: + +1. Implement `Pass` with a stable `name()`. +2. Decide `enabled_by_default` and `allow_disable`. +3. Parse pass-specific options in `init`. +4. Define order in `register_pass`. +5. Use `IrBuilder`/CFG helpers for mutations. +6. Add tests. +7. Run `cargo test --release`. diff --git a/docs/core_design.png b/docs/core_design.png deleted file mode 100644 index 74822a2b..00000000 --- a/docs/core_design.png +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2353446b85a4e8021f7688cfc21187a4403dac5a127b886b08af2f071caf78c5 -size 7638 diff --git a/docs/overview.png b/docs/overview.png deleted file mode 100644 index 814708f8..00000000 --- a/docs/overview.png +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:bd7d261c18cfceed6cd0f9af065a54480a92e3b5d8d066728702a10e128b4020 -size 3619 diff --git a/ePass-kernel b/ePass-kernel deleted file mode 160000 index b10011a9..00000000 --- a/ePass-kernel +++ /dev/null @@ -1 +0,0 @@ -Subproject commit b10011a9969a695892ef9dfda1be3e6b217d5408 diff --git a/test/Makefile b/test/Makefile index 6d0d6718..e3942228 100644 --- a/test/Makefile +++ b/test/Makefile @@ -1,10 +1,42 @@ -output/evaluation_div_by_zero_div_by_zero2.o: ./evaluation/div_by_zero/div_by_zero2.c +output/evaluation_counter_loop1med.o: ./evaluation/counter/loop1med.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_div_by_zero_div_by_zero2.ll: ./evaluation/div_by_zero/div_by_zero2.c +output/evaluation_counter_loop1med.ll: ./evaluation/counter/loop1med.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_div_by_zero_div_by_zero.o: ./evaluation/div_by_zero/div_by_zero.c +output/evaluation_counter_loopnested.o: ./evaluation/counter/loopnested.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_div_by_zero_div_by_zero.ll: ./evaluation/div_by_zero/div_by_zero.c +output/evaluation_counter_loopnested.ll: ./evaluation/counter/loopnested.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_loop2.o: ./evaluation/counter/loop2.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_loop2.ll: ./evaluation/counter/loop2.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_complex.o: ./evaluation/counter/complex.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_complex.ll: ./evaluation/counter/complex.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_loop1sim.o: ./evaluation/counter/loop1sim.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_loop1sim.ll: ./evaluation/counter/loop1sim.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_loop1.o: ./evaluation/counter/loop1.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_loop1.ll: ./evaluation/counter/loop1.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_loop3.o: ./evaluation/counter/loop3.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_loop3.ll: ./evaluation/counter/loop3.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_loopwithif.o: ./evaluation/counter/loopwithif.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_loopwithif.ll: ./evaluation/counter/loopwithif.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_complex2.o: ./evaluation/counter/complex2.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_complex2.ll: ./evaluation/counter/complex2.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/evaluation_counter_loop4.o: ./evaluation/counter/loop4.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_counter_loop4.ll: ./evaluation/counter/loop4.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/evaluation_masking_map_val_rejected.o: ./evaluation/masking/map_val_rejected.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ @@ -22,141 +54,137 @@ output/evaluation_masking_masksimple.o: ./evaluation/masking/masksimple.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/evaluation_masking_masksimple.ll: ./evaluation/masking/masksimple.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_compile_speed_speed_20.o: ./evaluation/compile_speed/speed_20.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_compile_speed_speed_20.ll: ./evaluation/compile_speed/speed_20.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_compile_speed_speed_50.o: ./evaluation/compile_speed/speed_50.c +output/evaluation_msan_msan3.o: ./evaluation/msan/msan3.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_compile_speed_speed_50.ll: ./evaluation/compile_speed/speed_50.c +output/evaluation_msan_msan3.ll: ./evaluation/msan/msan3.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_compile_speed_speed_30.o: ./evaluation/compile_speed/speed_30.c +output/evaluation_msan_simpl1.o: ./evaluation/msan/simpl1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_compile_speed_speed_30.ll: ./evaluation/compile_speed/speed_30.c +output/evaluation_msan_simpl1.ll: ./evaluation/msan/simpl1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_compile_speed_speed_100.o: ./evaluation/compile_speed/speed_100.c +output/evaluation_msan_simpl2.o: ./evaluation/msan/simpl2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_compile_speed_speed_100.ll: ./evaluation/compile_speed/speed_100.c +output/evaluation_msan_simpl2.ll: ./evaluation/msan/simpl2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_compile_speed_speed_70.o: ./evaluation/compile_speed/speed_70.c +output/evaluation_msan_msan2.o: ./evaluation/msan/msan2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_compile_speed_speed_70.ll: ./evaluation/compile_speed/speed_70.c +output/evaluation_msan_msan2.ll: ./evaluation/msan/msan2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_compile_speed_speed_120.o: ./evaluation/compile_speed/speed_120.c +output/evaluation_msan_simpl3.o: ./evaluation/msan/simpl3.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_compile_speed_speed_120.ll: ./evaluation/compile_speed/speed_120.c +output/evaluation_msan_simpl3.ll: ./evaluation/msan/simpl3.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/evaluation_msan_msan1.o: ./evaluation/msan/msan1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/evaluation_msan_msan1.ll: ./evaluation/msan/msan1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_msan_simpl2.o: ./evaluation/msan/simpl2.c - clang -O0 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_msan_simpl2.ll: ./evaluation/msan/simpl2.c +output/evaluation_compile_speed_speed_120.o: ./evaluation/compile_speed/speed_120.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_compile_speed_speed_120.ll: ./evaluation/compile_speed/speed_120.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_msan_msan2.o: ./evaluation/msan/msan2.c +output/evaluation_compile_speed_speed_30.o: ./evaluation/compile_speed/speed_30.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_msan_msan2.ll: ./evaluation/msan/msan2.c +output/evaluation_compile_speed_speed_30.ll: ./evaluation/compile_speed/speed_30.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_msan_msan3.o: ./evaluation/msan/msan3.c +output/evaluation_compile_speed_speed_70.o: ./evaluation/compile_speed/speed_70.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_msan_msan3.ll: ./evaluation/msan/msan3.c +output/evaluation_compile_speed_speed_70.ll: ./evaluation/compile_speed/speed_70.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_msan_simpl1.o: ./evaluation/msan/simpl1.c +output/evaluation_compile_speed_speed_50.o: ./evaluation/compile_speed/speed_50.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_msan_simpl1.ll: ./evaluation/msan/simpl1.c +output/evaluation_compile_speed_speed_50.ll: ./evaluation/compile_speed/speed_50.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_msan_simpl3.o: ./evaluation/msan/simpl3.c - clang -O0 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_msan_simpl3.ll: ./evaluation/msan/simpl3.c +output/evaluation_compile_speed_speed_100.o: ./evaluation/compile_speed/speed_100.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/evaluation_compile_speed_speed_100.ll: ./evaluation/compile_speed/speed_100.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_complex2.o: ./evaluation/counter/complex2.c +output/evaluation_compile_speed_speed_20.o: ./evaluation/compile_speed/speed_20.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_complex2.ll: ./evaluation/counter/complex2.c +output/evaluation_compile_speed_speed_20.ll: ./evaluation/compile_speed/speed_20.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loop1sim.o: ./evaluation/counter/loop1sim.c +output/evaluation_div_by_zero_div_by_zero2.o: ./evaluation/div_by_zero/div_by_zero2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loop1sim.ll: ./evaluation/counter/loop1sim.c +output/evaluation_div_by_zero_div_by_zero2.ll: ./evaluation/div_by_zero/div_by_zero2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loop1med.o: ./evaluation/counter/loop1med.c +output/evaluation_div_by_zero_div_by_zero.o: ./evaluation/div_by_zero/div_by_zero.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loop1med.ll: ./evaluation/counter/loop1med.c +output/evaluation_div_by_zero_div_by_zero.ll: ./evaluation/div_by_zero/div_by_zero.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loopnested.o: ./evaluation/counter/loopnested.c +output/progs_map1.o: ./progs/map1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loopnested.ll: ./evaluation/counter/loopnested.c +output/progs_map1.ll: ./progs/map1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loop1.o: ./evaluation/counter/loop1.c +output/progs_latency.o: ./progs/latency.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loop1.ll: ./evaluation/counter/loop1.c +output/progs_latency.ll: ./progs/latency.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loop4.o: ./evaluation/counter/loop4.c +output/progs_tengjiang_simple_trace.o: ./progs/tengjiang/simple_trace.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loop4.ll: ./evaluation/counter/loop4.c +output/progs_tengjiang_simple_trace.ll: ./progs/tengjiang/simple_trace.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loop3.o: ./evaluation/counter/loop3.c +output/progs_tengjiang_unbounded_loop.o: ./progs/tengjiang/unbounded_loop.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loop3.ll: ./evaluation/counter/loop3.c +output/progs_tengjiang_unbounded_loop.ll: ./progs/tengjiang/unbounded_loop.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_complex.o: ./evaluation/counter/complex.c +output/progs_tengjiang_access_control.o: ./progs/tengjiang/access_control.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_complex.ll: ./evaluation/counter/complex.c +output/progs_tengjiang_access_control.ll: ./progs/tengjiang/access_control.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loop2.o: ./evaluation/counter/loop2.c +output/progs_tengjiang_xdp.o: ./progs/tengjiang/xdp.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loop2.ll: ./evaluation/counter/loop2.c +output/progs_tengjiang_xdp.ll: ./progs/tengjiang/xdp.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/evaluation_counter_loopwithif.o: ./evaluation/counter/loopwithif.c +output/progs_tengjiang_syscount.o: ./progs/tengjiang/syscount.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/evaluation_counter_loopwithif.ll: ./evaluation/counter/loopwithif.c +output/progs_tengjiang_syscount.ll: ./progs/tengjiang/syscount.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_map1.o: ./progs/map1.c +output/progs_test_asm.o: ./progs/test_asm.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_map1.ll: ./progs/map1.c +output/progs_test_asm.ll: ./progs/test_asm.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_Thejokr_ebpf-playground_probe.o: ./progs/Thejokr_ebpf-playground/probe.c +output/progs_fn_nonrejected_uninit_var_access.o: ./progs/fn_nonrejected_uninit_var_access.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_Thejokr_ebpf-playground_probe.ll: ./progs/Thejokr_ebpf-playground/probe.c +output/progs_fn_nonrejected_uninit_var_access.ll: ./progs/fn_nonrejected_uninit_var_access.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_str.o: ./progs/str.c +output/progs_hashmap.o: ./progs/hashmap.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_str.ll: ./progs/str.c +output/progs_hashmap.ll: ./progs/hashmap.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_loop1.o: ./progs/loop1.c +output/progs_loop2.o: ./progs/loop2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_loop1.ll: ./progs/loop1.c +output/progs_loop2.ll: ./progs/loop2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_simple2.o: ./progs/simple2.c +output/progs_mask.o: ./progs/mask.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_simple2.ll: ./progs/simple2.c +output/progs_mask.ll: ./progs/mask.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_fn_oob.o: ./progs/fn_oob.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_fn_oob.ll: ./progs/fn_oob.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_loop3.o: ./progs/loop3.c +output/progs_str.o: ./progs/str.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_loop3.ll: ./progs/loop3.c +output/progs_str.ll: ./progs/str.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_mask.o: ./progs/mask.c +output/progs_counter.o: ./progs/counter.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_mask.ll: ./progs/mask.c +output/progs_counter.ll: ./progs/counter.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_libbpf_lsm.bpf.o: ./progs/libbpf/lsm.bpf.c +output/progs_mem2.o: ./progs/mem2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_libbpf_lsm.bpf.ll: ./progs/libbpf/lsm.bpf.c +output/progs_mem2.ll: ./progs/mem2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_libbpf_task_iter.bpf.o: ./progs/libbpf/task_iter.bpf.c +output/progs_simple2.o: ./progs/simple2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_libbpf_task_iter.bpf.ll: ./progs/libbpf/task_iter.bpf.c +output/progs_simple2.ll: ./progs/simple2.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_libbpf_minimal_legacy.bpf.o: ./progs/libbpf/minimal_legacy.bpf.c +output/progs_libbpf_fentry.bpf.o: ./progs/libbpf/fentry.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_libbpf_minimal_legacy.bpf.ll: ./progs/libbpf/minimal_legacy.bpf.c +output/progs_libbpf_fentry.bpf.ll: ./progs/libbpf/fentry.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_libbpf_uprobe.bpf.o: ./progs/libbpf/uprobe.bpf.c +output/progs_libbpf_task_iter.bpf.o: ./progs/libbpf/task_iter.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_libbpf_uprobe.bpf.ll: ./progs/libbpf/uprobe.bpf.c +output/progs_libbpf_task_iter.bpf.ll: ./progs/libbpf/task_iter.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_libbpf_minimal_ns.bpf.o: ./progs/libbpf/minimal_ns.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ @@ -166,104 +194,76 @@ output/progs_libbpf_sockfilter.bpf.o: ./progs/libbpf/sockfilter.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_libbpf_sockfilter.bpf.ll: ./progs/libbpf/sockfilter.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/progs_libbpf_lsm.bpf.o: ./progs/libbpf/lsm.bpf.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/progs_libbpf_lsm.bpf.ll: ./progs/libbpf/lsm.bpf.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_libbpf_ksyscall.bpf.o: ./progs/libbpf/ksyscall.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_libbpf_ksyscall.bpf.ll: ./progs/libbpf/ksyscall.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_libbpf_fentry.bpf.o: ./progs/libbpf/fentry.bpf.c +output/progs_libbpf_minimal_legacy.bpf.o: ./progs/libbpf/minimal_legacy.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_libbpf_fentry.bpf.ll: ./progs/libbpf/fentry.bpf.c +output/progs_libbpf_minimal_legacy.bpf.ll: ./progs/libbpf/minimal_legacy.bpf.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ +output/progs_libbpf_profile.bpf.o: ./progs/libbpf/profile.bpf.c + clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ +output/progs_libbpf_profile.bpf.ll: ./progs/libbpf/profile.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_libbpf_bootstrap.bpf.o: ./progs/libbpf/bootstrap.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_libbpf_bootstrap.bpf.ll: ./progs/libbpf/bootstrap.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_libbpf_minimal.bpf.o: ./progs/libbpf/minimal.bpf.c +output/progs_libbpf_uprobe.bpf.o: ./progs/libbpf/uprobe.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_libbpf_minimal.bpf.ll: ./progs/libbpf/minimal.bpf.c +output/progs_libbpf_uprobe.bpf.ll: ./progs/libbpf/uprobe.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_libbpf_profile.bpf.o: ./progs/libbpf/profile.bpf.c +output/progs_libbpf_minimal.bpf.o: ./progs/libbpf/minimal.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_libbpf_profile.bpf.ll: ./progs/libbpf/profile.bpf.c +output/progs_libbpf_minimal.bpf.ll: ./progs/libbpf/minimal.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_libbpf_kprobe.bpf.o: ./progs/libbpf/kprobe.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_libbpf_kprobe.bpf.ll: ./progs/libbpf/kprobe.bpf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_counter.o: ./progs/counter.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_counter.ll: ./progs/counter.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_filter.o: ./progs/filter.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_filter.ll: ./progs/filter.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_mem2.o: ./progs/mem2.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_mem2.ll: ./progs/mem2.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_loop2.o: ./progs/loop2.c +output/progs_empty.o: ./progs/empty.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_loop2.ll: ./progs/loop2.c +output/progs_empty.ll: ./progs/empty.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_ringbuf.o: ./progs/ringbuf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_ringbuf.ll: ./progs/ringbuf.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_tengjiang_syscount.o: ./progs/tengjiang/syscount.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_tengjiang_syscount.ll: ./progs/tengjiang/syscount.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_tengjiang_simple_trace.o: ./progs/tengjiang/simple_trace.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_tengjiang_simple_trace.ll: ./progs/tengjiang/simple_trace.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_tengjiang_unbounded_loop.o: ./progs/tengjiang/unbounded_loop.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_tengjiang_unbounded_loop.ll: ./progs/tengjiang/unbounded_loop.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_tengjiang_xdp.o: ./progs/tengjiang/xdp.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_tengjiang_xdp.ll: ./progs/tengjiang/xdp.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_tengjiang_access_control.o: ./progs/tengjiang/access_control.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_tengjiang_access_control.ll: ./progs/tengjiang/access_control.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_empty.o: ./progs/empty.c +output/progs_simple1.o: ./progs/simple1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_empty.ll: ./progs/empty.c +output/progs_simple1.ll: ./progs/simple1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ output/progs_mem1.o: ./progs/mem1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ output/progs_mem1.ll: ./progs/mem1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_hashmap.o: ./progs/hashmap.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_hashmap.ll: ./progs/hashmap.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_compact_opt.o: ./progs/compact_opt.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_compact_opt.ll: ./progs/compact_opt.c - clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_simple1.o: ./progs/simple1.c +output/progs_alu64.o: ./progs/alu64.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_simple1.ll: ./progs/simple1.c +output/progs_alu64.ll: ./progs/alu64.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_latency.o: ./progs/latency.c +output/progs_loop1.o: ./progs/loop1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_latency.ll: ./progs/latency.c +output/progs_loop1.ll: ./progs/loop1.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_alu64.o: ./progs/alu64.c +output/progs_Thejokr_ebpf-playground_probe.o: ./progs/Thejokr_ebpf-playground/probe.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_alu64.ll: ./progs/alu64.c +output/progs_Thejokr_ebpf-playground_probe.ll: ./progs/Thejokr_ebpf-playground/probe.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_test_asm.o: ./progs/test_asm.c +output/progs_loop3.o: ./progs/loop3.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_test_asm.ll: ./progs/test_asm.c +output/progs_loop3.ll: ./progs/loop3.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -output/progs_fn_nonrejected_uninit_var_access.o: ./progs/fn_nonrejected_uninit_var_access.c +output/progs_compact_opt.o: ./progs/compact_opt.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -c $< -o $@ -output/progs_fn_nonrejected_uninit_var_access.ll: ./progs/fn_nonrejected_uninit_var_access.c +output/progs_compact_opt.ll: ./progs/compact_opt.c clang -O2 -I./progs -I/usr/include/x86_64-linux-gnu -D__TARGET_ARCH_x86 -target bpf -g -emit-llvm -S -c $< -o $@ -all:output/progs_fn_nonrejected_uninit_var_access.o output/progs_fn_nonrejected_uninit_var_access.ll output/progs_test_asm.o output/progs_test_asm.ll output/progs_alu64.o output/progs_alu64.ll output/progs_latency.o output/progs_latency.ll output/progs_simple1.o output/progs_simple1.ll output/progs_compact_opt.o output/progs_compact_opt.ll output/progs_hashmap.o output/progs_hashmap.ll output/progs_mem1.o output/progs_mem1.ll output/progs_empty.o output/progs_empty.ll output/progs_tengjiang_access_control.o output/progs_tengjiang_access_control.ll output/progs_tengjiang_xdp.o output/progs_tengjiang_xdp.ll output/progs_tengjiang_unbounded_loop.o output/progs_tengjiang_unbounded_loop.ll output/progs_tengjiang_simple_trace.o output/progs_tengjiang_simple_trace.ll output/progs_tengjiang_syscount.o output/progs_tengjiang_syscount.ll output/progs_ringbuf.o output/progs_ringbuf.ll output/progs_loop2.o output/progs_loop2.ll output/progs_mem2.o output/progs_mem2.ll output/progs_filter.o output/progs_filter.ll output/progs_counter.o output/progs_counter.ll output/progs_libbpf_kprobe.bpf.o output/progs_libbpf_kprobe.bpf.ll output/progs_libbpf_profile.bpf.o output/progs_libbpf_profile.bpf.ll output/progs_libbpf_minimal.bpf.o output/progs_libbpf_minimal.bpf.ll output/progs_libbpf_bootstrap.bpf.o output/progs_libbpf_bootstrap.bpf.ll output/progs_libbpf_fentry.bpf.o output/progs_libbpf_fentry.bpf.ll output/progs_libbpf_ksyscall.bpf.o output/progs_libbpf_ksyscall.bpf.ll output/progs_libbpf_sockfilter.bpf.o output/progs_libbpf_sockfilter.bpf.ll output/progs_libbpf_minimal_ns.bpf.o output/progs_libbpf_minimal_ns.bpf.ll output/progs_libbpf_uprobe.bpf.o output/progs_libbpf_uprobe.bpf.ll output/progs_libbpf_minimal_legacy.bpf.o output/progs_libbpf_minimal_legacy.bpf.ll output/progs_libbpf_task_iter.bpf.o output/progs_libbpf_task_iter.bpf.ll output/progs_libbpf_lsm.bpf.o output/progs_libbpf_lsm.bpf.ll output/progs_mask.o output/progs_mask.ll output/progs_loop3.o output/progs_loop3.ll output/progs_fn_oob.o output/progs_fn_oob.ll output/progs_simple2.o output/progs_simple2.ll output/progs_loop1.o output/progs_loop1.ll output/progs_str.o output/progs_str.ll output/progs_Thejokr_ebpf-playground_probe.o output/progs_Thejokr_ebpf-playground_probe.ll output/progs_map1.o output/progs_map1.ll output/evaluation_counter_loopwithif.o output/evaluation_counter_loopwithif.ll output/evaluation_counter_loop2.o output/evaluation_counter_loop2.ll output/evaluation_counter_complex.o output/evaluation_counter_complex.ll output/evaluation_counter_loop3.o output/evaluation_counter_loop3.ll output/evaluation_counter_loop4.o output/evaluation_counter_loop4.ll output/evaluation_counter_loop1.o output/evaluation_counter_loop1.ll output/evaluation_counter_loopnested.o output/evaluation_counter_loopnested.ll output/evaluation_counter_loop1med.o output/evaluation_counter_loop1med.ll output/evaluation_counter_loop1sim.o output/evaluation_counter_loop1sim.ll output/evaluation_counter_complex2.o output/evaluation_counter_complex2.ll output/evaluation_msan_simpl3.o output/evaluation_msan_simpl3.ll output/evaluation_msan_simpl1.o output/evaluation_msan_simpl1.ll output/evaluation_msan_msan3.o output/evaluation_msan_msan3.ll output/evaluation_msan_msan2.o output/evaluation_msan_msan2.ll output/evaluation_msan_simpl2.o output/evaluation_msan_simpl2.ll output/evaluation_msan_msan1.o output/evaluation_msan_msan1.ll output/evaluation_compile_speed_speed_120.o output/evaluation_compile_speed_speed_120.ll output/evaluation_compile_speed_speed_70.o output/evaluation_compile_speed_speed_70.ll output/evaluation_compile_speed_speed_100.o output/evaluation_compile_speed_speed_100.ll output/evaluation_compile_speed_speed_30.o output/evaluation_compile_speed_speed_30.ll output/evaluation_compile_speed_speed_50.o output/evaluation_compile_speed_speed_50.ll output/evaluation_compile_speed_speed_20.o output/evaluation_compile_speed_speed_20.ll output/evaluation_masking_masksimple.o output/evaluation_masking_masksimple.ll output/evaluation_masking_map_val_accepted.o output/evaluation_masking_map_val_accepted.ll output/evaluation_masking_mask.o output/evaluation_masking_mask.ll output/evaluation_masking_map_val_rejected.o output/evaluation_masking_map_val_rejected.ll output/evaluation_div_by_zero_div_by_zero.o output/evaluation_div_by_zero_div_by_zero.ll output/evaluation_div_by_zero_div_by_zero2.o output/evaluation_div_by_zero_div_by_zero2.ll +all:output/progs_compact_opt.o output/progs_compact_opt.ll output/progs_loop3.o output/progs_loop3.ll output/progs_Thejokr_ebpf-playground_probe.o output/progs_Thejokr_ebpf-playground_probe.ll output/progs_loop1.o output/progs_loop1.ll output/progs_alu64.o output/progs_alu64.ll output/progs_mem1.o output/progs_mem1.ll output/progs_simple1.o output/progs_simple1.ll output/progs_ringbuf.o output/progs_ringbuf.ll output/progs_empty.o output/progs_empty.ll output/progs_filter.o output/progs_filter.ll output/progs_libbpf_kprobe.bpf.o output/progs_libbpf_kprobe.bpf.ll output/progs_libbpf_minimal.bpf.o output/progs_libbpf_minimal.bpf.ll output/progs_libbpf_uprobe.bpf.o output/progs_libbpf_uprobe.bpf.ll output/progs_libbpf_bootstrap.bpf.o output/progs_libbpf_bootstrap.bpf.ll output/progs_libbpf_profile.bpf.o output/progs_libbpf_profile.bpf.ll output/progs_libbpf_minimal_legacy.bpf.o output/progs_libbpf_minimal_legacy.bpf.ll output/progs_libbpf_ksyscall.bpf.o output/progs_libbpf_ksyscall.bpf.ll output/progs_libbpf_lsm.bpf.o output/progs_libbpf_lsm.bpf.ll output/progs_libbpf_sockfilter.bpf.o output/progs_libbpf_sockfilter.bpf.ll output/progs_libbpf_minimal_ns.bpf.o output/progs_libbpf_minimal_ns.bpf.ll output/progs_libbpf_task_iter.bpf.o output/progs_libbpf_task_iter.bpf.ll output/progs_libbpf_fentry.bpf.o output/progs_libbpf_fentry.bpf.ll output/progs_simple2.o output/progs_simple2.ll output/progs_mem2.o output/progs_mem2.ll output/progs_counter.o output/progs_counter.ll output/progs_str.o output/progs_str.ll output/progs_fn_oob.o output/progs_fn_oob.ll output/progs_mask.o output/progs_mask.ll output/progs_loop2.o output/progs_loop2.ll output/progs_hashmap.o output/progs_hashmap.ll output/progs_fn_nonrejected_uninit_var_access.o output/progs_fn_nonrejected_uninit_var_access.ll output/progs_test_asm.o output/progs_test_asm.ll output/progs_tengjiang_syscount.o output/progs_tengjiang_syscount.ll output/progs_tengjiang_xdp.o output/progs_tengjiang_xdp.ll output/progs_tengjiang_access_control.o output/progs_tengjiang_access_control.ll output/progs_tengjiang_unbounded_loop.o output/progs_tengjiang_unbounded_loop.ll output/progs_tengjiang_simple_trace.o output/progs_tengjiang_simple_trace.ll output/progs_latency.o output/progs_latency.ll output/progs_map1.o output/progs_map1.ll output/evaluation_div_by_zero_div_by_zero.o output/evaluation_div_by_zero_div_by_zero.ll output/evaluation_div_by_zero_div_by_zero2.o output/evaluation_div_by_zero_div_by_zero2.ll output/evaluation_compile_speed_speed_20.o output/evaluation_compile_speed_speed_20.ll output/evaluation_compile_speed_speed_100.o output/evaluation_compile_speed_speed_100.ll output/evaluation_compile_speed_speed_50.o output/evaluation_compile_speed_speed_50.ll output/evaluation_compile_speed_speed_70.o output/evaluation_compile_speed_speed_70.ll output/evaluation_compile_speed_speed_30.o output/evaluation_compile_speed_speed_30.ll output/evaluation_compile_speed_speed_120.o output/evaluation_compile_speed_speed_120.ll output/evaluation_msan_msan1.o output/evaluation_msan_msan1.ll output/evaluation_msan_simpl3.o output/evaluation_msan_simpl3.ll output/evaluation_msan_msan2.o output/evaluation_msan_msan2.ll output/evaluation_msan_simpl2.o output/evaluation_msan_simpl2.ll output/evaluation_msan_simpl1.o output/evaluation_msan_simpl1.ll output/evaluation_msan_msan3.o output/evaluation_msan_msan3.ll output/evaluation_masking_masksimple.o output/evaluation_masking_masksimple.ll output/evaluation_masking_map_val_accepted.o output/evaluation_masking_map_val_accepted.ll output/evaluation_masking_mask.o output/evaluation_masking_mask.ll output/evaluation_masking_map_val_rejected.o output/evaluation_masking_map_val_rejected.ll output/evaluation_counter_loop4.o output/evaluation_counter_loop4.ll output/evaluation_counter_complex2.o output/evaluation_counter_complex2.ll output/evaluation_counter_loopwithif.o output/evaluation_counter_loopwithif.ll output/evaluation_counter_loop3.o output/evaluation_counter_loop3.ll output/evaluation_counter_loop1.o output/evaluation_counter_loop1.ll output/evaluation_counter_loop1sim.o output/evaluation_counter_loop1sim.ll output/evaluation_counter_complex.o output/evaluation_counter_complex.ll output/evaluation_counter_loop2.o output/evaluation_counter_loop2.ll output/evaluation_counter_loopnested.o output/evaluation_counter_loopnested.ll output/evaluation_counter_loop1med.o output/evaluation_counter_loop1med.ll diff --git a/test/gen_tests.sh b/test/gen_tests.sh index f95f924b..d47e044e 100755 --- a/test/gen_tests.sh +++ b/test/gen_tests.sh @@ -4,7 +4,7 @@ mkdir -p output -files=$(find . \( -path ./env -o -path ./pass \) -prune -o -name '*.c' -print) +files=$(find . \( -path ./env -o -path ./pass -o -path ./kmtest \) -prune -o -name '*.c' -print) rm Makefile all_objs="" diff --git a/test/evaluation/km/.gitignore b/test/kmtest/km/.gitignore similarity index 100% rename from test/evaluation/km/.gitignore rename to test/kmtest/km/.gitignore diff --git a/test/evaluation/km/Makefile b/test/kmtest/km/Makefile similarity index 100% rename from test/evaluation/km/Makefile rename to test/kmtest/km/Makefile diff --git a/test/evaluation/km/benchuser.c b/test/kmtest/km/benchuser.c similarity index 100% rename from test/evaluation/km/benchuser.c rename to test/kmtest/km/benchuser.c diff --git a/test/evaluation/km/llbench.c b/test/kmtest/km/llbench.c similarity index 100% rename from test/evaluation/km/llbench.c rename to test/kmtest/km/llbench.c diff --git a/test/evaluation/km/llbench_bpf.c b/test/kmtest/km/llbench_bpf.c similarity index 100% rename from test/evaluation/km/llbench_bpf.c rename to test/kmtest/km/llbench_bpf.c diff --git a/test/evaluation/km/llbench_delete_bpf.c b/test/kmtest/km/llbench_delete_bpf.c similarity index 100% rename from test/evaluation/km/llbench_delete_bpf.c rename to test/kmtest/km/llbench_delete_bpf.c diff --git a/test/evaluation/km/llbench_lookup_bpf.c b/test/kmtest/km/llbench_lookup_bpf.c similarity index 100% rename from test/evaluation/km/llbench_lookup_bpf.c rename to test/kmtest/km/llbench_lookup_bpf.c diff --git a/test/evaluation/km/loader.c b/test/kmtest/km/loader.c similarity index 100% rename from test/evaluation/km/loader.c rename to test/kmtest/km/loader.c diff --git a/test/evaluation/km/syscall.c b/test/kmtest/km/syscall.c similarity index 100% rename from test/evaluation/km/syscall.c rename to test/kmtest/km/syscall.c diff --git a/test/evaluation/km/syscall_user.c b/test/kmtest/km/syscall_user.c similarity index 100% rename from test/evaluation/km/syscall_user.c rename to test/kmtest/km/syscall_user.c diff --git a/test/evaluation/kmbst/Makefile b/test/kmtest/kmbst/Makefile similarity index 100% rename from test/evaluation/kmbst/Makefile rename to test/kmtest/kmbst/Makefile diff --git a/test/evaluation/kmbst/bst_bpf.c b/test/kmtest/kmbst/bst_bpf.c similarity index 100% rename from test/evaluation/kmbst/bst_bpf.c rename to test/kmtest/kmbst/bst_bpf.c diff --git a/test/evaluation/kmbst/bstkm.c b/test/kmtest/kmbst/bstkm.c similarity index 100% rename from test/evaluation/kmbst/bstkm.c rename to test/kmtest/kmbst/bstkm.c diff --git a/test/evaluation/kmbst/syscall_user.c b/test/kmtest/kmbst/syscall_user.c similarity index 100% rename from test/evaluation/kmbst/syscall_user.c rename to test/kmtest/kmbst/syscall_user.c diff --git a/test/evaluation/kmbst/vmlinux.h b/test/kmtest/kmbst/vmlinux.h similarity index 100% rename from test/evaluation/kmbst/vmlinux.h rename to test/kmtest/kmbst/vmlinux.h diff --git a/third-party/ePass-bpftool b/third-party/ePass-bpftool index 267493ac..ee8e0d8d 160000 --- a/third-party/ePass-bpftool +++ b/third-party/ePass-bpftool @@ -1 +1 @@ -Subproject commit 267493ac4285cd3927a2e1d53fd82e82144552bf +Subproject commit ee8e0d8dd800a9184a5722388128c4786b582480 diff --git a/third-party/ePass-libbpf b/third-party/ePass-libbpf index 6297d296..31b632bc 160000 --- a/third-party/ePass-libbpf +++ b/third-party/ePass-libbpf @@ -1 +1 @@ -Subproject commit 6297d2969531f1310461d4ef1178c7e8ba41ec9b +Subproject commit 31b632bc5a06a663aecaf5e936977e78bc4c4766 diff --git a/tools/.gitignore b/tools/.gitignore deleted file mode 100644 index 68359a78..00000000 --- a/tools/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -*.c -*.h diff --git a/tools/helper.py b/tools/helper.py deleted file mode 100755 index 83cadfa4..00000000 --- a/tools/helper.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python3 -import subprocess -import re - -with open("helper.h", "r") as f: - content = f.read() - -res = re.compile(r"FN\((.*?), (.*?),").findall(content) - -kernel_path = "/home/linsy/Projects/ebpf/ePass-kernel" - -mapping = {} - -for name, i in res: - # print(name, int(i)) - - process = subprocess.Popen(['/usr/bin/rg', '-rn', kernel_path, '-e', f"BPF_CALL_.\\(bpf_{name}", "--json"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - out, err = process.communicate() - # print(err) - # print(out) - num = re.compile(r"BPF_CALL_(.*?)\(bpf_" + name + r"(,|\))").findall(out.decode()) - # print(len(num)) - ctnum = -1 - err = 0 - for it, _ in num: - if ctnum == -1: - ctnum = int(it) - if ctnum != int(it): - err = 1 - break - if err == 1 or ctnum == -1: - print("Error", name, i) - continue - mapping[int(i)] = (ctnum, name) - -header = "" -body = "" - -for k, (vn, name) in mapping.items(): - header += f"{k}," - body += f"[{k}] = {vn}, // {name}\n" - -with open("helper.c", "w") as f: - f.write(f"{header}\n\n{body}\n") diff --git a/tools/initvm.sh b/tools/initvm.sh deleted file mode 100755 index a1223906..00000000 --- a/tools/initvm.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash - -VMSSHPORT=2222 - -GIT_REPO=$(git rev-parse --show-toplevel) - -DRIVE_PATH=$GIT_REPO/test/vm/disk.qcow2 - -qemu-system-x86_64 -enable-kvm -drive file=${DRIVE_PATH},format=qcow2 --nographic -net nic -net user,hostfwd=tcp::2222-:22 -cpu host -smp 16 -m 8G & - -sleep 10 -ssh -p $VMSSHPORT root@localhost -C "apt-get install -y cloud-guest-utils" -ssh -p $VMSSHPORT root@localhost -C "growpart /dev/sda 1" -ssh -p $VMSSHPORT root@localhost -C "resize2fs /dev/sda1" - -scp -P $VMSSHPORT ~/.ssh/* root@localhost:~/.ssh/ diff --git a/tools/prepvm.sh b/tools/prepvm.sh deleted file mode 100755 index 8edd7078..00000000 --- a/tools/prepvm.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -GIT_REPO=$(git rev-parse --show-toplevel) - -mkdir -p $GIT_REPO/test/vm -cd $GIT_REPO/test/vm - -wget https://cdimage.debian.org/images/cloud/trixie/latest/debian-13-nocloud-amd64.qcow2 -O disk.qcow2 - -qemu-img resize disk.qcow2 +20G - -# Run inside VM: -# apt update; apt-get install -y openssh-server; echo 'PermitEmptyPasswords yes' >> /etc/ssh/sshd_config; echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config; systemctl restart sshd diff --git a/tools/startvm.sh b/tools/startvm.sh deleted file mode 100755 index 2988b7a4..00000000 --- a/tools/startvm.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -GIT_REPO=$(git rev-parse --show-toplevel) - -KERNEL_PATH=$GIT_REPO/ePass-kernel - -DRIVE_PATH=$GIT_REPO/test/vm/disk.qcow2 - -screen -S epass -dm bash -c "qemu-system-x86_64 -enable-kvm -drive file=${DRIVE_PATH},format=qcow2 -kernel ${KERNEL_PATH}/arch/x86_64/boot/bzImage -append 'root=/dev/sda1' --nographic -net nic -net user,hostfwd=tcp::2222-:22 -cpu host -smp 16 -m 8G" - -# To start with default kernel, run: - -# qemu-system-x86_64 -enable-kvm -drive file=${DRIVE_PATH},format=qcow2 --nographic -net nic -net user,hostfwd=tcp::2222-:22 -cpu host -smp 16 -m 8G diff --git a/tools/verifier.py b/tools/verifier.py deleted file mode 100755 index f604b488..00000000 --- a/tools/verifier.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python3 - -import regex as re - -gid = 0 -gmap = dict() - - -def fun(match): - global gid, gmap - old = match.group() - newitem = f"verbose_err({gid}, " - gmap[gid] = match.group(1) - gid += 1 - return old.replace("verbose(", newitem) - - -content = "" -with open("verifier.c", "r") as f: - content = f.read() - -rec = r'verbose\(env[^"]*?"([^"]*?)"[^;]*?;[\s\n]*return' -rec = re.compile(rec) - -nc = re.sub(rec, fun, content) - -with open("verifier_modified.c", "w") as f: - f.write(nc) - -data = "" -for i in gmap: - comment = gmap[i] - if comment[-2:] == "\\n": - comment = comment[:-2] - data += f"BPF_VERIFIER_ERR_{i} = {i}, // {comment} \n" - -header = f""" -enum bpf_verifier_error{{ - {data} -}}; -""" - -with open("error.h", "w") as f: - f.write(header)