Skip to content

test: add class-width compile-time benchmark for issue 559 - #3118

Open
carloscasellas wants to merge 1 commit into
glideapps:masterfrom
carloscasellas:agent/issue-559
Open

test: add class-width compile-time benchmark for issue 559#3118
carloscasellas wants to merge 1 commit into
glideapps:masterfrom
carloscasellas:agent/issue-559

Conversation

@carloscasellas

Copy link
Copy Markdown

Merging this PR is not necessary, I wanted to provide the scripts in case anyone looking at issue 559 wanted to look at them.

Description

Adds a reproducible benchmark (script/benchmark-class-width/) measuring whether narrowing the widest generated Rust struct — as issue #559 proposes — actually improves compile time. It doesn't add or change any generated-code behavior; it's a standalone investigation with scripts, a manual (README.md), and a results write-up (RESULTS.md).

Related Issue

Investigates #559. Recommends closing it as obsolete based on the measurements below.

Motivation and Context

#559 claims blns-object.json, keywords.json, and keyword-unions.schema have objects large enough to slow down compilation, "particularly Rust," citing profiling from #516 (cargo-expand + -Ztime-passes on rustc ~1.24, showing 30k lines concentrated in one visit_map function from serde's derive).

That's an eight-year-old measurement. Before doing the work of splitting these fixtures, I wanted to confirm the underlying cost still exists on a current toolchain.

Previous Behaviour / Output

No benchmark existed for this — the issue's justification was 2018-era profiling with no way to re-verify it against today's rustc.

New Behaviour / Output

script/benchmark-class-width/reshape-blns.py and reshape-keyword-unions.py regroup the fixtures' keys into progressively narrower classes while holding the total field count constant, so any compile-time change is attributable to width rather than data volume. bench-rust.sh times cargo build against each candidate with serde prebuilt in a shared CARGO_TARGET_DIR, isolating the compile of the generated module itself.

Result on rustc/cargo 1.97.1 (median of 10 builds per candidate):

widest class blns compile keyword-unions compile
276 / 230 (today) 0.73s 1.98s
138 / 116 0.74s 1.97s
56 / 59 0.78s 2.00s
28 / 30 0.82s 2.02s

keyword-unions is flat within noise. blns gets monotonically slower as classes narrow (+12%), tracking the extra code from more, smaller types. Neither supports the issue's premise on current rustc.

Separately, ~91% of a Rust sample's cost in the real fixture harness turns out to be rebuilding serde from scratch per sample (no setupCommand, no shared CARGO_TARGET_DIR in test/languages.ts/test/fixtures.ts) — noted in RESULTS.md as a lead worth its own follow-up spike, not a verified fix.

Full write-up, including the keyword-unions.schema → Python 255-argument limit finding and the keywords.json generator-drift bug found along the way, is in script/benchmark-class-width/RESULTS.md.

How Has This Been Tested?

This PR is the test — a benchmark, not a code change. To reproduce:

npm run build
export PATH="$HOME/.cargo/bin:$PATH"   # if using rustup
export BENCH=/tmp/bench
mkdir -p $BENCH/inputs
B=script/benchmark-class-width

for n in 1 2 4 8;  do python3 $B/reshape-blns.py $n           > $BENCH/inputs/blns-$n.json
for n in 1 2 5 10; do python3 $B/reshape-keyword-unions.py $n > $BENCH/inputs/ku-$n.schema; done

REPS=10 bash $B/bench-rust.sh --src-lang json   $BENCH/inputs/blns-{1,2,4,8}.json
REPS=10 bash $B/bench-rust.sh --src-lang schema $BENCH/inputs/ku-{1,2,5,10}.schema

Full instructions, including reference output and troubleshooting, are in script/benchmarkm run test:unit and npm run lint are unaffected since this only adds new files underscript/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant