[codex] add RCL dominance arena scorecards - #57
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
CI note: the three required workflows were rejected before any step started. GitHub reports runner_id=0 and steps=[], and recent main runs show the same condition. This is the repository/account billing or spending-limit infrastructure blocker documented in the existing evidence ledgers, not a test assertion failure. Local verification remains recorded in the PR body; do not treat this PR as CI-passed until a real hosted runner executes the workflows. |
|
继续推进已完成并推送到 commit 7501bef。\n\n新增:\n- provider-evidence comparison contract:同一 inputRoot 才允许比较,缺失/不匹配保持 UNVERIFIED。\n- compiler-microbench-v0.1:同一算术链语料的真实 RCL compileRealityToBytecode + native VM,以及真实 rustc + executable reference。\n- 原始指标:correctness、compileBuildSpeed、runtimeMs、artifactFootprintBytes;无加权平均。\n- 2 个真实执行回归测试和 Arena 动态契约测试。\n\n最终本地证据:\n- reportRoot: f358014806b02bcef135d50b8f2b0a401cca3222706e8f21af3b30754bc69017\n- inputRoot: b6a5b9512d04d884eae20d61c543255252c377c79307261ee204de714f9a7363\n- candidate/reference: PASS/PASS\n- narrow microbenchmark dominance: PASS\n- RCL compile ms: 6.251; rustc compile ms: 494.236\n- RCL runtime ms: 47.803; reference runtime ms: 89.279\n- RCL artifact: 2948 bytes; reference artifact: 147968 bytes\n- RCL RBC SHA-256: CBF8B4BBF38921EBEC8B7A63E1344A96E68135BA4903E5D4F35204C4B313688E\n- reference EXE SHA-256: 7E41A38CE3A942F6846258A4A4A678FD418D027E550F3E50853057B8E5FB9BAB\n\n验证:\n- Arena 11/11\n- microbench execution 2/2\n- K01/K02/K03 focused 8/8\n- full npm test: 805 total, 796 pass, 8 pre-existing Windows/frontier failures, 1 skip\n\n边界:该 PASS 只证明固定 arithmetic-chain microbenchmark 的 raw metrics,不证明 whole-language/ecosystem/memory/authorability/commercial superiority。\n\nCI:latest runs for this SHA still fail before steps due the repository account billing/spending-limit; Vercel passes. PR remains open/draft until CI can actually execute. |
|
三轮重复基准已完成(commit 7501bef)。\n\nWorkload: arithmetic-chain-24\ninputRoot: b6a5b9512d04d884eae20d61c543255252c377c79307261ee204de714f9a7363\n所有 RCL/rustc runs: PASS,correctness=1。\n\n三轮平均:\n- RCL compileBuildSpeed: 7.294 ms;Rust: 547.248 ms\n- RCL runtime: 53.736 ms;Rust: 95.596 ms\n- RCL artifact: 2948 bytes;Rust: 147968 bytes\n- RCL/Rust ratios: compile 0.0133x,runtime 0.5621x,artifact 0.0199x\n\nRCL RBC 三轮 SHA-256 均为 CBF8B4B...313688E,说明该产物在重复构建中稳定;rustc EXE 每轮哈希不同,但输出和指标均通过。\n\n汇总证据:\noutput/benchmark-runs/2026-08-16/benchmark-summary.json\nsummaryRoot: 05bf033c50bc5a7ffa5123dabf55d44d0e1a13b74d4a0147977b8ea720c5bd71\n\n边界仍然是固定 arithmetic-chain 微基准;不等同于 whole-language、生态、内存或商业产品优势。 |
Cross-language benchmark update (2026-08-16)\n\nSource revision: 196f718\nWorkload: �rithmetic-chain-24; identical input root �6a5b9512d04d884eae20d61c543255252c377c79307261ee204de714f9a7363; 3 repetitions; all correctness checks PASS.\n\n| Provider | Build/prep mean | Runtime mean | Artifact |\n| --- | ---: | ---: | ---: |\n| RCL native VM | 5.687 ms | 49.330 ms | 2,948 B |\n| rustc 1.97.1 | 510.740 ms | 88.401 ms | 147,968 B |\n| CPython 3.11.6 | 228.922 ms* | 181.773 ms | 1,204 B |\n\nRCL is faster than both references on this workload. RCL loses CPython on artifact footprint (2.4485x), so the non-compensatory overall dominance result is FAIL, not a whole-language superiority claim. * CPython build/prep is py_compile preparation, not machine-code compilation.\n\nEvidence: output/benchmark-runs/2026-08-16/cross-language/benchmark-summary.json; summaryRoot |
Multi-language workload matrix update\n\nSource revision: 3588c3c\nCommand:pm run evidence:dominance-arena:matrix\nWorkloads: 4; providers: RCL native VM, rustc 1.97.1, CPython 3.11.6; 12/12 provider executions passed correctness.\n\n| Workload | RCL vs rustc | RCL vs CPython |\n| --- | --- | --- |\n| text normalization | PASS | FAIL: artifact 622 B vs 532 B |\n| sequence aggregation | PASS | FAIL: artifact 1,091 B vs 374 B |\n| UTF-8 byte length | PASS | PASS |\n| JSON-shaped text scan | PASS | FAIL: artifact 735 B vs 712 B |\n\nThe matrix uses per-workload non-compensatory raw comparisons. RCL therefore has 4/4 against rustc and 1/4 against CPython; overall matrix status is FAIL, not a whole-language superiority claim. Build/prep and cold runtime were wins in the listed comparisons, but CPython wins artifact footprint on three workloads.\n\nSeparate capability gaps remain explicitly BLOCKED: native JSON parsing, declared file I/O, and declared concurrency. The JSON-shaped workload is only textual scanning and does not count as parser parity.\n\nEvidence: output/dominance-arena/compiler-workload-matrix-v0.1/compiler-workload-matrix-report.json; reportRoot 18c821046f51279c52c6abe4c71ca1c09dc09b9a68b2f47918e2cd491d293921. Focused matrix regression: 1/1 PASS. CI remains failing on the current SHA, so no merge was performed. |
Native JSON compact sliceImplemented and verified the first real RCL JSON parse-and-compact path.
Evidence
Local verification
GitHub Actions currently reports |
|
修复已推送:77d9a5f (fix: close native selfhost and Windows verification gaps) 本地 Windows 验证:
远端 CI 当前不是代码失败:RCL Canonical Verification、RCL Authority Contract、RCL Universal Program Stress 三个 job 都在 runner 启动前被 GitHub 账户 Billing & plans / spending limit 阻断,annotation 原文为: 因此本 PR 暂不宣称 CI 通过,也不执行合并。请账户侧修复 Billing & plans 后重新运行必需检查,再进行最终差异审阅和合并。 |
What this changes
RCL's Universal Stress program answers whether a task is expressible and executable across environments. Its legacy nine-gate cell currently lets an independently unverified
AI_GENERATEgate collapse an otherwise evidenced compiler result intoBLOCKED, while the repository has no executable vertical comparison contract for the question "who is stronger?"This PR adds an evidence-bound RCL Dominance Arena v0.1 with three independent scorecard axes:
Capability: all Universal Stress gates exceptAI_GENERATE.Dominance: comparable raw candidate/reference metrics with no weighted compensation.Authorability: the independentAI_GENERATEcontract.The first arena executes the production K01 self-host verification path as a real candidate and records optional reference tool probes. It deliberately reports
Dominance: UNVERIFIEDuntil a semantically matched reference corpus is available.Root cause and fix
The K01 adapter counted the historical
stage0proxy boundary as a compiler correctness stage.stage0explicitly reports that the core runtime is still proxy-hosted and is outside the self-hosting compiler proposition, so a valid40/40compiler-stage run was reported as40/41and failed K01 correctness. The adapter now excludes onlystage0from K01 compiler correctness while preserving its report and boundary; a run with no compiler stages still fails closed.The arena command runner also records command exit status, timeout/tool-not-found failure types, output hashes, artifact hashes and the checked-out Git revision. Missing comparable references remain visible rather than becoming a false win.
Validation
npm run test:dominance-arena: 10/10 pass.npm run evidence:dominance-arenaon the committed source: candidate execution PASS; K01 capability PASS; legacy K01BLOCKEDonly onAI_GENERATE; dominance and authorability UNVERIFIED.30d6f5a13d319982dca87ecb2eec812c198859568c1124c5d82a6a1f6c53cc27.npm test: 793 pass, 8 unrelated existing Frontier/environment failures, 1 skipped. The failures are outside this PR's changed modules and include Windows temporary-path composition, host interaction detection, continuous-field contract classification and self-record count assumptions.The PR does not claim RCL is already the strongest compiler language. It supplies the executable measurement boundary needed to make that claim falsifiable.