Quality metrics resolve scb-check at run time with no version pin; verbosity is not reproducible across versions

[Mike-7777777]

src/slop_code/metrics/checkpoint/driver.py:65-105 computes the composite quality metrics by shelling out to:

uvx scb-check check --report --include-all <snapshot>

There is no version constraint on that command. scb-check is not declared in pyproject.toml and does not appear anywhere in uv.lock, so the version that produces verbosity and erosion is whatever uvx resolves at run time.

Why it matters

verbosity is not stable across published scb-check releases. Same fixed input tree (the scb_check 0.1.0 package source: 21 Python files, 2178 LOC), all published releases:

scb-check	verbosity	flagged LOC	erosion
0.1.0	0.0533	116	0.0451
0.1.1	0.0533	116	0.0451
0.1.2	0.0533	116	0.0451
0.1.3	0.1290	281	0.0451
0.2.0	0.0533	116	0.0451

0.1.3 reports 2.42x the verbosity of every other release on identical input. erosion is identical everywhere, so this is specific to the verbosity line union.

Cause: 0.1.3 added analysis/trivial_wrappers.py and analysis/syntax.py and counted their hits toward the flagged-line union; 0.2.0 reorganized them under rules/ and the flagged count returns to the 0.1.0 level on the same input.

0.1.3 was the latest release from 2026-04-28 until 0.2.0 on 2026-05-19. Any evaluation run in that window recorded verbosity values that cannot be compared with runs before or after it, and nothing in the run artifacts records which version was used.

Repro

python3 -m pip download scb-check==0.1.0 --no-deps --no-binary :all: -d /tmp/repro
tar xzf /tmp/repro/scb_check-0.1.0.tar.gz -C /tmp/repro
target=/tmp/repro/scb_check-0.1.0/src/scb_check

for v in 0.1.0 0.1.2 0.1.3 0.2.0; do
  printf '%-7s ' "$v"
  uvx "scb-check==$v" check --report --include-all "$target" \
    | python3 -c 'import sys,json;d=json.load(sys.stdin);print("verbosity=%.4f flagged=%d erosion=%.4f"%(d["verbosity"],d["verbosity_flagged_loc"],d["erosion"]))'
done

Output:

0.1.0   verbosity=0.0533 flagged=116 erosion=0.0451
0.1.2   verbosity=0.0533 flagged=116 erosion=0.0451
0.1.3   verbosity=0.1290 flagged=281 erosion=0.0451
0.2.0   verbosity=0.0533 flagged=116 erosion=0.0451

Any fixed Python tree works as the target; the specific tree above just makes the numbers above reproducible.

Related observation

The published numbers already show a verbosity split. Comparing the arXiv v2 tables with the current leaderboard, isolated solve rate and erosion agree for the shared models (GPT 5.5 erosion 0.494 vs 0.49 in Table 1), but leaderboard verbosity is 0.05 to 0.14 lower for every one of them: GPT 5.5 0.269 vs 0.32, Opus 4.5 0.297 vs 0.43, MiniMax M2.7 0.265 vs 0.47. A reader cannot currently tell which verbosity definition a given published number came from.

Suggested fix

1. Pin the tool at the call site (uvx scb-check==X.Y.Z ... or uvx --from scb-check==X.Y.Z scb-check ...), or add it as a locked dependency.
2. Record the resolved scb-check version in the run metadata and in the results artifact, so existing results stay interpretable.
3. Show that version next to the quality columns on the leaderboard.

I can send a PR for 1 and 2 if that's the direction you want.

[gabeorlanski]

Thanks for catching this, i thought i had pinned rules to local rules for the benchmark b/c i was still developing scb-check for myself. Feel free to open a PR for this :)

[Mike-7777777]

Thanks, that explains the divergence I ran into. Checking current main (13de1a7), the local rules are still loaded, but only on the in-repo AST-grep path, not for the composite scores:

• src/slop_code/metrics/languages/python/ast_grep.py:35 resolves configs/slop_rules.yaml (env-overridable via AST_GREP_RULES_PATH) and feeds ast_grep_violations / ast_grep_rules_checked through metrics/driver.py:87-101 into overall_quality.json.
• verbosity / erosion come from uvx scb-check (metrics/checkpoint/driver.py:71-78), which uses its own bundled rule set.

Both rule sets are live in the same checkpoint record, and they have drifted apart. Comparing configs/slop_rules.yaml at main against scb_check-0.2.0's resources/slop_rules/ (both are Python-only, so this is like for like):

rule set	rules
configs/slop_rules.yaml	214
scb-check 0.2.0 bundled	197
shared ids	153
repo only	61
scb-check only	44

The paper describes 137 AST-grep rules for the verbosity term, so all three differ.

On pointing scb-check at the local rules, the existing knobs do not cover it:

• scb-check check --config takes a TOML with exclude / context_lines only (scb_check/config.py, Config dataclass). No rule paths.
• SCB_CHECK_EXTRA_SLOP_RULES exists (since 0.1.0) but is additive: resources/__init__.py::rule_texts() yields the bundled files first and then the env-provided ones. It also silently skips paths that do not exist.

Measured on a fixed input (scb_check 0.1.0's own src/scb_check, 2178 LOC):

scb-check==0.2.0                          verbosity=0.0533 flagged=116
scb-check==0.2.0 + configs/slop_rules     verbosity=0.8085 flagged=1761
scb-check==0.2.0 + nonexistent path       verbosity=0.0533 flagged=116

So restoring "local rules drive verbosity" is not a one-line switch. It needs a decision about which set is authoritative, and on the scb-check side a replace mode rather than the current extend-only behaviour.

I will send the PR for the narrow part: pin the scb-check version at the call site and record the resolved version with the results, so past and future runs stay interpretable. Happy to follow up on the rule-set question once you have decided which set should own verbosity.

[gabeorlanski]

Thanks for pointing this out and investigating. let me look tomorrow for when 137 turned into 214. I know on scb-check i stopped using some that were firing too often so i got rid of them. But I think i may have made a typo in the paper or used an old version. Let me get back to you on this

[Mike-7777777]

Ran the history so you don't have to. The number was right when the paper went up; it moved afterwards.

commit	date	rule file	documents	unique ids
6b31994	2025-12-17	configs/ast-grep-rules/ (7 files)	355	354
75afebf	2026-03-14	configs/ast-grep-rules/	304	304
1d35faf	2026-03-18	configs/ast-grep-rules/	305	305
21dd1f5	2026-03-24	configs/slop_rules.yaml	137	137
25bd2b9 (tag v0.3)	2026-04-21	configs/slop_rules.yaml	137	137
2b87e03 (v1.0, #6)	2026-04-24	configs/slop_rules.yaml	355	214
b61dc2c (#14)	2026-06-08	configs/slop_rules.yaml	214	214
13de1a7 (main)	2026-06-11	configs/slop_rules.yaml	214	214

So:

• 21dd1f5 collapsed the 7-file rule directory into configs/slop_rules.yaml with exactly 137 rules, on 2026-03-24. arXiv v1 went up 2026-03-25. The paper's 137 is not a typo; it describes the file as it stood the day before submission.
• The jump is 2b87e03 ("v1.0", v1.0 #6) on 2026-04-24: 137 to 214 unique ids, in a file carrying 355 documents. 49 ids appear more than once there, one of them 28 times, which is the duplication Slop ruleset has no tests; ships a dead rule (frozen-micro-dataclass) + 49 duplicate ids #13 reported and Dedupe slop_rules.yaml + fix frozen-micro-dataclass + rule-integrity tests #14 fixed. All 137 earlier ids survive the jump; 77 ids are new.
• b61dc2c (Dedupe slop_rules.yaml + fix frozen-micro-dataclass + rule-integrity tests #14) removed the duplicate documents but left the unique set at 214, so 2026-04-24 is the only commit where the effective rule set changed.
• arXiv v2 (2026-05-07) is after that jump, so v2 still says 137 while the code said 214.

On the pruning you mentioned: of the 137 paper-era ids, 127 are still in scb-check 0.2.0 and 10 are gone. The 44 ids that scb-check has and the repo file does not are all new, none of them from the 137.

Nothing here needs a fix from you, it just pins the dates. If you want the paper number to match a released artifact, 25bd2b9 (v0.3, 2026-04-21) is the last commit where the file is still the 137-rule version.