When scb-check cannot produce a report for a checkpoint, that checkpoint drops out of the quality means with no visible trace, and the printed summary looks the same as a fully measured run.
Path
src/slop_code/metrics/checkpoint/driver.py:87-104 catches CalledProcessError, OSError and JSONDecodeError, logs a warning, and returns {}. driver.py:68 also returns {} when the snapshot directory is missing.
driver.py:144 merges that empty dict, so the checkpoint record simply has no verbosity / erosion keys. tests/metrics/checkpoint_results_test.py:892 encodes exactly this (assert "verbosity" not in result).
src/slop_code/metrics/summary/stats.py:44 filters those checkpoints out (if key in chkpt).
src/slop_code/metrics/summary/aggregators.py:307-325 averages whatever is left.
src/slop_code/entrypoints/utils.py:363-372 prints Mean verbosity score and Mean erosion score as mean +/- stddev, and never prints MetricStats.count.
Consequence
A run where scb-check failed on many checkpoints prints a quality mean that looks indistinguishable from a run where it succeeded on all of them. The denominator is not shown anywhere in the summary, and the only signal is a WARNING line in the log.
There is also reason to expect the missingness to correlate with checkpoint state rather than be random: the failure mode the repo's own test simulates is stderr="no Python files found", which is a property of a degenerate workspace, not of the tool. Checkpoints like that are exactly the ones an agent-quality comparison should not drop without saying so. Two runs of the same model can end up reporting quality over different denominators with nothing in the output indicating it.
count_expected_checkpoints (utils.py:378) already exists for the solve-rate side; the quality side has no equivalent.
Not asking for a hard failure by default
The intent behind test_scb_check_failure_does_not_drop_other_checkpoint_metrics looks right: one metric failing should not discard the rest of the checkpoint. The gap is visibility, not tolerance.
Suggested fix
- Record a per-checkpoint quality status (
ok / tool_failed / no_snapshot) instead of only omitting the keys, so the results artifact distinguishes "measured as clean" from "not measured".
- Print
count next to Mean verbosity score / Mean erosion score, ideally as n/expected, so a shrunken denominator is visible without reading logs.
- Optionally add a threshold (for example, fail or clearly mark the run when quality coverage falls below some fraction of ran checkpoints).
I can send a PR for 1 and 2 if that's useful.
When
scb-checkcannot produce a report for a checkpoint, that checkpoint drops out of the quality means with no visible trace, and the printed summary looks the same as a fully measured run.Path
src/slop_code/metrics/checkpoint/driver.py:87-104catchesCalledProcessError,OSErrorandJSONDecodeError, logs a warning, and returns{}.driver.py:68also returns{}when the snapshot directory is missing.driver.py:144merges that empty dict, so the checkpoint record simply has noverbosity/erosionkeys.tests/metrics/checkpoint_results_test.py:892encodes exactly this (assert "verbosity" not in result).src/slop_code/metrics/summary/stats.py:44filters those checkpoints out (if key in chkpt).src/slop_code/metrics/summary/aggregators.py:307-325averages whatever is left.src/slop_code/entrypoints/utils.py:363-372printsMean verbosity scoreandMean erosion scoreas mean +/- stddev, and never printsMetricStats.count.Consequence
A run where
scb-checkfailed on many checkpoints prints a quality mean that looks indistinguishable from a run where it succeeded on all of them. The denominator is not shown anywhere in the summary, and the only signal is a WARNING line in the log.There is also reason to expect the missingness to correlate with checkpoint state rather than be random: the failure mode the repo's own test simulates is
stderr="no Python files found", which is a property of a degenerate workspace, not of the tool. Checkpoints like that are exactly the ones an agent-quality comparison should not drop without saying so. Two runs of the same model can end up reporting quality over different denominators with nothing in the output indicating it.count_expected_checkpoints(utils.py:378) already exists for the solve-rate side; the quality side has no equivalent.Not asking for a hard failure by default
The intent behind
test_scb_check_failure_does_not_drop_other_checkpoint_metricslooks right: one metric failing should not discard the rest of the checkpoint. The gap is visibility, not tolerance.Suggested fix
ok/tool_failed/no_snapshot) instead of only omitting the keys, so the results artifact distinguishes "measured as clean" from "not measured".countnext toMean verbosity score/Mean erosion score, ideally asn/expected, so a shrunken denominator is visible without reading logs.I can send a PR for 1 and 2 if that's useful.