Skip to content

scb-check failures silently shrink the quality denominator; summary shows no count #20

Description

@Mike-7777777

When scb-check cannot produce a report for a checkpoint, that checkpoint drops out of the quality means with no visible trace, and the printed summary looks the same as a fully measured run.

Path

  1. src/slop_code/metrics/checkpoint/driver.py:87-104 catches CalledProcessError, OSError and JSONDecodeError, logs a warning, and returns {}. driver.py:68 also returns {} when the snapshot directory is missing.
  2. driver.py:144 merges that empty dict, so the checkpoint record simply has no verbosity / erosion keys. tests/metrics/checkpoint_results_test.py:892 encodes exactly this (assert "verbosity" not in result).
  3. src/slop_code/metrics/summary/stats.py:44 filters those checkpoints out (if key in chkpt).
  4. src/slop_code/metrics/summary/aggregators.py:307-325 averages whatever is left.
  5. src/slop_code/entrypoints/utils.py:363-372 prints Mean verbosity score and Mean erosion score as mean +/- stddev, and never prints MetricStats.count.

Consequence

A run where scb-check failed on many checkpoints prints a quality mean that looks indistinguishable from a run where it succeeded on all of them. The denominator is not shown anywhere in the summary, and the only signal is a WARNING line in the log.

There is also reason to expect the missingness to correlate with checkpoint state rather than be random: the failure mode the repo's own test simulates is stderr="no Python files found", which is a property of a degenerate workspace, not of the tool. Checkpoints like that are exactly the ones an agent-quality comparison should not drop without saying so. Two runs of the same model can end up reporting quality over different denominators with nothing in the output indicating it.

count_expected_checkpoints (utils.py:378) already exists for the solve-rate side; the quality side has no equivalent.

Not asking for a hard failure by default

The intent behind test_scb_check_failure_does_not_drop_other_checkpoint_metrics looks right: one metric failing should not discard the rest of the checkpoint. The gap is visibility, not tolerance.

Suggested fix

  1. Record a per-checkpoint quality status (ok / tool_failed / no_snapshot) instead of only omitting the keys, so the results artifact distinguishes "measured as clean" from "not measured".
  2. Print count next to Mean verbosity score / Mean erosion score, ideally as n/expected, so a shrunken denominator is visible without reading logs.
  3. Optionally add a threshold (for example, fail or clearly mark the run when quality coverage falls below some fraction of ran checkpoints).

I can send a PR for 1 and 2 if that's useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions