Skip to content

feat(ci): export stable bounded SARIF evidence - #62

Open
DivyamTalwar wants to merge 2 commits into
mainfrom
codex/sarif-evidence-export-20260824
Open

feat(ci): export stable bounded SARIF evidence#62
DivyamTalwar wants to merge 2 commits into
mainfrom
codex/sarif-evidence-export-20260824

Conversation

@DivyamTalwar

Copy link
Copy Markdown
Owner

Summary

  • export persisted Structorium findings as GitHub-compatible SARIF 2.1.0
  • map tiers to SARIF levels and detector identities to rule descriptors
  • use stable, line-independent partial fingerprints to prevent duplicate alerts after source movement
  • bound and deterministically prioritize output; exclude resolved/suppressed findings by default
  • document a ready-to-use upload-sarif@v4 workflow

Why this is high leverage

Structorium already finds architecture and maintainability defects, but those findings cannot currently participate in GitHub's native code-scanning review surface. This PR closes that delivery gap without adding a runtime dependency or rerunning detectors.

Competitive research and provenance

This is an original Structorium implementation informed by public product patterns, not copied source:

Verification

  • PYTHONPATH=/tmp/structorium-test-deps python -m pytest -q tests/core/test_sarif.py tests/commands/test_sarif_cmd.py — 5 passed
  • python -m compileall -q core/sarif.py app/commands/sarif_cmd.py tests/core/test_sarif.py tests/commands/test_sarif_cmd.py — passed
  • Ruff on every changed Python file — passed
  • git diff --check — passed

Safety

  • no network upload is performed by Structorium
  • output defaults to active, unsuppressed findings
  • output is capped at 5,000 results and sorted by tier/confidence/identity
  • no existing PR or branch was changed

This PR intentionally remains open for review.

Copy link
Copy Markdown
Owner Author

Implementation evidence

Contract Evidence
GitHub-compatible envelope SARIF 2.1.0 schema/version, one deterministic run, tool/rule descriptors, relative artifact URIs
Alert continuity SHA-256 partialFingerprints["structoriumFinding/v1"] excludes source line and is regression-tested across line moves
Bounded CI output default 5,000-result cap; deterministic tier → confidence → ID priority
Honest defaults only open, unsuppressed findings unless explicit audit flags are passed
Source precision detector line-shape normalization supports line, line_number, start_line, lineno, and lines
Operational adoption documented github/codeql-action/upload-sarif@v4 example with a dedicated category
Local evidence 5 focused tests passed; compileall, Ruff, and diff hygiene passed

The implementation uses only Python's standard library and Structorium's existing atomic writer. No competitor code, SDK, upload credential, or new dependency is introduced.

Latest reviewed head: 5dedae385a1f9159e6f97db030d20b7c7b04a9c9.

I am leaving this PR open as requested and will treat GitHub CI/Security on this exact head as authoritative.

Copy link
Copy Markdown
Owner Author

CI follow-up: pre-existing flaky review assertion fixed

The first authoritative CI run exposed one failure shared by tests-core (job 97253106135) and tests-full (job 97253106234): the test globbed two generated prompt files without sorting, then asserted that filesystem-dependent element [0] contained retrospective evidence. Only the relevant batch is expected to carry that evidence.

A narrow second commit now sorts the two prompt paths and joins both generated prompts before asserting cross-batch evidence. The exact regression test passes locally:

pytest -q tests/review/test_review_commands.py::TestCmdReviewPrepare::test_do_run_batches_dry_run_generates_packet_and_prompts — 1 passed.

This defect existed on main and is unrelated to SARIF, but the branch now carries the deterministic fix so its own authoritative CI can be green.

Security note: dependency-review job 97253106099 fails because the repository dependency graph is disabled. That is an external repository setting, not an implementation failure. Secret scan, pip-audit, and CodeQL were green.

Latest head: bcceccf4cb0454ffadf54a83a7779e7b12c83c8f. The PR remains open and unmerged.

Copy link
Copy Markdown
Owner Author

Final authoritative verification

Latest head: bcceccf4cb0454ffadf54a83a7779e7b12c83c8f

  • CI run 32663686214: success (all CI jobs green)
  • Security run 32663686223:
    • CodeQL job 97253615739: success
    • pip-audit job 97253615760: success
    • secret-scan job 97253615772: success
    • dependency-review job 97253615866: externally blocked because the repository dependency graph is disabled

The Security workflow's aggregate failure is therefore repository configuration, not a source, dependency, secret, or CodeQL failure. The implementation-authoritative CI and security analyzers are green on the exact latest head.

PR state: open, unmerged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant