Sequential CDM Triage

Event-level risk-controlled triage for conjunction data message streams.

Research question

Can a sequential policy reduce manual review of low-final-risk conjunction events while controlling the probability that a high-final-calculated-risk event is ever assigned SAFE-EXCLUDE during the 2–7 day decision window?

Status

Frozen-policy research prototype with one completed confirmation run. Not for operational collision avoidance or maneuver decisions.

Decisions

• SAFE-EXCLUDE: no current manual review; routine automated ingestion continues.
• MONITOR: priority automated watchlist; reassess at every new CDM.
• ESCALATE: manual analysis required.

Current evidence

Current risk, logistic snapshot, CatBoost snapshot, and CatBoost dynamic scores were compared on identical event-level out-of-fold splits. CatBoost snapshot led at the strict end of the development frontier.

The single locked evaluation run observed 4/73 dangerous exclusions (5.48%; one-sided 95% upper bound 12.10%), 70.64% SAFE-EXCLUDE coverage among low-final-risk events, and a median first decision time of 5.53 days before TCA. The pre-specified 10% evaluation upper-bound criterion was not met. Immutable run artifacts are in artifacts/confirmation_v1/.

The policy module supports two event-level calibration statements:

• marginal rank calibration for average risk over the random calibration sample;
• PAC rank calibration with an explicit confidence level over the calibration sample.

Development results are in reports/DEVELOPMENT_REPORT_002_RU.md, reports/DEVELOPMENT_NOTES_CALIBRATION.md, and reports/DEVELOPMENT_NOTES_ROBUSTNESS.md.

Subgroup diagnostics cover mission, history length, and entry-message completeness. The three-CDM minimum-history gate is frozen. PAC calibration on 73 positive events selected rank 3 and threshold 0.002248533976580822; the single evaluation run is complete.

A split-conformal applicability gate is implemented for numeric event features. It blocks SAFE-EXCLUDE on non-finite inputs and on events outside the calibrated robust-deviation region. Its false-flag statement is marginal and requires event-level exchangeability; it is not a guarantee under arbitrary distribution shift. Offline sequential evaluation accepts the same fitted and calibrated gate as the runtime policy, reports blocked threshold crossings, and only counts a decision when both the calibrated score rule and gate permit SAFE-EXCLUDE.

The runtime policy combines the calibrated lower threshold, minimum-history rule, applicability gate, and the calibrated 2–7 day decision window. Updates outside the window remain auditable but cannot receive SAFE-EXCLUDE; the minimum-history counter includes only updates inside the window. Every accepted update records its sequence number, eligible-history count, score, gate result, decision, and reason. Out-of-order event updates are rejected. The ESCALATE threshold is an operational prioritisation setting and is not covered by the dangerous-exclusion guarantee. Runtime state can be checkpointed atomically with SequentialTriagePolicy.checkpoint(path) and restored with SequentialTriagePolicy.restore(path, shift_gate=gate). The checkpoint contains the policy configuration, per-event sequence and window-history counters, last accepted TCA, and the append-only audit log. Its embedded SHA-256 and optional shift-gate fingerprint are verified before state is restored.

Offline diagnostics now use eligible_history_count, which starts when the 2–7 day decision window opens. This matches the runtime counter; total pre-window history remains available separately as n_cdm_so_far.

The first nested gate-aware development diagnostic used current risk as the only gate feature and event-level gate alpha 0.05. It blocked no held-out events and matched its ungated control.

A follow-up diagnostic restored the feature-rich development partition from the checksum-verified ESA archive and evaluated a pre-specified four-feature gate: risk, max_risk_estimate, miss_distance, and mahalanobis_distance. At gate alpha 0.05 it flagged 358/7,146 held-out events (5.01%), all from the low-final-risk class. It did not prevent any dangerous exclusion at minimum histories 1, 3, 4, or 5, while reducing correct SAFE-EXCLUDE decisions by 33--46 events. The multivariate gate therefore behaved as a conservative applicability filter but did not improve the development safety frontier. This exploratory result is in reports/development_multivariate_shift_gate_v7.csv.

A development-only event-aligned CatBoost candidate now focuses half of each positive event's training mass on its lowest-scored quarter of prefixes. Hard-prefix scores are generated by nested inner out-of-fold models, so the outer evaluation fold is never used to define training weights. At the frozen PAC alpha 0.10 and minimum history 3, it retained the snapshot baseline's 12/192 dangerous exclusions (UCB95 9.93%) while increasing correct SAFE-EXCLUDE decisions from 5,117 to 5,286 of 6,954 negatives. The paired coverage gain was 2.43 percentage points (bootstrap 95% CI 1.77--3.12; exact McNemar p = 7.33e-13), and median first SAFE-EXCLUDE moved from 4.98 to 5.33 days before TCA. This is exploratory development evidence, not a replacement confirmation result.

Paired subgroup diagnostics show that the pooled coverage gain is not fold-stable. Coverage improved in folds 1--3 by 4.60--6.97 percentage points but decreased in folds 0 and 4 by 2.16 and 2.66 points. Pooled dangerous exclusions remained 12/192, although individual dangerous events moved between missions and folds. Coverage improved in all three history groups; the largest gain was 4.25 points for events with 1--4 CDMs in the decision window. Missingness groups none and low improved by 2.28 and 3.00 points; the high group estimate was +1.51 points with a bootstrap 95% interval crossing zero. These results are exploratory and do not establish subgroup-specific safety guarantees.

A no-retraining score ensemble study compared arithmetic mean, geometric mean, pointwise maximum, and pointwise minimum of the snapshot and tail-aligned OOF scores. Lower scores mean safer predictions, so the maximum is pointwise pessimistic and the minimum pointwise optimistic before calibration. After independent PAC recalibration, those pointwise subset relations no longer determine the decision frontier. Arithmetic mean, geometric mean, and maximum each produced 13/192 dangerous exclusions (UCB95 10.55%) and failed the 10% development criterion. The recalibrated minimum produced 10/192 (UCB95 8.67%) with 5,237/6,954 safe negatives (75.31%). The tail-aligned score remained on the development Pareto frontier with higher coverage (76.01%) at 12/192 danger; the minimum ensemble formed a second, more conservative frontier point. This is exploratory development evidence only.

A 250-repeat stratified 50/50 calibration-stability diagnostic was then run on the fixed OOF scores. These repeats vary the event-level calibration/test split only; they are correlated stability estimates, not independent model replications. Median safe-negative rates were 70.19% for snapshot, 73.94% for tail-aligned, and 74.81% for the minimum ensemble. Tail-aligned and minimum improved coverage over snapshot in 80% and 82% of repeats, but their 5th-percentile paired changes were negative (-3.46 and -2.56 percentage points). Mean dangerous-event rates were 5.24%, 5.24%, and 5.31%, while dangerous-event count was no worse than snapshot in 59.2% and 58.0% of repeats. The minimum ensemble's apparent pooled safety advantage therefore did not persist across calibration splits.

Development tuning is closed after v11. The next-study candidate is frozen in artifacts/next_validation_preregistration_v12.json with an exclusive lock. The selected score is catboost_tail_aligned with hard fraction 0.25, hard mass 0.50, 500 CatBoost iterations, PAC alpha 0.10, minimum history 3, and the 2–7 day decision window. The next study requires genuinely new, disjoint calibration and evaluation events. Planning targets 100 positive calibration events and 200 positive evaluation events; this gives PAC rank 5 with bound 8.92% and about 79.6% probability of passing the 10% evaluation UCB criterion if the true dangerous-exclusion rate is 5%.

The preregistered score model is now trained on the full development partition and stored as artifacts/catboost_tail_aligned_final_v13.cbm. Its SHA-256 is 9607c64be397229a9814131d5011e80a7df28d62a51abe514e8b7814e6d67e72. Fresh five-fold inner-OOF dynamic scores were generated on the development events to define the frozen positive-tail weights before the final fit. The manifest records 66,592 prefixes, 7,146 events, 192 positives, 500 trees, and no calibration or evaluation access. The model has no decision threshold; PAC calibration on genuinely new events is still required.

Quick historical demo

Build the locked confirmation_v1 replay audit and self-contained operator console with one command:

python scripts/run_demo.py --output-dir /tmp/cdm-sequential-triage-demo

The command writes replay-audit.parquet, runtime-state.json, operator-console.html, evidence-dashboard.html, and summary.json into a fresh output directory outside source and artifact folders. The evidence dashboard keeps exploratory development results, the locked confirmation_v1 result, and the preregistered v13 study in separate, color-coded tiers and verifies their SHA-256 links before rendering. It refuses non-empty or protected destinations and removes staged output if either replay or dashboard generation fails. The demo is label-blind historical runtime evidence only: confirmation_v1 did not meet its pre-specified 10% UCB criterion and does not validate the v13 candidate.

Reproducibility

• Dataset source/checksums: data/manifest.json
• Statistical protocol: PROTOCOL.md
• Causal feature builder: src/prefix_features.py
• Event-level policy and calibration utilities: src/policy.py
• Conformal applicability gate: src/shift_gate.py
• Stateful three-decision policy and audit log: src/triage.py
• Replay a pre-scored stream through the calibrated runtime: python scripts/replay_scores.py --scores artifacts/confirmation_v1/evaluation_scores.parquet --calibration artifacts/confirmation_v1/calibration.json --checkpoint runtime-state.json --output replay-audit.parquet. Add --gate when the calibration artifact is bound to a shift gate, and --model-manifest when it is bound to a model manifest. The command verifies model, calibration, gate, and manifest identities; refuses calibration-event overlap and duplicate updates; and resumes only from a matching checkpoint. Each output contains decisions accepted from that input batch, with its own artifact SHA-256 lineage and an is_current_decision marker. Audit output is committed before the staged checkpoint replaces the previous state, so a failed audit write cannot advance runtime state. The included confirmation_v1 files are suitable for a historical runtime demonstration only; they do not validate the v13 candidate.
• Build the self-contained operator console: python scripts/operator_dashboard.py --audit replay-audit.parquet --calibration artifacts/confirmation_v1/calibration.json --confirmation artifacts/confirmation_v1/confirmation.json --output operator-console.html. Multiple --audit files may be supplied in chronological order after checkpoint resume. The dashboard verifies calibration/model lineage, rejects overlapping event sequence numbers, derives current state from the latest sequence per event, escapes event identifiers, limits the rendered active queue with --max-events, and embeds no external assets. Message-level runtime counts and event-level confirmation metrics are displayed separately. The historical confirmation failure is shown explicitly and is not presented as validation of v13.
• Shift-gate artifacts are fitted on training events, calibrated from the maximum nonconformity over each calibration event's 2–7 day path, and fingerprinted before use in confirmation.
• Calibration diagnostics: python scripts/calibration_diagnostics.py --output reports/calibration.csv
• Event-aligned development model: python scripts/event_aligned_diagnostics.py --training data/processed/partitions/training.parquet --oof-scores artifacts/development_oof_model_comparison_v2.parquet --hard-fraction 0.25 --hard-mass 0.50 --minimum-history 3 --alpha 0.10 --mode pac --iterations 500 --scores-output artifacts/development_event_aligned_oof_v8.parquet --report-output reports/development_event_aligned_v8.csv --manifest-output artifacts/development_event_aligned_v8.json. The two-stage training uses nested inner OOF scores and preserves total weight one per event.
• Event-aligned robustness: python scripts/event_aligned_robustness.py --training data/processed/partitions/training.parquet --baseline-scores artifacts/development_oof_model_comparison_v2.parquet --candidate-scores artifacts/development_event_aligned_oof_v8.parquet --minimum-history 3 --alpha 0.10 --mode pac --decisions-output artifacts/development_event_aligned_decisions_v9.parquet --report-output reports/development_event_aligned_robustness_v9.csv. The report uses like-for-like cross-fitted thresholds and separately summarizes missions, window-local history, first-message missingness, and outer folds.
• Score ensemble diagnostics: python scripts/score_ensemble_diagnostics.py --baseline-scores artifacts/development_oof_model_comparison_v2.parquet --candidate-scores artifacts/development_event_aligned_oof_v8.parquet --minimum-history 3 --alpha 0.10 --mode pac --scores-output artifacts/development_score_ensemble_oof_v10.parquet --decisions-output artifacts/development_score_ensemble_decisions_v10.parquet --summary-output reports/development_score_ensemble_v10.csv --fold-output reports/development_score_ensemble_folds_v10.csv --manifest-output artifacts/development_score_ensemble_v10.json. Every combination is recalibrated separately; the output marks safety-feasible and Pareto-efficient development points.
• Repeated calibration stability: python scripts/repeated_calibration_stability.py --scores artifacts/development_score_ensemble_oof_v10.parquet --repeats 250 --test-fraction 0.5 --seed-base 24072026 --minimum-history 3 --alpha 0.10 --mode pac --detail-output artifacts/development_score_ensemble_repeated_v11.parquet --summary-output reports/development_score_ensemble_repeated_v11.csv --manifest-output artifacts/development_score_ensemble_repeated_v11.json. The repeats share fixed OOF scores and must not be described as independent retraining runs.
• Freeze the next validation plan: python scripts/freeze_next_validation.py --terminal-artifact artifacts/development_event_aligned_v8.json --terminal-artifact artifacts/development_score_ensemble_v10.json --terminal-artifact artifacts/development_score_ensemble_repeated_v11.json --output artifacts/next_validation_preregistration_v12.json --lock artifacts/next_validation_preregistration_v12.lock --planning-output reports/next_validation_sample_size_v12.csv. The command is one-shot and refuses to overwrite the lock.
• Train the frozen score model: python scripts/train_final_tail_aligned.py --training data/processed/partitions/training.parquet --fold-table artifacts/development_oof_model_comparison_v2.parquet --preregistration artifacts/next_validation_preregistration_v12.json --lock artifacts/next_validation_preregistration_v12.lock --model artifacts/catboost_tail_aligned_final_v13.cbm --manifest artifacts/catboost_tail_aligned_final_v13.json. The trainer verifies the preregistration and development-partition hashes, generates fresh inner-OOF base scores, and refuses to overwrite an existing final model.
• Freeze the genuinely new study before opening outcomes: python scripts/freeze_new_study.py --calibration-features data/new/calibration_features.parquet --evaluation-features data/new/evaluation_features.parquet --preregistration artifacts/next_validation_preregistration_v12.json --preregistration-lock artifacts/next_validation_preregistration_v12.lock --output artifacts/new_study.json --lock artifacts/new_study.lock. The command requires disjoint event rosters and creates a one-shot lock over exact files and event IDs. For a preregistered decision window it also enforces an outcome firewall: feature cohorts may contain pre-window history but no rows after the window closes, and columns explicitly named as final outcomes are rejected. The manifest records columns, TCA range, decision-window coverage, and the exact roster of events that enter the decision window. Calibration and confirmation reject score files that omit or add any event from this roster. This mechanical check reduces accidental outcome leakage and silent scoring attrition; it cannot prove that an externally prepared, innocuously named feature was created without outcome access.
• Score genuinely new, label-blind CDM histories with the frozen model: python scripts/score_final_tail_aligned.py --features data/new/calibration_features.parquet --model artifacts/catboost_tail_aligned_final_v13.cbm --manifest artifacts/catboost_tail_aligned_final_v13.json --output artifacts/new_calibration_scores.parquet --study-manifest artifacts/new_study.json --study-lock artifacts/new_study.lock --cohort calibration [--gate-features risk max_risk_estimate miss_distance mahalanobis_distance]. Repeat with the evaluation feature file and --cohort evaluation. If a shift gate will be used, pass the same feature list for both cohorts; the columns are retained in the score files for confirm_policy.py --gate. The command rejects input containing y, verifies the model and study hashes, builds causal dynamic features, and refuses to overwrite scores. Each Parquet score artifact is written to a temporary file in the destination directory, flushed with fsync, and committed with an atomic replacement; interrupted writes leave neither a published partial artifact nor a stale temporary file.
• Minimum-history/timing diagnostics: python scripts/history_gate_diagnostics.py --output reports/history_gate.csv
• Nested gate-aware diagnostics: python scripts/history_gate_diagnostics.py --scores artifacts/development_oof_model_comparison_v2.parquet --gate-feature-scores data/processed/partitions/training.parquet --score-col catboost_snapshot --minimum-history 1 3 4 5 --gate-features risk max_risk_estimate miss_distance mahalanobis_distance --gate-alpha 0.05 --output reports/development_multivariate_shift_gate_v7.csv. Each held-out fold uses disjoint folds for gate fitting, event-level gate calibration, policy calibration, and evaluation. The output separately reports any gate flag, threshold crossings actually blocked by the gate, and an ungated control under the same fold allocation.
• Build frozen event partitions: python scripts/make_partitions.py --archive data/raw/train_data.zip --output-dir data/processed/partitions --manifest data/processed/partitions.json --expected-sha256 68362fe5629cc80f17291f2d73f733bf4e922675e37b91a8ee79afadb46f3edc
• Train and score frozen snapshot model: python scripts/train_snapshot.py --training data/processed/partitions/training.parquet --calibration data/processed/partitions/calibration.parquet --evaluation-features data/processed/partitions/evaluation_features.parquet --model artifacts/catboost_snapshot.cbm --calibration-scores artifacts/calibration_scores.parquet --evaluation-scores artifacts/evaluation_scores.parquet --manifest artifacts/snapshot_model.json
• Gate-aware training and scoring: add --gate-features risk miss_distance mahalanobis_distance --gate-output artifacts/shift_gate.json --gate-alpha 0.05; the selected numeric columns are retained in both score files.
• Frozen calibration: python scripts/confirm_policy.py calibrate --scores artifacts/new_calibration_scores.parquet --labels data/new/calibration_labels.parquet --model-manifest artifacts/catboost_tail_aligned_final_v13.json --study-manifest artifacts/new_study.json --study-lock artifacts/new_study.lock --output calibration.json [--gate artifacts/shift_gate.json]. The artifact stores both the declared calibration event roster and its canonical SHA-256 digest. Confirmation recomputes this digest and rejects a missing, duplicated, or modified roster before evaluating event overlap or outcomes.
• One-shot confirmation: python scripts/confirm_policy.py confirm --scores artifacts/new_evaluation_scores.parquet --labels data/new/evaluation_labels.parquet --model-manifest artifacts/catboost_tail_aligned_final_v13.json --study-manifest artifacts/new_study.json --study-lock artifacts/new_study.lock --calibration calibration.json --output confirmation.json --lock confirmation.lock [--gate artifacts/shift_gate.json]. Before creating the irreversible confirmation lock, the command performs a label-blind preflight over the evaluation scores, frozen-study binding, calibration/model/gate identities, required columns, model hash, duplicate updates, and calibration/evaluation event overlap. Invalid label-blind inputs therefore do not consume the one permitted confirmation run. Label-roster validation remains after lock acquisition because it accesses evaluation outcomes.
• Subgroup diagnostics: python scripts/robustness_diagnostics.py --group mission_id --output reports/mission.csv
• Tests: pytest -q

The partition command reproduces the frozen 60/20/20 event split and keeps complete calibration/evaluation label rosters, including events with no CDM in the decision window. The training command scores evaluation features without loading their outcomes. The confirmation command creates an exclusive lock before loading event-level labels, rejects calibration/evaluation event overlap, verifies the frozen policy, and records input checksums. Keep the lock and result under version control after the first confirmatory run.

Data

ESA Collision Avoidance Challenge, DOI: 10.5281/zenodo.4463683, CC BY 4.0. Raw data are not redistributed in this repository.

Limitations

The target is high final calculated collision risk, not collision occurrence. Statistical control relies on event-level exchangeability with calibration data and does not constitute an operational safety guarantee under arbitrary distribution shift.