fix(skill): harden phase delivery quality loop - #122
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe phase-delivery workflow now requires global ChangesPhase delivery quality loop
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant PhaseDelivery
participant offload
participant Reviewer
participant Implementer
participant CandidateReview
PhaseDelivery->>offload: plan semantic implementer and reviewer routes
offload-->>PhaseDelivery: return compatible route metadata
PhaseDelivery->>Reviewer: request must-cover bullets and authority result
Reviewer-->>PhaseDelivery: return stable-ID bullets
PhaseDelivery->>Implementer: pass route, bullets, and authority
Implementer-->>PhaseDelivery: return proof mappings and verification evidence
PhaseDelivery->>CandidateReview: provide exact candidate and recorded evidence
CandidateReview-->>PhaseDelivery: return grouped findings and re-review scope
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 950ad97fb0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/orchestrate-phase-delivery/evals/evals.json:
- Around line 6-13: Update the evaluation’s expected_output and related
assertions so offload resolves only model, effort, compatible runtime type,
isolated context, budget, and fallback routing; require the local Jig workflow
to assign and record verification ownership in the dispatch trace. Remove
verification ownership from global offload planning while preserving all other
risk classification, authority, fallback, and admission requirements.
In @.agents/skills/orchestrate-phase-delivery/SKILL.md:
- Around line 45-55: The workflow must block all writes for an admitted story
when required authority is unresolved. In
.agents/skills/orchestrate-phase-delivery/SKILL.md lines 45-55, remove
permission for mixed-scope bookkeeping after OWNER_DECISION_REQUIRED; in
implementer-review-design.md lines 195-198, allow authorized bookkeeping only
when no required authority remains unresolved; in references/phase-protocol.md
lines 55-65, make OWNER_DECISION_REQUIRED stop every write for that story while
allowing independent ready stories to continue; and in evals/evals.json lines
5-18, require the GF-020-like story to stop until authority is resolved or scope
is explicitly reauthorized.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5443ea9f-bca0-4297-8e7f-25e77c4884d9
📒 Files selected for processing (6)
.agents/skills/orchestrate-phase-delivery/README.md.agents/skills/orchestrate-phase-delivery/SKILL.md.agents/skills/orchestrate-phase-delivery/evals/evals.json.agents/skills/orchestrate-phase-delivery/evals/trigger_queries.json.agents/skills/orchestrate-phase-delivery/implementer-review-design.md.agents/skills/orchestrate-phase-delivery/references/phase-protocol.md
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
.agents/skills/orchestrate-phase-delivery/evals/evals.json (2)
41-54: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winPreserve the full
OWNER_DECISION_REQUIREDcontract.The protocol also allows material scope, dependency, realization, provider-reachability, and accepted-trade-off decisions. The phrase “only for a material authority decision” is narrower than that contract. State that this scenario has no such decision, or list the full allowed decision categories.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.agents/skills/orchestrate-phase-delivery/evals/evals.json around lines 41 - 54, Update the expected_output and related assertions for worktree recovery to preserve the full OWNER_DECISION_REQUIRED contract: state that this scenario contains no material authority decision, or explicitly include material scope, dependency, realization, provider-reachability, and accepted-trade-off decisions alongside authority decisions. Do not narrow the contract to authority decisions alone.
41-50: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winScope incompatible-route replacement to implementers.
The protocol permits implementer replacement for an incompatible runtime route. Reviewer replacement is limited to repeated boundary breach, mutation, evidence writing, or loss of availability. The unqualified route condition here can authorize reviewer replacement for a route mismatch. Mark this condition as implementer-only, or update the protocol and design together if reviewer route mismatch is intended.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.agents/skills/orchestrate-phase-delivery/evals/evals.json around lines 41 - 50, Update the incompatible approved runtime route replacement condition in the protocol represented by the eval prompt and assertions so it applies only to implementers, not reviewers. Preserve reviewer replacement eligibility exclusively for repeated boundary breaches, mutation, evidence writing, or loss of availability, and keep the existing implementer replacement behavior unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/orchestrate-phase-delivery/evals/evals.json:
- Around line 6-7: The expected output and its matching assertion should
distinguish runtime agent type from planned model and effort: reject fixed
runtime type Codex because it forces the incompatible Terra model at medium
effort, rather than describing Terra/medium as the fixed type. Preserve the
separate planned_model Luna and high-effort requirements.
- Around line 6-7: Update the evaluation prompt to provide explicit offload
route and fallback inputs for semantic reviewer roles, not only risky
implementers, before asserting that global offload plans every implementer and
reviewer. Preserve the expected requirement that both roles use planned model,
effort, compatible runtime type, isolated context, budget, and fallback routing.
---
Outside diff comments:
In @.agents/skills/orchestrate-phase-delivery/evals/evals.json:
- Around line 41-54: Update the expected_output and related assertions for
worktree recovery to preserve the full OWNER_DECISION_REQUIRED contract: state
that this scenario contains no material authority decision, or explicitly
include material scope, dependency, realization, provider-reachability, and
accepted-trade-off decisions alongside authority decisions. Do not narrow the
contract to authority decisions alone.
- Around line 41-50: Update the incompatible approved runtime route replacement
condition in the protocol represented by the eval prompt and assertions so it
applies only to implementers, not reviewers. Preserve reviewer replacement
eligibility exclusively for repeated boundary breaches, mutation, evidence
writing, or loss of availability, and keep the existing implementer replacement
behavior unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 65eda84c-45b7-49d0-9c10-be15f76337d3
📒 Files selected for processing (5)
.agents/skills/orchestrate-phase-delivery/README.md.agents/skills/orchestrate-phase-delivery/SKILL.md.agents/skills/orchestrate-phase-delivery/evals/evals.json.agents/skills/orchestrate-phase-delivery/implementer-review-design.md.agents/skills/orchestrate-phase-delivery/references/phase-protocol.md
🚧 Files skipped from review as they are similar to previous changes (1)
- .agents/skills/orchestrate-phase-delivery/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Phase 2 showed that reviewer preparation and grouped findings improved review quality, but planned Luna/high work could still run through fixed Terra/medium runtime types, incomplete proof claims could freeze candidates, authority gaps reached implementation, and pair replacements discarded useful story context.
Scope
Verification
pnpm check— passed, 18/18 taskspnpm delivery:check— passed, 48 stories / 7 phasespython3 .agents/skills/orchestrate-phase-delivery/scripts/validate_evals.py— passed, 3 output evals / 20 balanced trigger queriesvalidate_skill.py— passedgit diff --check— passedSummary by CodeRabbit
Documentation
Tests