UiPath · mohsen-uipath · Jul 9, 2026 · Jul 9, 2026 · Jul 9, 2026 · Jul 9, 2026
@@ -5,21 +5,27 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
 
 ## From code review 260701-1954 (fix-review-top5 run) — deferred to a dedicated guardrail plan
 
-- **CE024** — workflow-YAML rule: forbid any `uses:` step pinned to a floating ref
+> Numbering note: these are *proposed* ids. `CE024` (discriminated-unions) and
+> `CE025` (live-verdict consistency) have since been **implemented** for other
+> rules, so the candidates below were renumbered to the next free ids. Always
+> claim the next unused number in `tests/lint/rules/` — the id-uniqueness assert
+> in `tests/lint/runner.py` is the source of truth.
+
+- **CE026** — workflow-YAML rule: forbid any `uses:` step pinned to a floating ref
   (`@v3`, `@main`) rather than a 40-hex commit SHA. Would have caught
   `mxschmitt/action-tmate@v3` (fixed manually in this run). >30 min: needs a
   non-Python file-walk branch in `tests/lint/runner.py`.
-- **CE025** — retired-token grep gate: fail when a removed-subsystem token
+- **CE027** — retired-token grep gate: fail when a removed-subsystem token
   (`LLMGW_`, `API_BACKEND=proxy`, `uipath_llmgw_client`) reappears outside an
   allowlist across docs/config/src. Would have caught the LLM-Gateway residue
   swept in this run. >30 min: needs an allowlist + repo-wide text scan.
-- **CE026** — assert the Makefile `lint:` help does not hardcode a stale `CE0NN`
+- **CE028** — assert the Makefile `lint:` help does not hardcode a stale `CE0NN`
   upper bound (use `CE001+`). Would have caught the `CE001–CE005` drift fixed here.
 - **docs-vs-harness smoke test** — execute the CI tutorial's `coder-eval run`
   command against a NoOp task and assert the produced tree matches the documented
   globs. Would have caught the `--run-dir runs` layout bug. Not statically
   reachable (needs a live run).
-- [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CE025), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
+- [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CExxx at the next free id), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
 
 ## From 2026-07-03 open-source docs cleanup
 

@@ -36,7 +36,7 @@ coder_eval/
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
 │   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
-│   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, CriterionAggregate, ThresholdCheck, SuiteRollup
+│   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
 │   ├── tasks.py                   # TaskDefinition, AgentConfig, Dataset (dataset fan-out + sample)
@@ -79,6 +79,7 @@ coder_eval/
 ├── orchestration/                 # Batch execution utilities
 │   ├── batch.py                   # Parallel task execution (run_batch + run_batch_resolved)
 │   ├── config.py                  # Batch run configuration
+│   ├── early_stop.py              # validate_early_stop guardrails + EarlyStopWatcher (armed live-verdict observer)
 │   ├── evaluation.py              # Evaluation helpers
 │   ├── experiment.py              # ExperimentRunner, resolve_task_for_variant, load_experiment
 │   └── task_loader.py             # YAML task loading
@@ -138,6 +139,7 @@ templates/                         # Sandbox template directories
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided`; only criteria that can decide from a partial trajectory may arm (they declare a non-empty `live_stop_polarities` ClassVar and override `live_verdict` on `BaseCriterion` — currently `skill_triggered`, `command_executed`; CE025 enforces the two stay consistent). It uses a cooperative `should_stop` seam on the Claude agent's between-messages guard (tool-call granularity, no SIGKILL) driven by `orchestration/early_stop.py::EarlyStopWatcher` (own `EventCollector` + stop rule). Live verdicts only *trigger* the stop; the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use (non-observable criterion, non-Claude agent, wrong polarity, no armed criterion, simulation mode) is an error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (reason + deciding criterion + when), report notes/badges, `stopped_early` run.json rows, and `EarlyStopped`/`EarlyStopReason` telemetry dims. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
@@ -158,7 +160,7 @@ templates/                         # Sandbox template directories
 | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) |
 | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. |
 
-All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
+All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
 
 ## Evaluation Flow
 
@@ -191,11 +193,11 @@ make format      # ruff format
 make check       # ruff check (lint)
 make typecheck   # pyright
 make test        # pytest
-make lint        # custom architectural lint rules (CE001–CE013)
+make lint        # custom architectural lint rules (CE001–CE025)
 make verify      # All of the above + coverage check (CI equivalent)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE013 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE025 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
 
 ## Configuration
 

@@ -17,6 +17,7 @@ to the orchestrator are involved.
 - [Recipe: A/B a Skill](#recipe-ab-a-skill)
 - [Recipe: A/B a Model](#recipe-ab-a-model)
 - [Recipe: A/B a Prompt](#recipe-ab-a-prompt)
+- [Recipe: Smoke vs. e2e Flavors (Early Stop)](#recipe-smoke-vs-e2e-flavors-early-stop)
 - [Replicates (Statistical Power)](#replicates-statistical-power)
 - [Measuring the Difference](#measuring-the-difference)
 - [CLI Reference](#cli-reference)
@@ -222,6 +223,39 @@ variants:
 The full mutation catalog (prefix / suffix / replace / template / rephrase) is
 defined in `coder_eval/models/mutations.py`.
 
+## Recipe: Smoke vs. e2e Flavors (Early Stop)
+
+Run the **same** task file as both a fast `smoke` flavor and a full `e2e` flavor
+by flipping one boolean per variant — `run_limits.stop_early`. Arm the criteria
+that define "the interesting thing happened" with `stop_when` in the task file;
+the `smoke` variant cuts off as soon as they're decided, while `e2e` runs to
+completion. Because the field merge is per-key, the variant sets only
+`stop_early` without disturbing the task's `max_turns`.
+
+```yaml
+experiment_id: early-stop-ab
+description: "Smoke vs. e2e from one file via opt-in early stop"
+
+variants:
+  - variant_id: e2e
+    run_limits:
+      stop_early: false # full run to completion (the reference flavor)
+  - variant_id: smoke
+    run_limits:
+      stop_early: true # cut off once the armed criteria are decided
+```
+
+The task file supplies the arming (`stop_when` on the criteria that gate the
+flavor) and a `max_turns` generous enough for `e2e`; see
+[`stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop). This recipe
+ships as `experiments/early-stop-ab.yaml`.
+
+Expect **identical pass/fail verdicts** between the two variants — an
+early-stopped run is gated on the armed subset only, and the non-armed criteria
+become advisory (clearly marked in the report), so the `smoke` flavor can't
+"pass for free" — with the `smoke` variant significantly lower on turns,
+duration, and tokens.
+
 ## Replicates (Statistical Power)
 
 Agents are stochastic — a single run per arm is noise, not signal. Set `repeats`

@@ -192,6 +192,56 @@ budgets consistently across a suite — the headline % is only comparable when
 tasks are measured against realistic, like-for-like targets. Omit it (the
 default) to exclude a task from the metric entirely.
 
+### `stop_early` (opt-in early stop)
+
+`run_limits.stop_early` (default `false`) ends a single-shot run **early** once
+the run's **armed** criteria are decided — so you can raise `max_turns` for the
+full-run flavor without paying for turns the smoke flavor doesn't need. A
+criterion is *armed* by giving it a `stop_when` (see the criterion-fields table);
+`stop_early` is the master switch that turns arming on for the run.
+
+```yaml
+run_limits:
+  max_turns: 30
+  stop_early: true            # opt in; default false leaves behavior unchanged
+success_criteria:
+  - type: skill_triggered
+    skill_name: date-teller
+    expected_skill: date-teller
+    stop_when: decided        # arm on pass OR definitive fail
+  - type: file_exists         # not armed → advisory on an early-stopped run
+    path: report.md
+```
+
+Semantics:
+
+- **Opt-in, per run.** With `stop_early: false` (the default) the run behaves
+  exactly as before — `stop_when` is inert and every criterion gates normally.
+- **Polarity.** `stop_when: pass` stops the moment all armed criteria are decided
+  in the pass direction; `stop_when: fail` stops on a definitive wrong-signal
+  fail; `stop_when: decided` stops on either. Only criteria that can decide from a
+  partial trajectory (currently `skill_triggered`, `command_executed`) may be
+  armed — arming any other criterion is a hard error at resolution (plan *and*
+  run), never a silent no-op.
+- **Verdict.** An early-stopped run is gated on the **armed subset only**; the
+  non-armed criteria become **advisory** and are clearly marked (report badge +
+  per-criterion note + `stopped_early` row). A run that completes naturally is
+  gated on the **full** set, as always. This is what lets one file serve both a
+  `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`)
+  with identical verdicts — see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md).
+- **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) —
+  it can never silently disable a criterion or cause a false early stop.
+
+Observability (every early-stopped run is flagged everywhere so analysis never
+compares a truncated run against a full one):
+
+| Surface | Field / marker |
+|---------|----------------|
+| `run.json` row | `stopped_early`, `early_stop_reason`, `turns_remaining_at_stop` |
+| `run.md` | `> **NOTE:** […] stopped early (<reason>); <= N turn(s) avoided …` |
+| `task.html` | header badge `stopped early (<reason>)` + `advisory — not gated` markers |
+| Telemetry | `EarlyStopped` / `EarlyStopReason` dimensions on `CoderEval.Task.End` |
+
 ## Sandbox Configuration
 
 The `sandbox` block is optional. When omitted, it defaults to `driver: "tempdir"` with standard Python environment.
@@ -319,6 +369,7 @@ All criteria share these fields:
 | `description` | — | Human-readable description (required) |
 | `weight` | 1.0 | Relative importance for weighted score |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
+| `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 **Scoring types:**
 - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`

@@ -95,6 +95,7 @@ created are preserved under `runs/latest/<variant>/<task>/<NN>/` (`task.json`,
 ## Where to go deeper
 
 - **All 14 criterion types, weights, thresholds** → [Task Definition Guide](../TASK_DEFINITION_GUIDE.md)
+- **Stop a run early once the key criteria are decided** (opt-in `run_limits.stop_early` + `stop_when` on a criterion) → [Task Definition Guide → `stop_early`](../TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
 - **Fan one task out over a dataset of rows** → [Bring Your Own Data](../BYOD.md)
 - **Full CLI & config reference** → [User Guide](../USER_GUIDE.md)
 - **Compare two configurations on this task** → [Tutorial 05](05-comparing-models.md)
@@ -0,0 +1,23 @@
+experiment_id: early-stop-ab
+description: |
+  Smoke vs. e2e flavors from ONE task file via opt-in early-stop. Both variants
+  share the same tasks, criteria, and max_turns; they differ only in the
+  run_limits.stop_early boolean (field-merged, so neither replaces the task's
+  run_limits block). The task's armed criteria (those carrying `stop_when`)
+  decide when the `smoke` variant cuts off — as soon as the designated criteria
+  are decided — while `e2e` runs every task to completion.
+
+  Expect identical pass/fail verdicts between the two variants (the armed subset
+  gates an early-stopped run; other criteria are advisory and clearly marked),
+  with the `smoke` variant significantly lower on turns, duration, and tokens.
+
+variants:
+  - variant_id: e2e
+    description: "Full run to completion — no early stop (the reference flavor)."
+    run_limits:
+      stop_early: false
+
+  - variant_id: smoke
+    description: "Opt-in early stop — cut off once the armed criteria are decided."
+    run_limits:
+      stop_early: true