From f326b717f636c7e61da61773df04e291d34b57e0 Mon Sep 17 00:00:00 2001 From: Reid Baker Date: Mon, 27 Jul 2026 12:11:54 -0400 Subject: [PATCH 1/3] Genericize run-evals prompts and enforce lint cheating rubric --- tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md | 8 ++++---- .../run-evals/resources/baseline_execution_prompt.md | 1 + .../run-evals/resources/with_skill_execution_prompt.md | 1 + tool/dart_skills_lint/evals/code_quality_rubric.json | 7 +++++++ .../dart_skills_lint/evals/code_quality_rubric_evals.json | 3 ++- 5 files changed, 15 insertions(+), 5 deletions(-) diff --git a/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md b/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md index 776354f9..127fe6cc 100644 --- a/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md +++ b/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md @@ -7,13 +7,13 @@ metadata: # Run Skill Evals -1. **Read Framework**: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals. -2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `tool/dart_skills_lint/evals/`. +1. **Read Framework**: Read `/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals (where `` is the directory containing the `.agents` or `skills` folder). +2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `/evals/`. 3. **Determine Agent Configuration**: Check the `agent_config` field in the target target JSON file to determine the environment/harness to spawn. If `agent_config` is `"bare-agent"`, spawn a subagent with the `bare-agent` profile. If it is a specific contributor profile (e.g. `"reidbaker-agent"`), use that profile to provide the necessary contributor context. 4. **Orchestrate**: By default, run an Integration Test by spawning a single **With-Skill** subagent using `Workspace: branch` and the identified `agent_config`. - - Provide the task prompt. See `resources/with_skill_execution_prompt.md` for the template. When filling in ``, you MUST use a relative path from the repository root, not an absolute path. If you are running a cross-skill evaluation, fill in `` with `"none (cross-skill meta-eval)"`. + - Provide the task prompt. See `resources/with_skill_execution_prompt.md` for the template. When filling in ``, you MUST use a relative path from the repository root, not an absolute path. If you are running a cross-skill evaluation, fill in `` with `"none (cross-skill meta-eval)"`. Also, replace `` with the actual directory path. - **Only if the user explicitly requests a comparison or benchmark**, also spawn a **Baseline** subagent. See `resources/baseline_execution_prompt.md` for the template. - Instruct the subagent(s) to return their `git diff` and verification outputs (`dart format`, `dart analyze`, `dart test`) without committing. + Instruct the subagent(s) to return their `git diff` and verification outputs (`dart format`, `dart analyze`, `dart test`) without committing. Ensure you instruct them to run these commands exclusively from within the `` directory to avoid analyzing unrelated packages. **CRITICAL**: You must explicitly warn the subagent(s) to confine all file edits strictly to their current working directory and avoid using absolute paths to modify the parent workspace. **WORKSPACE LIMITATION WARNING**: If the user has multiple active workspaces mounted, the `Workspace: branch` feature will fail. In this situation, you MUST warn the user that running concurrent evaluations in `Workspace: inherit` mode will cause git state bleed and cross-eval pollution (e.g., changes made by a failure scenario will be visible to a success scenario running simultaneously in the same shared directory). Instruct the user to fix this by closing all workspaces except the primary package workspace, and then re-run the evaluations. Do NOT silently fallback to `Workspace: inherit` for concurrent tasks. 5. **Grade**: Parse the combined rubric (resolving `repo_criteria` + `evals.json` expectations). Use the grading instructions in `resources/agent_judge_prompt.md`. When an expectation fails, you MUST explicitly list both the expectation and what was actually found that caused the failure. diff --git a/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md b/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md index a9a4a98d..7e6747bc 100644 --- a/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md +++ b/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md @@ -5,4 +5,5 @@ Execute this task: WARNING: You are executing in an isolated branch workspace. Confine all file modifications strictly to your current working directory. Do NOT use absolute paths to modify files in the parent workspace. Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart format`, `dart analyze`, `dart test`). +CRITICAL: You must explicitly `cd` into the `` directory before running any verification commands to avoid analyzing unrelated packages in the workspace! NOTE: If your task is strictly to grade, review, or evaluate code, do NOT fix the issues you find. Leave the code exactly as it is, even if verification commands fail. Your job is only to report the evaluation results. diff --git a/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md b/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md index d9341be8..0f66468a 100644 --- a/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md +++ b/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md @@ -6,4 +6,5 @@ Execute this task: WARNING: You are executing in an isolated branch workspace. Confine all file modifications strictly to your current working directory. Do NOT use absolute paths to modify files in the parent workspace. Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart format`, `dart analyze`, `dart test`). +CRITICAL: You must explicitly `cd` into the `` directory before running any verification commands to avoid analyzing unrelated packages in the workspace! NOTE: If your task is strictly to grade, review, or evaluate code, do NOT fix the issues you find. Leave the code exactly as it is, even if verification commands fail. Your job is only to report the evaluation results. diff --git a/tool/dart_skills_lint/evals/code_quality_rubric.json b/tool/dart_skills_lint/evals/code_quality_rubric.json index e4514096..02d21e6f 100644 --- a/tool/dart_skills_lint/evals/code_quality_rubric.json +++ b/tool/dart_skills_lint/evals/code_quality_rubric.json @@ -30,6 +30,13 @@ "Source code must be placed in 'lib/', tests in 'test/', and executable scripts in 'bin/' or 'tool/'.", "No temporary scratch files or orphaned directories are left behind after execution." ] + }, + { + "id": "lint_cheating", + "expected_repo_state": [ + "No lint configurations (e.g., analysis_options.yaml or its rules) are removed or disabled.", + "No file-level Dart ignores (e.g., // ignore_for_file:) are added to any Dart files." + ] } ] } \ No newline at end of file diff --git a/tool/dart_skills_lint/evals/code_quality_rubric_evals.json b/tool/dart_skills_lint/evals/code_quality_rubric_evals.json index 789c0d2b..60fbab09 100644 --- a/tool/dart_skills_lint/evals/code_quality_rubric_evals.json +++ b/tool/dart_skills_lint/evals/code_quality_rubric_evals.json @@ -12,7 +12,8 @@ "expected_repo_state": [ "The evaluation artifact explicitly flags a failure for directory placement hygiene.", "The evaluation artifact explicitly flags a failure for effective Dart idioms.", - "The evaluation artifact explicitly flags a failure for cross platform compatibility." + "The evaluation artifact explicitly flags a failure for cross platform compatibility.", + "The evaluation artifact explicitly flags a failure for lint cheating." ], "agent_config": "reidbaker-agent" }, From b8202c93800e789b48ee58732d04f5a0d92ee0c3 Mon Sep 17 00:00:00 2001 From: Reid Baker Date: Mon, 27 Jul 2026 12:20:48 -0400 Subject: [PATCH 2/3] chore: explicitly suggest dart pub get in eval execution prompts --- .../skills/run-evals/resources/baseline_execution_prompt.md | 2 +- .../skills/run-evals/resources/with_skill_execution_prompt.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md b/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md index 7e6747bc..8e65b34b 100644 --- a/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md +++ b/tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md @@ -4,6 +4,6 @@ Execute this task: WARNING: You are executing in an isolated branch workspace. Confine all file modifications strictly to your current working directory. Do NOT use absolute paths to modify files in the parent workspace. -Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart format`, `dart analyze`, `dart test`). +Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart pub get`, `dart format`, `dart analyze`, `dart test`). CRITICAL: You must explicitly `cd` into the `` directory before running any verification commands to avoid analyzing unrelated packages in the workspace! NOTE: If your task is strictly to grade, review, or evaluate code, do NOT fix the issues you find. Leave the code exactly as it is, even if verification commands fail. Your job is only to report the evaluation results. diff --git a/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md b/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md index 0f66468a..8a8b5464 100644 --- a/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md +++ b/tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md @@ -5,6 +5,6 @@ Execute this task: WARNING: You are executing in an isolated branch workspace. Confine all file modifications strictly to your current working directory. Do NOT use absolute paths to modify files in the parent workspace. -Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart format`, `dart analyze`, `dart test`). +Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart pub get`, `dart format`, `dart analyze`, `dart test`). CRITICAL: You must explicitly `cd` into the `` directory before running any verification commands to avoid analyzing unrelated packages in the workspace! NOTE: If your task is strictly to grade, review, or evaluate code, do NOT fix the issues you find. Leave the code exactly as it is, even if verification commands fail. Your job is only to report the evaluation results. From e5d7c88c359a466aec6f9ef4b0be2cadc809934f Mon Sep 17 00:00:00 2001 From: Reid Baker Date: Mon, 27 Jul 2026 13:21:16 -0400 Subject: [PATCH 3/3] chore: remove self referential code_quality_rubric.json from run-evals evals.json and clarify prompt template placeholder instructions --- tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md | 4 ++-- .../.agents/skills/run-evals/evals/evals.json | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md b/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md index 127fe6cc..a7a180f4 100644 --- a/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md +++ b/tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md @@ -11,9 +11,9 @@ metadata: 2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `/evals/`. 3. **Determine Agent Configuration**: Check the `agent_config` field in the target target JSON file to determine the environment/harness to spawn. If `agent_config` is `"bare-agent"`, spawn a subagent with the `bare-agent` profile. If it is a specific contributor profile (e.g. `"reidbaker-agent"`), use that profile to provide the necessary contributor context. 4. **Orchestrate**: By default, run an Integration Test by spawning a single **With-Skill** subagent using `Workspace: branch` and the identified `agent_config`. - - Provide the task prompt. See `resources/with_skill_execution_prompt.md` for the template. When filling in ``, you MUST use a relative path from the repository root, not an absolute path. If you are running a cross-skill evaluation, fill in `` with `"none (cross-skill meta-eval)"`. Also, replace `` with the actual directory path. + - Provide the task prompt. See `resources/with_skill_execution_prompt.md` for the template. When filling in ``, you MUST use a relative path from the repository root, not an absolute path. If you are running a cross-skill evaluation, fill in `` with `"none (cross-skill meta-eval)"`. Also, replace `` with the actual directory path in both templates. - **Only if the user explicitly requests a comparison or benchmark**, also spawn a **Baseline** subagent. See `resources/baseline_execution_prompt.md` for the template. - Instruct the subagent(s) to return their `git diff` and verification outputs (`dart format`, `dart analyze`, `dart test`) without committing. Ensure you instruct them to run these commands exclusively from within the `` directory to avoid analyzing unrelated packages. + Instruct the subagent(s) to return their `git diff` and verification outputs (`dart pub get`, `dart format`, `dart analyze`, `dart test`) without committing. Ensure you instruct them to run these commands exclusively from within the `` directory to avoid analyzing unrelated packages. **CRITICAL**: You must explicitly warn the subagent(s) to confine all file edits strictly to their current working directory and avoid using absolute paths to modify the parent workspace. **WORKSPACE LIMITATION WARNING**: If the user has multiple active workspaces mounted, the `Workspace: branch` feature will fail. In this situation, you MUST warn the user that running concurrent evaluations in `Workspace: inherit` mode will cause git state bleed and cross-eval pollution (e.g., changes made by a failure scenario will be visible to a success scenario running simultaneously in the same shared directory). Instruct the user to fix this by closing all workspaces except the primary package workspace, and then re-run the evaluations. Do NOT silently fallback to `Workspace: inherit` for concurrent tasks. 5. **Grade**: Parse the combined rubric (resolving `repo_criteria` + `evals.json` expectations). Use the grading instructions in `resources/agent_judge_prompt.md`. When an expectation fails, you MUST explicitly list both the expectation and what was actually found that caused the failure. diff --git a/tool/dart_skills_lint/.agents/skills/run-evals/evals/evals.json b/tool/dart_skills_lint/.agents/skills/run-evals/evals/evals.json index 498e12b3..0e83e89d 100644 --- a/tool/dart_skills_lint/.agents/skills/run-evals/evals/evals.json +++ b/tool/dart_skills_lint/.agents/skills/run-evals/evals/evals.json @@ -1,6 +1,5 @@ { "repo_criteria": [ - "evals/code_quality_rubric.json" ], "evals": [ {