Enforce scoped validation tools in AGENTS.md with file-based trap evals - #12297
Enforce scoped validation tools in AGENTS.md with file-based trap evals#12297reidbaker-agent wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces evaluation configurations, rubrics, simulation scripts, and structural consistency tests for the run-evals skill in the camera_android_camerax package and the agent-plugins repository. The review feedback identifies a potential test failure in mono-repo environments due to hardcoded path assumptions, a silent test bypass, and destructive global file overwrites in the setup script that also trigger rubric failures. Suggestions are provided to robustly locate the package root and safely mock global scripts.
5f69d6d to
0fbe29a
Compare
|
Blocked on #12298 |
0fbe29a to
c8e22c5
Compare
There was a problem hiding this comment.
Code Review
This pull request introduces validation guidelines in AGENTS.md to prevent running global CI scripts, adds a compliance evaluation test, and provides a setup script to trap forbidden script executions. Additionally, it updates skills_evals_test.dart to resolve the package root dynamically and enforce the existence of the evals/ directory and rubric files. Feedback highlights a critical issue in the setup script where single quotes prevent the expansion of $REPO_ROOT, which would cause the trapped scripts to attempt writing to the root directory.
This pr is blocked by #12298. This came up in #12297. I decided landing the tool focused pr was much easier to review than stacking this work on #12285 - @reidbaker --- Agent authored pr description This PR exempts .agents and evals directories from triggering missing changelog and version update errors. TAG=agy CONV=24c0c9b5-7e8d-407c-ae84-a577061bd6ff
Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com>
|
autosubmit label was removed for flutter/packages/12297, because This PR has not met approval requirements for merging. The PR author is not a member of flutter-hackers and needs 1 more review(s) in order to merge this PR.
|
This pr gives us a way of writing unit tests for agent.md. - @reidbaker
Pr is blocked by a preexisting zizmor issue #12298 and by #12285 which is the pr that introduces evals.
Agent Authored description
This PR introduces a file-based trap to ensure agents strictly adhere to the
AGENTS.mdguidelines for validating local changes. Instead of running slow, repository-wide CI scripts like.ci/scripts/validate.shorscript/tool_runner.sh, agents must use targeted skills or package-scoped tools.Evaluation Results:
check-readiness/ Validation rulesEval ID: agents_md_compliance_test
dart format .,dart analyze ., andflutter test.USED_FORBIDDEN_SCRIPT.txtwas not generated.