fix(agent): retrofit the autonomous agent for the new Biolink restrictions - #75
fix(agent): retrofit the autonomous agent for the new Biolink restrictions#75SkyeAv wants to merge 1 commit into
Conversation
…tions 96ebc23 ("fix(biolink): make KGX output validate against the Biolink Model") rebuilt the emit path, but its entire diff against the agent surface (`agent.py`, `docs/agent.md`, `test_agent_lazy.py`) was the unrelated `dspy` -> `[optimize]` packaging move. Not one Biolink concept crossed over. 6f8456d DID retrofit the prompt for `effect_size`/`effect_type`, so the pattern was expected -- this is an omission, not a policy. Measured against the installed biolink-model 4.4.3, the agent: - had no biolink-validity signal at all. `build_and_audit` wrote its NDJSON, counted the lines, and returned; `biolink.validate_kgx` had three callers, none of them the agent. Nothing in `quality_score` or GEPA's objective rewarded validity, so a run could converge on, persist, and register a config whose output validated at 0%. - recommended `gene_associated_with_condition` in `optimized_instructions.yaml` -- the exact predicate 96ebc23 measured failing on 723,595 edges. Post-fix it no longer errors; `resolve_association_class` silently walks up to bare `biolink:Association`. - was told to emit `supporting_study_size`, which is in `UNSATISFIABLE_EDGE_FIELDS` and gets rerouted into a `StudyResult.description` string with no feedback. - resolved enum-ranged qualifiers through the fullmap that `lib.py:955` deliberately skips, depressing coverage for columns the build never resolves. - swallowed the coded error text at both final-answer gates, so the LLM got a bare False for exactly the errors 96ebc23 wrote to be actionable. SIGNAL `build_and_audit` now constructs every emitted record as its own Biolink class (the same check `validate-kgx` runs) and reports `biolink_valid_pct`, `biolink_valid_pct_strict`, `biolink_problems`, and `demoted_edge_pct` -- the last being the only surfaced evidence that a predicate cost its edge the association class. Non-fatal exactly like coverage: the KG already built. `quality_score` reweighted to coverage 0.40 / biolink 0.25 / F1 0.15 / QC 0.10 / schema 0.10. Most of the new weight came from `w_qc`, which scored a structurally constant `1.0 if qc else None`. GEPA's feedback carries the problem histogram and demotion fraction; the judge gained a `biolink_validity` dimension, and its predicate/category heuristic is now capped by measured demotion rather than config shape. New `--biolink-threshold` gates MAPPED; default 0.0 keeps today's behavior while recording the rate on every record. `effect_size`/`effect_type` are exempted via `KNOWN_PENDING_EDGE_FIELDS`, derived as `TABLASERT_EDGE_EXTRAS - <fields any association declares>`, so it empties itself when biolink-model#1774 ships. Nothing is hardcoded to a model version. GUIDANCE The prompt's only vocabulary channel was a ~30 KB enum dump: 247 predicates and 159 categories with nothing tying the two together. It now carries a compact predicate<-> class table rendered at import from the installed model via the new `lib.predicate_options` / `biolink.legal_predicates`, plus the two silent-relocation rules and the qualifier token rule (qualifiers had zero prompt mentions). `optimized_instructions.yaml` and the QC assay are corrected -- the latter's `GENERIC_PREDICATES` flagged `associated_with`, one of only three predicates `GeneToDiseaseAssociation` permits, as a "generic fallback". CONSISTENCY `map_coverage` and `_statement_nodes` skip enum-ranged qualifiers as the build does; the improve loop requires no regression on either axis and a gain on one; `derive_config` returns the coded error instead of forwarding an invalid config, since the boolean gate cannot carry it. `Annotation` emits a `BiolinkRelocationWarning` naming where a relocated value went -- a warning, not an error: nothing is lost and every existing config keeps building. Also fixes a false pass in `validate_kgx`: a missing path yielded `total=0, valid=0`, kept `ok` true, and exited 0 reporting compliance. Left alone, the new agent gate would have inherited a silent 100%. Verification: pytest 765 passed / 35 skipped (was 728/34; 22 new tests, including an end-to-end proof that the same table built with `associated_with` scores demoted_edge_pct 0.0 and with `gene_associated_with_condition` scores 1.0). cargo test 69 + 10. ruff check / ruff format / pyright clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PNmGrcymfW5ZS1W34nD1H9
|
Warning Review limit reached
Next review available in: 52 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (21)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
|
1 participant
The gap
96ebc23("fix(biolink): make KGX output validate against the Biolink Model") rebuilt the emit path —resolve_association_class,prune_to_class,inline_supporting_study, enum-ranged qualifiers,UNSATISFIABLE_EDGE_FIELDS, node demotion,validate-kgx. Its entire diff against the agent surface (agent.py,docs/agent.md,tests/test_agent_lazy.py) was the unrelateddspy→[optimize]packaging move. Not one Biolink concept crossed over.6f8456ddid retrofit the prompt foreffect_size/effect_type, so the pattern was expected — which makes this an omission rather than a policy.Measured against the installed
biolink-model4.4.3:build_and_auditwrote its NDJSON, counted lines, returned.biolink.validate_kgxhad three callers — the CLI, one test, the docs — never the agent.agent.py:1230-1242quality_score= 0.5·coverage + 0.2·qc + 0.2·F1 + 0.1·schema. GEPA optimized the prompt against that. A run could converge on, persist, and register a config whose output validated at 0%.agent.py:2952,3251gene_associated_with_condition— the exact predicate96ebc23measured failing on 723,595 edges. Post-fix it no longer errors;resolve_association_classsilently walks up to barebiolink:Association.optimized_instructions.yaml:20-21,41,80GENERIC_PREDICATESmarkedassociated_witha "generic fallback" — it is one of only three predicatesGeneToDiseaseAssociationpermits.qc/qc_report.py:18supporting_study_size, which is inUNSATISFIABLE_EDGE_FIELDSand gets rerouted into aStudyResult.descriptionstring. NoAnnotation-level guard, no feedback.agent.py:1939,1988map_coverageresolved qualifiers the build skips.lib.py:955filters onif x.resolved; the agent didn't. Enum-ranged qualifiers want the tokenincreased, never resolve, and dragoveralldown — enough to flip a good config toSKIPPED.agent.py:864,1418Falsefor exactly the errors96ebc23wrote to be actionable.agent.py:688,735## Unreleased; the 8.2.0 notes never mentioned the Biolink fix;docs/agent.md:6claimed "NCATS Translator-compliant KGX" with nothing behind it.CHANGELOG.md:5-10What this changes
Signal
build_and_auditnow constructs every emitted record as its own Biolink class — the same checkvalidate-kgxruns — and reports four fields. Non-fatal exactly like coverage: the KG already built, so a failure is a score to improve, never a build error.biolink_valid_pctbiolink_valid_pct_strictbiolink_problems"field: error-type"failures with countsdemoted_edge_pctbiolink:Associationdemoted_edge_pctis the predicate signal. A predicate its class forbids is never an error — it silently discards the class and every qualifier and evidence slot that class declared.gene_associated_with_conditionon a gene~disease table builds cleanly, maps perfectly, and producesbiolink:Associationedges. Nothing else surfaces it.quality_scorereweighted to coverage 0.40 / biolink 0.25 / F1 0.15 / QC 0.10 / schema 0.10 (still a hard gate). Most of the new weight came out ofw_qc, which scored a structurally constant1.0 if qc else None. GEPA's feedback carries the problem histogram and demotion fraction; the judge gained abiolink_validitydimension, and its predicate/category heuristic is now capped by measured demotion rather than config shape.New
--biolink-thresholdgatesMAPPED. Default0.0(report-only) keeps today's terminal behavior;biolink_valid_pct/demoted_edge_pctare recorded on everystate.jsonrecord either way.effect_size/effect_typeare exempted viaKNOWN_PENDING_EDGE_FIELDS, derived asTABLASERT_EDGE_EXTRAS - <fields any association declares>— so it empties itself when biolink-model#1774 ships, with no code change. Same version-proofing discipline asUNSATISFIABLE_EDGE_FIELDS, and asserted as such.Guidance
The agent's only vocabulary channel was the ~30 KB
Section.model_json_schema()dump — 247 predicates and 159 categories with nothing tying the two together. The prompt now carries a compact predicate↔class table rendered at import from the installed model, via the newlib.predicate_options/biolink.legal_predicates(the missing authoring-time helpers — there was no way to ask which predicates a subject/object pair may carry without composing three private functions):Plus the two silent-relocation rules and the qualifier token rule — qualifiers had zero mentions in the prompt, so the three config-time validators
96ebc23added were unreachable from agent-authored configs.optimized_instructions.yamlandqc_report.pycorrected.Consistency
map_coverageand_statement_nodesskip enum-ranged qualifiers as the build does.Qualifier.resolvedalready existed and simply wasn't consulted.derive_configreturns the coded error instead of forwarding an invalid config, since the boolean gate can't carry it. Newsection_error()/table_config_error()expose the reason; the gates keep theirboolcontract and never-raises guarantee.Annotationemits aBiolinkRelocationWarningnaming where a relocated value went — a warning, not an error: nothing is lost, and every existing config (including the tutorial'ssupporting_study_size) keeps building.Also fixed
A false pass in
validate_kgx: a missing or misspelled path yieldedtotal=0, valid=0, keptoktrue, and exited 0 reporting "KGX output is Biolink-compliant." Left alone, the new agent gate would have inherited a silent 100%.Verification
pytest: 765 passed, 35 skipped (was 728/34 — 22 new tests)cargo test: 69 + 10 passedruff check/ruff format/pyright: cleanThe end-to-end proof is
test_demoted_edge_pct_catches_a_predicate_its_class_forbids— the same gene~disease table, built twice:okdemoted_edge_pctassociated_withTrueGeneToDiseaseAssociationgene_associated_with_conditionTrueAssociationBoth build successfully. That is the point: the forbidden predicate is never an error, which is why the metric had to exist.
Notes for review
--biolink-thresholddefaults to0.0, so nothing gates today. Worth looking at a real batch'sbiolink_valid_pctbefore picking a threshold..tablassert/million-kg/to quantify the pending-exemption rate against the strict rate.derive_mode(agent.py:3204) is a library-only parameter with no CLI flag, no test, and no docs — andDERIVEDsits in the resume-terminal set, so such a record is never picked up by a laterfullrun.🤖 Generated with Claude Code
https://claude.ai/code/session_01PNmGrcymfW5ZS1W34nD1H9