Skip to content

feat(savings): support NER/extraction traffic in rulechef-savings#28

Open
AZERDSQ131 wants to merge 1 commit into
KRLabsOrg:mainfrom
AZERDSQ131:feat/savings-ner-traffic
Open

feat(savings): support NER/extraction traffic in rulechef-savings#28
AZERDSQ131 wants to merge 1 commit into
KRLabsOrg:mainfrom
AZERDSQ131:feat/savings-ner-traffic

Conversation

@AZERDSQ131

Copy link
Copy Markdown

Problem

rulechef-savings only understood classification traffic — it compares a rule's label string to the observed llm_label. A traffic row shaped {"text": ..., "llm_entities": [...]} was silently treated as unanswered, even though that's exactly the TAB-style NER use case the tool targets.

Fix

  • Auto-detect NER traffic from the first row's llm_entities key.
  • For NER traffic: "answered" means the rules produced a non-empty entity list. "Agreement" is corpus-level micro-F1 against llm_entities, computed by reusing _match_entities from rulechef.evaluation (mode "text": type + text match, position-agnostic) instead of exact string comparison, per the issue's suggestion.
  • Per-rule "agreement" column becomes that rule's precision (share of its produced entities that matched gold) — recall isn't attributable per-rule since an unmatched gold entity has no producing rule to blame.
  • Classification path (llm_label) is unchanged.

Tests

test_savings_cli_ner_traffic in tests/test_reports.py, next to the existing test_savings_cli: a 2-row NER traffic file (one true positive, one false positive) asserting 100% coverage and 67% fidelity (micro-F1: TP=1, FP=1, FN=0 → P=.5, R=1, F1=.667).

pytest tests/ -q → 312 passed, 1 skipped. ruff check and mypy rulechef/savings.py clean.

Closes #19

rulechef-savings only understood classification traffic (comparing a
rule's label to llm_label). A traffic row with {text, llm_entities}
was silently treated as unanswered, even though that's exactly the
TAB-style NER use case the tool targets.

Auto-detect NER traffic from the first row's llm_entities key. For
NER: 'answered' means the rules produced a non-empty entity list,
and 'agreement' is corpus-level micro-F1 against llm_entities (reusing
_match_entities from rulechef.evaluation, mode='text') instead of
exact string comparison. Per-rule 'agreement' is that rule's
precision (share of its produced entities that matched gold) --
recall isn't attributable per-rule since unmatched gold entities have
no producing rule.

Closes KRLabsOrg#19
@adaamko

adaamko commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Reviewed — the NER path looks right: micro-F1 vs llm_entities as the fidelity number is the honest way to score span traffic, and the FP test with hand-computed 67% is exactly the kind of test I want here. Two notes, neither blocking: (1) first-row auto-detection is fine but worth a warning if later rows have the other shape; (2) gold_label support for NER traffic can be a follow-up. Now that #21 is merged, chef.export_traffic() produces NER traffic natively, so this closes the loop end-to-end. Blocking item is just the rights checkbox in the PR body (the failing required check) — tick it and this merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rulechef-savings: support NER/extraction traffic, not just classification

2 participants