feat: add custom:mrr metric for RAG retrieval quality evaluation - #302
feat: add custom:mrr metric for RAG retrieval quality evaluation#302x86girl wants to merge 1 commit into
Conversation
Add Mean Reciprocal Rank (MRR) as a new custom metric that measures how high the first relevant context appears in the ranked list of retrieved contexts. This is a deterministic, non-LLM metric that complements existing Ragas context metrics. - Add `expected_contexts` field to TurnData for ground-truth contexts - Implement MRR evaluation with normalized containment matching - Register metric in CustomMetrics, validator, and system config - Add 26 comprehensive tests covering all edge cases Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WalkthroughAdds a turn-level Mean Reciprocal Rank metric for retrieval evaluation, including expected-context data, input validation, normalized containment matching, reciprocal-rank scoring, metric registration, metadata, and unit tests. ChangesMRR Retrieval Metric
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Evaluation
participant CustomMetrics
participant evaluate_mrr
participant TurnData
Evaluation->>CustomMetrics: request custom:mrr
CustomMetrics->>evaluate_mrr: evaluate turn data
evaluate_mrr->>TurnData: read contexts and expected_contexts
evaluate_mrr-->>CustomMetrics: score and reason
CustomMetrics-->>Evaluation: MRR result
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lightspeed_evaluation/core/metrics/custom/mrr_eval.py`:
- Around line 21-23: Update the containment check in the custom MRR matching
function around `_normalize_text` so it returns `False` whenever
`norm_retrieved` or `norm_expected` is empty before evaluating substring
containment. Add regression tests covering whitespace-only retrieved and
expected contexts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 065e6a3d-25ef-4df1-b84e-dac605a54fd1
📒 Files selected for processing (7)
config/system.yamlsrc/lightspeed_evaluation/core/metrics/custom/__init__.pysrc/lightspeed_evaluation/core/metrics/custom/custom.pysrc/lightspeed_evaluation/core/metrics/custom/mrr_eval.pysrc/lightspeed_evaluation/core/models/data.pysrc/lightspeed_evaluation/core/system/validator.pytests/unit/core/metrics/custom/test_mrr_eval.py
| norm_retrieved = _normalize_text(retrieved) | ||
| norm_expected = _normalize_text(expected) | ||
| return norm_expected in norm_retrieved or norm_retrieved in norm_expected |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Reject empty normalized contexts before containment matching.
Line 23 treats "" as matching every context. Thus expected_contexts=[" "] or contexts=[" "] can produce a perfect MRR score. Return False when either normalized value is empty, and add regression tests for both inputs.
Proposed fix
def _is_context_match(retrieved: str, expected: str) -> bool:
norm_retrieved = _normalize_text(retrieved)
norm_expected = _normalize_text(expected)
+ if not norm_retrieved or not norm_expected:
+ return False
return norm_expected in norm_retrieved or norm_retrieved in norm_expected📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| norm_retrieved = _normalize_text(retrieved) | |
| norm_expected = _normalize_text(expected) | |
| return norm_expected in norm_retrieved or norm_retrieved in norm_expected | |
| norm_retrieved = _normalize_text(retrieved) | |
| norm_expected = _normalize_text(expected) | |
| if not norm_retrieved or not norm_expected: | |
| return False | |
| return norm_expected in norm_retrieved or norm_retrieved in norm_expected |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lightspeed_evaluation/core/metrics/custom/mrr_eval.py` around lines 21 -
23, Update the containment check in the custom MRR matching function around
`_normalize_text` so it returns `False` whenever `norm_retrieved` or
`norm_expected` is empty before evaluating substring containment. Add regression
tests covering whitespace-only retrieved and expected contexts.
asamal4
left a comment
asamal4
left a comment
There was a problem hiding this comment.
Thanks !!
Please check coderabbit's comment.. and PTAL my inline comment.
Primarily I have concern about the custom implementation especially checking the relevancy part. We should either adopt a standard package or replace the matching logic with something more robust.
| return re.sub(r"\s+", " ", text.lower().strip()) | ||
|
|
||
|
|
||
| def _is_context_match(retrieved: str, expected: str) -> bool: |
There was a problem hiding this comment.
This approach (sub-string) is very naive. If we're building our own logic, then we must have a better deterministic relevancy or context match logic.
| expected_contexts: Optional[list[str]] = Field( | ||
| default=None, | ||
| description="Expected contexts for retrieval evaluation (ground truth)", | ||
| ) |
There was a problem hiding this comment.
Let's add min_length for early validation
| description: "LLM judge of agentic remediation workflow quality (diagnosis, actions, risk, verification)" | ||
| default: false | ||
|
|
||
| "custom:mrr": |
There was a problem hiding this comment.
| "custom:mrr": | |
| "nlp:mrr": |
This is custom, but yet NLP related.. I would suggest to keep this under NLP metrics
2 participants
Mean Reciprocal Rank (MRR) is a well-established information retrieval metric introduced by Voorhees (1999) in the TREC-8 Question Answering Track Report. It measures the rank position of the first relevant result in a ranked list, computed as 1/rank, yielding 1.0 when the
first retrieved item is relevant, 0.5 for the second, and so on.
▎ "The reciprocal rank of a query response is the multiplicative
▎ inverse of the rank of the first correct answer."
▎ Voorhees, E.M. (1999). The TREC-8 Question Answering Track Report.
▎ Proceedings of the 8th Text REtrieval Conference (TREC-8), NIST
▎ Special Publication 500-246, pp. 77–82.
In the context of RAG (Retrieval-Augmented Generation) evaluation, MRR provides a deterministic, non-LLM signal that directly measures retrieval ranking quality
It mesuares how quickly the system surfaces a relevant context chunk. This complements the existing LLM-judge-based Ragas metrics (context_recall, context_precision, context_relevance) by adding a fast, reproducible, and cost-free retrieval quality indicator.
Summary by CodeRabbit
New Features
Tests