Skip to content

fix(eval): handle variable retrieval depths in MKQA recall - #1590

Open
BingH225 wants to merge 1 commit into
FlagOpen:masterfrom
BingH225:fix/mkqa-variable-depth-recall
Open

fix(eval): handle variable retrieval depths in MKQA recall#1590
BingH225 wants to merge 1 commit into
FlagOpen:masterfrom
BingH225:fix/mkqa-variable-depth-recall

Conversation

@BingH225

Copy link
Copy Markdown

Summary

  • Size MKQA recall buckets using the maximum retrieval depth across queries.
  • Add a regression test for queries with different numbers of retrieved contexts.

Problem

evaluate_qa_recall currently initializes its recall buckets using only the
first query's context count. If a later query has more retrieved contexts and
its first answer occurs beyond that initial depth, the hit is silently omitted.

For example, with retrieval depths [1, 2] and the second query matching at
rank 2, the previous implementation returns:

[0.0, 0.0]

The expected Recall@1 and Recall@2 values are:

[0.0, 0.5]

Testing

  • Added test_evaluate_qa_recall_supports_variable_retrieval_depths.
  • Executed the regression scenario directly against the modified module.
  • Python AST parsing passed for the implementation and test.
  • git diff --check and targeted codespell passed.

The complete pytest invocation was not run locally because the available
Python environment does not include FlagEmbedding's torch dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant