fix: divide DocumentMAPEvaluator average precision by all relevant documents

[anxkhn]

Related Issues

• No existing issue; found while comparing DocumentMAPEvaluator output against the standard mean average precision definition.

Proposed Changes:

DocumentMAPEvaluator computed average precision as sum(precision@k) / number_of_retrieved_relevant_documents. Average precision is defined as sum(precision@k) / number_of_relevant_documents, where relevant documents that were never retrieved contribute a precision of zero (see the TREC measure definitions at https://trec.nist.gov/pubs/trec16/appendices/measures.pdf and the trec_eval reference implementation, which accumulates precision and then divides by the total number of relevant documents).

Because of the wrong denominator, a query with two relevant documents where only one was retrieved scored 1.0 instead of 0.5. In addition, the ground-truth values were never consumed once matched, so retrieving the same relevant document twice was credited twice and could push the score back up while a relevant document was still missing.

This PR:

• collects the ground-truth comparison values into a set, so the denominator is the number of unique valid ground-truth documents;
• divides the accumulated precision by that total instead of by the hit count;
• removes a value from the pending set once it has been credited, so duplicate retrievals of the same document are not counted again.

Documented behaviour of the class (the usage example in the docstring) is unaffected: its expected scores are unchanged. Scores for evaluations with missed or duplicated relevant documents will change, which is called out in the release note under upgrade.

How did you test it?

• Added a regression test covering a missed relevant document and a duplicated retrieval; both now yield 0.5 for a query with two relevant documents and one distinct hit.
• Updated test_run_with_complex_data expectations, which previously encoded the inflated values (1.0 for a query where only one of two relevant documents was retrieved, and 0.805… for a query where a duplicate retrieval was credited).
• Ran the evaluator's unit test module locally; all tests pass.

Notes for the reviewer

This is a behaviour change for users who track absolute MAP values, so a release note with an upgrade section is included. Happy to split the duplicate-retrieval part out if you would rather land the denominator fix on its own.

Disclosure: this change was drafted with AI assistance and reviewed by a human before submission.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:
• I documented my code
• I ran pre-commit hooks and fixed any issue

[vercel]

@anxkhn is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[julian-risch]

Thank you for opening this PR and congrats on this contribution to Haystack @anxkhn ! It looks good to me and it's ready to be merged.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/evaluators
document_map.py					134
Project Total

This report was generated by python-coverage-comment-action