Cohort filters cannot express "original data only"

[fedorov]

Author: Prepared during an interactive analysis session with Claude
Date: 2026-08-06
Server tested: IDC MCP - prod and api.imaging.datacommons.cancer.gov — api_version 3.0.0b2, build 968e137
Data release: IDC v24 (idc-index 24.2.2)

---

Filtering for an analysis result is easy; filtering it out is impossible through the
cohort endpoints, and the attribute metadata actively points the wrong way. Found while
documenting the collection_id / analysis_result_id relationship for IDC-Docs
(ImagingDataCommons/IDC-Docs#83).

The gap

Derived series carry the collection_id of the images they analyze, so a bare collection
filter returns both axes. This surprises people, and the amount involved is not marginal:

SELECT COUNT(*) FROM index WHERE collection_id = 'nlst';                             -- 590,572
SELECT COUNT(*) FROM index WHERE collection_id = 'nlst' AND analysis_result_id IS NULL; -- 204,346

Two thirds of what collection_id = 'nlst' returns was contributed by analysis results.
55 of the 176 collections contain derived series.

In SQL the fix is one predicate, and it works on every SQL surface (idc-index,
POST /v3/sql, run_sql, BigQuery). Through the cohort API there is no fix at all:

# rejected
curl -s .../v3/cohort/counts -H 'Content-Type: application/json' \
  -d '{"terms": {"collection_id": ["nlst"], "analysis_result_id": [null]}}'
# {"detail":[{"type":"string_type","loc":["body","terms","analysis_result_id",0],
#             "msg":"Input should be a valid string","input":null}]}

analysis_result_id is advertised as a filterable categorical term, but its value domain
contains only the 24 analysis result ids — no value denotes "none". So build_cohort,
/v3/cohort/*, and the manifest endpoints can all select derived data and none of them
can exclude it. The workaround is to abandon cohort filters and rewrite the whole query in
SQL, which also means giving up manifest generation from that cohort.

Two supporting defects

1. The attribute description is wrong. list_attributes says of analysis_result_id:

this string is not empty if the specific series is part of an analysis results collection

The column is NULL for original data, not empty — 560,965 NULLs and zero empty strings
in v24. Anyone who trusts that wording writes analysis_result_id != '', which returns
nothing, or = '', which also returns nothing. This is worth fixing on its own; LLM agents
read these descriptions as ground truth and cannot see the underlying storage.

2. get_attribute_values hides the majority of the index. It returns the 24 ids,
totalling 471,946 series, with truncated: false — implying a complete picture. The
560,965 series with no analysis result (54.3% of index) are simply absent, with nothing
signalling their existence. Same NULL-visibility theme as #31 §4.2.

Suggested fix

Preferred — expose source_type as a filterable term. IDC already defines this
vocabulary: collections_index.sources[].source_type is original_data or
analysis_result. Promoting it to a filter attribute would give:

{"terms": {"collection_id": ["nlst"], "source_type": ["original_data"]}}

This needs no null semantics on the wire, is symmetric (either side is selectable), is
self-documenting through get_attribute_values, reuses IDC's own nomenclature rather than
inventing a flag, and keeps working if further source types are added later.

Cheaper fallback — accept null in a term list, so {"analysis_result_id": [null]}
means "unset". Smaller change, but it leans on a wire-level convention that clients have to
know about, and it stays invisible in get_attribute_values.

A general negation facility would also solve it, but that is a much larger design change
and this case does not require it.

Regardless of which is chosen, the description fix in (1) is worth doing immediately.

Acceptance

• A cohort filter selects NLST's 204,346 original series without SQL, and a manifest can be
  generated from it.
• get_attribute_values reveals that unset/original data exists as a selectable option.
• No attribute description describes a NULL column as "not empty".

Counts are IDC v24.