Skip to content

feat(benchmark): expose demographics and standings breakdown in SDK#702

Draft
RapidPoseidon wants to merge 3 commits into
mainfrom
feat(benchmark)/expose-demographics-breakdown
Draft

feat(benchmark): expose demographics and standings breakdown in SDK#702
RapidPoseidon wants to merge 3 commits into
mainfrom
feat(benchmark)/expose-demographics-breakdown

Conversation

@RapidPoseidon

Copy link
Copy Markdown
Contributor

What

Exposes the two new leaderboard demographic endpoints through the Python SDK so programmatic consumers can pull voter composition and per-segment standings for a benchmark.

Adds two methods to RapidataBenchmarkManager (client.mri):

  • get_demographics(benchmark_id, run_id=None, filters=None)BenchmarkDemographics
    Voter composition per dimension (ageBucket, gender, occupation, country, language) — raw votes and share, including the "unknown" bucket; shares sum to 1.
  • get_standings_breakdown(benchmark_id, dimension, run_id=None, filters=None, use_weighted_scoring=False)BenchmarkStandingsBreakdown
    Global standings plus one segment per bucket of dimension (incl. "unknown"), each with that segment's votes and standings.

Both accept the same demographic/tag/run filters as standings/query and a use_weighted_scoring flag (default False = raw counts).

How it respects the three-tier structure

generated client → service wrapper → manager. The backend endpoints aren't in the checked-in generated api_client yet, so this adds a thin typed wrapper (BenchmarkDemographicsApi) that talks to the same low-level RapidataApiClient the generated APIs use (param_serialize / call_api / response_deserialize), mirroring their filter-serialization and error-wrapping. It's exposed on LeaderboardService as benchmark_demographics_api. Once the backend ships and the OpenAPI client is regenerated, the generated methods can supersede the wrapper.

Honesty

Field names stay faithful to the API contract: votes, share, the "unknown" buckets are all preserved (not dropped into a flattened DataFrame). Docstrings note that ageBucket/gender/occupation are estimated (inferred), not self-declared, and that segments are raw vote counts by default.

Backend contract built against:

  • GET /benchmark/{benchmarkId}/demographics
  • GET /benchmark/{benchmarkId}/standings/breakdown?dimension=AgeBucket

Validation

  • python -c "from rapidata import RapidataClient" → OK; new public classes re-exported from both __init__.py files.
  • pyright src/rapidata/rapidata_client → 0 errors.
  • End-to-end unit exercise with a mocked API client: query-param explosion (country[in]=US, dimension, useWeightedScoring), model parsing (incl. unknown buckets + confidenceInterval), and filter validation all verified.
  • mkdocs build → succeeds; docs added under mri_advanced.md (Voter Demographics section).

Scope

This is the SDK leg of the "Expose Demographic Information for Benchmarks" RFC (phase 4). Sibling PRs build in parallel:

  • rapidata-backend — the AllowAnonymous endpoints
  • app-frontend — the UI

This PR only touches rapidata-python-sdk.

🔗 Session: https://node-bfba29d9.poseidon.rapidata.internal/

RapidPoseidon and others added 2 commits July 21, 2026 13:16
Add RapidataBenchmarkManager.get_demographics and
get_standings_breakdown, surfacing the new leaderboard demographic
endpoints so programmatic consumers can pull voter composition and
per-segment standings for a benchmark.

The endpoints are not in the checked-in generated OpenAPI client yet, so
a thin typed wrapper (BenchmarkDemographicsApi) talks to the same
low-level RapidataApiClient the generated APIs use, mirroring their
request/response handling; it is superseded once the backend ships and
the client is regenerated. Results are pydantic models faithful to the
API contract — vote counts, shares, and the 'unknown' buckets are kept,
and docstrings note that age/gender/occupation are estimated (inferred)
and that segments are raw counts unless weighted scoring is requested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: luca <25279790+LucStr@users.noreply.github.com>
Reconcile with the merged backend leg (rapidata-backend#4821), which
locked both new endpoints as raw-count only:

- Remove use_weighted_scoring from get_standings_breakdown and the thin
  wrapper — neither /demographics nor /standings/breakdown exposes a
  weighting param (the pre-existing standings query still does).
- Model the full locked standings item shape: add proprietaryName, logo,
  status (StandingStatus) and isDisabled alongside the existing fields.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: luca <25279790+LucStr@users.noreply.github.com>
@RapidPoseidon

Copy link
Copy Markdown
Contributor Author

Folded in contract update (from parent — backend leg merged as RapidataAI/rapidata-backend#4821)

Pushed a follow-up commit reconciling with the locked endpoint shapes:

  • Removed use_weighted_scoring from get_standings_breakdown(...) and the thin wrapper. Correction from the parent: neither new endpoint exposes weighting — both are raw-count only. (The pre-existing standings/query still has it; these two new endpoints do not.) get_demographics(...) never had a weighting param.
  • Completed the standings item shape to match the merged contract: added proprietaryName (optional), logo (optional), status (StandingStatus), and isDisabled (bool) alongside id/name/family/score/wins/totalMatches/confidenceInterval. Vote counts, shares, and the always-present "unknown" buckets are preserved.

No conflict with anything already shipped — the wrapper stays (per repo convention) until #4821 is deployed and the OpenAPI client is regenerated. Re-validated: pyright 0 errors, mocked end-to-end parse of the full item shape, mkdocs build clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: luca <25279790+LucStr@users.noreply.github.com>
@RapidPoseidon

Copy link
Copy Markdown
Contributor Author

⚠️ CI not scheduling on the follow-up commits — needs a human nudge

The type-check (pyright) workflow ran green on the first commit, but GitHub has not scheduled a run for the follow-up commits (9e1bdcc, 17734b5). No github-actions check suite is being created for the branch's synchronize events — verified over a 30-minute automated watch plus an empty-commit re-trigger. Other PRs on the repo get Actions runs, so this is a GitHub-side scheduling hiccup specific to this branch, not a code failure. My session token lacks the scope to close/reopen the PR or poke Actions.

To kick CI: close + reopen this PR (fires a reopened event, which the workflow listens for), or push any commit from an account with full scopes.

Local verification of the follow-up (same checks CI runs):

  • pyright src/rapidata/rapidata_client0 errors, 0 warnings
  • Mocked end-to-end parse of the full locked standings-item shape (status, isDisabled, proprietaryName, logo, confidenceInterval) + confirmation that use_weighted_scoring is gone from both the manager method and the wrapper
  • mkdocs build → clean

The last commit (17734b5) is an empty ci: re-trigger checks commit; feel free to drop it on squash-merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant