feat(benchmark): expose demographics and standings breakdown in SDK#702
feat(benchmark): expose demographics and standings breakdown in SDK#702RapidPoseidon wants to merge 3 commits into
Conversation
Add RapidataBenchmarkManager.get_demographics and get_standings_breakdown, surfacing the new leaderboard demographic endpoints so programmatic consumers can pull voter composition and per-segment standings for a benchmark. The endpoints are not in the checked-in generated OpenAPI client yet, so a thin typed wrapper (BenchmarkDemographicsApi) talks to the same low-level RapidataApiClient the generated APIs use, mirroring their request/response handling; it is superseded once the backend ships and the client is regenerated. Results are pydantic models faithful to the API contract — vote counts, shares, and the 'unknown' buckets are kept, and docstrings note that age/gender/occupation are estimated (inferred) and that segments are raw counts unless weighted scoring is requested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: luca <25279790+LucStr@users.noreply.github.com>
Reconcile with the merged backend leg (rapidata-backend#4821), which locked both new endpoints as raw-count only: - Remove use_weighted_scoring from get_standings_breakdown and the thin wrapper — neither /demographics nor /standings/breakdown exposes a weighting param (the pre-existing standings query still does). - Model the full locked standings item shape: add proprietaryName, logo, status (StandingStatus) and isDisabled alongside the existing fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: luca <25279790+LucStr@users.noreply.github.com>
Folded in contract update (from parent — backend leg merged as RapidataAI/rapidata-backend#4821)Pushed a follow-up commit reconciling with the locked endpoint shapes:
No conflict with anything already shipped — the wrapper stays (per repo convention) until |
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: luca <25279790+LucStr@users.noreply.github.com>
|
What
Exposes the two new leaderboard demographic endpoints through the Python SDK so programmatic consumers can pull voter composition and per-segment standings for a benchmark.
Adds two methods to
RapidataBenchmarkManager(client.mri):get_demographics(benchmark_id, run_id=None, filters=None)→BenchmarkDemographicsVoter composition per dimension (
ageBucket,gender,occupation,country,language) — rawvotesandshare, including the"unknown"bucket; shares sum to 1.get_standings_breakdown(benchmark_id, dimension, run_id=None, filters=None, use_weighted_scoring=False)→BenchmarkStandingsBreakdownGlobal standings plus one segment per bucket of
dimension(incl."unknown"), each with that segment'svotesand standings.Both accept the same demographic/tag/run
filtersasstandings/queryand ause_weighted_scoringflag (defaultFalse= raw counts).How it respects the three-tier structure
generated client → service wrapper → manager. The backend endpoints aren't in the checked-in generated
api_clientyet, so this adds a thin typed wrapper (BenchmarkDemographicsApi) that talks to the same low-levelRapidataApiClientthe generated APIs use (param_serialize/call_api/response_deserialize), mirroring their filter-serialization and error-wrapping. It's exposed onLeaderboardServiceasbenchmark_demographics_api. Once the backend ships and the OpenAPI client is regenerated, the generated methods can supersede the wrapper.Honesty
Field names stay faithful to the API contract:
votes,share, the"unknown"buckets are all preserved (not dropped into a flattened DataFrame). Docstrings note thatageBucket/gender/occupationare estimated (inferred), not self-declared, and that segments are raw vote counts by default.Backend contract built against:
GET /benchmark/{benchmarkId}/demographicsGET /benchmark/{benchmarkId}/standings/breakdown?dimension=AgeBucketValidation
python -c "from rapidata import RapidataClient"→ OK; new public classes re-exported from both__init__.pyfiles.pyright src/rapidata/rapidata_client→ 0 errors.country[in]=US,dimension,useWeightedScoring), model parsing (incl.unknownbuckets +confidenceInterval), and filter validation all verified.mkdocs build→ succeeds; docs added undermri_advanced.md(Voter Demographics section).Scope
This is the SDK leg of the "Expose Demographic Information for Benchmarks" RFC (phase 4). Sibling PRs build in parallel:
AllowAnonymousendpointsThis PR only touches
rapidata-python-sdk.🔗 Session: https://node-bfba29d9.poseidon.rapidata.internal/