Extraction: PAHO Oropouche + USDA APHIS Tableau scrapers#74
Merged
Conversation
Two custom scrapers that turn dashboard-backed surveillance data into the compact HTML summaries the insight pipeline consumes: - usda_aphis_livestock: exports the USDA APHIS HPAI livestock Tableau crosstab via a headless browser (Playwright), then renders monthly and state-level analytics with linear/exponential fits. Verified live end to end (the export returns ~1,168 detection rows and renders HTML). If the browser is unavailable the fetch returns None and extraction falls back to the generic fetcher. - paho_oropouche_portal: renders weekly/country analytics from the PAHO ARBO Oropouche CSV via an injectable csv_fetcher seam, registered under specific_pathogen_sources in sources.yaml. Its export endpoint is session-scoped, so live fetch currently degrades to the generic fetcher; a headless-browser port (as done for USDA) is a follow-up. Both scrapers return None on any failure, so a dead source never breaks extraction. Unit tests inject fixture data and need no network or browser. requirements.txt gains playwright and pandas; README documents the one-time `playwright install chromium` step. The unused TableauScraper dependency from the source branch is intentionally not carried over. Extracted from @rapsoj's #66/#70 (USDA Playwright fetch) and #67 (PAHO). Co-Authored-By: Jess Rapson <jessicakristenr@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 17, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extracted from @rapsoj's #66/#70 (USDA Playwright fetch) and #67 (PAHO).
What
Two custom scrapers that turn dashboard-backed surveillance data into the compact HTML summaries the insight pipeline consumes:
usda_aphis_livestock— exports the USDA APHIS HPAI livestock Tableau crosstab via a headless browser (Playwright), then renders monthly/state-level analytics with linear/exponential fits. Verified live end-to-end: the export returns ~1,168 detection rows and renders HTML. If the browser is unavailable the fetch returnsNoneand extraction falls back to the generic fetcher.paho_oropouche_portal— renders weekly/country analytics from the PAHO ARBO Oropouche CSV via an injectablecsv_fetcherseam; registered underspecific_pathogen_sourcesinsources.yaml. Its export endpoint is session-scoped, so live fetch currently degrades to the generic fetcher; a headless-browser port (as done for USDA) is a sensible follow-up.Safety
Both scrapers return
Noneon any failure, so a dead source never breaks extraction. Unit tests inject fixture data and need no network or browser.Dependencies
requirements.txtgainsplaywrightandpandas; the README documents the one-timeplaywright install chromiumstep. The unusedTableauScraperdependency from the source branch is intentionally not carried over.