Extraction: live PAHO Oropouche fetch + USDA date-parse fix#75
Open
smodee wants to merge 2 commits into
Open
Conversation
…parse fix PAHO: replace the dead session-scoped vudcsv URL with a headless-browser fetch. It redeems a fresh Tableau trusted ticket via the public ARBO wrapper, captures the vizql session token, and pulls the "Full Data" underlying CSV (the view/table ids are stable across sessions; only the session token is per-render). Retries the whole flow on a transient miss, and returns None (generic-fetch fallback) if Playwright or the browser is unavailable. Verified live and reliable across repeated runs. Access note: the public embed disables the download toolbar and every row of the underlying table is labelled "PAHO internal use only"; this reconstructs the export endpoint the UI would otherwise expose. Recorded in the module docstring. USDA: pass pd.to_datetime format="mixed" so it no longer warns "Could not infer format" on the crosstab's "14-Jul-26" dates (identical per-element parsing, warning suppressed). Also: the PAHO test patches the new default fetcher and adds a vudcsv column-superset parse test; the README notes both scrapers need `playwright install chromium`. Co-Authored-By: Jess Rapson <jessicakristenr@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds paho_oropouche_documents (the topic-filtered documents listing at paho.org/en/documents/topics/oropouche-virus-disease) to the oropouche family. This is the public, unrestricted source that carries the official country-count resolution figure (stated in the epidemiological-update PDFs) -- the authoritative counterpart to the internal-use-only dashboard. Generic fetch, no custom scraper; it routes alongside the existing Oropouche sources for Oropouche questions (e.g. bfg_q20). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the scraper work in #74 (extracted from @rapsoj's #66/#67/#70).
PAHO Oropouche — live fetch
Replaces the dead session-scoped
vudcsvURL with a headless-browser fetch that works reliably:ais.paho.org/.../AME_OROV_Viz.asp), which mints a fresh single-use Tableau ticket and establishes an authenticated vizql session.vudcsv). The view ID and underlying-table ID are stable across sessions — only the session token is per-render.Robustness: whole-flow retry on a transient miss; returns
None→ generic-fetch fallback if Playwright/the browser is unavailable, so a failure never breaks extraction. Verified live and reliable (4/4 back-to-back + 6/6 spaced runs); currently returns 2026 data (Brasil + Panamá, 17 lab-confirmed, latest epi-week 2026-W15).The public embed disables the download toolbar, and every row of the underlying table is labelled "PAHO internal use only." This scraper reconstructs the export endpoint the UI would otherwise expose — i.e. it reads data PAHO has flagged internal and gated. This was a deliberate decision to include; flagging it here so it's visible for review / a terms-of-use check. It is also recorded in the module docstring. (Easy to revert to graceful-degrade-only if that call changes.)
PAHO Oropouche — official resolution source registered
Also registers
paho_oropouche_documents(the topic-filtered documents listing,paho.org/en/documents/topics/oropouche-virus-disease) in the oropouche family. This is the public, unrestricted source carrying the official country-count figure thatbfg_q20("how many Americas countries report confirmed Oropouche in 2026") resolves on — the authoritative counterpart to the internal-use-only dashboard. Generic fetch, no custom scraper.The two are complementary, not redundant: the dashboard is fresher (live 2026 data) but gated and undercounts (excludes imported-only countries); the epi-update PDFs behind the documents listing are authoritative but lag (latest is Aug 2025 — no 2026 Oropouche update published yet). Both now route to Oropouche questions.
USDA — date-parse warning fix
pd.to_datetime(..., format="mixed")on the crosstab's14-Jul-26dates, so it no longer emits the "Could not infer format" warning. Identical per-element parsing, just without the warning. Verified live (still 1,168 rows).Tests / deps
playwright install chromiumfor both scrapers.