feat: OpenDataSUS catalog client and unified metadata layer

.gitignore

@@ -191,3 +191,4 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 .idea/
 pyrightconfig.json
+roadmap.md

CHANGELOG.md

@@ -3,6 +3,11 @@ Release Notes
 
 ## [2.9.0](https://github.com/AlertaDengue/PySUS/compare/2.8.0...2.9.0) (2026-08-17)
 
+### Features
+
+* **metadata:** unified metadata layer (`MetadataBag`, `MetadataExtractor`) with per-client extractors for FTP, DadosGov, DuckLake and Saude; every remote entity now exposes a merged `.metadata` bag
+* **saude:** add OpenDataSUS catalog client (`pysus.api.saude`) with Next.js buildId discovery, catalog listing, full CKAN package metadata and resource downloads (stage 1 of the dadosabertos.saude.gov.br integration)
+
 ## [2.8.0](https://github.com/AlertaDengue/PySUS/compare/2.7.0...2.8.0) (2026-08-14)
 
 ## [2.7.0](https://github.com/AlertaDengue/PySUS/compare/2.6.5...2.7.0) (2026-07-13)

docs/source/guides/index.rst

@@ -8,7 +8,9 @@ Guides
    pysus-orchestrator
    ftp
    dadosgov
+   saude
    ducklake
+   metadata
    files-and-formats
    datasets
    web-ui

docs/source/guides/metadata.rst

@@ -0,0 +1,102 @@
+========
+Metadata
+========
+
+Every remote entity in PySUS — clients, datasets, groups and files —
+exposes a unified metadata surface via the ``.metadata`` property:
+
+.. code-block:: python
+
+   from pysus.api.saude import SaudeClient
+
+   async with SaudeClient() as c:
+       pkg = await c.fetch_dataset("arboviroses-dengue")
+       # pkg.metadata is a MetadataBag
+
+The MetadataBag
+---------------
+
+:class:`pysus.api.metadata.models.MetadataBag` organizes metadata into
+eight typed facets:
+
+- ``identity`` — name, slug, aliases, and ``cross_origin_id`` (the
+  shared CKAN UUID that links the same dataset across the
+  dados.gov.br and dadosabertos.saude.gov.br portals);
+- ``description`` — title, long name, description, tags, themes;
+- ``temporal`` — created/modified timestamps, periodicity, year/month;
+- ``spatial`` — geographic scope, UF list, municipalities, state;
+- ``provenance`` — origin, organization, author, contact, license;
+- ``structure`` — columns, row count, file count, format;
+- ``access`` — URL, format, size, auth requirement, policy;
+- ``quality`` — content fingerprint, integrity, freshness.
+
+Every bag serializes to JSON (``bag.to_dict()`` /
+``MetadataBag.from_dict(...)``) so it can be persisted in the DuckLake
+catalogs or snapshot files.
+
+Extractors
+----------
+
+Each client ships its own extractors, one per entity type
+(:class:`~pysus.api.metadata.extractors.MetadataExtractor`):
+
++-------------+------------------------------------------+
+| Client      | Extractors                               |
++=============+==========================================+
+| FTP         | ``FtpDatasetExtractor``,                 |
+|             | ``FtpGroupExtractor``,                   |
+|             | ``FtpFileExtractor``                     |
++-------------+------------------------------------------+
+| DadosGov    | ``DadosGovDatasetExtractor``,            |
+|             | ``DadosGovGroupExtractor``,              |
+|             | ``DadosGovFileExtractor``                |
++-------------+------------------------------------------+
+| DuckLake    | ``DuckLakeDatasetExtractor``,            |
+|             | ``DuckLakeGroupExtractor``,              |
+|             | ``DuckLakeFileExtractor``                |
++-------------+------------------------------------------+
+| Saude       | ``SaudeDatasetExtractor``,               |
+|             | ``SaudeGroupExtractor``,                 |
+|             | ``SaudeFileExtractor``                   |
++-------------+------------------------------------------+
+
+Concrete model classes declare their extractors via the
+``extractor_types`` class attribute; the base classes only know that
+*some* extractor exists:
+
+.. code-block:: python
+
+   file = await some_dataset.search(name="DENGBR25.csv.zip")[0]
+   bag = file.metadata          # merged bag, cached
+   print(bag.temporal.year)     # 2025 (from the filename formatter)
+   print(bag.access.size_bytes)
+
+Merging across origins
+----------------------
+
+:func:`~pysus.api.metadata.models.merge_bags` combines bags from
+different origins with a documented per-facet precedence
+(``roadmap_saude.md`` §1.7): Saude wins for descriptive fields,
+DuckLake for structure and content fingerprints, and so on:
+
+.. code-block:: python
+
+   from pysus.api.metadata.models import merge_bags
+
+   merged = merge_bags([ftp_file.metadata, saude_file.metadata])
+
+   # Descriptive metadata comes from Saude, structure from DuckLake:
+   print(merged.description.title)
+   print(merged.structure.row_count)
+
+Local files
+-----------
+
+Tabular local files (Parquet, CSV, DBF, ...) also expose ``.metadata``
+computed from their content — columns, row count and size — without
+any network access:
+
+.. code-block:: python
+
+   local = await file.download()
+   print(local.metadata.structure.columns)

docs/source/guides/saude.rst

@@ -0,0 +1,102 @@
+==============
+OpenDataSUS
+==============
+
+:class:`pysus.api.saude.client.SaudeClient` talks to the portal of the
+Brazilian Ministry of Health — `dadosabertos.saude.gov.br
+<https://dadosabertos.saude.gov.br/>`_ — a Next.js frontend over a CKAN
+backend. It exposes the catalog of 138 health datasets, their full CKAN
+metadata, and the resource (file) downloads.
+
+.. note::
+   This is the *catalog* client (Stage 1 of ``roadmap_saude.md``). The
+   structured DEMAS REST API (``apidadosabertos.saude.gov.br``) and the
+   DuckLake sync integration ship in later stages.
+
+No token required
+-----------------
+
+The portal is public — no authentication header is needed. Instantiate
+the client and use it as an async context manager:
+
+.. code-block:: python
+
+   import asyncio
+   from pysus.api.saude import SaudeClient
+
+   async def main():
+       async with SaudeClient() as client:
+           ...
+
+Listing datasets
+----------------
+
+.. code-block:: python
+
+   # One page (20 entries) of the catalog
+   page = await client.list_datasets(group="arboviroses")
+
+   # All pages, lazily
+   async for entry in client.iter_datasets(group="arboviroses"):
+       print(entry.name, entry.title)
+
+   # Filters: q (text), group, tag, fmt (resource format)
+   page = await client.list_datasets(q="dengue", fmt="CSV")
+
+Groups and tags
+---------------
+
+.. code-block:: python
+
+   groups = await client.list_groups()   # 14 themes
+   tags = await client.list_tags()
+
+Fetching full metadata
+----------------------
+
+.. code-block:: python
+
+   package = await client.fetch_dataset("arboviroses-dengue")
+   print(package.title)              # "Sinan/Dengue"
+   print(package.id)                 # CKAN UUID (shared with dados.gov.br)
+   print(package.license_title)      # "Creative Commons Atribuição"
+   print(package.periodicity)        # "Semanal"  (from extras[])
+   print(package.contact)            # "arboviroses@saude.gov.br"
+
+   resources = package.resources     # 19 fields each
+   for res in resources[:5]:
+       print(res.name, res.format, res.size)
+
+Downloading resources
+---------------------
+
+.. code-block:: python
+
+   # All CSV resources of the dengue dataset
+   paths = await client.download_dataset(
+       "arboviroses-dengue", fmt="CSV", dest_dir="./data"
+   )
+
+   # A single resource, by id or name
+   path = await client.download_resource(
+       "arboviroses-dengue", name="Dengue - 2024", dest_dir="./data"
+   )
+
+Resources with format ``API`` are documentation links and are skipped
+automatically.
+
+Caching
+-------
+
+The portal's Next.js ``buildId`` and every catalog response are cached
+on disk (default TTL 24 h). Override with ``cache_dir`` and
+``cache_ttl``:
+
+.. code-block:: python
+
+   from datetime import timedelta
+
+   client = SaudeClient(
+       cache_dir="./.cache/saude",
+       cache_ttl=timedelta(hours=6),
+   )

pysus/api/dadosgov/metadata.py

@@ -0,0 +1,132 @@
+"""Metadata extractors for the DadosGov (dados.gov.br) client.
+
+Consumes the client's own models (``File``, ``Group``, ``Dataset`` in
+:mod:`pysus.api.dadosgov.models`) and produces
+:class:`~pysus.api.metadata.models.MetadataBag` instances.
+
+The filename-derived ``year`` / ``month`` / ``state`` metadata (from
+the per-dataset formatters) maps into the temporal and spatial facets.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from pysus.api.metadata.extractors import MetadataExtractor
+from pysus.api.metadata.models import (
+    AccessFacet,
+    DescriptionFacet,
+    IdentityFacet,
+    MetadataBag,
+    ProvenanceFacet,
+    SpatialFacet,
+    StructureFacet,
+    TemporalFacet,
+)
+
+
+class DadosGovDatasetExtractor(MetadataExtractor):
+    """Build a dataset-level bag from a DadosGov ``Dataset``."""
+
+    origin = "dadosgov"
+
+    def supported_facets(self) -> set[str]:
+        return {"identity", "description", "provenance"}
+
+    def _extract(self, obj: Any) -> MetadataBag:
+        dataset = obj
+        return MetadataBag(
+            identity=IdentityFacet(
+                name=dataset.name,
+                slug=getattr(dataset, "slug", "") or dataset.name,
+            ),
+            description=DescriptionFacet(
+                title=dataset.long_name,
+                long_name=dataset.long_name,
+                description=getattr(dataset, "description", "") or "",
+            ),
+            provenance=ProvenanceFacet(origin=self.origin),
+        )
+
+
+class DadosGovGroupExtractor(MetadataExtractor):
+    """Build a group-level bag from a DadosGov ``Group``."""
+
+    origin = "dadosgov"
+
+    def supported_facets(self) -> set[str]:
+        return {"identity", "description", "provenance"}
+
+    def _extract(self, obj: Any) -> MetadataBag:
+        group = obj
+        return MetadataBag(
+            identity=IdentityFacet(
+                name=group.name,
+                slug=getattr(group, "slug", "") or group.name,
+            ),
+            description=DescriptionFacet(
+                title=group.long_name,
+                long_name=group.long_name,
+                description=getattr(group, "description", "") or "",
+            ),
+            provenance=ProvenanceFacet(origin=self.origin),
+        )
+
+
+class DadosGovFileExtractor(MetadataExtractor):
+    """Build a file-level bag from a DadosGov ``File``."""
+
+    origin = "dadosgov"
+
+    def supported_facets(self) -> set[str]:
+        return {
+            "identity",
+            "temporal",
+            "spatial",
+            "access",
+            "structure",
+            "provenance",
+        }
+
+    def _extract(self, obj: Any) -> MetadataBag:
+        file = obj
+        modified = None
+        try:
+            modified = file.modify
+        except (ValueError, AttributeError):
+            modified = None
+        state = None
+        try:
+            state = file.state
+        except (ValueError, AttributeError):
+            state = None
+        return MetadataBag(
+            identity=IdentityFacet(
+                name=file.basename,
+                slug=file.basename,
+            ),
+            temporal=TemporalFacet(
+                modified=modified,
+                year=file.year,
+                month=file.month,
+            ),
+            spatial=SpatialFacet(state=state),
+            structure=StructureFacet(
+                format=(file.extension or "").lstrip("."),
+            ),
+            access=AccessFacet(
+                url=str(file.path),
+                format=(file.extension or "").lstrip("."),
+                size_bytes=file.size or 0,
+                download_strategy="http",
+                requires_auth=True,
+            ),
+            provenance=ProvenanceFacet(origin=self.origin),
+        )
+
+
+__all__ = [
+    "DadosGovDatasetExtractor",
+    "DadosGovFileExtractor",
+    "DadosGovGroupExtractor",
+]