Add Amendement model and preserve analysis tables on rebuild#4
Closed
DavidNO wants to merge 3 commits into
Closed
Conversation
Introduce the raw `amendements` table (fields sourced directly from the Tricoteuses /amendements endpoint, grouped into identity, text, author, attachment, status and dates). Column names mirror the JSON keys so the existing schema-driven ETL loads them without changes. Scope the rebuild to ETL-managed tables (ETL_TABLES) so future analysis tables are neither dropped by `create_db` nor treated as source files by the ETL loop. This lets analysis results survive a re-download without introducing Alembic. Add `.env.example` wiring the ETL to the local docker-compose Postgres.
getenv returns a string, so bool(getenv("PG_ECHO", False)) was truthy for
any non-empty value including "False", leaving SQL echo permanently on.
Parse the flag explicitly instead.
# Conflicts: # models/__init__.py
DavidNO
force-pushed
the
add-amendements-model
branch
from
July 11, 2026 18:05
627d6a4 to
495c3df
Compare
This was referenced Jul 12, 2026
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contexte
Premier socle DB pour le POC d'analyse des amendements. Objectif : pouvoir ingérer les amendements bruts et pouvoir y greffer plus tard des tables d'analyse (mentions « travaillé avec… », similarité, sémantique) sans perdre les résultats d'analyse à chaque rebuild
Changements
Amendement(models/amendement.py) : table bruteamendements. Champs issus directement de l'endpoint Tricoteuses/amendements, regroupés en 6 catégories (identité, texte, auteur/signataires, rattachement, statut/sort, dates). Les noms de colonnes sont en camelCase pour matcher 1:1 les clés du JSON → chargés tels quels par l'ETL existant. Seuluidest non nullable ; le reste est nullable (amendements hétérogènes : budgétaires, sous-amendements).etl/database.py) :create_db()ne drop/recrée que les tables listées dansETL_TABLES(dossiers,amendements). Les futures tables d'analyse ne sont ni détruites au rebuild, ni parcourues par la boucle ETL (get_tables_definition()est également restreint).create_allreste idempotent..env.example: câblage de l'ETL sur le Postgres local dudocker-compose.yml.Notes de revue
*RefUid(acteur, organe, dossier, document…) sont conservés en colonnes texte non contraintes (références molles), volontairement : on n'ingère les entités liées que quand une analyse le demandera.Vérifications
uidPK).dossiers,amendements).exposeSommairerenseigné.ruff check+ruff format.uid, idempotence du re-ETL confirmée (on_conflict_do_nothing), drop sélectif OK.Correctif inclus
PG_ECHO(etl/database.py) :bool(getenv("PG_ECHO", False))était toujours vrai (getenv renvoie une chaîne,bool("False") == True), l'echo SQL restait donc toujours actif. Parsing explicite du flag. Vérifié sur données réelles : ETL de bout en bout OK (500 dossiers + 1000 amendements chargés, idempotence confirmée, 0 doublon).