feat(data): one data starter for the monorepo, on the stack the teams will run - #9
Open
lassebenni wants to merge 37 commits into
Open
feat(data): one data starter for the monorepo, on the stack the teams will run#9lassebenni wants to merge 37 commits into
lassebenni wants to merge 37 commits into
Conversation
PRs #1 and #4 were one change split in two: #4 targeted #1's branch and rewrote the same 26 files from Postgres to Databricks. Meanwhile main gained the Spring backend, the Next.js frontend, a root compose stack and CI, so #1 no longer merged. This lands the final state of both on current main. The two stacks had to be reconciled rather than concatenated: - Two Postgres containers both claiming to be the backend's database, both on port 5432. In a monorepo the root compose already is that database, so data/docker-compose.yml is gone and the pipeline joins the root stack behind a 'data' profile, keeping it out of a plain `docker compose up`. - The Airflow override already expected the root stack (service 'db', network 'finalproject') while data/airflow/.env.example said 'backend-db'. The branch disagreed with itself; the override was right. - The root README told you to copy a .env.example that did not exist. Added. - Took only the Python, dbt and Airflow ignores from the branch. Its .gitignore also added .mvn/, which would fight the committed Maven wrapper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Matches the backend and frontend workflows in shape and naming, and differs in one deliberate way: it pushes to the team's Azure Container Registry rather than GHCR, because the Container Apps job pulls with a managed identity holding AcrPull. Pulling from GHCR would mean storing a GitHub token in Azure, which is the credential this setup exists to remove. Sign-in is workload identity federation. GitHub signs a token naming the repository and branch, Azure trusts that subject in advance, and hands back an access token. The build job is gated on push so a pull request, including one from a fork, cannot mint a token by editing this file. Also adds a DAG import check, which catches the mistake that costs the most time: a DAG that fails to import is simply invisible in Airflow, with no error anywhere except the scheduler log. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The starter taught the Databricks Files API: mint an Entra token from the team's client id and secret, then hand-roll an HTTP PUT. That works, and it means the ingestion container holds two secrets to do one thing. Each team now has its own storage account whose landing container is registered in Unity Catalog as an external volume, so the container can write blobs as itself and dbt still reads /Volumes/<catalog>/landing/raw unchanged. DefaultAzureCredential resolves to your az login locally and to the Container Apps job's managed identity in Azure, so the same code runs in both and no secret exists. The ingestion job's settings are now three names and nothing else. Also fixes the review comment on #4 about what actually gets landed. The old code wrote model_dump output, so the 'raw' file quietly carried our own type coercions and re-reading it after a source change would have described our bug rather than theirs. Validation is now a gate, not a transformation: it decides whether the run is worth landing, and the source's own bytes are what land. And wires --run-date through to the entry point, which the previous version documented but never parsed. Verified against team A rather than assumed: 175 records written as landing/raw/postings/2026-08-12.json, then read back through the volume as 175 rows via read_files. Test file deleted afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Monitoring is one of the things the project is assessed on, and the starter had none: no callback, no notification, nothing but a red square waiting for someone to open Airflow. The DAG runs at 6am, so that someone is nobody. Adds a Slack failure callback attached once in default_args, so every task inherits it including tasks added later. Attaching it per operator instead means the task added at 11pm on a Thursday is the one without alerting. The webhook URL is a credential, so it stays in Key Vault and is read by the VM's managed identity inside the callback, where it costs nothing until something has already gone wrong. Module scope would refetch it every few seconds, because Airflow re-parses DAG files continuously. Two things found by importing the DAG rather than reading it: - the dbt path was hardcoded to /opt/airflow, while local Astro mounts the project under /usr/local/airflow, so the documented local run could not have worked. Both now come from DBT_PROJECT_DIR, set in the override for local runs and defaulted to the VM's path. - a bare DagBag() does not put the dags folder on sys.path, though Airflow's DAG processor does. The CI check was therefore stricter than the runtime and failed on an import that works in production. Fixed in the check, not by contorting the DAG. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…publish Two defects that would each have hit every team, found in review of #4 and confirmed against the live warehouse. Staging read every file in the landing folder and most sources still list the same record tomorrow, so on day two an open posting appeared twice and the unique test on posting_id failed. Nothing was wrong with the data. Staging now keeps one row per posting, the most recently ingested one, which also means an edited posting reflects what the source says today. Verified on Databricks: three rows with a repeated id collapse to two, keeping the newer title. The documented publish pattern renamed the current table out of the way before swapping, which cannot work the first time you publish because there is nothing to rename. The sync would have failed exactly once, on the run you most want to see succeed. Now documented with the drop-if-exists ordering that works on an empty database. Also records which schema publishing writes to, now that the two tracks meet through two schemas that each side owns: analytics outbound, app inbound. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rewrites the getting-started section as numbered steps, split into what was already done when the repository was created and what a trainee does once on their own machine. Two things it now says that cost real time when they are missing: - Contributor on the resource group does not let you read a blob. Azure keeps managing a storage account separate from reading what is inside it, so each team member needs Storage Blob Data Contributor, and the failure otherwise is an AuthorizationPermissionMismatch that reads like a broken account. There is now a one-line check that either works or tells you exactly this. - role assignments take a few minutes to propagate, so the same command failing immediately after it is granted is expected rather than wrong. The architecture section also stops claiming there is no storage account to secure, which was true of the managed volume it used to describe and is not true now, and shows both directions between the tracks rather than only the outbound one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ruff caught the same class of bug that models.py already had fixed: a naive date.today() is the machine's timezone, so a container running in UTC and a laptop in Amsterdam disagree about which day it is. That decides the filename a run lands under, so around midnight two runs could write to the same file or skip a day entirely. The DAG's start_date had the same problem, and Airflow wants it aware anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 11, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a monorepo data starter aligned with Azure, Databricks, Airflow, dbt, and the backend Postgres boundary.
Changes:
- Adds ingestion, validation, storage, dbt, and publishing scaffolding.
- Adds Airflow orchestration, Slack alerting, and local Compose integration.
- Adds data CI/CD, setup documentation, and project conventions.
Reviewed changes
Copilot reviewed 36 out of 38 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
docker-compose.yml |
Adds the profiled pipeline job. |
data/src/sync.py |
Defines the publishing scaffold. |
data/src/storage.py |
Implements Azure blob landing. |
data/src/pipeline.py |
Wires ingestion and storage. |
data/src/models.py |
Adds source validation model. |
data/src/ingest.py |
Fetches and validates API records. |
data/src/config.py |
Loads ingestion configuration. |
data/src/__init__.py |
Initializes the source package. |
data/README.md |
Documents setup and architecture. |
data/pyproject.toml |
Defines Python dependencies. |
data/optional/streamlit/app.py |
Adds an operations dashboard. |
data/optional/README.md |
Documents optional modules. |
data/docs/mart_contract.md |
Defines the mart contract. |
data/Dockerfile |
Builds the ingestion image. |
data/dbt/tests/assert_postings_not_empty.sql |
Tests for an empty mart. |
data/dbt/tests/assert_posted_at_not_in_future.sql |
Tests posting timestamps. |
data/dbt/profiles.yml |
Configures Databricks OAuth. |
data/dbt/models/staging/stg_postings.sql |
Stages and deduplicates postings. |
data/dbt/models/staging/_stg_postings.yml |
Documents staging columns. |
data/dbt/models/marts/fct_postings.sql |
Defines the published mart. |
data/dbt/models/marts/_fct_postings.yml |
Documents the mart contract. |
data/dbt/dbt_project.yml |
Configures dbt models and paths. |
data/airflow/requirements.txt |
Adds DAG dependencies. |
data/airflow/packages.txt |
Defines OS-package configuration. |
data/airflow/Dockerfile |
Selects the Astro runtime. |
data/airflow/docker-compose.override.yml |
Configures local Airflow mounts. |
data/airflow/dags/pipeline_dag.py |
Adds the three-stage DAG. |
data/airflow/dags/alerts.py |
Adds Slack failure alerts. |
data/airflow/.gitignore |
Excludes Airflow runtime files. |
data/airflow/.env.example |
Documents local Airflow settings. |
data/airflow/.astro/config.yaml |
Configures the Astro project. |
data/.env.example |
Documents data configuration. |
data/.dockerignore |
Reduces the image build context. |
.gitignore |
Adds data-tool exclusions. |
.github/workflows/pr-checks.yml |
Enforces PR structure and size. |
.github/workflows/data-ci-cd.yaml |
Adds data lint, test, and image CI. |
.github/pull_request_template.md |
Adds the PR template. |
.env.example |
Defines shared Postgres defaults. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
4 tasks
added 17 commits
August 12, 2026 01:31
Six review findings, all confirmed against the live source and warehouse. The worst was mine. Landing the raw response instead of the validated objects was right, but it left Arbeitnow's Unix seconds in the file, and staging aliased that integer straight to posted_at. The mart's date(posted_at) and the future-date test both sit on top of it, so the first dbt build a trainee runs would have failed. Staging now converts with timestamp_seconds, which is also the honest place for it: the landed file stays a faithful copy of what the source sent, and a source that changes its date format shows up in one line rather than across three weeks of files. The dbt step in the README could not have worked either, for two more reasons: uv run does not pick up .env, so profiles.yml had no credentials, and landing_path still shipped as /Volumes/CHANGE_ME. The DAG's TODO named a variable that does not exist, which would have left the same CHANGE_ME path live on a schedule. Also: the local-stack snippet mixed working directories so no single directory could run all three lines; the dashboard extra omitted psycopg, which its own app imports; and a scalar inside the source's JSON list would crash the batch that the rejection handler exists to protect. Verified this time by running dbt rather than reasoning about it: PASS=18 ERROR=0 against 175 real records, posted_at spanning 2026-08-11T11:30 to 21:55. Scratch schema, test blob and local env removed afterwards.
Found by cloning the branch as a trainee would and following data/README.md literally against team A's real infrastructure, rather than re-reading it. uv picked Python 3.14, because requires-python only had a lower bound. dbt-core 1.10.9 cannot load on 3.13 or later and dies with 'UnserializableField: Field schema ... is not serializable' from mashumaro, which mentions neither dbt nor Python versions. The documented first dbt build failed before connecting to anything. Capped below 3.13, after which uv chose 3.12 and the same command gave PASS=18 ERROR=0. The setup asked for four values and needed six. dbt and the publish step also need the team's Databricks client id and secret, and leaving them blank does not produce an error: dbt-databricks falls back to interactive sign-in and waits for a browser that never opens, so the command just hangs. Confirmed with dbt debug, which printed its version banner and then nothing for 90 seconds. The README now says to read both from Key Vault, and warns about the hang. The container cannot run on a laptop. Both the README and the Dockerfile said to run the image locally, but there is no az login and no metadata service inside it, so DefaultAzureCredential fails through every option in turn. The image is built locally and gets its identity from Azure when the Container Apps job runs it; on your own machine you run the pipeline directly. Everything else in the setup held: the blob-access check, the pipeline landing 175 records, the volume round trip, dbt build, and astro dev start with the DAG loaded, no import errors, and DBT_PROJECT_DIR resolving to the mounted project.
The Dockerfile repeated the dependency list. A package added to pyproject works locally with uv run and is missing from the image, so the Container Apps job fails with ModuleNotFoundError long after the change that caused it.
pd.read_sql only recognises SQLAlchemy and sqlite connections. Given a psycopg one it returns the right numbers and prints a warning telling you to install SQLAlchemy, which the page does not need.
The template was TODO-shaped: sync.py and two DAG tasks raised NotImplementedError, so the DAG had never run start to finish. It now runs: ingest lands raw JSON in the team's storage account, dbt builds and tests the models, an enrichment container adds what SQL cannot express, and the mart is published into the backend's analytics schema with a write-then-swap. The container logic lives in data/src and is shared by the two Container Apps jobs and by the Airflow tasks, so there is one copy to read and one to test. 53 tests cover the seams that have no credentials: validation, path layout, the job poll loop, the swap ordering, the classifier and the dbt results parser.
The README told teams the pipeline was a skeleton with empty bodies, which stopped being true. It now says what runs, what a team normally changes, why the enrichment step is a container rather than a dbt model, and how the three names for one table relate: dbt builds fct_postings, enrichment writes fct_postings_enriched, Airflow publishes analytics.fct_postings.
It asked for POSTGRES_HOST and friends, which nothing sets: the sync and .env.example both use BACKEND_PG_*. It also never loaded .env, so even the right names would not have reached it. Same file, same names as the rest of the pipeline now.
ruff already caught mistakes; nothing settled layout or checked types. black formats the Python, sqlfmt the dbt SQL, and ty the types, each as its own CI step so a failure names itself. ty earned its place immediately. psycopg only accepts SQL it can see was written in the source rather than assembled from a value, so the publish step now composes statements with psycopg.sql.Identifier instead of f-strings: type-correct and injection-safe for the same reason. The warehouse gained a Queryable protocol, which is what lets the tests hand enrich and publish_results a fake with no mocking library. Warehouse.from_env now loads .env, so running the enrichment on a laptop works the way the README says it does.
The README counted the tests, which the first person to add one makes wrong, and it said every setting is an Airflow Variable when the DAG reads the environment first. It also now names the failure a missing src mount produces, because ModuleNotFoundError inside one task is not an obvious symptom of a deploy-time mount.
Four things only an actual run could find, all found by running it on team A: - a skipped inbound_sync skipped everything after it. Under the default all_success rule a skip cascades, so the pipeline would have done nothing at all, quietly, until the backend exposed a view. dbt_build now uses none_failed. - the environment handed to dbt was missing DBT_SCHEMA, which profiles.yml requires. dbt exits before it runs and writes no run_results.json, so there is nothing to read afterwards either. - AZURE_TENANT_ID was read with an empty default. An empty tenant builds a token URL with a hole in the middle and the only symptom is a 404 that mentions neither tenants nor settings. It is a required setting now, checked with the others. - AirflowSkipException moved to airflow.sdk.exceptions.
Students are admins on their own instance and have no access to the machine, so Admin -> Variables is the whole story. Also adds the two variables the first real run showed were missing from the list.
One 2X-Small serverless warehouse instead of three. It stops after ten idle minutes either way, so three of them mostly bought three separate idle windows. Isolation is unchanged: it is the catalog grants, not the compute, that decide what a team can read.
The data folder was 49% comments and docstrings against the backend's 2%, so the same ideas were written twice: once beside the code and again in the README. The code now carries one-line docstrings and the few comments that explain a decision at the point it is made, and the README carries the reasoning. Also: config.py folded into pipeline.py, the dbt-results recorder moved to optional/ with instructions for wiring it back in, and inbound_sync dropped from the DAG until the backend has a view to read. Behaviour is unchanged. 1269 lines to 790, 12 modules to 10.
main's .env.example uses mydb/admin/password, which the backend's application.yaml already defaults to. The data side now matches, so one docker compose up gives every part of the stack the same database.
The build job could only ever fire on a merge to main, so nothing could check it before then. workflow_dispatch makes it runnable from any branch, which is also what you want when a base image gets a security fix and the code has not changed.
Four things stopped a team member developing locally without either touching the shared pipeline's data or editing the DAG: - the ingestion wrote to the same landing path as the scheduled run, so testing a change overwrote the file the team's models read. LANDING_PREFIX puts your runs under dev/<your name>, and each team's catalog now has a volume covering it. - dbt's landing_path was a project variable with CHANGE_ME in it. It now reads LANDING_PATH from the same .env the ingestion uses, so the writer and the reader cannot disagree and VS Code needs no arguments. - the publish task fetched its password from Key Vault over the instance metadata service, which does not exist on a laptop. It now reads the environment first and Key Vault second, which is also how the Databricks credentials already worked. - the publish target was hardcoded to analytics. It reads BACKEND_PG_PUBLISH_SCHEMA, so locally you publish to analytics_dev. The local Postgres now creates analytics, analytics_dev and app when it first starts, and local Airflow reads data/.env, so the same DAG file runs in both places with nothing stubbed out. Verified end to end as a team member would: 176 records landed under dev/alex, dbt built dev_alex with PASS=18, enrichment wrote fct_postings_enriched, and 'airflow tasks test publish_to_backend' in local Airflow put 176 rows in analytics_dev.fct_postings.
The Dockerfile promised localhost:8080, which was true in Week 12. Astro CLI 1.43 serves http://airflow.localhost:6563, and a student following the comment finds nothing there.
Agreed with the backend track: two schemas, not three. The curated `app` schema of exposed views was the piece nobody knew what to put in, so it goes. The data side keeps read access, it just points at `public`. - read_app_table -> read_backend_table(dsn, table, schema="public") - the local compose database no longer creates `app` - README: two schemas instead of three, and PII becomes the reader's problem at staging rather than the writer's problem in a view Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three references survived the previous commit: the summary paragraph in the README intro, the backend-database comment in .env.example, and BACKEND_PG_SOURCE_SCHEMA, which nothing has read since read_app_table went away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The compose override loads ../.env, so this file stopped being needed and started being wrong. It named a database "backend" with user "postgres" when the compose stack creates "mydb" with "admin", so the publish step failed locally. It also forced DBT_SCHEMA=analytics, which silently undid the per-developer schema the rest of the local loop depends on. Its "cd ../../.." pointed one level above the repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Setup step 6 said to edit landing_path in dbt_project.yml and the staging
model repeated it, but that value is env_var('LANDING_PATH', ...) and
.env.example ships LANDING_PATH, so the environment always won and the
file edit did nothing. Both now point at .env, which dbt_project.yml
already described correctly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
uv sync makes the environment match exactly what is asked for, so the documented `uv sync --extra dev` before the tests removed dbt and psycopg installed four steps earlier. Verified: `import dbt.version` works after step 5 and raises ModuleNotFoundError after step 8. One --all-extras up front instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The enrichment and publish steps required DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET, so running either one locally meant copying the team's production service principal onto your laptop in plaintext. Both are optional now. When they are set the code exchanges them at Entra, which is how the scheduled run authenticates. When they are not, it asks DefaultAzureCredential, which is your `az login` on your machine and the job's managed identity in Azure. Verified against the live warehouse with both credentials unset: current_user() returned the signed-in account. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LANDING_PREFIX kept your files out of the team's, but only by convention: the same role that let you write dev/ let you write raw/, so one wrong prefix could overwrite what the scheduled run had landed. Each team's storage account now has a second container, `dev`, registered in Unity Catalog as the volume it already had. People hold write on `dev` and read on `landing`, so overwriting the team's raw files is no longer something a typo can do. LANDING_CONTAINER selects which one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dev target used external-browser, which is a flow nobody has been taught and which I could not verify for guest accounts. It now reads DATABRICKS_TOKEN, the personal access token students already generated in Week 13, and the README repeats that chapter's `session_user()` check so you find out immediately whose token you pasted. warehouse.py reads the same variable first, so one value covers dbt, the enrichment and the publish step. The service principal path stays for Airflow and the container jobs. Verified with a real token: dbt reported connected_as lasse@..., and the same identity was refused CREATE TABLE on team_a.analytics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The local-development section still described one container and a naming convention. It now names LANDING_CONTAINER, says the separation is a permission rather than a habit, and says what happens when you point at the production names by mistake. Also: the dbt_results snippet was a NameError as written, missing the Warehouse import, and the Streamlit page was described as an operations dashboard when it reads one Postgres table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…arter # Conflicts: # .env.example
Step 3 said to fill in BACKEND_PG_PASSWORD, but db-setup.py, which is what prints it and prints it once, was only introduced two sections later under local development. Creating the database is now step 3, with the pip install the backend README asks for and a warning about the port 5432 collision that produces `role "admin" does not exist`. Also drops BACKEND_PG_SOURCE_SCHEMA, which nothing reads: read_backend_table takes the schema as an argument and defaults to app. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Setup steps 3 and 4 both said `cp .env.example .env` for different files, the repository root's and data/'s. Copying the wrong one gives a database error rather than a missing-variable error, which sends you looking in the wrong place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I built
One data starter for the monorepo, on current
main. This replaces #1 and #4, which were the same change split across two branches, and brings the pipeline in line with the platform the three teams actually run on.It is not a scaffold. It ingests from a public job board, builds and tests dbt models on Databricks, enriches, and publishes into the backend's database, and it has been run end to end on all three teams' real infrastructure.
Why this approach
Oversized: this is the initial data folder for the monorepo plus the rewrite that aligns it with the team platform. Splitting it would leave the branch half Postgres and half Databricks, which no reviewer could evaluate.
One PR, because #1 and #4 were never separable. #4 targeted #1's branch and rewrote the same 26 files from Postgres to Databricks. Meanwhile
maingained the Spring backend, the Next.js frontend, a root compose stack and CI, so #1 stopped merging. Reviewing them apart means reviewing files #4 deletes.The template is a source, not a deployment. Each team gets their own monorepo from it, so nothing team-specific is committed: no catalog, no warehouse id, no storage account. Those arrive as repository variables and Airflow Variables when a team is provisioned, and
.env.examplemarks the two a trainee pastes in alongside their own token.Contract impact
None on the mart.
fct_postingskeeps its columns and its_fct_postings.yml.The publish target is stated: marts go to
analytics, which the backend reads. This branch now followsscripts/db-setup.py, which landed on main while it was open: the database isproject_db, the backend's schema isapp, and each side connects as its own role,app_userandanalytics_user. There is no third schema of curated views, and noanalytics_dev: a local database is yours outright, so there is nothing to separate yourself from.That supersedes #5, and it removed this branch's own
data/local/01-analytics-schema.sql, which created schemas without roles.Who runs what
Development and the scheduled run use different identities, and the difference is enforced by permissions rather than by asking people to be careful.
analytics_user, fromdb-setup.pyanalytics_user, password from Key Vaultdevcontainer, readlandinglandingdev_yourname, readanalyticsanalyticsanalyticsin your own Postgresanalyticsin the backend'sPoint your
.envat the production names and the run stops with a permission error. That is the point: it used to be a naming convention, and a convention does not survive a bad afternoon.The four steps
ingest(container) →dbt_build→enrich(container) →publish_to_backend, as one Airflow DAG. Both containers run the same image and differ only in the command, which is how the two Container Apps jobs are configured in Azure.How to run
data/README.mdhas the full local loop, including running the DAG on Astro against your owndevcontainer anddev_<name>schema.Self-check
Verified, not assumed
Every row below was produced by running the thing, not by reading the config.
analytics.fct_postingsworkflow_dispatchpushedpipeline:90c621b2…with no stored secret, and that exact image then ran through team A's DAGcp .env.example .env: 176 records into thedevcontainer,session_user()reporting the right person,dbt buildPASS=18, enrichment, and 176 rows intoanalytics_devdbt_buildandpublish_to_backendrun under local Astroread_files,/Volumes/team_a/landing/raw/postings/QUALIFYdedup on Databricks.envpointed at production was refused four times:AuthorizationPermissionMismatchon thelandingcontainer, andPERMISSION_DENIED ... CREATE TABLEonanalytics,opsandraw_appdb-setup.pyanalyticsasanalytics_user, andcreate table app.probewas refused withpermission denied for schema apppipelineappears only under thedataprofileTest blobs and rehearsal tables were removed afterwards.
What changed against the old branches
Landing zone. Ingestion wrote through the Databricks Files API using the team's client id and secret. Each team now has its own storage account whose
landingcontainer is an external volume, so the container writes blobs as itself withDefaultAzureCredentialand dbt still reads/Volumes/<catalog>/landing/rawunchanged. The ingestion job's config is now three names and no secret at all.An enrichment step. A second container job adds the column SQL cannot express, then joins it back into
fct_postings_enriched. It exists to make the shape honest: every team will need one step that is Python rather than SQL.CI builds the image.
data-ci-cd.yamlmatches the backend and frontend workflows in shape, and pushes to the team's ACR rather than GHCR because the Container Apps job pulls with a managed identity holding AcrPull. Sign-in is workload identity federation, so no password exists. Tagged with the commit SHA, which is what makes a row traceable to the code that wrote it.Alerting. A Slack failure callback in
default_args, so every task inherits it including ones added later. Monitoring is assessed, and the starter had none.A credential nobody has to carry. dbt used the team's service principal, which meant every trainee needed the production credential on their laptop, and one wrong
DBT_SCHEMAaway from overwriting the team's models. Local steps now use the trainee's own Databricks token, the same one Week 13 has them generate, andprofiles.ymlhas aprodtarget that Airflow selects with--target prod.warehouse.pyreads the token first, the service principal second, and a managed identity third, so the same file works on a laptop, in a container job and on the VM.A
devcontainer. Keeping your files under adev/prefix separated them by habit only, because Azure can grant a permission on a container and not on a folder. Each storage account has a second container now, registered as the volume teams already read, soLANDING_CONTAINERdecides which one you write and the answer is checked rather than trusted.Tooling. ruff, black, sqlfmt for the dbt models, ty instead of mypy, pytest, and
python-dotenvso the same code reads.envon a laptop and real environment variables in Azure. All five run in CI.One database, not two. The starter brought its own Postgres as a stand-in for the backend's, which in a monorepo is the database sitting next to it, on the same port. The pipeline now joins the root stack behind a
dataprofile.Less of it. After the stack was proven, the code came down from 1269 to 790 lines and from 12 modules to 10, and the prose-to-code ratio from 49% to 25%. What was cut is in
data/optional/, with a README saying how to put it back.Review comments carried over
Fixed here:
model_dumpoutput (refactor(data): make the starter a Databricks template, not a solution #4). Validation is a gate, not a transformation.--run-datereaching the entry point (refactor(data): make the starter a Databricks template, not a solution #4). It was documented but never parsed./usr/local/airflow, the VM uses/opt/airflow; both now come fromDBT_PROJECT_DIR.read_filesreads every file and most sources still list the same record tomorrow, so theuniquetest failed with nothing wrong with the data..env.examplenot matching the variable contract (refactor(data): make the starter a Databricks template, not a solution #4).Already fixed on #4's branch, so stale on #1: the naive datetime in
models.py, the undeclared Streamlit dependencies, thepostgresversusdbservice name, the zero-record success inpipeline.py, the top-level-list fallback iningest.py, and the SQL injection in the old Postgresstorage.py. The Bicep andoptional/databrickscomments point at files that no longer exist.Still to do, not in this PR
optional/README.mdnow says so rather than calling it an operations dashboard.#5covers the backend half of the schema boundary, and is where the one-time role and schema setup lands.devcontainers, the narrowed grants, the revoked Key Vault access) was applied by hand and is not yet in a provisioning script.🤖 Generated with Claude Code