refactor(data): make the starter a Databricks template, not a solution

[lassebenni]

What I built

Rebases the data starter onto the stack every team will actually use (Databricks, ACA jobs, Airflow) and turns it from a working pipeline into a template with TODOs. Targets feat/data-starter so it lands as a review on top of #1 rather than a competing branch.

Oversized: it rewrites the same starter #1 introduced, so splitting it would leave the branch in a half-Postgres, half-Databricks state that no reviewer could evaluate.

Why this approach

Three decisions taken after #1 was opened:

1. All three teams use Databricks, ACA jobs and Airflow. The starter was Postgres end to end, and the two-stack choice it assumed no longer exists.
2. Airflow also publishes from the lakehouse into the backend's Postgres. That step was missing, and it is the one that turns a pipeline into a data product.
3. Trainees get starter templates, not full solutions.

Postgres stays, but in its real role: the backend's database, which the pipeline publishes into, not where raw data lives. Raw lands as files in the team's Unity Catalog volume, because a file is exactly what the source sent and can be re-read when a column changes shape.

Staging reads the volume with read_files rather than a declared dbt source, so a new day's file needs no code change.

Considered and rejected: keeping a Postgres path for local development alongside Databricks. It gives every concept two implementations and doubles what a trainee has to hold in their head.

Contract impact

None. fct_postings keeps the same columns and the same _fct_postings.yml. One description changed: tags no longer mentions Postgres JSONB operators, since it is a Databricks array until the sync writes it out.

How to run

gh pr checkout 4
cd data
python3 -m py_compile src/*.py airflow/dags/pipeline_dag.py   # imports cleanly
grep -rn "NotImplementedError" src/ airflow/dags/            # what trainees write

Self-check

• I ran this and it works
• Tests pass locally
• No secrets, tokens, or connection strings in the diff
• This pull request does one thing

What changed

Databricks first. src/storage.py lands raw JSON in /Volumes/<catalog>/landing/raw/<source>. dbt/profiles.yml targets dbt-databricks against the team catalog.

Three-step DAG. ingest (trigger the ACA job and wait) → dbt_build (uvx on Python 3.11 with dbt-databricks) → publish_to_backend. Publishing runs after dbt, so a failed test never reaches the backend.

Template, not solution. Finished: config.py, ingest.py, the pipeline.py wiring, the Dockerfile, the contract .yml, the two custom tests. Left to the trainees, each with a docstring saying what it must do: land_raw_json, landing_file_name, read_mart, publish, and the three DAG task bodies.

Local dev. docker compose starts one Postgres named backend-db, standing in for the backend database so the sync can be built before the real one exists.

Removed optional/databricks/, which described Databricks as an optional upgrade from Postgres.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Refactors the data starter into a Databricks-first training template using ADLS, ACA jobs, Airflow, dbt, and backend Postgres publishing.

Changes:

• Replaces Postgres raw storage with ADLS and Databricks models.
• Adds three-stage Airflow orchestration and outbound synchronization scaffolding.
• Promotes Azure infrastructure and removes obsolete optional modules.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
data/src/sync.py	Adds backend publishing template.
data/src/storage.py	Replaces Postgres writes with ADLS landing template.
data/src/pipeline.py	Wires ingestion to raw-file landing.
data/src/config.py	Adds storage-oriented configuration.
data/README.md	Documents the new architecture and workflow.
data/pyproject.toml	Adds Azure, Databricks, and sync dependencies.
data/optional/README.md	Removes promoted optional components.
data/optional/databricks/README.md	Removes obsolete Databricks upgrade guide.
data/optional/bicep/modules/storage.bicep	Removes old optional storage module.
data/optional/bicep/main.bicep	Removes old optional deployment entry point.
data/infra/README.md	Documents required Azure infrastructure.
data/infra/modules/storage.bicep	Provisions secured ADLS storage.
data/infra/main.bicep	Adds deployment and role assignment wiring.
data/Dockerfile	Installs Azure storage dependencies.
data/docker-compose.yml	Reframes Postgres as the backend database.
data/dbt/tests/assert_posted_at_not_in_future.sql	Adapts timestamp syntax for Databricks.
data/dbt/profiles.yml	Switches dbt to Databricks OAuth.
data/dbt/models/staging/stg_postings.sql	Reads raw files through read_files.
data/dbt/models/staging/_sources.yml	Removes the Postgres source declaration.
data/dbt/models/marts/fct_postings.sql	Updates the mart for Databricks staging.
data/dbt/models/marts/_fct_postings.yml	Updates the tags contract description.
data/dbt/dbt_project.yml	Adds the landing-volume path variable.
data/airflow/requirements.txt	Replaces embedded dbt dependencies.
data/airflow/dags/pipeline_dag.py	Adds ingest, dbt, and publish orchestration.
data/.env.example	Adds storage, Databricks, and backend settings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[lassebenni]

Superseded by #9, which lands the final state of both branches on current main.

This one no longer merges: main has since gained the Spring backend, the Next.js frontend, the root compose stack and CI, all of which postdate this branch.

#9 carries the six live review comments from #4 as fixes, and notes in its description which comments here are already stale (the naive datetime, the undeclared Streamlit dependencies, the postgres versus db service name, the zero-record success, the top-level-list fallback, and the SQL injection in the old Postgres storage.py, plus two comments on files that no longer exist).

Leaving this open for you to close.