feat(data): working end-to-end data pipeline starter

[lassebenni]

What I built

A working end-to-end starter for the data/ folder: fetch from a source API, validate, store, shape with dbt, publish a mart the backend reads. Plus the PR template and checks this description is now demonstrating.

Why this approach

One composed starter, not one template per tool. Week 15 is synthesis: a team ships one pipeline. Five parallel scaffolds (dbt, Airflow, Databricks, Docker, Python) would make the two data trainees spend day one picking tools instead of building, and would duplicate the ingestion, config, and secrets layer five times. Databricks, Bicep, and Streamlit sit under optional/, matching how week_15__2_project_requirements.md already marks them.

It runs on clone with no credentials. The default source is the Arbeitnow job board, which needs no key, and docker compose up -d db gives a local Postgres. A team's first hour goes into their product rather than setup.

The mart is the contract with backend/. dbt/models/marts/_fct_postings.yml documents every published column, and docs/mart_contract.md sets out which changes are safe alone (add a column) and which need agreement (rename, remove, retype). That is what lets the backend write endpoints before the pipeline is finished.

Contract impact

None. This is new scaffolding in an empty folder; no existing mart or endpoint changes.

How to run

cd data
cp .env.example .env
docker compose up -d db
uv venv && uv pip install -e ".[dbt]"
uv run python -m src.pipeline
cd dbt && uv run dbt build --profiles-dir .

# the way Azure will run it
docker compose run --rm pipeline

# orchestrated
cd airflow && cp .env.example .env && astro dev start

Self-check

• I ran this and it works
• Tests pass locally
• No secrets, tokens, or connection strings in the diff
• This pull request does one thing

Verified end to end, not assumed:

Aspect	Result
docker build	builds
Pipeline from host	175 records, 0 rejected
Pipeline in container	175 rows written
Idempotency (second run)	still 175 rows
dbt build from host	PASS=18 ERROR=0
dbt build inside Airflow	PASS=18 ERROR=0
astro dev start	stack boots
Full DAG run	state=success, both tasks

Four real defects were found by running it rather than reading it: the container could not reach Postgres via localhost; the DAG used Airflow 2 imports against a pinned Airflow 3 runtime; data/airflow/ was not a valid Astro project and my own .gitignore made it permanently unfixable; and the DAG was silently connecting to Airflow's own metadata database, because Astro ships a service also called postgres. All fixed, details in the comments below.

Oversized: initial scaffold for an empty folder, 32 files that only make sense together. Splitting it would produce several PRs that individually do nothing.

🤖 Generated with Claude Code

[lassebenni]

Verification update: the two unverified items are now done, and both had defects

Docker was unavailable when this PR was opened. Re-ran both checks and each one found a real bug, now fixed in b2ab975.

1. docker build passes, but the documented run command did not

The image builds. Then docker run --env-file .env final-project-data failed:

connection to server at "127.0.0.1", port 5432 failed: Connection refused

Inside a container localhost is the container, not the host, so POSTGRES_HOST=localhost from .env.example could never reach the compose database. This is exactly the day-one blocker the template exists to remove.

Fixed by adding a pipeline service to docker-compose.yml that joins the compose network and overrides the host. Verified from inside the container:

POSTGRES_HOST resolved to: postgres
rows written: 1
CONTAINER -> POSTGRES OK

The README now shows both paths and explains why they differ.

2. The DAG parsed, but against a deprecated API

Parsed in the Astro runtime image: import_errors: {}, so it was never broken. But it emitted three deprecation warnings, because the pinned runtime:3.3-2 is Airflow 3:

• airflow.decorators.dag and .task are now airflow.sdk
• airflow.operators.bash.BashOperator is now airflow.providers.standard.operators.bash.BashOperator

Shipping deprecated imports in a starter teaches them to every trainee who copies it. Switched to the current API. Re-parsed clean:

import_errors: {}
DAG: final_project_pipeline | tasks: ['dbt_build', 'ingest']

Still not verified

The DAG parses and its tasks are correct, but I have not executed a scheduled run end to end. Worth one astro dev start before trainees see this.

🤖 Generated with Claude Code

[lassebenni]

astro dev start run: three more defects, all fixed in 5f41c9d

Parsing the DAG was not enough. Actually booting the stack found three things that would each have stopped a trainee on day one.

1. data/airflow/ was not an Astro project.

Error: this is not an Astro project directory.

It had no .astro/config.yaml. Worse, the .gitignore I added in b503f40 ignored .astro/ wholesale, so the file could never be committed and the folder was permanently broken. Root .gitignore now ignores only .astro/config.yaml.lock.

2. The image could not build without packages.txt.

ERROR: failed to build: "/packages.txt": not found

The Astro runtime ONBUILD-copies packages.txt, so it is mandatory even when empty. Added, with a comment saying OS packages go there and Python packages go in requirements.txt.

3. The DAG referenced code the containers could not see.

ingest does from src.pipeline import run and dbt_build reads /usr/local/airflow/include/dbt. Neither existed inside the Astro project. Rather than duplicate the pipeline and dbt project into airflow/ (which drifts), added a docker-compose.override.yml that bind-mounts ../src and ../dbt and sets PYTHONPATH. One copy of each.

Verified with the stack actually running

dag_id                 | fileloc                                 | owners
final_project_pipeline | /usr/local/airflow/dags/pipeline_dag.py | data-team
import errors: No data found

src package importable inside Airflow: /usr/local/airflow/src/pipeline.py
include/dbt mounted: dbt_project.yml  models  profiles.yml  seeds  macros

dbt_build task command:
Done. PASS=18 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=18

Still not verified, and why

The ingest task has not run inside Airflow. Containers in my environment have no outbound DNS, so the fetch fails with Failed to resolve 'www.arbeitnow.com' before it reaches the database. The same code path is verified from the host, where it fetched 175 live records. On a normal machine with working container networking this should pass, but it is worth one astro dev start plus a manual DAG trigger to confirm.

🤖 Generated with Claude Code

[lassebenni]

Everything is now tested, including the ingest task. One more real defect found.

My earlier note said containers had no outbound DNS. That was wrong: the Docker daemon had only just started when I tested it. Retested and DNS resolves fine, so I ran the whole thing properly.

The defect: the DAG was talking to Airflow's own metadata database

With the containers wired onto the data stack's network, the DAG failed:

connection to server at "172.20.0.2", port 5432 failed:
FATAL: database "finalproject" does not exist

The Astro stack ships its own service named postgres. A DAG configured with POSTGRES_HOST=postgres resolves to Airflow's metadata database, not the project's. Silent, and it would have cost a team hours.

Fixed in 4cb4d6a:

• Data service renamed postgres to db, with a comment explaining why so nobody renames it back.
• The data stack now has a fixed network name, finalproject, and the Airflow containers attach to it. Better than host.docker.internal, which breaks on any machine already running something on 5432 (mine does).
• Added airflow/.env.example. The folder shipped no environment file at all, so the DAG had nothing to read.

Full end-to-end run

DagRun Finished: dag_id=final_project_pipeline, state=success, run_duration=8.53s

raw.postings                -> 175
analytics.fct_postings      -> 175
fct_postings.ingested_at    -> 2026-08-03 21:28:21

The run window was 21:28:18 to 21:28:27, so ingested_at at 21:28:21 proves the ingest task fetched live data during that run rather than reusing an earlier load. dbt then built the mart from it: PASS=18 WARN=0 ERROR=0.

Verification status: complete

Aspect	Status
docker build	✅ builds
Pipeline from host	✅ 175 records, 0 rejected
Pipeline in container (docker compose run)	✅ 175 rows written
Idempotency (second run)	✅ still 175 rows, upsert works
dbt build from host	✅ PASS=18 ERROR=0
dbt build inside Airflow	✅ PASS=18 ERROR=0
astro dev start	✅ stack boots
DAG parse	✅ no import errors
Full DAG run, both tasks	✅ state=success

Nothing on the data side is unverified now.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Introduces an end-to-end starter data pipeline spanning ingestion, Postgres storage, dbt transformations, Airflow orchestration, containers, optional cloud tooling, and PR checks.

Changes:

• Adds Python ingestion, validation, storage, and dbt mart layers.
• Adds Docker/Airflow orchestration and optional Azure, Databricks, and Streamlit scaffolding.
• Adds contributor documentation, PR templates, and automated description/size checks.

Reviewed changes

Copilot reviewed 36 out of 38 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
data/src/storage.py	Adds Postgres schema creation and upserts.
data/src/pipeline.py	Wires ingestion, validation, and storage.
data/src/models.py	Defines the posting validation model.
data/src/ingest.py	Fetches and validates API records.
data/src/config.py	Loads environment-based configuration.
data/src/__init__.py	Initializes the Python package.
data/README.md	Documents setup and pipeline usage.
data/pyproject.toml	Defines Python dependencies and tooling.
data/optional/streamlit/app.py	Adds an operational dashboard.
data/optional/README.md	Describes optional modules.
data/optional/databricks/README.md	Documents a Databricks target.
data/optional/bicep/modules/storage.bicep	Defines Azure storage infrastructure.
data/optional/bicep/main.bicep	Composes the Bicep deployment.
data/docs/mart_contract.md	Documents the backend mart contract.
data/Dockerfile	Builds the ingestion image.
data/docker-compose.yml	Runs Postgres and the pipeline locally.
data/dbt/tests/assert_postings_not_empty.sql	Tests for an empty mart.
data/dbt/tests/assert_posted_at_not_in_future.sql	Tests posting timestamps.
data/dbt/profiles.yml	Configures dbt’s Postgres target.
data/dbt/models/staging/stg_postings.sql	Stages raw postings.
data/dbt/models/staging/_stg_postings.yml	Documents and tests staging columns.
data/dbt/models/staging/_sources.yml	Declares the raw source and freshness.
data/dbt/models/marts/fct_postings.sql	Publishes the postings mart.
data/dbt/models/marts/_fct_postings.yml	Defines the mart contract and tests.
data/dbt/dbt_project.yml	Configures dbt model materialization.
data/airflow/requirements.txt	Adds DAG runtime dependencies.
data/airflow/packages.txt	Provides the Astro OS-package manifest.
data/airflow/Dockerfile	Selects the Astro runtime.
data/airflow/docker-compose.override.yml	Connects Astro to code and Postgres.
data/airflow/dags/pipeline_dag.py	Schedules ingestion and dbt builds.
data/airflow/.gitignore	Excludes Astro runtime artifacts.
data/airflow/.env.example	Documents Airflow environment settings.
data/airflow/.astro/config.yaml	Declares the Astro project.
data/.env.example	Provides local configuration defaults.
data/.dockerignore	Excludes development artifacts from images.
.gitignore	Adds data-track generated-file exclusions.
.github/workflows/pr-checks.yml	Enforces PR descriptions and size limits.
.github/pull_request_template.md	Adds the repository PR template.

Suppressed comments (1)

data/src/ingest.py:48

• A malformed non-object item does raise ValidationError, but this error handler then calls .get on that item and aborts the whole batch. That contradicts the function's promise to count and skip malformed records.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[lassebenni]

Superseded by #9, which lands the final state of both branches on current main.

This one no longer merges: main has since gained the Spring backend, the Next.js frontend, the root compose stack and CI, all of which postdate this branch.

#9 carries the six live review comments from #4 as fixes, and notes in its description which comments here are already stale (the naive datetime, the undeclared Streamlit dependencies, the postgres versus db service name, the zero-record success, the top-level-list fallback, and the SQL injection in the old Postgres storage.py, plus two comments on files that no longer exist).

Leaving this open for you to close.