A local-first data warehouse built around DuckDB. Databox ingests public data with dlt, coordinates concurrent writes through Quack, transforms it with SQLMesh, validates it with Soda, and orchestrates the workflow with Dagster—without always-on infrastructure.
flowchart LR
sources[Public sources] --> dlt[dlt]
dlt -->|writes through Quack| duckdb[(DuckDB)]
duckdb --> sqlmesh[SQLMesh]
soda[Soda] -. validates .-> duckdb
dagster[Dagster] -. orchestrates .-> dlt
dagster -. orchestrates .-> sqlmesh
dagster -. asset checks .-> soda
New dlt sources move through a reviewable, agent-guided modeling workflow:
flowchart LR
schema[dlt schema] --> annotate["annotate-sources<br/>annotations + taxonomy"]
annotate --> ontology[create-ontology]
ontology --> cdm["generate-cdm<br/>Kimball CDM"]
cdm --> transform["create-transformation<br/>SQLMesh models"]
The project skills—annotate-sources,
create-ontology,
generate-cdm, and
create-transformation—turn raw
schemas into business-aware warehouse models before transformation SQL is
written. See the workflow.
The included Rufous bird app is a reference consumer of the warehouse, not the core of the project.
Prerequisites: Python 3.12+, uv, and Task. Node.js 22+ and npm are only needed for Rufous.
git clone https://github.com/Doctacon/databox.git
cd databox
task install # creates .env from .env.example when absent
task ciAfter the initial dependency install, source tests replay recorded responses, so
task ci needs neither provider credentials nor a populated warehouse.
Configure the source credentials in .env. For a new database, bootstrap the
pinned AVONET snapshot once, then refresh the routine sources:
$EDITOR .env
mkdir -p data .dagster
DAGSTER_HOME="$PWD/.dagster" PYTHONPATH="$PWD" \
uv run dg launch --target-path packages/databox --job avonet_ingest
task full-refresh # ingest and transform into data/databox.duckdb
task dagster:dev # inspect assets at http://localhost:3000AVONET is intentionally excluded from routine refreshes. See the operations runbook.