diff --git a/.gitignore b/.gitignore index ad2324ada..6823a7d29 100644 --- a/.gitignore +++ b/.gitignore @@ -153,3 +153,6 @@ Libs/ios/ # Claude Code session data .claude/ + +# Linux dev sysroot (see linux/scripts/dev-env.sh) +.local-deps/ diff --git a/linux/README.md b/linux/README.md index bd8467afd..9f7b5cce7 100644 --- a/linux/README.md +++ b/linux/README.md @@ -4,17 +4,17 @@ Native Linux database client. Sister product to the macOS TablePro app, sharing ## Status -Phase 0 — foundation. The technology stack was validated by a 2-day spike in April 2026: Rust + GTK4 + libadwaita + sqlx + GtkColumnView built and scrolled 100,000 rows with no perceptible lag. Real codebase scaffolding has not started yet. +Phases 2 and 3 in progress (see [ROADMAP.md](ROADMAP.md)). The stack (Rust + GTK4 + libadwaita + Relm4 + sqlx / tiberius) runs as an app you can build and use: PostgreSQL, MySQL, SQLite, and MSSQL drivers, workspace tabs, structure editing, SSH tunnels, and query history. It is not beta-shippable yet. Flatpak / Flathub distribution and the remaining hardening items are open. ## Stack | Layer | Pick | |---|---| | Language | Rust 1.93+ | -| GUI toolkit | GTK4 4.14+ + libadwaita 1.5+ | +| GUI toolkit | GTK4 4.14+ + libadwaita 1.6+ + GtkSourceView 5.12+ | | App architecture | [Relm4](https://relm4.org) — Elm-style components on gtk4-rs | | Async | tokio (DB drivers) bridged to glib main loop (UI) | -| DB drivers | sqlx (PG / MySQL / SQLite), fred (Redis), official mongodb crate, clickhouse-arrow, etc. | +| DB drivers | sqlx (PG / MySQL / SQLite), tiberius (MSSQL); planned: fred (Redis), official `clickhouse` / mongodb / duckdb crates, etc. | | Persistence | libsecret (passwords), gio::Settings (prefs), JSON files (connection metadata) | | Distribution | Flathub primary, .deb / .rpm / AppImage secondary | @@ -33,29 +33,43 @@ System dependencies: ```bash # Ubuntu / Debian -sudo apt install -y build-essential pkg-config libgtk-4-dev libadwaita-1-dev libssl-dev libsecret-1-dev +sudo apt install -y build-essential pkg-config libgtk-4-dev libadwaita-1-dev libgtksourceview-5-dev libssl-dev libsecret-1-dev # Fedora -sudo dnf install -y gcc pkg-config gtk4-devel libadwaita-devel openssl-devel libsecret-devel +sudo dnf install -y gcc pkg-config gtk4-devel libadwaita-devel gtksourceview5-devel openssl-devel libsecret-devel # Arch -sudo pacman -S --needed base-devel pkg-config gtk4 libadwaita openssl libsecret +sudo pacman -S --needed base-devel pkg-config gtk4 libadwaita gtksourceview5 openssl libsecret ``` Verify the right versions are present: ```bash -pkg-config --modversion gtk4 libadwaita-1 # need 4.14+ / 1.5+ -rustc --version # need 1.93+ +pkg-config --modversion gtk4 libadwaita-1 gtksourceview-5 # need 4.14+ / 1.6+ / 5.12+ +rustc --version # need 1.93+ ``` -Build and run (once `crates/app/` exists): +Build and run: ```bash cd linux cargo run -p tablepro-app ``` +Local CI mirror (fmt + clippy + build + unit tests): + +```bash +./scripts/ci-local.sh +``` + +Driver smoke against a Postgres you already run, no Docker needed: + +```bash +./scripts/smoke-postgres.sh +``` + +Optional: if the system `-dev` packages above are missing, extract the package payloads under `../.local-deps/root/` (so headers land in `../.local-deps/root/usr/include`) and `source scripts/dev-env.sh` before cargo. Debian-family layouts only. + ## Documentation index | Topic | File | diff --git a/linux/ROADMAP.md b/linux/ROADMAP.md index 19f972c45..552e1a2e5 100644 --- a/linux/ROADMAP.md +++ b/linux/ROADMAP.md @@ -1,26 +1,26 @@ # Roadmap -## Where we are (2026-04-26) +## Where we are (2026-07-27) -**Phase 0 and Phase 1 are complete in spirit but not in production sense.** +**Phase 0 is complete. Phase 1 is complete. Large parts of Phase 2 and Phase 3 are already in the tree.** -The current build connects to PostgreSQL / MySQL / SQLite, browses tables with a virtualized 100k-row grid, supports in-place cell edit, has a SQL editor, and persists connections to JSON + libsecret. 31 unit tests, CI green, Relm4 architecture clean. +The app connects to PostgreSQL, MySQL, SQLite, and Microsoft SQL Server; browses tables in a virtualized `GtkColumnView`; supports in-place cell edit with transactional save; hosts an `AdwTabView` workspace (editor / table / structure); filters rows; tunnels over SSH via `russh`; stores passwords in the Secret Service; and records query history in SQLite + FTS5. Drivers ship with testcontainers integration tests. Relm4 architecture is intact. -This is **demo-grade**. It is not beta-shippable. The gap between "works on the developer's machine" and "I would install this from Flathub and use it daily" is substantial — see [`docs/production-audit.md`](docs/production-audit.md) for the detailed gap analysis. The big-ticket missing pieces: +This is **past demo-grade**, but still **not beta-shippable**. The gap between "works on the developer's machine" and "I would install this from Flathub and use it daily" remains. See [`docs/production-audit.md`](docs/production-audit.md) for the detailed gap analysis. Remaining big-ticket items: | Concern | Status | |---|---| -| Type system | 6 of ~20 needed types (no Date / Time / Decimal / Uuid / Json) | -| Result scaling | Materializes full result into memory; OOM at 1M+ rows | -| Connection management | One global connection at a time; no multi-tab, no multi-window | -| Network security | TLS UI absent, SSH tunnelling absent | -| Distribution | Flatpak manifest never built; no AppStream metainfo, icons, or screenshots | -| Internationalization | 100% English hardcoded | +| Type system | Done. `Value` covers Null/Bool/Int/Float/Text/Bytes/Date/Time/DateTime/TimestampTz/Decimal/Uuid/Json | +| Result scaling | Streams via sqlx `fetch` with `MAX_QUERY_ROWS` cap; full result still held in the grid model | +| Connection management | `DatabaseService` + `AdwTabView` workspace tabs done; one active connection at a time, multi-window still open | +| Network security | SSH tunnelling + TLS toggle present; cert-path / verify-mode UI still thin | +| Distribution | Flatpak manifest + metainfo + desktop + icon present; never built end-to-end on CI | +| Internationalization | gettext + `tr!` macro + `po/` template; the template has 227 strings against 390 in the app, and `POTFILES.in` is stale | | Accessibility | Untested with Orca / keyboard nav | -| Integration tests | One ignored testcontainers test; zero per-driver matrix | -| Recovery | No reconnect on connection loss; no cancel running query | +| Integration tests | Postgres and MySQL suites run in CI; the MSSQL suite exists but no CI job runs it; SQLite has none | +| Recovery | `connection_monitor` ping + reconnect loop; cancel drops the client future, the server-side query keeps running | -**What "production-ready" means for this project**: a user on Fedora 41 or Ubuntu 24.04 can install from Flathub, connect to their everyday Postgres or MySQL database, browse and edit data correctly across all native types, run SQL queries, see schema, and trust the app to handle errors gracefully. Demo-grade does not meet this bar. +**What "production-ready" means for this project**: a user on Fedora 41 or Ubuntu 24.04 can install from Flathub, connect to their everyday Postgres or MySQL database, browse and edit data correctly across all native types, run SQL queries, see schema, and trust the app to handle errors gracefully. ## Phase legend @@ -32,11 +32,11 @@ Phases are ordered by **maturity**, not feature count. Each phase has a single c **Status**: complete. -- [x] Cargo workspace with `app`, `core`, `storage`, `drivers/{postgres,sqlite,mysql}` +- [x] Cargo workspace with `app`, `core`, `storage`, `ssh`, `drivers/{postgres,sqlite,mysql,mssql}` - [x] `core::DatabaseDriver`, `core::Connection`, `core::DriverRegistry` traits - [x] `storage::connections` (JSON, atomic writes, schema versioning) -- [x] `storage::secrets` (libsecret via `oo7`) -- [x] CI on Ubuntu 24.04 (build, clippy `-D warnings`, fmt check, tests) +- [x] `storage::secrets` (Secret Service via `oo7`) +- [x] CI (`build-linux.yml`: fmt, clippy `-D warnings`, build, unit tests + driver integration) - [x] `rustfmt.toml`, `clippy.toml`, `rust-toolchain.toml` - [x] Flatpak manifest skeleton (not yet validated end-to-end — see Phase 3) - [x] Architecture decision records for stack picks @@ -49,10 +49,10 @@ Exit criterion: a fresh contributor can `cargo run -p tablepro-app` and reach a **Status**: complete. -- [x] Three drivers wired (PostgreSQL, SQLite, MySQL) via `sqlx` +- [x] Drivers wired: PostgreSQL / SQLite / MySQL via `sqlx`, MSSQL via `tiberius` - [x] `AdwNavigationSplitView` shell with header bar + Connect/Open/Edit/Disconnect - [x] Multi-driver Connect dialog with engine picker + per-driver form -- [x] Saved connection list popover with delete + reconnect +- [x] Saved connection list with delete + reconnect - [x] Browse paginated table results in `GtkColumnView` (100k rows scroll smoothly) - [x] Sidebar table search (case-insensitive substring filter) - [x] SQL editor pane (GtkSourceView 5 + Run button) @@ -66,68 +66,72 @@ Exit criterion: a fresh contributor can `cargo run -p tablepro-app` and reach a Exit criterion: a developer can demo the basic flows (connect, browse, edit, query) without crashes on their own machine. **Met.** -This is where the project sits today. The app is interesting but **not production-ready**. - --- -## Phase 2 — Production hardening (4 weeks) +## Phase 2 — Production hardening (in progress) **Goal**: handle real-world data and real-world failure modes correctly. -### Type system expansion (~1.5 weeks) +### Type system expansion ✅ -- [ ] Add to `core::Value`: `Date`, `Time`, `DateTime`, `TimestampTz`, `Decimal`, `Uuid`, `Json` -- [ ] Map driver-specific types: PG `numeric`/`uuid`/`jsonb`/`timestamptz`, MySQL `datetime`/`json`, SQLite `julianday`/text variants -- [ ] Add `chrono` and `rust_decimal` and `serde_json` to workspace -- [ ] Update `RowObject` to handle full `Value` set (already typed as `Vec`, just needs the new variants) -- [ ] `value_to_display_text` / `value_to_edit_text` per type with locale-aware formatting +- [x] Add to `core::Value`: `Date`, `Time`, `DateTime`, `TimestampTz`, `Decimal`, `Uuid`, `Json` +- [x] Map driver-specific types across PG / MySQL / SQLite / MSSQL +- [x] `chrono`, `rust_decimal`, `serde_json` in the workspace +- [x] `RowObject` / display helpers consume the full `Value` set -### Streaming results (~1 week) +### Streaming results (partial) -- [ ] Replace `fetch_rows` materialization with sqlx `fetch` stream -- [ ] Backpressure model: read up to `PAGE_SIZE` rows, hold the stream open for next-page reads -- [ ] Memory-bounded grid: drop rows outside viewport (already virtualized in GTK, but model holds all rows) +- [x] sqlx `fetch` stream into a bounded collector (`MAX_QUERY_ROWS`) +- [ ] Backpressure model: hold the stream open for next-page reads +- [ ] Memory-bounded grid: drop rows outside viewport (GTK virtualizes paint; model still holds all loaded rows) - [ ] Cancellation: drop the stream when user navigates away -### Multi-connection / multi-tab architecture (~1 week) +### Multi-connection / multi-tab architecture (partial) -- [ ] `DatabaseService` actor (worker component) owning `Vec>` keyed by connection id -- [ ] Replace `connection_holder` static singleton with service handle -- [ ] App state: `active_connections: HashMap` -- [ ] `AdwTabView` for multiple open tables / queries within one connection +- [x] `DatabaseService` owning active connection(s) +- [x] `AdwTabView` for multiple open tables / queries within one connection +- [x] Workspace tab persistence (`workspace_state.json`) +- [ ] Switch the active connection without reconnecting (`DatabaseService` has no `set_active`) - [ ] Multi-window: each window holds its own active connection via `gtk::Application::add_window` -### Security baseline (~1 week) +### Security baseline (partial) -- [ ] TLS configuration UI: cert path, verify mode, SNI override -- [ ] SSH tunnelling via `russh` crate (host, port, key path, optional jump host) -- [ ] Read-only mode toggle per connection (blocks UPDATE/DELETE/DROP from UI) -- [ ] Cancel running query (button + Esc shortcut + `Connection::cancel` driver method) -- [ ] Connection lost recovery: detect disconnect, show modal, retry button +- [x] TLS toggle on connect options +- [x] SSH tunnelling via `russh` (host, port, key / password auth) +- [ ] SSH jump host +- [x] Read-only mode toggle per connection +- [x] Cancel running query: button + Esc shortcut +- [ ] `Connection::cancel` driver method, so cancelling stops the server-side query instead of dropping the client future +- [x] Connection lost recovery: ping monitor + reconnect loop - [ ] Statement timeout configurable per connection +- [ ] TLS cert path / verify mode / SNI override UI -### Integration tests (~0.5 weeks) +### Integration tests (partial) -- [ ] Per-driver `tests/integration.rs` using `testcontainers-rs` -- [ ] Each driver: connect, list_tables, fetch_columns (with PK detection), fetch_rows pagination, execute_params CRUD, query -- [ ] Run in CI gated behind `--ignored` to avoid Docker dependency in fast-checks job +- [x] `tests/integration.rs` using `testcontainers-rs` for Postgres, MySQL, MSSQL +- [ ] SQLite suite (the crate has no `tests/` directory) +- [x] Connect, list_tables, fetch_columns (PK detection), pagination, value round-trip, bad SQL +- [x] CI integration job gated behind `--include-ignored`, for Postgres and MySQL +- [ ] Run the MSSQL suite in CI +- [x] `smoke_local` test + `scripts/smoke-postgres.sh` for a Docker-free driver check **Exit criterion**: Connect to a 10M-row Postgres table, scroll, edit a date column, lose network mid-query, see a recoverable error, reconnect via the same UI flow. --- -## Phase 3 — Beta release (4 weeks) +## Phase 3 — Beta release (in progress) **Goal**: shippable to Flathub. A user installs and uses for real work. -### Browse UX (~1 week) +### Browse UX (partial) -- [ ] Where-filter UI: per-column filter row above grid with operators (=, !=, LIKE, NULL, IN) -- [ ] ORDER BY wired to `GtkColumnView` header click → server sort -- [ ] Multi-row select via shift-click + Ctrl-click -- [ ] Bulk delete with confirmation -- [ ] Right-click context menu (copy cell, copy row as INSERT, copy column, set NULL, delete row) -- [ ] Save column widths and order per (connection, table) +- [x] Where-filter UI (`filter_strip`) with per-column operators +- [x] ORDER BY wired to `GtkColumnView` header click → server sort +- [x] Multi-row select via shift-click + Ctrl-click +- [x] Bulk delete with confirmation +- [x] Right-click context menu (copy cell, copy row as INSERT, copy column, set NULL, delete row) +- [x] Save column widths per (connection, table) +- [ ] Save column order per (connection, table) ### Export / import (~1 week) @@ -136,22 +140,25 @@ This is where the project sits today. The app is interesting but **not productio - [ ] Import CSV → table (with column mapping dialog) - [ ] Run SQL file (load + execute via SQL editor) -### Schema browser (~1 week) +### Schema browser (partial) -- [ ] Sidebar tabs: Tables, Views, Indexes, Foreign Keys, Triggers, Functions, Sequences +- [x] Structure tab: columns, indexes, foreign keys (edit + DDL diff) +- [x] Column metadata: type, nullable, default +- [ ] Column comments (`ColumnInfo` has no `comment` field and no driver reads one) +- [ ] Sidebar tabs: Views, Triggers, Functions, Sequences - [ ] Click view → SELECT * FROM view (re-uses browse view) - [ ] Click index → show CREATE INDEX DDL + which columns - [ ] Click FK → highlight columns + jump to referenced table -- [ ] Column metadata: type, nullable, default, comment -### Query history + saved queries (~0.5 weeks) +### Query history + saved queries (partial) -- [ ] SQLite FTS5 store at `$XDG_DATA_HOME/tablepro/history.db` -- [ ] Every executed query recorded with timestamp, duration, success, connection name -- [ ] History pane with full-text search +- [x] SQLite FTS5 store at `$XDG_CONFIG_HOME/tablepro/history.db` +- [x] SQL editor runs recorded with timestamp, duration, success, connection name +- [ ] Record the SQL the app runs outside the editor (Structure tab DDL saves, grid row saves) +- [x] History pane with full-text search - [ ] Saved queries: name + SQL, organized by connection -### Connection management (~0.5 weeks) +### Connection management (partial) - [ ] Connection groups (folders in saved-connections list) - [ ] Color tags per connection @@ -161,18 +168,20 @@ This is where the project sits today. The app is interesting but **not productio ### Distribution scaffolding (~1 week) -- [ ] `com.tablepro.linux.metainfo.xml` (AppStream metadata, screenshots refs, license SPDX) -- [ ] App icon set: 16/32/48/64/128/256/512 PNG + scalable SVG +- [x] `com.tablepro.linux.metainfo.xml` skeleton +- [x] App icon: scalable SVG +- [ ] Icon set: 16/32/48/64/128/256/512 PNG - [ ] 4–5 high-resolution screenshots showing key flows -- [ ] Long description + short description in metainfo -- [ ] ContentRating +- [ ] Long description + short description polish in metainfo +- [x] ContentRating (`oars-1.1` in the metainfo) - [ ] `cargo-sources.json` generation via `flatpak-builder-tools/cargo` - [ ] CI job: `flatpak-builder` builds the manifest end-to-end on each PR - [ ] Submit to Flathub `flathub/flathub` PR → review → first publish ### Observability (~0.5 weeks) -- [ ] Structured JSON logging via `tracing-subscriber` JSON layer (env-toggleable) +- [x] Structured logging via `tracing-subscriber` (env-filter) +- [ ] JSON log layer, env-toggleable (the `json` feature is not enabled, so `.json()` is not compiled in) - [ ] Crash reporter: panic hook captures backtrace, writes to log, optional anonymous upload (with explicit opt-in) - [ ] "Help → Report bug" UI helper that opens the issue tracker pre-filled with sanitized log excerpt @@ -184,11 +193,14 @@ This is where the project sits today. The app is interesting but **not productio **Goal**: shipped beta, accepting external bug reports, ready for first wave of public users. -### Internationalization setup (~1 week) +### Internationalization setup (partial) -- [ ] `gettext` integration via `gettext-rs` or `cargo-i18n` -- [ ] Extract all user-facing strings to `.pot` template -- [ ] Locale detection from `LANG` env +- [x] `gettext` integration via `gettext-rs` +- [x] `tr!` macro + locale bind in `i18n::init` +- [x] `po/` scaffolding: `tablepro.pot`, `POTFILES.in`, `LINGUAS` +- [x] Locale detection: `setlocale(LC_ALL, "")` in `i18n::init` +- [ ] Extract all user-facing strings: 218 of the app's 390 `tr!` strings are missing from the template, and 55 of its 227 entries no longer exist in the source +- [ ] Refresh `POTFILES.in`: it lists 3 files that no longer exist and misses 17 that call `tr!` - [ ] Build pipeline integrates `.po` → `.mo` compilation - [ ] Ship English-only at first; structure ready for translators @@ -206,7 +218,7 @@ This is where the project sits today. The app is interesting but **not productio - [ ] KDE Plasma 6 visual smoke test (Adwaita styling acceptable; do not adopt KDE styling) - [ ] Wayland-specific bug fixes (HiDPI fractional scaling, drag handles) - [ ] X11 fallback works on older distros -- [ ] Manual install + smoke on Fedora 41, Ubuntu 24.04, Arch (latest), Debian 12 +- [ ] Manual install + smoke on Fedora 41, Ubuntu 24.04, Arch (latest), Debian 12/13 ### Distribution variants (~0.5 weeks) @@ -232,7 +244,7 @@ This is where the project sits today. The app is interesting but **not productio ### Additional drivers (parallelizable, ~1 week each) -- [ ] ClickHouse via `clickhouse-arrow` +- [ ] ClickHouse via official `clickhouse` crate (or `clickhouse-arrow`) - [x] MSSQL via `tiberius` - [ ] Oracle via `oracle` crate (ODPI-C) - [ ] Redis via `fred` @@ -256,10 +268,11 @@ This is where the project sits today. The app is interesting but **not productio ### Schema editor (~2 weeks) -- [ ] Create / alter / drop table via UI -- [ ] Add / remove / rename columns -- [ ] Add / remove indexes -- [ ] Add / remove foreign keys +- [x] Create / alter / drop table via Structure tab + DDL materialization +- [x] Add / remove columns +- [ ] Rename columns (`build_rename_column` exists but has no caller; needs a `RenameColumn` op in `diff_to_ops` / `materialize_ops`) +- [x] Add / remove indexes +- [x] Add / remove foreign keys - [ ] Drag-drop column reordering with ALTER TABLE preview ### ER diagram (~3 weeks) @@ -343,11 +356,11 @@ This is a separate undertaking and is not blocking any phase. It moves only afte | Stage | Effort | Calendar | |---|---|---| -| Phase 0 + 1 (current) | done | done | -| Phase 2 — Production hardening | 4 weeks FT | Sprint 1 | -| Phase 3 — Beta release | 4 weeks FT | Sprint 2 | -| Phase 4 — Beta polish | 3 weeks FT | Sprint 3 | -| **Beta on Flathub** | **11 weeks total** | **~3 months from start of Phase 2** | +| Phase 0 + 1 | done | done | +| Phase 2 remainder (streaming backpressure, driver-side query cancel, multi-window, TLS UI polish) | ~2 weeks FT | next | +| Phase 3 remainder — Beta release | ~3 weeks FT | following | +| Phase 4 — Beta polish | 3 weeks FT | after Beta | +| **Beta on Flathub** | **~8 weeks FT from this revision** | **after Phase 4** | | Phase 5 — GA expansion | 6 months FT | rolling | | Phase 6 — Parity ambitions | 12+ months FT | optional | @@ -355,12 +368,12 @@ At 50% effort (part-time), double everything. At 25% effort (side project), 4x. --- -## What changed in this revision - -The previous version of this file had Phase 0 and Phase 1 as 4–6 weeks and 2–3 months respectively, then Phase 2 "Parity push" at 3–4 months and Phase 3 "Polish + distribution" at 1–2 months. Total ~8–12 months. +## What changed in this revision (2026-07-27) -That sequence underestimated production hardening. Adding more drivers ("parity push") before fixing type system, streaming, multi-connection, and security would have shipped a feature-rich-but-fragile app. The new sequence prioritizes hardening first, then ships Beta on a smaller-but-solid surface, then expands. +The previous "Where we are (2026-04-26)" section still described the project as Phase 0/1 demo-grade with a type system of 6 variants, no multi-tab, no SSH, and a single ignored integration test. The tree has moved on: -The `Phase 1 — MVP (2–3 months)` checklist with mostly-checked items in the old roadmap was misleading — it implied "MVP done means shippable," when in fact our MVP is demo-grade and Beta is 11 weeks of focused work away. +- Four drivers (including MSSQL), full `Value` set, `DatabaseService`, workspace tabs, Structure tab + DDL, filters, SSH, read-only mode, query history, connection monitor, gettext scaffolding, and CI integration jobs are all present. +- Phase 2 and Phase 3 checklists were marked to match the code. Unchecked items are the real remaining work. +- Timeline shortened: Beta is roughly 8 weeks of focused work from this baseline, not 11 weeks from a stale Phase 1. -This revision makes that explicit: **Phase 1 is the demo state. Beta is Phases 2 + 3 + 4.** +This revision does not claim Beta readiness. It stops the roadmap from under-selling what already ships. diff --git a/linux/crates/drivers/postgres/tests/smoke_local.rs b/linux/crates/drivers/postgres/tests/smoke_local.rs new file mode 100644 index 000000000..24d174e3c --- /dev/null +++ b/linux/crates/drivers/postgres/tests/smoke_local.rs @@ -0,0 +1,104 @@ +//! Local smoke against an already-running Postgres (no Docker required). +//! +//! Ignored by default so a plain `cargo test` stays green without a database. +//! Run it through `scripts/smoke-postgres.sh`, or directly: +//! +//! ```text +//! cargo test -p tablepro-driver-postgres --test smoke_local -- --include-ignored +//! ``` +//! +//! Env: SMOKE_PG_HOST, SMOKE_PG_PORT, SMOKE_PG_USER, SMOKE_PG_PASS, SMOKE_PG_DB. +//! Defaults match `scripts/smoke-postgres.sh`. `docs/testing.md` has the +//! container one-liner that serves those defaults. +//! +//! The test creates, truncates and drops its own table, so point it at a +//! scratch database. + +use drivers_postgres::PgDriver; +use tablepro_core::{ConnectOptions, DatabaseDriver, Value}; + +const TABLE: &str = "tablepro_smoke_items"; +const DEFAULT_PORT: u16 = 54329; + +fn opts_from_env() -> ConnectOptions { + ConnectOptions { + host: std::env::var("SMOKE_PG_HOST").unwrap_or_else(|_| "127.0.0.1".into()), + port: port_from_env(), + database: std::env::var("SMOKE_PG_DB").unwrap_or_else(|_| "tablepro".into()), + username: std::env::var("SMOKE_PG_USER").unwrap_or_else(|_| "tablepro".into()), + password: secrecy::SecretString::new( + std::env::var("SMOKE_PG_PASS") + .unwrap_or_else(|_| "tablepro".into()) + .into(), + ), + use_tls: false, + } +} + +fn port_from_env() -> u16 { + let Ok(raw) = std::env::var("SMOKE_PG_PORT") else { + return DEFAULT_PORT; + }; + raw.parse() + .unwrap_or_else(|e| panic!("SMOKE_PG_PORT={raw} is not a port number: {e}")) +} + +#[tokio::test] +#[ignore = "requires a local postgres; run via scripts/smoke-postgres.sh"] +async fn connect_browse_and_edit_cell() { + let opts = opts_from_env(); + let conn = PgDriver.connect(opts).await.expect("connect to smoke postgres"); + + conn.execute(&format!( + "CREATE TABLE IF NOT EXISTS {TABLE} ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL, + qty INT DEFAULT 0 + )" + )) + .await + .expect("create table"); + + conn.execute(&format!("DELETE FROM {TABLE}")).await.expect("clear"); + conn.execute(&format!( + "INSERT INTO {TABLE} (name, qty) VALUES ('alpha', 1), ('beta', 2)" + )) + .await + .expect("seed"); + + let tables = conn.list_tables().await.expect("list_tables"); + assert!( + tables.iter().any(|t| t.name == TABLE), + "{TABLE} missing from {:?}", + tables.iter().map(|t| &t.name).collect::>() + ); + + let cols = conn.fetch_columns(Some("public"), TABLE).await.expect("fetch_columns"); + assert!(cols.iter().any(|c| c.name == "id" && c.primary_key)); + + let before = conn + .fetch_rows(Some("public"), TABLE, 0, 100) + .await + .expect("fetch_rows"); + assert_eq!(before.rows.len(), 2); + + let updated = conn + .execute_params( + &format!("UPDATE {TABLE} SET qty = $1 WHERE name = $2"), + &[Value::Int(99), Value::Text("alpha".into())], + ) + .await + .expect("edit cell"); + assert_eq!(updated.rows_affected, 1); + + let after = conn + .query(&format!("SELECT qty FROM {TABLE} WHERE name = 'alpha'")) + .await + .expect("verify"); + assert_eq!(after.rows.len(), 1); + assert_eq!(after.rows[0][0], Value::Int(99)); + + conn.execute(&format!("DROP TABLE {TABLE}")).await.expect("drop table"); + + conn.close().await.expect("close"); +} diff --git a/linux/docs/testing.md b/linux/docs/testing.md index 2a776bd4d..38f1289b8 100644 --- a/linux/docs/testing.md +++ b/linux/docs/testing.md @@ -9,6 +9,8 @@ Three layers, three tools. Each crate's test policy follows from its position in | `drivers/` | Real engines | Unit tests + `testcontainers-rs` integration tests | Yes | | `app` | GTK4 + Relm4 components | Limited; pure logic in `services/` is unit-tested | No | +Two helper scripts sit on top: `scripts/ci-local.sh` runs the fast CI checks, `scripts/smoke-postgres.sh` runs the driver smoke against a Postgres you already have. + ## Unit tests In-crate, in `#[cfg(test)] mod tests` next to the code they cover. Standard Rust idiom. @@ -30,9 +32,11 @@ mod tests { Run all unit tests: ```bash -cargo test --workspace --lib +cargo test --workspace --lib --bins ``` +`--bins` is not optional: `tablepro-app` has no `lib.rs`, so `--lib` alone skips every test in the app crate. `scripts/ci-local.sh` runs this command; the CI workflow still passes `--lib` only. + ## Integration tests Per-crate `tests/` directory. One file per scenario. @@ -42,19 +46,18 @@ For `storage`, integration tests use `tempfile::TempDir` to run against an isola For drivers, integration tests use [`testcontainers`](https://docs.rs/testcontainers/latest/testcontainers/) to spin up a real database. The pattern is identical for every driver: ```rust -use testcontainers::{clients::Cli, images::generic::GenericImage}; +use testcontainers::ImageExt; +use testcontainers_modules::postgres::Postgres; +use testcontainers_modules::testcontainers::runners::AsyncRunner; #[tokio::test] +#[ignore = "requires docker"] async fn list_tables_returns_seeded_tables() { - let docker = Cli::default(); - let image = GenericImage::new("postgres", "16") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_exposed_port(5432); - let node = docker.run(image); - let port = node.get_host_port_ipv4(5432); + let container = Postgres::default().with_tag("16-alpine").start().await.unwrap(); + let host = container.get_host().await.unwrap().to_string(); + let port = container.get_host_port_ipv4(5432).await.unwrap(); - let driver = PgDriver; - let conn = driver.connect(opts_for(port)).await.unwrap(); + let conn = PgDriver.connect(opts_for(host, port)).await.unwrap(); conn.execute("CREATE TABLE foo (id INT)").await.unwrap(); let tables = conn.list_tables().await.unwrap(); @@ -62,7 +65,41 @@ async fn list_tables_returns_seeded_tables() { } ``` -Integration tests run in CI. Locally they require Docker and a user that can reach the daemon (see the macOS `CLAUDE.md` notes about Docker group permissions). +Keep the container alive for the whole test: dropping the handle stops it. + +Integration tests run in CI. Locally they require a Docker-compatible API socket. + +### Docker or Podman + +Upstream CI uses Docker. Fedora ships Podman instead, and on Debian it is the easier install; either way, point testcontainers at Podman's rootless socket: + +```bash +sudo dnf install -y podman # or: sudo apt install -y podman +systemctl --user enable --now podman.socket +export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock +cargo test --test integration -p tablepro-driver-postgres -- --include-ignored --test-threads=1 +cargo test --test integration -p tablepro-driver-mysql -- --include-ignored --test-threads=1 +``` + +Do not bother with `TESTCONTAINERS_RYUK_DISABLED`. That is a testcontainers-java / go setting; the Rust crate has no Ryuk reaper and stops each container when its handle drops. + +`curl --unix-socket "${DOCKER_HOST#unix://}" http://localhost/_ping` should print `OK` before you run the suite. `--unix-socket` takes a filesystem path, so the `unix://` prefix has to come off. + +A test that panics hard can still leave a container behind. `podman container prune` clears them. + +### Local smoke without a container + +`crates/drivers/postgres/tests/smoke_local.rs` runs connect, list tables, fetch rows, edit a cell against a Postgres that is already up. It is `#[ignore]`d like the container suites, so it never runs during a plain `cargo test`. + +```bash +podman run -d --name tablepro-smoke -p 54329:5432 \ + -e POSTGRES_USER=tablepro -e POSTGRES_PASSWORD=tablepro -e POSTGRES_DB=tablepro \ + docker.io/library/postgres:16-alpine + +./scripts/smoke-postgres.sh +``` + +Point it somewhere else with `SMOKE_PG_HOST`, `SMOKE_PG_PORT`, `SMOKE_PG_USER`, `SMOKE_PG_PASS`, `SMOKE_PG_DB`. The test creates, clears and drops `tablepro_smoke_items`, so use a scratch database. Mark slow integration tests with `#[ignore]` if they take more than ~5 seconds: @@ -92,16 +129,16 @@ If a UI bug ships and a regression test would have caught it, write the test the ## End-to-end smoke test -`crates/app/tests/smoke.rs` runs at the top of the CI pipeline. It launches the app under `xvfb-run`, sends a fixed sequence of D-Bus actions through `gtk::Application`'s registered actions, and asserts that the app reaches "connected to PostgreSQL, table list visible" without panicking. +There is no app-level end-to-end test yet. Driving the GTK app under `xvfb-run` through its registered `gtk::Application` actions is the intended shape when we add one. -The smoke test is not exhaustive. Its job is to fail loudly when something fundamental is broken before the slower per-driver integration tests run. +What exists today is the driver-level smoke described above: `scripts/smoke-postgres.sh` against a Postgres you already run. ## CI -GitHub Actions, Linux runner, two jobs: +GitHub Actions (`.github/workflows/build-linux.yml`), Ubuntu runner, two jobs: -1. **Fast** (~2 min) — `cargo fmt --check`, `cargo clippy --all -- -D warnings`, `cargo build --workspace`, `cargo test --workspace --lib`. -2. **Full** (~10 min) — runs after fast passes. Boots Docker, runs all integration tests including the smoke test. +1. **Fast checks**: `cargo fmt --all -- --check`, `cargo clippy --all-targets -- -D warnings`, `cargo build --workspace`, `cargo test --workspace --lib`. Runs in an `ubuntu:25.10` container, which ships the glib version libadwaita 1.6 needs. `scripts/ci-local.sh` runs the same steps, but with `--lib --bins` so the app crate's tests actually run. The workflow should pick up `--bins` too. +2. **Driver integration tests**: runs after fast checks pass. Boots Docker on the host runner and runs the Postgres and MySQL suites with `--include-ignored`. The MSSQL suite exists but is not wired in yet. PRs only merge when both jobs are green. diff --git a/linux/scripts/ci-local.sh b/linux/scripts/ci-local.sh new file mode 100755 index 000000000..a87115a44 --- /dev/null +++ b/linux/scripts/ci-local.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Mirror .github/workflows/build-linux.yml "Fast checks" job locally. +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$ROOT" + +if [[ -f "$ROOT/scripts/dev-env.sh" ]]; then + # shellcheck source=/dev/null + source "$ROOT/scripts/dev-env.sh" +fi + +echo "==> cargo fmt --check" +cargo fmt --all -- --check + +echo "==> cargo clippy" +cargo clippy --all-targets -- -D warnings + +echo "==> cargo build --workspace" +cargo build --workspace + +echo "==> cargo test --workspace --lib --bins" +# --bins matters: tablepro-app has no lib.rs, so --lib alone skips its tests. +cargo test --workspace --lib --bins + +echo "All fast checks passed." +echo "Driver integration tests are a separate CI job; see docs/testing.md to run them locally." diff --git a/linux/scripts/dev-env.sh b/linux/scripts/dev-env.sh new file mode 100644 index 000000000..e23c31d4c --- /dev/null +++ b/linux/scripts/dev-env.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Source this when system -dev packages for gtksourceview5/libsecret are unavailable. +# Extract the package payloads under /.local-deps/root first, then: +# source scripts/dev-env.sh +# Every variable below is namespaced and unset again so sourcing does not +# clobber the caller's ROOT or leave state behind. +# ${BASH_SOURCE[0]} is empty under zsh, where $0 holds the sourced path instead. +_dev_env_self="${BASH_SOURCE[0]:-$0}" +_dev_env_root="$(cd "$(dirname "$_dev_env_self")/../.." && pwd)" +_dev_env_deps="$_dev_env_root/.local-deps/root" +if [[ -d "$_dev_env_deps" ]]; then + if command -v dpkg-architecture >/dev/null 2>&1; then + _dev_env_multiarch="$(dpkg-architecture -qDEB_HOST_MULTIARCH)" + else + _dev_env_multiarch="$(uname -m)-linux-gnu" + fi + _dev_env_lib="$_dev_env_deps/usr/lib/$_dev_env_multiarch" + export PKG_CONFIG_PATH="$_dev_env_lib/pkgconfig:$_dev_env_deps/usr/share/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}" + export LD_LIBRARY_PATH="$_dev_env_lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + export LIBRARY_PATH="$_dev_env_lib${LIBRARY_PATH:+:$LIBRARY_PATH}" + export CPATH="$_dev_env_deps/usr/include${CPATH:+:$CPATH}" + unset _dev_env_multiarch _dev_env_lib +fi +unset _dev_env_root _dev_env_deps diff --git a/linux/scripts/smoke-postgres.sh b/linux/scripts/smoke-postgres.sh new file mode 100755 index 000000000..07372a51b --- /dev/null +++ b/linux/scripts/smoke-postgres.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Driver-level smoke: connect, list tables, fetch rows, edit a cell. +# Needs a Postgres already listening on SMOKE_PG_HOST:SMOKE_PG_PORT. +# docs/testing.md has a one-liner that starts one. +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$ROOT" + +SMOKE_HOST="${SMOKE_PG_HOST:-127.0.0.1}" +SMOKE_PORT="${SMOKE_PG_PORT:-54329}" +SMOKE_USER="${SMOKE_PG_USER:-tablepro}" +SMOKE_PASS="${SMOKE_PG_PASS:-tablepro}" +SMOKE_DB="${SMOKE_PG_DB:-tablepro}" + +if [[ -f "$ROOT/scripts/dev-env.sh" ]]; then + # shellcheck source=/dev/null + source "$ROOT/scripts/dev-env.sh" +fi + +export SMOKE_PG_HOST="$SMOKE_HOST" +export SMOKE_PG_PORT="$SMOKE_PORT" +export SMOKE_PG_USER="$SMOKE_USER" +export SMOKE_PG_PASS="$SMOKE_PASS" +export SMOKE_PG_DB="$SMOKE_DB" + +echo "Smoke against postgres://${SMOKE_USER}@${SMOKE_HOST}:${SMOKE_PORT}/${SMOKE_DB}" +cargo test -p tablepro-driver-postgres --test smoke_local -- --include-ignored --nocapture +echo "Smoke passed."