From bfe3d6d76431428aef211e3050cba1427f83c81e Mon Sep 17 00:00:00 2001 From: Hans Lemm Date: Mon, 3 Aug 2026 15:45:47 +0200 Subject: [PATCH] release: 0.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps __version__ and dates the changelog section. The changelog section is rewritten rather than merely retitled. It had accumulated across five development batches and was written as a diff against unreleased intermediate states: bullets said "ADV001 *now* requires…" and "ADV003 *no longer* proposes…", which is meaningless to someone meeting `advise` for the first time in this release, and the whole `### Fixed` section described bugs in code that has never shipped — implying users experienced them. Four of its entries were not fixes at all but features (`--dry-run`, connection resolution, the extras, cursor and COPY unwrapping), and two were internal test infrastructure. So: `Fixed` is gone, with a one-line note saying why, since a first release of a command has no previously-released behaviour to fix. Its features moved to `Added`. The two shipped behaviours worth keeping from it — the `--ddl` line-boundary guarantee and ADV004's coverage disclosure — are stated as properties of what ships rather than as repairs. A `Known limitations` section now carries the thing a user most needs at install time: Redshift's introspection SQL has never run against a live cluster, what *is* verified, and the invitation to check it with `--dry-run`. Two accuracy fixes while there: the reconciliation bullet said "the eight rules" when there are now sixteen across three families, and it described only Postgres's prefix collapse — Redshift's strategy collapse (DISTSTYLE ALL withholding DISTKEY) was undocumented. And the `co_occurring_fingerprints` evidence key, a public `--json` field, was only mentioned in a bullet whose delta framing I dropped, so it is now stated on its own terms. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 153 ++++++++++++++++--------------------- src/sqlquality/__init__.py | 2 +- 2 files changed, 68 insertions(+), 87 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dc35a79..31106c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,10 +5,19 @@ All notable changes to this project are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.3.0] - 2026-08-03 + +`advise` is new in this release. Everything below describes it as it ships; it has no +previously-released behaviour, so there is no "Fixed" section — bugs found and fixed while +building it are in the git history, not here. ### Added +- `sqlquality advise` — reads Postgres query history (`pg_stat_statements`) and catalog + metadata over a read-only connection and proposes indexes, index removals, partial + indexes, sargability fixes and `SELECT *` cleanups (ADV001–ADV008), with a `--json` + and `--markdown` report and a reviewable `--ddl` script. + - `sqlquality advise --engine redshift` — reads Redshift query history (`sys_query_history`) and catalog metadata (`svv_columns`, `svv_table_info`, `svv_alter_table_recommendations`) over a read-only connection and proposes SORTKEY @@ -25,32 +34,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 dbt-enrichment path Postgres's index proposals use. **Redshift's introspection SQL has not been executed against a live cluster** — see the README's `advise` section for what is and is not verified, and run `--dry-run` before pointing this at production. -- `sqlquality advise` — reads Postgres query history (`pg_stat_statements`) and catalog - metadata over a read-only connection and proposes indexes, index removals, partial - indexes, sargability fixes and `SELECT *` cleanups (ADV001–ADV008), with a `--json` - and `--markdown` report and a reviewable `--ddl` script. + - `advise` supports multiple `--schema` flags: every catalog fact (table sizes, NDV, index lists, generated DDL) is keyed by `schema.table`, so same-named tables in different introspected schemas no longer alias into one another. + - ADV007 proposes an index on a hot, unindexed join key; ADV008 proposes a composite index for a hot `GROUP BY`. -- Overlapping proposals are reconciled before the report is written, so the eight rules - cannot contradict each other: two rules reaching identical DDL collapse into one entry, - and a proposed index whose columns are a leading prefix of another proposed index for the - same table collapses into the wider one (creating both would have produced a pair ADV003 - flags as redundant on the next run). The absorbed proposal's rationale and confidence are - folded into the survivor's, attributed by rule code; its `evidence` block is **not** - merged and is discarded. Two proposals covering the same columns in a different order are - both kept, each naming the other. Consequence for `--json` consumers: a rule can fire and - contribute no entry of its own to `proposals`, so counting entries by `code` is not a - count of which rules matched. -- ADV001 now requires the columns of a composite candidate to co-occur in at least one query - group, and reports that joint count as `co_occurring_fingerprints` in place of the former - per-column `fingerprints`. Previously a near-free query could contribute a column to the - middle of an otherwise correct composite, producing an index that no query used and that - could no longer satisfy the hot query's `ORDER BY`. -- ADV003 is scoped to the tables the workload was observed using, like ADV002 — it no longer - proposes `DROP INDEX` for a relation the run never analysed. + +- Overlapping proposals are reconciled before the report is written, so no two rules can + advise contradictory work on one relation. On Postgres: two rules reaching identical DDL + collapse into one entry, and a proposed index whose columns are a leading prefix of another + proposed index for the same table collapses into the wider one (creating both would have + produced a pair ADV003 flags as redundant on the next run). On Redshift, where the conflict + is between whole strategies rather than prefixes, an ADV103 `DISTSTYLE ALL` withholds the + ADV102 `DISTKEY` proposal for the same relation, since `ALL` subsumes it — otherwise an + operator would run one hours-long table rewrite and then a second undoing it. The absorbed + proposal's rationale and confidence are folded into the survivor's, attributed by rule code; + its `evidence` block is **not** merged and is discarded. Two proposals covering the same + columns in a different order are both kept, each naming the other. Consequence for `--json` + consumers: a rule can fire and contribute no entry of its own to `proposals`, so counting + entries by `code` is not a count of which rules matched. + +- A composite index proposal requires its columns to co-occur in at least one query group, and + reports that joint count as `co_occurring_fingerprints` rather than a per-column + `fingerprints` — so the number beside a multi-column proposal is the support for *that + combination*, not the highest support any one column has. ADV001, ADV004 and ADV008 all + report it; the single-candidate rules (ADV005, ADV007) keep a per-column count, which for + them is the whole truth. + - Optional dbt enrichment for `advise`: `--project-dir` (reads `/target/manifest.json`) or `--manifest ` layers dbt model metadata onto the same analysis, and every manifest-free `advise` invocation is proven @@ -86,75 +98,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 dropped from matching (not guessed at) and counted in the CLI disclosure and the JSON payload's `dbt.dropped_collisions`. -### Fixed - -- CI now runs the 23 live-Postgres integration tests, in a job with a `postgres:16` service - container that has `pg_stat_statements` preloaded. Until now they ran only on a contributor's - own machine, while this feature's live suite was repeatedly the only thing that caught a - whole class of bug — a `reltuples = -1` sentinel that suppressed every proposal, redaction - dismembering `$N` placeholders, a `toplevel` filter that produced confidently-wrong advice, - and a workload statement that failed on the wire for every default run. The job fails if the - suite skipped or executed nothing, since every one of those tests skips itself when no server - answers and `pytest` exits 0 on a skip. -- The integration compose file publishes host port 27432 instead of 55432, which collided with - an unrelated container in practice — and because `docker compose up` neither binds nor fails - in that case, the suite silently talked to whatever else was listening. The fixture now also - verifies which server answered (database name, and `pg_stat_statements` in - `shared_preload_libraries`) and fails naming a port collision as the likely cause, rather - than trusting that a successful connection reached the right database. -- `--ddl`'s guarantee that every line of the generated script is either an intended statement - or a `--` comment now holds for all ten codepoints `str.splitlines()` treats as a line - boundary, on both the Postgres and Redshift renderers. The guard tested only `\n` and `\r`, - while everything that splits the text uses `splitlines()`, so an introspected identifier - containing `\v`, `\f`, `\x1c`, `\x1d`, `\x1e`, `\x85`, `U+2028` or `U+2029` — all legal - inside a quoted Postgres identifier — produced a second physical line the guard never - examined, and the tail of the statement was emitted looking like a bare statement of its own. -- ADV004 (partial index) now consults the existing-index list, which it was alone among the - index-creating rules in never doing. A plain index leading with the guarded column now - suppresses the proposal — that index already serves the lookup, and the partial index's only - advantage is a size saving this tool cannot measure against a second index's write cost (and - which ADV003 would never flag as redundant, since its prefix check is restricted to plain - indexes). Where a check genuinely could not run it is now stated rather than skipped: an - unreadable existing-index list caps confidence at LOW and says so, and an existing *partial* - or expression index that leads with the same column is named, since sqlquality does not - compare index predicates and so cannot tell whether the proposal is already applied. New - evidence keys `partial_indexes_not_compared` and `expression_indexes`; deliberately not - ADV001's `partial_indexes_skipped`, which records a different fact. -- dbt enrichment now discloses itself in the terminal on **every** engine. The stderr - disclosure line counted only ADV302's config-block rewrite, which no Redshift proposal can - reach (nothing Redshift emits is a `CREATE INDEX`), so a `--project-dir` run on Redshift - warned in `rationale` and in the `--ddl` note that `dbt run` may undo an hours-long - full-table rewrite while the terminal row stayed byte-identical to a dbt-free run. Any - proposal whose DDL cannot be expressed as dbt config is now counted and reported too, as is - an index *drop* (ADV002, ADV003) on a dbt-managed relation — the case reachable on Postgres, - where a run proposing only drops enriched every one of them and still said nothing, so an - operator applied a drop that the next `dbt run` recreated from the model's `indexes:` config. - Each of the three outcomes is reported as its own clause, because each calls for a different - action: paste a config block, expect a runnable statement not to survive the next rebuild, or - delete a config entry as well as running the drop. -- `IS NOT NULL` predicates were classified as `IS NULL` when sqlglot 30.13 or newer was - installed, because that release moved the negation from a wrapping `Not` node onto a - `negate` flag on the `Is` node itself. Both encodings are now read. This was not cosmetic: - ADV004 turns these roles directly into a partial index's `WHERE` clause, so it proposed - `WHERE col IS NULL` for a workload filtering `WHERE col IS NOT NULL` — an index over exactly - the complement of the intended rows. `uv.lock` pins 30.12, so development and CI never saw - it while any fresh `pip install sqlquality` resolved a newer 30.x and did. CI now also runs - the suite against the highest versions the declared dependency ranges allow. -- `advise` unwraps `DECLARE ... CURSOR FOR` and `COPY (...) TO` reads to their inner - query before filtering, so server-side-cursor and `COPY`-based workloads (what - psycopg2, Django and SQLAlchemy emit for large result sets) reach the analysis - instead of being discarded as maintenance statements. -- Connections resolve from `--dsn`, `SQLQUALITY_DSN`, or a dbt `profiles.yml`, in that - order. dbt is optional throughout. -- `advise --dry-run` prints every introspection statement without connecting. -- Optional extras `sqlquality[postgres]` and `sqlquality[warehouse]`. +- `advise` resolves connections from `--dsn`, the `SQLQUALITY_DSN` environment variable, or a + dbt `profiles.yml`, in that order. dbt is optional throughout: every `advise` invocation + behaves identically without a manifest, which is verified by comparing stdout, JSON, markdown + and the DDL file byte-for-byte. +- `advise --dry-run` prints every introspection statement the chosen engine can issue, and + exits without connecting — so the exact reads can be audited, or handed to a DBA, before any + credential is used. +- `advise` unwraps `DECLARE ... CURSOR FOR` and `COPY (...) TO` reads to their inner query + before filtering, so server-side-cursor and `COPY`-based workloads — what psycopg2, Django + and SQLAlchemy emit for large result sets — reach the analysis instead of being discarded as + maintenance traffic. Note that on Postgres a cursor's cost is attributed to its `FETCH` + statements, so such a read contributes its predicate columns but little of its cost. +- Optional extras `sqlquality[postgres]` and `sqlquality[warehouse]`, both currently psycopg. + Without one, `advise` exits with an install hint rather than a traceback. +- Every line of the `--ddl` script is either a statement `advise` intends or a `--` comment, + including when an introspected identifier contains a character `str.splitlines()` treats as a + line boundary. sqlquality never executes the script. ### Changed -- The "static tool, never connects" claim is now scoped: sqlquality never executes your - SQL, and only `advise` opens a (read-only, metadata-only) connection. +- The "static tool, never connects" claim is now scoped: sqlquality never executes your SQL, + and only `advise` opens a connection — read-only, metadata-only, with a statement timeout. - Query literals are redacted at ingest by default; `--keep-literals` opts back in. +### Known limitations + +- **Redshift's introspection SQL has never been executed against a live cluster.** Its + connection path is verified against a real server (Redshift speaks the PostgreSQL wire + protocol) and every statement is syntax-checked and proven bindable, but the column names and + proposal semantics come from AWS documentation. Run `advise --engine redshift --dry-run` and + check the statements against your cluster before relying on the output — and please report + what you find. +- Snowflake is designed but not implemented; `advise` supports `postgres` and `redshift`. +- `advise`'s remaining engine-specific caveats — cursor cost attribution, `COPY` and PL/pgSQL + double-counting under `pg_stat_statements.track = all`, Redshift's `--limit` semantics, and + what `svv_table_info` absence cannot distinguish — are documented in the README's Limitations + section rather than repeated here. + ## [0.2.0] - 2026-07-20 ### Added diff --git a/src/sqlquality/__init__.py b/src/sqlquality/__init__.py index 521a656..c7c5010 100644 --- a/src/sqlquality/__init__.py +++ b/src/sqlquality/__init__.py @@ -1,3 +1,3 @@ """sqlquality — measure dbt model performance and complexity.""" -__version__ = "0.2.0" +__version__ = "0.3.0"