diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bbcc7a..81acd25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,8 +11,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `sqlquality advise` — reads Postgres query history (`pg_stat_statements`) and catalog metadata over a read-only connection and proposes indexes, index removals, partial - indexes, sargability fixes and `SELECT *` cleanups (ADV001–ADV006), with a `--json` + indexes, sargability fixes and `SELECT *` cleanups (ADV001–ADV008), with a `--json` and `--markdown` report and a reviewable `--ddl` script. +- `advise` supports multiple `--schema` flags: every catalog fact (table sizes, NDV, + index lists, generated DDL) is keyed by `schema.table`, so same-named tables in + different introspected schemas no longer alias into one another. +- ADV007 proposes an index on a hot, unindexed join key; ADV008 proposes a composite + index for a hot `GROUP BY`. +- Overlapping proposals are reconciled before the report is written, so the eight rules + cannot contradict each other: two rules reaching identical DDL collapse into one entry, + and a proposed index whose columns are a leading prefix of another proposed index for the + same table collapses into the wider one (creating both would have produced a pair ADV003 + flags as redundant on the next run). The absorbed proposal's rationale and confidence are + folded into the survivor's, attributed by rule code; its `evidence` block is **not** + merged and is discarded. Two proposals covering the same columns in a different order are + both kept, each naming the other. Consequence for `--json` consumers: a rule can fire and + contribute no entry of its own to `proposals`, so counting entries by `code` is not a + count of which rules matched. +- ADV001 now requires the columns of a composite candidate to co-occur in at least one query + group, and reports that joint count as `co_occurring_fingerprints` in place of the former + per-column `fingerprints`. Previously a near-free query could contribute a column to the + middle of an otherwise correct composite, producing an index that no query used and that + could no longer satisfy the hot query's `ORDER BY`. +- ADV003 is scoped to the tables the workload was observed using, like ADV002 — it no longer + proposes `DROP INDEX` for a relation the run never analysed. +- `advise` unwraps `DECLARE ... CURSOR FOR` and `COPY (...) TO` reads to their inner + query before filtering, so server-side-cursor and `COPY`-based workloads (what + psycopg2, Django and SQLAlchemy emit for large result sets) reach the analysis + instead of being discarded as maintenance statements. - Connections resolve from `--dsn`, `SQLQUALITY_DSN`, or a dbt `profiles.yml`, in that order. dbt is optional throughout. - `advise --dry-run` prints every introspection statement without connecting. diff --git a/README.md b/README.md index 9e26d80..1e1c59c 100644 --- a/README.md +++ b/README.md @@ -312,10 +312,10 @@ missing driver degrades with an install hint instead of a traceback. | `--profile` | — | dbt profile name, read from `profiles.yml`. | | `--target` | — | dbt target within the profile. | | `--profiles-dir` | `~/.dbt` | Directory holding `profiles.yml`. | -| `--schema` | `public` | Schema to introspect. **One at a time** — passing two exits 2, see Limitations. | +| `--schema` | `public` | Schema to introspect. Repeat for several: `--schema public --schema sales`. See Limitations for the ambiguity caveat. | | `--since` | — | Window, e.g. `7d`. **Not honored on Postgres** — see Prerequisites below. | | `--limit` | `500` | Max query-history rows to read. | -| `--min-cost-share` | `0.01` | Suppress proposals below this share of workload cost. Applies to the **cost-weighted** rules (ADV001, ADV004, ADV005, ADV006); the index-hygiene rules **ADV002 and ADV003 carry no cost evidence and are always reported**, whatever the threshold. | +| `--min-cost-share` | `0.01` | Suppress proposals below this share of workload cost. Applies to the **cost-weighted** rules (ADV001, ADV004, ADV005, ADV006, ADV007, ADV008); the index-hygiene rules **ADV002 and ADV003 carry no cost evidence and are always reported**, whatever the threshold. | | `--keep-literals` | off | Do **not** redact literal values from query text. | | `--timeout` | `30` | Statement timeout in seconds (rejected outside 1–3600). | | `--dry-run` | off | Print every statement the adapter would issue, then exit 0 **without connecting**. | @@ -378,12 +378,14 @@ statement is not valid SQL to copy out and run. | Code | Proposal | Evidence | |---|---|---| -| ADV001 | Composite index candidate: hot equality columns, then one range/sort column, arity ≤ 3 | cost share, NDV, row estimate, absence of a covering index | +| ADV001 | Composite index candidate: hot equality columns, then one range/sort column, arity ≤ 3, and only columns some single query group filters on *together* | cost share, NDV, row estimate, joint co-occurring fingerprint count, absence of a covering index | | ADV002 | Drop an index with zero recorded scans (excludes unique/primary-key indexes) | scans since last stats reset, size | | ADV003 | Drop an index whose column list is a strict prefix of a wider index | both column lists | | ADV004 | Partial index: a hot equality column guarded by a hot, co-occurring `IS [NOT] NULL` check | cost share, co-occurring fingerprint count | | ADV005 | Non-sargable predicate — a cast/function on a column, or a leading-wildcard `LIKE` | cost share | | ADV006 | Hot `SELECT *` on a wide table (≥15 columns) | cost share, column count | +| ADV007 | Add index on a hot join key with no existing index leading with it | cost share, NDV, row estimate, absence of a covering index | +| ADV008 | Composite index for a hot `GROUP BY`, column order inferred from cost, capped at MEDIUM | cost share, row estimate, absence of a covering index | **Confidence model**, mechanical rather than judgment-based: @@ -392,6 +394,11 @@ statement is not valid SQL to copy out and run. - **MEDIUM** — cost evidence is solid but a catalog input is missing or stale. ADV002 is capped at MEDIUM unconditionally: `idx_scan` only accumulates since the last statistics reset, so zero scans can never prove an index is unused across a full business cycle. + ADV008 is also capped at MEDIUM unconditionally, for a different reason: whether Postgres + uses the index for grouping depends on its choice between `GroupAggregate` and + `HashAggregate`, a planner decision driven by `work_mem` and group cardinality that no + catalog view exposes — HIGH would claim to know the planner's choice, not the catalog's + state, so ADV008 never reaches it. - **LOW** — thin evidence, and specifically **any check that could not be run**: the row count is unknown so the small-table floor could not be applied, or the existing-index list was denied so "no index already covers this" could not be confirmed. Absent @@ -400,6 +407,32 @@ statement is not valid SQL to copy out and run. Every proposal's evidence renders inline (cost share, calls, fingerprints, row estimate, NDV, existing index state) so it can be judged from the report alone. +**How overlapping proposals are reconciled.** The rules above are evaluated independently, +but their output is not shipped independently: two of them can reach the same index from +different evidence, and following both would mean creating a redundant pair that ADV003 then +advises dropping on the next run. So before anything is reported: + +- **Identical DDL collapses to one proposal.** The higher confidence wins; on a tie a fixed + rule preference decides, never list order. +- **A narrower index collapses into a wider one.** If one proposal's columns are a leading + prefix of another's for the same table, only the wider survives — it serves every lookup + the narrower would. Partial (`WHERE`) proposals never participate: a partial index is a + different object even when its column list is a prefix. +- **Same columns in a different order are both kept**, each disclosing the other. `(status, + region)` and `(region, status)` serve different probes, so neither is redundant — but + creating both means indexing the same columns twice, and the report says so. + +Two consequences worth knowing before you consume the output: + +- **A rule can fire and still contribute no proposal.** A `--json` consumer counting + `ADV007` entries can legitimately see zero on a run where ADV007 did propose something + that was absorbed. The absorbed proposal's rule code, confidence and rationale appear in + the surviving proposal's `rationale`, attributed — that is where to look for it. +- **The absorbed proposal's `evidence` is discarded, not merged.** Its rationale (the + constraint an operator needs) is preserved verbatim; its numbers (`leading_ndv`, + `partial_indexes_skipped`, `co_occurring_fingerprints`) are not carried into the + survivor's evidence block. + `--ddl` writes a standalone, commented script — never executed by sqlquality: ```sql @@ -430,6 +463,7 @@ $ sqlquality advise --dsn postgresql://readonly@db.internal/analytics --json { "analyzed": { "query_groups": 3, + "query_groups_in_window": 3, "tables": [ "orders" ], @@ -444,11 +478,11 @@ $ sqlquality advise --dsn postgresql://readonly@db.internal/analytics --json "ddl": "CREATE INDEX ON \"orders\" (\"status\");", "evidence": { "calls": 15000, + "co_occurring_fingerprints": 1, "columns": [ "status" ], "cost_share": 0.6702702702702703, - "fingerprints": 1, "leading_ndv": 500.0, "roles": [ "equality" @@ -465,7 +499,8 @@ $ sqlquality advise --dsn postgresql://readonly@db.internal/analytics --json "skipped": { "noise": 0, "unparseable": 0, - "unqualifiable": 0 + "unqualifiable": 0, + "ambiguous": 0 }, "window": "since stats reset at 2026-07-19 03:00:00+00" } @@ -475,11 +510,12 @@ $ sqlquality advise --dsn postgresql://readonly@db.internal/analytics --json JSON paths all print how many query groups were actually understood: ```console -analyzed 2 of 3 query group(s); skipped 0 unparseable, 0 filtered, 1 unresolvable +analyzed 2 of 3 query group(s); skipped 0 unparseable, 0 filtered, 1 unresolvable, 0 ambiguous low coverage: 33% of candidate statements could not be analyzed (0 unparseable, 1 -unresolvable against the schema). Cost shares are computed against the whole window, so -they are diluted and --min-cost-share is effectively stricter — few or no proposals may -reflect coverage rather than a healthy workload. +unresolvable against the schema, 0 ambiguous across the introspected schemas). Cost shares +are computed against the whole window, so they are diluted and --min-cost-share is +effectively stricter — few or no proposals may reflect coverage rather than a healthy +workload. ``` This matters because `cost_share` is **not** a partition of the workload: a query @@ -489,6 +525,16 @@ denominator always includes queries that could not be parsed or resolved against schema. Poor coverage silently dilutes every proposal's share rather than inflating it — read the skip counts alongside every proposal. +Running with more than one `--schema`, a bare, unqualified table name held by two or more +of the introspected schemas is genuinely ambiguous — attributing it to either would be a +guess, so it is dropped and counted rather than guessed at: + +```console +2 statement(s) named a table held by more than one of the introspected schemas without +qualifying it, so they could not be attributed and were dropped. Qualify the table in the +query, or run advise once per --schema. +``` + A missing grant degrades one capability at a time rather than aborting the whole run: ```console @@ -768,40 +814,89 @@ LLM suggestions unavailable: The 'anthropic' package is required for AnthropicPr statements that could not be parsed or resolved against the schema, so poor coverage dilutes every share and makes `--min-cost-share` effectively stricter; the CLI warns when coverage is poor, and the report always prints the skip counts. -- **Join keys and grouping columns are measured and then ignored.** `advise` classifies - eight column roles and cost-weights all of them, but only five are read by the proposal - rules. A hot unindexed foreign-key join produces `orders.customer_id join cost_share - 1.0` and **no proposal at all** — arguably the most valuable index recommendation in a - relational workload, measured and discarded. Same for `GROUP BY` columns. Compounding - it: any column under a `JOIN` is classified as a join key, so a predicate you placed in - an `ON` clause (as `LEFT JOIN` semantics require) is not treated as a filter and drops - out of ADV001's reach too. Proposing on join keys is a follow-up, not a bug fix. -- **`DECLARE` and `COPY` statements are discarded whole.** The noise filter matches on the - leading keyword, so `DECLARE cur CURSOR FOR SELECT ... WHERE ...` and - `COPY (SELECT ... WHERE ...) TO STDOUT` are dropped along with the session-control and - DDL traffic the filter is for — predicates and all. Django's `QuerySet.iterator()` and - every psycopg2 server-side cursor emit the first form, so on a Django codebase this can - be most of your hot reads. They are counted, and the skip line calls them `filtered` - rather than pretending they were introspection or DDL, but they are not analyzed. - Unwrapping to the inner `SELECT` is a follow-up. +- **Join keys and grouping columns are measured and read, not just cost-weighted.** + `advise` classifies eight column roles; join keys are read by ADV007 (a hot unindexed + foreign-key join produces a proposal, not just an `orders.customer_id join cost_share + 1.0` line that goes nowhere) and `GROUP BY` columns are read by ADV008, as one composite + index rather than one per column — `GROUP BY a, b` needs input sorted by `(a, b)`, and + two single-column indexes cannot provide that. Any column under a `JOIN` is classified + as a join key, so a predicate you placed in an `ON` clause (as `LEFT JOIN` semantics + require) is not treated as a filter and drops out of ADV001's reach — it is ADV007's + candidate instead. ADV008's column order within the composite is inferred from cost, + not read from the query, because redaction does not preserve each column's position in + the `GROUP BY` clause — check it against the actual grouping before applying. +- **A declared cursor's predicates are analyzed, but its cost usually reads as zero.** + `DECLARE cur CURSOR FOR SELECT ... WHERE ...` and `COPY (SELECT ... WHERE ...) TO STDOUT` + are unwrapped to their inner query before the noise filter runs, so both reach + `aggregate` — Django's `QuerySet.iterator()` and every psycopg2 server-side cursor emit + the first form, so on a Django codebase this can be most of your hot reads. `COPY (...) + TO` attributes correctly: Postgres charges the whole execution's time and rows to the + `COPY` statement. A `DECLARE`, measured on PostgreSQL 16, does not — opening a cursor + does no scanning, so while it is counted accurately by *call count* (one call per cursor + opened), its time and rows read as near-zero; the actual work is charged to the `FETCH` + statements that follow, which carry no query text and stay filtered as noise. So a + cursor read's columns can still join an index candidate, but the read cannot earn a + proposal on cost alone, and the default `--min-cost-share` can suppress it outright. +- **A `COPY (...) TO` execution can be counted twice under `pg_stat_statements.track = + all`.** That setting (not the default `track = top`) makes Postgres record both the + verbatim top-level `COPY` statement and its normalised nested query as separate rows for + the same execution, and `unwrap`/redaction give the pair different fingerprints (a real + literal in one, `$1` in the other) — so it lands in `aggregate` as two query groups at + roughly twice the execution's true cost, inflating both that group's `cost_share` and the + whole-window denominator. This is accepted rather than fixed, **for a stated price rather + than for want of a way to fix it**. A blanket `AND s.toplevel` is not the answer: + `toplevel = false` is also the *only* way Postgres ever exposes the SQL executed inside a + PL/pgSQL function body, and tried live it made a genuinely hot, function-wrapped query + disappear from evidence entirely (no predicates, no cost, no disclosure that anything was + dropped) while a much colder query took its place as a `high`-confidence proposal — + confidently wrong, which is strictly worse than an inflated `cost_share`. A *narrow* + predicate does exist, though, and was measured to work: on PostgreSQL 16 the COPY's nested + row keeps its wrapper (`COPY (SELECT ... $1) TO STDOUT`) while a function body is recorded + bare, so `NOT (s.toplevel = false AND s.query ~* '^\s*COPY\s*\(')` removes exactly the + duplicate and leaves function bodies alone. It is declined because *naming* `s.toplevel` + at all requires PostgreSQL 14 — the column does not exist on 13 — so adding it would cost + every PostgreSQL 13 user the entire workload capability in exchange for removing a 2× + over-count of one statement form under a non-default setting. If the supported floor ever + rises to 14, that is the predicate to add. +- **Every PL/pgSQL function call is counted twice under `pg_stat_statements.track = all`, + and no predicate can fix it.** That setting records both the calling statement and each + statement inside the function body: on one PostgreSQL 16 run, a single execution of + `SELECT lc.hot()` appeared as the call at 68.21 ms *and* its body at 67.67 ms — the two + durations track each other, so the absolute figures vary per machine. Both land in + the whole-window denominator, so on a function-heavy workload every `cost_share` is + roughly halved and `--min-cost-share` is correspondingly stricter than it looks. Unlike + the `COPY` case above there is no filter that helps: the call carries the cost while the + body carries the predicates a proposal is built from, so dropping either row loses + something real. On the default `track = top` neither this nor the `COPY` duplicate arises, + because Postgres records no nested statements at all — if you run `track = all`, read + `cost_share` as a lower bound. - **Expression indexes are read but not matched.** `advise` now sees that an index on `lower(status)` exists and names it in the proposal's evidence, but it cannot tell whether that index already serves a lookup on `status` — so it proposes and says so, rather than - suppressing or ignoring. Confirm before applying. + suppressing or ignoring. Confirm before applying. True of all three index-creating rules, + ADV001, ADV007 and ADV008. - **ADV003 only compares plain indexes.** A pair where either index carries a `WHERE` predicate or an indexed expression is skipped entirely rather than proposed at lower confidence: a partial index exists to serve a subset, so recommending its removal is likely wrong rather than merely uncertain. Plain pairs are reported at HIGH. +- **Both `DROP INDEX` rules only look at tables the workload actually used.** ADV002 and + ADV003 iterate the relations that appear in the analyzed query groups, so an index on a + table no observed statement touched is never proposed for removal — including when it sits + in a second `--schema` whose table happens to share a bare name with a hot one. - **A partial index does not suppress a proposal.** `idx ON orders(status) WHERE shipped_at IS NULL` does not serve `WHERE status = $1`, so it is not treated as covering - a candidate index — it is named in the evidence instead. -- **One schema per run.** Every catalog fact is keyed on the bare relation name — table - sizes, NDV statistics, index lists and the `qualify()` schema all merge across schemas — - so `orders` in two schemas would alias into one another and the last catalog row read - would silently win the row estimate. Rather than report that quietly, `advise` rejects - more than one `--schema` with exit 2. Run it once per schema. Generated DDL is qualified - with the schema you passed, so it does not depend on the applying session's - `search_path`. + a candidate index — it is named in the evidence instead. True of all three index-creating + rules, ADV001, ADV007 and ADV008. +- **Multiple `--schema` values are supported, with one honest caveat.** Every catalog fact + (table sizes, NDV statistics, index lists, the `qualify()` schema) is keyed by + `schema.table`, so `orders` in two introspected schemas no longer aliases into one + another. What remains is genuine ambiguity in the *query text* itself: a statement that + says `from orders` bare, when two of the introspected schemas both hold `orders`, cannot + be attributed to either without guessing — it is dropped and counted rather than guessed + at (see `ambiguous` in the skip counts, and the coverage warning that names the remedy). + Qualify the table in the query, or run `advise` once per `--schema`, to recover it. + Generated DDL is qualified with the schema it was read from, so it does not depend on the + applying session's `search_path`. - **Redshift, Snowflake and dbt enrichment are designed but not implemented.** `advise` supports Postgres only today; passing another `--engine` fails with a clear error rather than silently degrading. diff --git a/docs/superpowers/plans/2026-07-27-advise-batch-2.md b/docs/superpowers/plans/2026-07-27-advise-batch-2.md new file mode 100644 index 0000000..9517155 --- /dev/null +++ b/docs/superpowers/plans/2026-07-27-advise-batch-2.md @@ -0,0 +1,2392 @@ +# Advise Batch 2 Implementation Plan — schema-qualified keying, join/group rules, wrapper unwrapping + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `sqlquality advise` correct across multiple schemas, consume the two column +roles it already collects and throws away (JOIN, GROUP), and stop discarding the ordinary +reads that arrive wrapped in `DECLARE ... CURSOR FOR` or `COPY (...) TO`. + +**Architecture:** Three phases in dependency order. Phase C replaces the bare-table-name +key with a `Relation(schema, table)` value type threaded through extract → aggregate → +facts → rules → report, which is the enabling model change. Phase A adds ADV007 (join-key +index) and ADV008 (group-by index) on top of that model, and restores the `_dedupe_by_ddl` +tie-break those rules make reachable again. Phase B is independent: a pre-parse unwrap step +in ingest. + +**Tech Stack:** Python 3.11+, sqlglot 30.12 (`qualify`, `build_scope`), psycopg 3 +(integration only), typer, rich, pytest. + +## Global Constraints + +Every task's requirements implicitly include this section. + +- **All four CI gates must pass before every commit:** `uv run ruff check .`, + `uv run ruff format --check .`, `uv run mypy src/sqlquality`, `uv run pytest -q`. +- **The default test suite must need no extras and no Docker.** `uv run pytest` after a + plain `uv sync` must report `N passed, M deselected` — never `skipped`. Integration tests + are marked `integration` and deselected by default. +- **sqlquality never executes user SQL.** `advise` opens a read-only session + (`SET default_transaction_read_only = on`) with a statement timeout, runs only the + statements in `PostgresWorkloadAdapter.SQL`, and writes DDL to a file for human review. +- **No credential, and no user literal, may reach stdout, stderr, a report, or an exception + message.** Redaction happens at ingest; secrets are scrubbed via + `workload/secrets.py`'s `secrets_for`/`scrub`. +- **Confidence must never overstate evidence.** If a check could not run — denied grant, + unknown row count, unreadable index list — the proposal is emitted at LOW (or is + suppressed) and the rationale names the check that was skipped. "Probably wrong" is not a + confidence level. +- **Baseline is `main` at 442 default tests + 8 integration, all gates green.** Every task + must leave the suite green; the count only goes up. +- **A test that passes with the production change reverted is not a test.** Every new test + must be run against the un-fixed code, or against a deliberate mutation of the line it + claims to pin, and observed to FAIL. Report the mutation you used. +- Public identifiers get a docstring saying *why*, matching the density of the surrounding + module. + +--- + +## Phase C — schema-qualified keying + +### Task 1: `Relation`, and resolving it from the schema map rather than the AST + +**Files:** +- Modify: `src/sqlquality/models.py` (add `Relation`; change `ColumnUsage.table` → + `ColumnUsage.relation`) +- Modify: `src/sqlquality/workload/extract.py:84-149` +- Test: `tests/test_workload_extract.py` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `sqlquality.models.Relation` — `@dataclass(frozen=True, order=True)` with fields + `schema: str`, `table: str`, and `__str__` returning `f"{schema}.{table}"`. + - `sqlquality.models.ColumnUsage.relation: Relation` replacing `table: str`. + - `sqlquality.workload.extract.resolve_relation(table: exp.Table, schema: dict) -> Relation | None` + - `extract_usage(tree, dialect, schema) -> tuple[tuple[Relation, str, ColumnRole], ...]` + — first tuple element is now a `Relation`, not a `str`. + - `extract_usage` now also raises `UnqualifiableQuery` for `sqlglot.errors.SchemaError`. + - The `schema` argument to `extract_usage` is now **nested**: + `{schema_name: {table: {column: type}}}`. + +**The constraint that decides this task.** `qualify()` does *not* populate `Table.db` for a +bare table reference, even when the nested schema resolves it unambiguously. Verified +against sqlglot 30.12: + +``` +schema={'public': {'orders': {'id': 'int', 'status': 'text'}}} +sql=select id from orders where status='x' + -> SELECT "orders"."id" AS "id" FROM "orders" AS "orders" WHERE "orders"."status" = 'x' + source alias='orders' name='orders' db='' <-- db is EMPTY +``` + +Reading `table.db` and trusting it therefore keys almost every real workload under +`schema=""`, because production queries rely on `search_path` and say `from orders`, not +`from public.orders`. A phantom `Relation("", "orders")` matches no catalog fact, so every +table falls through the `facts.get(...)` lookup and **every proposal is silently +suppressed** — the same failure shape as the `reltuples = -1` bug. The schema must be +resolved from the schema map we introspected, with `table.db` used only when it is +non-empty. + +Also verified: `SchemaError` is **not** a subclass of `OptimizeError` +(`SchemaError.__mro__` is `SchemaError → SqlglotError → Exception`). `extract_usage` +catches only `OptimizeError` today, so an ambiguous bare name +(`SchemaError: Ambiguous mapping for orders: sales, staging.`) propagates out of +`aggregate()` and crashes the whole run with a traceback. Multi-schema is what makes that +reachable, so it must be caught in the same task that makes it reachable. + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/test_workload_extract.py`: + +```python +import pytest +from sqlglot import exp + +from sqlquality.models import ColumnRole, Relation +from sqlquality.sqlast import parse +from sqlquality.workload.extract import ( + UnqualifiableQuery, + extract_usage, + resolve_relation, +) + +ONE_SCHEMA = {"public": {"orders": {"id": "int", "status": "text", "shipped_at": "timestamp"}}} +TWO_SCHEMAS = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"items": {"sku": "text", "qty": "int"}}, +} +COLLIDING = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"orders": {"id": "int", "status": "text"}}, +} + + +def test_bare_table_resolves_to_its_only_owning_schema(): + """The common case: production SQL relies on search_path and says `from orders`. + + qualify() leaves Table.db empty here, so a `table.db`-only implementation keys this + under Relation("", "orders") and every catalog lookup misses. + """ + tree = parse("select id from orders where status = 'x'", "postgres") + usage = extract_usage(tree, "postgres", ONE_SCHEMA) + assert {relation for relation, _c, _r in usage} == {Relation("public", "orders")} + + +def test_explicitly_qualified_table_uses_the_schema_it_names(): + tree = parse("select id from staging.items where qty > 1", "postgres") + usage = extract_usage(tree, "postgres", TWO_SCHEMAS) + assert {relation for relation, _c, _r in usage} == {Relation("staging", "items")} + + +def test_two_schemas_distinct_names_attribute_to_the_right_one(): + """A join across schemas must not collapse both sides onto one relation.""" + tree = parse( + "select o.id, i.sku from orders o join items i on i.sku = o.status", "postgres" + ) + usage = extract_usage(tree, "postgres", TWO_SCHEMAS) + assert {relation for relation, _c, _r in usage} == { + Relation("sales", "orders"), + Relation("staging", "items"), + } + + +def test_ambiguous_bare_name_is_unqualifiable_not_a_crash(): + """sqlglot raises SchemaError, which is NOT an OptimizeError subclass.""" + tree = parse("select id from orders where status = 'x'", "postgres") + with pytest.raises(UnqualifiableQuery): + extract_usage(tree, "postgres", COLLIDING) + + +def test_resolve_relation_prefers_an_explicit_db_over_the_map(): + table = exp.Table(this=exp.to_identifier("orders"), db=exp.to_identifier("sales")) + assert resolve_relation(table, COLLIDING) == Relation("sales", "orders") + + +def test_resolve_relation_returns_none_when_ambiguous(): + """Two owners is not a guess we are entitled to make.""" + table = exp.Table(this=exp.to_identifier("orders")) + assert resolve_relation(table, COLLIDING) is None + + +def test_resolve_relation_returns_none_for_a_table_outside_the_map(): + table = exp.Table(this=exp.to_identifier("nowhere")) + assert resolve_relation(table, ONE_SCHEMA) is None + + +def test_dml_columns_attribute_to_the_qualified_target(): + tree = parse("update orders set status = 'y' where id = 1", "postgres") + usage = extract_usage(tree, "postgres", ONE_SCHEMA) + assert (Relation("public", "orders"), "id", ColumnRole.EQUALITY) in usage +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_workload_extract.py -x -q` + +Expected: FAIL — `ImportError: cannot import name 'Relation'`. + +- [ ] **Step 3: Add `Relation` to models.py** + +Insert immediately above `ColumnRole` in `src/sqlquality/models.py`: + +```python +@dataclass(frozen=True, order=True) +class Relation: + """A schema-qualified relation — the key every catalog fact is stored under. + + Bare table names were the key until multi-schema support landed, and they aliased: two + schemas each holding an `orders` merged into one entry, so the last catalog row won the + row estimate while `qualify()` resolved columns against the union of both column sets. + + ``order=True`` because the rules sort their output for canonical, run-to-run stable + report ordering, and a bare `sorted()` over relation keys has to work. Field order is + (schema, table) so that ordering groups a schema's tables together. + """ + + schema: str + table: str + + def __str__(self) -> str: + """`schema.table` — how the relation appears in a proposal title or JSON key.""" + return f"{self.schema}.{self.table}" +``` + +- [ ] **Step 4: Change `ColumnUsage.table` to `ColumnUsage.relation`** + +In `src/sqlquality/models.py`, replace the `table: str` field of `ColumnUsage` with: + +```python + relation: Relation +``` + +Leave every other field, and the whole `cost_share` docstring, exactly as it is. Do **not** +add a `table` compatibility property: this codebase has already paid for two fields +carrying one fact (see the `fingerprints` / `fingerprint_ids` note in the same class), and +a second spelling of the identity is how the two drift. + +- [ ] **Step 5: Resolve relations in extract.py** + +In `src/sqlquality/workload/extract.py`, add the `SchemaError` import and replace +`_scope_tables`, `_record` and `_collect_dml`: + +```python +from sqlglot.errors import OptimizeError, SchemaError +``` + +```python +def resolve_relation(table: exp.Table, schema: dict) -> Relation | None: + """The schema-qualified relation for one `exp.Table`, or None if it is not attributable. + + `table.db` is authoritative when present, but `qualify()` leaves it EMPTY for a bare + table reference even when the nested schema resolves the name unambiguously — and bare + references are the normal case, because production SQL relies on `search_path`. So the + fallback is a lookup in the schema map we actually introspected: + + * exactly one introspected schema holds the name -> that is the schema, no guess involved + * more than one -> ambiguous, and attributing it would be a coin flip. `qualify()` will + normally have raised `SchemaError` before we get here, but a table whose columns are + never referenced by name reaches this line, so the guard is real. + * none -> the table lives outside the introspected schemas; the caller drops the column. + """ + if table.db: + return Relation(schema=table.db, table=table.name) + owners = [name for name, tables in schema.items() if table.name in tables] + if len(owners) == 1: + return Relation(schema=owners[0], table=table.name) + return None + + +def _scope_relations(scope: Scope, schema: dict) -> dict[str, Relation]: + """Alias (or bare name) -> schema-qualified relation, for one scope only. + + A sub-scope source (CTE, derived table) maps to a ``Scope``, not an ``exp.Table``. + Columns resolving to one of those reference a projection rather than a base-table + column, so they are omitted here and skipped — the sub-scope contributes its own base + tables when ``traverse()`` reaches it. A source we cannot attribute to a schema is + omitted for the same reason: no key, no usage. + """ + resolved: dict[str, Relation] = {} + for name, source in scope.sources.items(): + if isinstance(source, exp.Table): + relation = resolve_relation(source, schema) + if relation is not None: + resolved[name] = relation + return resolved + + +def _record( + seen: set[tuple[Relation, str, ColumnRole]], + relation: Relation | None, + column: exp.Column, +) -> None: + """Add one (relation, column, role) triple, skipping unattributable or unused columns.""" + if relation is None or not column.name: + return + role = _role(column) + if role is None: + return + seen.add((relation, column.name, role)) + + +def _collect_dml( + qualified: exp.Expression, seen: set[tuple[Relation, str, ColumnRole]], schema: dict +) -> None: + """Attribute the columns of an UPDATE/DELETE to its sole target table. + + ``qualify()`` leaves DML columns bare (``column.table == ''``) rather than raising. + With exactly one table in the statement the target is unambiguous; with more than one + (``UPDATE ... FROM``) attribution would be a guess, so bare columns are dropped + instead of misattributed. + """ + tables = tuple(qualified.find_all(exp.Table)) + aliases: dict[str, Relation] = {} + for table in tables: + relation = resolve_relation(table, schema) + if relation is not None: + aliases[table.alias_or_name] = relation + sole = resolve_relation(tables[0], schema) if len(tables) == 1 else None + for column in qualified.find_all(exp.Column): + _record(seen, aliases.get(column.table) if column.table else sole, column) +``` + +Then update `extract_usage`'s body and signature docstring: + +```python +def extract_usage( + tree: exp.Expression, dialect: str, schema: dict +) -> tuple[tuple[Relation, str, ColumnRole], ...]: + """(relation, column, role) triples for one query, deduplicated. + + ``schema`` is nested — ``{schema_name: {table: {column: type}}}`` — because relations + are keyed by schema. Stars are not expanded: a projected star tells us nothing about + which columns are filtered, and expanding it would drown the rollup in projection noise. + + ``SchemaError`` is caught alongside ``OptimizeError`` and re-raised as + ``UnqualifiableQuery``. It is *not* an ``OptimizeError`` subclass — its bases are + ``SqlglotError``, ``Exception`` — so catching only ``OptimizeError`` let an ambiguous + bare table name (`Ambiguous mapping for orders: sales, staging.`) escape `aggregate()` + and abort the whole run with a traceback. + """ + try: + qualified = qualify(tree.copy(), dialect=dialect, schema=schema, expand_stars=False) + except (OptimizeError, SchemaError) as exc: + raise UnqualifiableQuery(str(exc)) from exc + + seen: set[tuple[Relation, str, ColumnRole]] = set() + root = build_scope(qualified) + if root is None: + # build_scope() returns None for UPDATE/DELETE — they are not SELECT-rooted. + _collect_dml(qualified, seen, schema) + else: + # Resolve aliases per scope, never with one flat map over the whole tree. Two + # different tables in different scopes can share an alias, and a flat map keeps + # whichever `find_all` visited last — silently attributing an outer filter to an + # inner table and losing the outer one entirely. + for scope in root.traverse(): + aliases = _scope_relations(scope, schema) + for column in scope.columns: + _record(seen, aliases.get(column.table), column) + return tuple( + sorted(seen, key=lambda triple: (triple[0], triple[1], triple[2].value)) + ) +``` + +Add `Relation` to the `sqlquality.models` import at the top of the module. + +- [ ] **Step 6: Run the new tests** + +Run: `uv run pytest tests/test_workload_extract.py -q` + +Expected: PASS. + +- [ ] **Step 7: Prove `test_bare_table_resolves_to_its_only_owning_schema` discriminates** + +Temporarily change `resolve_relation` to `return Relation(schema=table.db, table=table.name)` +— i.e. the naive `table.db`-only implementation this task exists to prevent. Run the test +file. Expected: that test FAILS with +`{Relation(schema='', table='orders')} != {Relation(schema='public', table='orders')}`. +Restore the real implementation. Report the mutation and the failure in your report. + +- [ ] **Step 8: Fix the rest of the suite's call sites mechanically** + +`uv run pytest -q` will now fail in `tests/test_workload_aggregate.py`, +`tests/test_workload_rules.py`, `tests/test_workload_postgres.py` and +`tests/test_models.py`. Do **not** fix them yet — Tasks 2-4 own those layers. For this +task's commit it is enough that `tests/test_workload_extract.py` and +`tests/test_workload_fingerprint.py` pass. Note the failing count in your report so the +next task can confirm it shrinks. + +If `mypy` reports errors in `aggregate.py` / `postgres.py` from the changed tuple type, +that is expected and Tasks 2-4 resolve it — say so in the report rather than papering over +it with `type: ignore`. + +- [ ] **Step 9: Commit** + +```bash +git add src/sqlquality/models.py src/sqlquality/workload/extract.py tests/test_workload_extract.py +git commit -m "feat(advise): key column usage by schema-qualified relation" +``` + +--- + +### Task 2: aggregate on relations, and count ambiguity separately + +**Files:** +- Modify: `src/sqlquality/workload/aggregate.py` +- Modify: `src/sqlquality/models.py` (`Aggregation.tables`, `Aggregation.skipped_ambiguous`) +- Test: `tests/test_workload_aggregate.py` + +**Interfaces:** +- Consumes: `Relation`, `extract_usage` returning `(Relation, str, ColumnRole)` triples, + nested `schema` (Task 1). +- Produces: + - `Aggregation.tables: frozenset[Relation]` (was `frozenset[str]`) + - `Aggregation.skipped_ambiguous: int = 0` — statements dropped specifically because a + table name was ambiguous across the introspected schemas. + - `star_tables(workload, schema) -> frozenset[Relation]` + - `aggregate(workload, schema, dialect)` unchanged in signature; `schema` is now nested. + +**Why ambiguity gets its own counter.** `skipped_unqualifiable` already exists, and folding +ambiguity into it would be defensible — except the remedy is different. An unresolvable +statement means the schema is incomplete (fetch more, or grant more); an ambiguous one means +*this* run introspected two schemas holding the same table name and the query did not say +which. The fix is "qualify the query, or run `advise` once per schema", and the report has +to be able to say that. A single bucket cannot. + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/test_workload_aggregate.py`: + +```python +from sqlquality.models import ColumnRole, QueryStat, Relation, Workload +from sqlquality.workload.aggregate import aggregate, star_tables + +ONE_SCHEMA = {"public": {"orders": {"id": "int", "status": "text"}}} +TWO_SCHEMAS = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"items": {"sku": "text", "qty": "int"}}, +} +COLLIDING = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"orders": {"id": "int", "status": "text"}}, +} + + +def _workload(*sql: str) -> Workload: + return Workload( + stats=tuple( + QueryStat(fingerprint=f"fp{i}", sql=s, calls=1, total_time_ms=100.0) + for i, s in enumerate(sql) + ), + window_description="test", + ) + + +def test_usage_is_keyed_by_relation(): + result = aggregate(_workload("select id from orders where status = 'x'"), ONE_SCHEMA, "postgres") + assert {u.relation for u in result.usage} == {Relation("public", "orders")} + assert result.tables == frozenset({Relation("public", "orders")}) + + +def test_same_table_name_in_two_schemas_does_not_alias(): + """The bug multi-schema keying exists to fix: two relations, not one merged entry.""" + result = aggregate( + _workload( + "select id from sales.orders where status = 'x'", + "select id from staging.orders where status = 'y'", + ), + COLLIDING, + "postgres", + ) + assert result.tables == frozenset( + {Relation("sales", "orders"), Relation("staging", "orders")} + ) + + +def test_ambiguous_bare_name_is_counted_not_crashed(): + result = aggregate(_workload("select id from orders where status = 'x'"), COLLIDING, "postgres") + assert result.skipped_ambiguous == 1 + assert result.usage == () + + +def test_a_plain_parse_failure_is_not_counted_as_ambiguous(): + """The two counters must not both fire for the same statement.""" + result = aggregate(_workload("this is not sql at all"), ONE_SCHEMA, "postgres") + assert result.skipped_ambiguous == 0 + assert result.skipped_unqualifiable == 1 + + +def test_star_tables_returns_qualified_relations(): + workload = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from items", + calls=1, + total_time_ms=1.0, + flags=frozenset({"select_star"}), + ), + ), + window_description="test", + ) + assert star_tables(workload, TWO_SCHEMAS) == frozenset({Relation("staging", "items")}) + + +def test_star_tables_skips_an_ambiguous_name(): + """Attributing a bare `select *` to one of two same-named tables would be a guess.""" + workload = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from orders", + calls=1, + total_time_ms=1.0, + flags=frozenset({"select_star"}), + ), + ), + window_description="test", + ) + assert star_tables(workload, COLLIDING) == frozenset() +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_workload_aggregate.py -x -q` + +Expected: FAIL — `AttributeError: 'ColumnUsage' object has no attribute 'relation'` or a +`TypeError` from the nested schema, depending on which test runs first. + +- [ ] **Step 3: Add the ambiguity signal to extract.py** + +`aggregate` must distinguish an ambiguity from any other resolution failure, and the only +place that knows is the raise site. Add a subclass to +`src/sqlquality/workload/extract.py`, right below `UnqualifiableQuery`: + +```python +class AmbiguousRelation(UnqualifiableQuery): + """A table name that two introspected schemas both hold, in a query that did not qualify it. + + A subclass, not a sibling: every caller that wants to treat all resolution failures + alike keeps working with one `except UnqualifiableQuery`, while `aggregate` can count + this case separately because its remedy is different — qualify the query or run once per + schema, rather than widen the schema. + """ +``` + +and raise it from `extract_usage` when sqlglot reports ambiguity: + +```python + try: + qualified = qualify(tree.copy(), dialect=dialect, schema=schema, expand_stars=False) + except SchemaError as exc: + # sqlglot has exactly one ambiguity message and no error code to match on, so the + # text is the only signal available. Matching it loosely (lowercased substring) + # rather than exactly, because a wording change upstream should degrade this to + # "counted as unqualifiable" — the pre-existing behaviour — not crash. + if "ambiguous mapping" in str(exc).lower(): + raise AmbiguousRelation(str(exc)) from exc + raise UnqualifiableQuery(str(exc)) from exc + except OptimizeError as exc: + raise UnqualifiableQuery(str(exc)) from exc +``` + +- [ ] **Step 4: Rewrite the aggregate internals** + +In `src/sqlquality/workload/aggregate.py`: + +- `_Key` becomes `tuple[Relation, str, ColumnRole]`. +- Import `AmbiguousRelation` and `Relation`. +- `star_tables` iterates the nested schema and resolves each mentioned name to a single + owning schema: + +```python +def star_tables(workload: Workload, schema: dict) -> frozenset[Relation]: + """Relations a `SELECT *` query group merely *mentions*, matched against ``schema``. + + A bare `select * from wide_t` filters nothing, so it contributes no column usage and + the relation never appears in ``Aggregation.tables``. Introspecting only the relations + that produced usage therefore left the star rule with no column counts to test — inert + for precisely the workload it exists to catch. These names are unioned in before catalog + facts are fetched. + + A name held by two introspected schemas is skipped rather than attributed to either or + to both: over-reporting would put a wide-table warning on a table the query never + touched. Consistent with `resolve_relation`, which declines the same guess. + + Deliberately *not* added to ``Aggregation.tables``: that set means "relations with + recorded column usage" and feeds the unused-index rule's notion of a hot table. + """ + found: set[Relation] = set() + for stat in workload.stats: + if FLAG_SELECT_STAR not in stat.flags: + continue + for table in _table_names(schema): + if not mentions_table(table, stat.sql): + continue + owners = [name for name, tables in schema.items() if table in tables] + if len(owners) == 1: + found.add(Relation(schema=owners[0], table=table)) + return frozenset(found) + + +def _table_names(schema: dict) -> frozenset[str]: + """Every bare table name in a nested schema map, deduplicated across schemas.""" + return frozenset(table for tables in schema.values() for table in tables) +``` + +- `aggregate` counts ambiguity separately and builds `ColumnUsage(relation=...)`: + +```python + skipped_unqualifiable = 0 + skipped_ambiguous = 0 + + for stat in workload.stats: + try: + tree = parse(stat.sql, dialect) + triples = extract_usage(tree, dialect, schema) + except AmbiguousRelation: + # Counted before the broader handler below, because AmbiguousRelation *is* an + # UnqualifiableQuery — ordering these the other way round makes the specific + # counter unreachable and the specific remedy unreportable. + skipped_ambiguous += 1 + continue + except (SqlParseError, UnqualifiableQuery): + skipped_unqualifiable += 1 + continue + for key in triples: + calls[key] += stat.calls + cost[key] += stat.total_time_ms + contributors[key].add(stat.fingerprint) + tables.add(key[0]) +``` + +with the `ColumnUsage` construction using `relation=relation` and the sort key becoming +`key=lambda u: (-u.cost_ms, u.relation, u.column, u.role.value)` (`Relation` is +`order=True`, so it sorts directly), and the `Aggregation` gaining +`skipped_ambiguous=skipped_ambiguous`. + +- [ ] **Step 5: Add the field to `Aggregation`** + +In `src/sqlquality/models.py`: + +```python +@dataclass(frozen=True) +class Aggregation: + usage: tuple[ColumnUsage, ...] + total_cost_ms: float + skipped_unqualifiable: int + tables: frozenset[Relation] + #: Statements dropped because a bare table name is held by two introspected schemas. + #: Separate from `skipped_unqualifiable` because the remedy differs: qualify the query + #: or run once per schema, rather than widen the schema. + skipped_ambiguous: int = 0 +``` + +- [ ] **Step 6: Run the tests** + +Run: `uv run pytest tests/test_workload_aggregate.py -q` + +Expected: PASS. + +- [ ] **Step 7: Prove the ordering of the two handlers matters** + +Swap the `except AmbiguousRelation` clause below the `except (SqlParseError, +UnqualifiableQuery)` clause. Run `tests/test_workload_aggregate.py`. Expected: +`test_ambiguous_bare_name_is_counted_not_crashed` FAILS with +`assert 0 == 1`, because the subclass is swallowed by the base handler. Restore the order. +Report this. + +- [ ] **Step 8: Commit** + +```bash +git add src/sqlquality/models.py src/sqlquality/workload/aggregate.py \ + src/sqlquality/workload/extract.py tests/test_workload_aggregate.py +git commit -m "feat(advise): aggregate per relation, count schema ambiguity separately" +``` + +--- + +### Task 3: catalog facts and indexes keyed by relation + +**Files:** +- Modify: `src/sqlquality/workload/base.py` (`fetch_table_facts` signature) +- Modify: `src/sqlquality/workload/postgres.py` (all six SQL statements, `fetch_schema`, + `fetch_table_facts`, `fetch_indexes`) +- Modify: `src/sqlquality/models.py` (`TableFacts.name` → `TableFacts.relation`) +- Test: `tests/test_workload_postgres.py` + +**Interfaces:** +- Consumes: `Relation` (Task 1), `Aggregation.tables: frozenset[Relation]` (Task 2). +- Produces: + - `TableFacts.relation: Relation` replacing `name: str`. + - `fetch_schema(schemas) -> dict` now nested: `{schema: {table: {column: type}}}`. + - `fetch_table_facts(schemas, relations: frozenset[Relation]) -> dict[Relation, TableFacts]` + - `fetch_indexes(schemas, relations: frozenset[Relation]) -> dict[Relation, tuple[PgIndex, ...]]` + - Every `SQL` statement that returns a relation now returns its schema as the **first** + column. + +**The SQL change.** Each of `CAP_SCHEMA`, `CAP_TABLE_FACTS`, `CAP_NDV` and `CAP_INDEXES` +already filters on `= ANY(%s)` over the schema list but does not *return* the schema, so a +row from `sales.orders` and one from `staging.orders` are indistinguishable in the result +set. Add the schema to the select list and to the grouping key. + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/test_workload_postgres.py`, following the existing `_FakeCursor`/querier +fixture style in that file (read it first and match it — do not invent a second harness): + +```python +def test_fetch_schema_is_nested_by_schema(): + rows = { + CAP_SCHEMA: [ + ("sales", "orders", "id", "integer"), + ("sales", "orders", "status", "text"), + ("staging", "orders", "id", "integer"), + ] + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + assert adapter.fetch_schema(("sales", "staging")) == { + "sales": {"orders": {"id": "integer", "status": "text"}}, + "staging": {"orders": {"id": "integer"}}, + } + + +def test_table_facts_do_not_alias_across_schemas(): + """Two same-named tables must keep their own row estimates.""" + rows = { + CAP_SCHEMA: [("sales", "orders", "id", "integer"), ("staging", "orders", "id", "integer")], + CAP_TABLE_FACTS: [("sales", "orders", 50_000, 1024), ("staging", "orders", 7, 64)], + CAP_NDV: [], + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + facts = adapter.fetch_table_facts( + ("sales", "staging"), + frozenset({Relation("sales", "orders"), Relation("staging", "orders")}), + ) + assert facts[Relation("sales", "orders")].row_estimate == 50_000 + assert facts[Relation("staging", "orders")].row_estimate == 7 + + +def test_ndv_does_not_leak_between_same_named_tables(): + rows = { + CAP_SCHEMA: [("sales", "orders", "id", "integer"), ("staging", "orders", "id", "integer")], + CAP_TABLE_FACTS: [("sales", "orders", 50_000, 1024), ("staging", "orders", 50_000, 1024)], + CAP_NDV: [("sales", "orders", "id", 5000.0), ("staging", "orders", "id", 3.0)], + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + facts = adapter.fetch_table_facts( + ("sales", "staging"), + frozenset({Relation("sales", "orders"), Relation("staging", "orders")}), + ) + assert facts[Relation("sales", "orders")].ndv["id"] == 5000.0 + assert facts[Relation("staging", "orders")].ndv["id"] == 3.0 + + +def test_indexes_do_not_alias_across_schemas(): + rows = { + CAP_INDEXES: [ + ("sales", "orders", "idx_a", "id", 1, False, False, 0, 100, False, None, False, "..."), + ("staging", "orders", "idx_b", "id", 1, False, False, 9, 200, False, None, False, "..."), + ] + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + indexes = adapter.fetch_indexes( + ("sales", "staging"), + frozenset({Relation("sales", "orders"), Relation("staging", "orders")}), + ) + assert [i.name for i in indexes[Relation("sales", "orders")]] == ["idx_a"] + assert [i.name for i in indexes[Relation("staging", "orders")]] == ["idx_b"] + assert indexes[Relation("staging", "orders")][0].scans == 9 + + +def test_every_relation_returning_statement_selects_its_schema(): + """A statement that filters on schema but does not return it cannot be keyed by it. + + This is the whole defect class of this task: the rows come back indistinguishable and + the last one silently wins. + """ + for capability in (CAP_SCHEMA, CAP_TABLE_FACTS, CAP_NDV, CAP_INDEXES): + sql = PostgresWorkloadAdapter.SQL[capability].lower() + assert "nspname" in sql or "schemaname" in sql or "table_schema" in sql, capability +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_workload_postgres.py -x -q` + +Expected: FAIL — the canned rows have one more element than the current unpacking expects +(`ValueError: too many values to unpack`). + +- [ ] **Step 3: Change the SQL to return the schema** + +In `PostgresWorkloadAdapter.SQL`, prepend the schema to each select list. `CAP_WORKLOAD` +and `CAP_STATS_RESET` are unchanged — neither returns a relation. + +```python + CAP_SCHEMA: """ + SELECT c.table_schema, c.table_name, c.column_name, c.data_type + FROM information_schema.columns c + WHERE c.table_schema = ANY(%s) + """, + CAP_TABLE_FACTS: """ + SELECT n.nspname, c.relname, c.reltuples::bigint, pg_total_relation_size(c.oid) + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE c.relkind = 'r' AND n.nspname = ANY(%s) AND c.relname = ANY(%s) + """, + CAP_NDV: """ + SELECT s.schemaname, s.tablename, s.attname, s.n_distinct + FROM pg_stats s + WHERE s.schemaname = ANY(%s) AND s.tablename = ANY(%s) + """, +``` + +and for `CAP_INDEXES` add `n.nspname` first and extend the `ORDER BY`, keeping every +existing comment in the statement verbatim: + +```python + CAP_INDEXES: """ + SELECT n.nspname, t.relname, i.relname, a.attname, k.ordinality, + ix.indisunique, ix.indisprimary, + COALESCE(psui.idx_scan, 0), pg_relation_size(i.oid), + ix.indpred IS NOT NULL, + pg_get_expr(ix.indpred, ix.indrelid), + ix.indexprs IS NOT NULL, + pg_get_indexdef(ix.indexrelid) + FROM pg_index ix + JOIN pg_class i ON i.oid = ix.indexrelid + JOIN pg_class t ON t.oid = ix.indrelid + JOIN pg_namespace n ON n.oid = t.relnamespace + JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS k(attnum, ordinality) ON TRUE + LEFT JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum + LEFT JOIN pg_stat_user_indexes psui ON psui.indexrelid = i.oid + WHERE n.nspname = ANY(%s) AND t.relname = ANY(%s) + ORDER BY n.nspname, t.relname, i.relname, k.ordinality + """, +``` + +Note the `= ANY(%s)` **table** parameter stays a list of bare names: Postgres filters on +`relname`, and narrowing per-schema would need one statement per schema. Passing the union +of bare names over-fetches slightly (a same-named table in a schema we do not care about +comes back) and the relation key then simply has no consumer. Say this in a comment. + +- [ ] **Step 4: Rework the fetch methods** + +`TableFacts.name` → `TableFacts.relation: Relation` in models.py, with a docstring note +that the field is the schema-qualified key rather than a display name. + +`fetch_schema` nests by schema: + +```python + def fetch_schema(self, schemas: tuple[str, ...]) -> dict: + """Nested schema mapping for sqlglot qualify(): {schema: {table: {column: type}}}. + + Nested rather than flat because `qualify()` needs to be able to *tell* two + same-named tables apart — a flat map resolves a column against the union of both + column sets, which is how a filter on a column that exists in only one of them was + silently accepted. + """ + schema: dict[str, dict[str, dict[str, str]]] = {} + for schema_name, table, column, data_type in self._schema_rows(schemas): + schema.setdefault(str(schema_name), {}).setdefault(str(table), {})[str(column)] = str( + data_type + ) + return schema +``` + +`fetch_table_facts` takes and returns relations. Keep the negative-`n_distinct` logic and +its whole comment intact, changing only the key from `str(table)` to +`Relation(schema=str(schema_name), table=str(table))`. Same for `fetch_indexes`: the +`grouped` dict key becomes `tuple[Relation, str]` and the result +`dict[Relation, tuple[PgIndex, ...]]`. + +Update the `fetch_table_facts` abstract signature in `base.py` to +`relations: frozenset[Relation]` and its docstring to say relations. + +- [ ] **Step 5: Run the tests** + +Run: `uv run pytest tests/test_workload_postgres.py -q` + +Expected: PASS. `tests/test_workload_rules.py` still fails — Task 4 owns it. + +- [ ] **Step 6: Prove the aliasing tests discriminate** + +Revert `fetch_table_facts`'s key to the bare `str(table)` while leaving everything else in +place (a `dict[str, ...]` keyed on bare name, looked up by `relation.table`). Run +`tests/test_workload_postgres.py`. Expected: `test_table_facts_do_not_alias_across_schemas` +FAILS — both relations report whichever row arrived last. Restore. Report the mutation. + +- [ ] **Step 7: Commit** + +```bash +git add src/sqlquality/models.py src/sqlquality/workload/base.py \ + src/sqlquality/workload/postgres.py tests/test_workload_postgres.py +git commit -m "feat(advise): key catalog facts and indexes by relation" +``` + +--- + +### Task 4: the six existing rules on relations, with schema-qualified DDL + +**Files:** +- Modify: `src/sqlquality/workload/postgres.py` (`_by_table`, `_covered` call sites, all six + `propose_*`, `propose`) +- Test: `tests/test_workload_rules.py` + +**Interfaces:** +- Consumes: everything from Tasks 1-3. +- Produces: + - `_by_relation(usage) -> dict[Relation, list[ColumnUsage]]` replacing `_by_table`. + - Every rule takes `facts: Mapping[Relation, TableFacts]` and + `existing: Mapping[Relation, Sequence[PgIndex]]`. + - Every rule's `evidence` dict gains `"schema": relation.schema` and keeps + `"table": relation.table` (the bare name, so existing JSON consumers still read the + same value from the same key). + - Rule titles render the relation as `schema.table`. + - The module-level `schema: str = DEFAULT_SCHEMA` keyword argument is **removed** from + every rule — each proposal now knows its own schema from its relation, so a + single run-wide schema is no longer meaningful. + +**Why `schema=` goes away.** `propose(...)` currently passes one `schema` to every rule and +`_qualified(schema, table)` stamps it onto the DDL. With more than one schema in play that +is wrong for every relation but one. Each rule now calls `_qualified(relation.schema, +relation.table)`. The `DEFAULT_SCHEMA` constant stays — `WorkloadAdapter.schemas` still +defaults to `("public",)`. + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/test_workload_rules.py` (match the existing fixture helpers in that file): + +```python +def test_adv001_ddl_is_qualified_with_the_relations_own_schema(): + usage = ( + _usage(Relation("sales", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + ) + facts = {Relation("sales", "orders"): _facts(Relation("sales", "orders"), rows=50_000)} + proposals = propose_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].ddl == 'CREATE INDEX ON "sales"."orders" ("status");' + assert proposals[0].evidence["schema"] == "sales" + assert proposals[0].evidence["table"] == "orders" + assert "sales.orders" in proposals[0].title + + +def test_two_same_named_relations_get_two_independent_proposals(): + """One proposal per relation, each stamped with its own schema.""" + usage = ( + _usage(Relation("sales", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + _usage(Relation("staging", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + ) + facts = { + Relation("sales", "orders"): _facts(Relation("sales", "orders"), rows=50_000), + Relation("staging", "orders"): _facts(Relation("staging", "orders"), rows=50_000), + } + ddls = {p.ddl for p in propose_indexes(usage, facts, {}, min_cost_share=0.01)} + assert ddls == { + 'CREATE INDEX ON "sales"."orders" ("status");', + 'CREATE INDEX ON "staging"."orders" ("status");', + } + + +def test_an_index_in_one_schema_does_not_cover_the_other_schemas_candidate(): + """The coverage check must not reach across schemas.""" + usage = ( + _usage(Relation("sales", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + _usage(Relation("staging", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + ) + facts = { + Relation("sales", "orders"): _facts(Relation("sales", "orders"), rows=50_000), + Relation("staging", "orders"): _facts(Relation("staging", "orders"), rows=50_000), + } + existing = { + Relation("sales", "orders"): ( + PgIndex(name="idx_status", columns=("status",), is_unique=False, + is_primary=False, scans=1, size_bytes=1), + ) + } + proposals = propose_indexes(usage, facts, existing, min_cost_share=0.01) + assert [p.evidence["schema"] for p in proposals] == ["staging"] + + +def test_adv002_drop_ddl_qualifies_the_index_with_its_relations_schema(): + existing = { + Relation("staging", "orders"): ( + PgIndex(name="idx_cold", columns=("note",), is_unique=False, + is_primary=False, scans=0, size_bytes=1), + ) + } + proposals = propose_unused_indexes( + existing, hot_tables=frozenset({Relation("staging", "orders")}) + ) + assert proposals[0].ddl == 'DROP INDEX "staging"."idx_cold";' +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_workload_rules.py -x -q` + +Expected: FAIL — `TypeError` on `_usage(...)` taking a `Relation`, or a `KeyError`/`ddl` +mismatch showing `"public"` where `"sales"` is expected. + +- [ ] **Step 3: Rework the rules** + +Rename `_by_table` and re-key it: + +```python +def _by_relation(usage: Sequence[ColumnUsage]) -> dict[Relation, list[ColumnUsage]]: + grouped: dict[Relation, list[ColumnUsage]] = {} + for item in usage: + grouped.setdefault(item.relation, []).append(item) + return grouped +``` + +Then, in each of `propose_indexes`, `propose_partial_indexes`, `propose_unused_indexes`, +`propose_redundant_indexes`, `propose_sargability`, `propose_select_star`: + +- iterate `for relation, items in sorted(_by_relation(usage).items())` (`Relation` is + `order=True`, so this sorts canonically without a key function); +- drop the `schema: str = DEFAULT_SCHEMA` keyword and build DDL with + `_qualified(relation.schema, relation.table)`; +- change `facts.get(table)` to `facts.get(relation)` and `existing.get(table, ())` to + `existing.get(relation, ())`; +- put `"schema": relation.schema` and `"table": relation.table` in `evidence`; +- render titles with `f"...on {relation}(...)"` — `Relation.__str__` gives `schema.table`. + +`propose_select_star` needs care: `wide` is built from `facts.items()` and matched against +statement text with `mentions_table`. Keep matching on the **bare** name (that is what the +SQL says) but carry the relation through, so the evidence reports qualified names: + +```python + wide = { + relation: fact for relation, fact in facts.items() if len(fact.columns) >= min_columns + } + ... + touched = sorted( + (relation for relation in wide if mentions_table(relation.table, stat.sql)), + ) + ... + "tables": tuple(str(relation) for relation in touched), + "column_counts": {str(relation): len(facts[relation].columns) for relation in touched}, +``` + +`propose_sargability`'s per-usage branch reports `item.relation`; its +leading-wildcard branch is statement-level and unchanged. + +Finally, in `PostgresWorkloadAdapter.propose`, delete the `schema = self.schemas[0] ...` +line and the `schema=schema` arguments, and update the surrounding comment: it currently +explains that only one schema is introspected because the CLI rejects more, which stops +being true in Task 5. + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest tests/test_workload_rules.py -q` + +Expected: PASS. + +- [ ] **Step 5: Prove the cross-schema coverage test discriminates** + +Change `_covered(columns, existing.get(relation, ()))` to +`_covered(columns, existing.get(Relation("sales", relation.table), ()))` — a deliberate +cross-schema lookup. Run `tests/test_workload_rules.py`. Expected: +`test_an_index_in_one_schema_does_not_cover_the_other_schemas_candidate` FAILS with +`[] != ["staging"]`. Restore. Report the mutation. + +- [ ] **Step 6: Run the whole suite** + +Run: `uv run pytest -q` + +Expected: PASS, except `tests/test_advise_cli.py` and `tests/test_report*.py` which Task 5 +owns. Report the remaining failure count. + +- [ ] **Step 7: Commit** + +```bash +git add src/sqlquality/workload/postgres.py tests/test_workload_rules.py +git commit -m "feat(advise): propose per relation, with schema-qualified DDL" +``` + +--- + +### Task 5: accept multiple `--schema`, and disclose ambiguity + +**Files:** +- Modify: `src/sqlquality/cli.py` (`_validate_schemas`, `_coverage_line`, + `_coverage_warning`, `_analyzed_count`, the `--schema` help text, the `advise` body) +- Modify: `src/sqlquality/report.py:149` (`"tables"` must be JSON-serializable) +- Modify: `README.md` +- Test: `tests/test_advise_cli.py`, `tests/test_report_markdown.py` + +**Interfaces:** +- Consumes: everything from Tasks 1-4. +- Produces: `advise --schema a --schema b` runs; `_validate_schemas` deduplicates and + returns `tuple[str, ...]` without rejecting; `Aggregation.skipped_ambiguous` surfaces in + the coverage line, the coverage warning, the JSON payload and the markdown report. + +- [ ] **Step 1: Write the failing tests** + +```python +def test_two_schemas_are_accepted(): + result = runner.invoke(app, ["advise", "--schema", "sales", "--schema", "staging", "--dry-run"]) + assert result.exit_code == 0 + + +def test_duplicate_schemas_are_deduplicated(): + assert _validate_schemas(["public", "public"]) == ("public",) + + +def test_schema_order_is_preserved(): + assert _validate_schemas(["b", "a"]) == ("b", "a") + + +def test_coverage_line_reports_ambiguous_separately(): + workload = _workload_with(stats=3, unparseable=1, noise=0) + aggregation = _aggregation_with(skipped_unqualifiable=1, skipped_ambiguous=2) + line = _coverage_line(workload, aggregation) + assert "2 ambiguous" in line + + +def test_ambiguity_warning_names_the_remedy(): + workload = _workload_with(stats=1, unparseable=0, noise=0) + aggregation = _aggregation_with(skipped_unqualifiable=0, skipped_ambiguous=4) + warning = _ambiguity_warning(aggregation) + assert warning is not None + assert "--schema" in warning + + +def test_no_ambiguity_means_no_warning(): + """The warning must not fire on the single-schema path, which is every existing run.""" + assert _ambiguity_warning(_aggregation_with(skipped_unqualifiable=3, skipped_ambiguous=0)) is None + + +def test_payload_tables_are_qualified_strings(): + payload = advise_payload( + [], _workload_with(stats=0, unparseable=0, noise=0), + _aggregation_with(tables=frozenset({Relation("sales", "orders")})), + engine="postgres", redacted=True, degraded=[], + ) + assert payload["tables"] == ["sales.orders"] + json.dumps(payload) # must not raise +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_advise_cli.py -x -q` + +Expected: FAIL — exit code 2 from `_validate_schemas` on the two-schema invocation, and +`ImportError`/`NameError` for `_ambiguity_warning`. + +- [ ] **Step 3: Replace `_validate_schemas`** + +```python +def _validate_schemas(values: list[str]) -> tuple[str, ...]: + """Deduplicate `--schema` values, preserving the order they were given in. + + Multiple schemas used to be rejected because every catalog fact was keyed on the bare + relation name, so two schemas each holding an `orders` aliased into one another. Facts, + NDV maps, index lists and the `qualify()` schema are all keyed by `Relation` now, so the + rejection is gone. What survives is a narrower caveat, surfaced by + `_ambiguity_warning`: a query that says `from orders` when two introspected schemas both + hold `orders` is genuinely ambiguous, and is counted and reported rather than guessed at. + """ + return tuple(dict.fromkeys(values)) +``` + +- [ ] **Step 4: Surface the new counter** + +Add `f"{aggregation.skipped_ambiguous} ambiguous"` to `_coverage_line`, add +`aggregation.skipped_ambiguous` to `_coverage_warning`'s `unexplained` sum (an ambiguous +statement is one we tried and failed to use, exactly like an unresolvable one), and add: + +```python +def _ambiguity_warning(aggregation: Aggregation) -> str | None: + """A warning naming the remedy for schema-ambiguous statements, or None. + + Separate from `_coverage_warning`, which fires on a *fraction* and says "coverage is + low". This fires on any occurrence at all, because the remedy is specific and + actionable — and because a handful of ambiguous statements can be the hottest ones in + the workload without moving the coverage fraction enough to trip a threshold. + """ + if not aggregation.skipped_ambiguous: + return None + return ( + f"{aggregation.skipped_ambiguous} statement(s) named a table held by more than one " + "of the introspected schemas without qualifying it, so they could not be attributed " + "and were dropped. Qualify the table in the query, or run advise once per --schema." + ) +``` + +Call it in the `advise` body immediately after the `_coverage_warning` block, echoing to +`err`. Also add `skipped_ambiguous` to the `advise_payload` counts dict and to the markdown +report's coverage section, alongside the existing skip counts — the JSON and markdown +reports carry every other counter, and this is the one a multi-schema user most needs. + +In `report.py:149`, change `"tables": sorted(aggregation.tables)` to +`"tables": sorted(str(relation) for relation in aggregation.tables)`. A `Relation` is not +JSON-serializable, so leaving it would make `--json` raise `TypeError` after the whole +analysis had already run. + +- [ ] **Step 5: Update the `--schema` help and the README** + +Help text: `"Schema to introspect. Repeat for several: --schema public --schema sales."` + +In `README.md`, find the `advise` section's statement that only one schema is supported at +a time and replace it with the multi-schema behaviour plus the ambiguity caveat. Also check +the Limitations section for the same claim and narrow it. + +- [ ] **Step 6: Run the whole suite** + +Run: `uv run pytest -q` and all four gates. + +Expected: PASS, 442 + the new tests. + +- [ ] **Step 7: Prove the JSON test discriminates** + +Revert `report.py:149` to `sorted(aggregation.tables)`. Run +`tests/test_report_markdown.py -k payload_tables`. Expected: FAIL with +`TypeError: Object of type Relation is not JSON serializable` from the `json.dumps` line. +Restore. Report the mutation. + +- [ ] **Step 8: Commit** + +```bash +git add src/sqlquality/cli.py src/sqlquality/report.py README.md \ + tests/test_advise_cli.py tests/test_report_markdown.py +git commit -m "feat(advise): accept multiple --schema, disclose ambiguous statements" +``` + +--- + +## Phase A — the two roles nothing consumed + +Context for the implementer: `ColumnRole.JOIN` and `ColumnRole.GROUP` are classified in +`extract.py:73` and `extract.py:77`, cost-weighted in `aggregate`, included in every +`cost_share` denominator — and then read by **no rule at all**. Verified: +`grep -rn "ColumnRole.JOIN\|ColumnRole.GROUP" src/` matches only the two lines that produce +them. This phase spends them. + +### Task 6: ADV007 — index the hot join key + +**Files:** +- Modify: `src/sqlquality/workload/postgres.py` +- Test: `tests/test_workload_rules.py` + +**Interfaces:** +- Consumes: `_by_relation`, `_covered`, `_is_prefix`, `MIN_ROWS_FOR_INDEX`, `SELECTIVE_NDV`, + `_UNKNOWN_ROWS_NOTE`, `_qualified` (Tasks 1-4). +- Produces: + `propose_join_keys(usage, facts, existing, *, min_cost_share, min_rows=MIN_ROWS_FOR_INDEX, have_index_data=True) -> list[Proposal]` + emitting code `"ADV007"`, wired into `PostgresWorkloadAdapter.propose`. + +**Why this is a separate rule and not a fourth role in ADV001.** ADV001's rationale is +"equality columns first so the range column can be scanned last" — the B-tree ordering +argument. A join key is not a filter predicate: it is probed once per outer row, and its +selectivity story is about the join's inner side, not about narrowing a scan. Folding JOIN +into ADV001's candidate list would make that rationale false for the resulting index while +leaving the text in place. Postgres also does not create an index on the *referencing* side +of a foreign key, so an unindexed hot join key is a common and genuinely costly gap. + +Confidence, following the house rule that a check which could not run caps the claim: +- LOW when `rows is None` or `not have_index_data`; +- HIGH when the join column's NDV is `>= SELECTIVE_NDV` (a selective key really does make a + nested-loop probe cheap); +- MEDIUM when NDV is unknown; +- LOW when NDV is below `SELECTIVE_NDV` — a low-cardinality join column is usually the + wrong thing to index, and saying so at LOW is more useful than suppressing it. + +- [ ] **Step 1: Write the failing tests** + +```python +def test_adv007_proposes_an_index_on_an_unindexed_hot_join_key(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4, cost_ms=400.0),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert [p.code for p in proposals] == ["ADV007"] + assert proposals[0].ddl == 'CREATE INDEX ON "public"."order_items" ("order_id");' + assert proposals[0].confidence is Confidence.HIGH + + +def test_adv007_is_silent_when_an_index_already_leads_with_the_join_key(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + existing = { + relation: ( + PgIndex(name="idx_oi_order", columns=("order_id", "sku"), is_unique=False, + is_primary=False, scans=5, size_bytes=1), + ) + } + assert propose_join_keys(usage, facts, existing, min_cost_share=0.01) == [] + + +def test_adv007_respects_the_small_table_floor(): + relation = Relation("public", "tiny") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.9),) + facts = {relation: _facts(relation, rows=10, ndv={"order_id": 5.0})} + assert propose_join_keys(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv007_caps_at_low_when_the_index_list_could_not_be_read(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01, have_index_data=False) + assert proposals[0].confidence is Confidence.LOW + assert "could not be read" in proposals[0].rationale + + +def test_adv007_caps_at_low_and_discloses_an_unknown_row_count(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=None, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.LOW + assert "small-table floor" in proposals[0].rationale + + +def test_adv007_is_low_for_a_low_cardinality_join_key(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "kind", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"kind": 3.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.LOW + + +def test_adv007_ignores_non_join_roles(): + """The rule must not re-propose what ADV001 already covers.""" + relation = Relation("public", "orders") + usage = (_usage(relation, "status", ColumnRole.EQUALITY, cost_share=0.9),) + facts = {relation: _facts(relation, rows=100_000)} + assert propose_join_keys(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv007_reports_the_hottest_join_key_per_relation(): + relation = Relation("public", "order_items") + usage = ( + _usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4, cost_ms=400.0), + _usage(relation, "sku", ColumnRole.JOIN, cost_share=0.1, cost_ms=100.0), + ) + facts = {relation: _facts(relation, rows=100_000)} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert [p.evidence["columns"] for p in proposals] == [("order_id",), ("sku",)] +``` + +Note the last test: one proposal per join column, ordered by cost descending. A composite +of two join keys is not proposed — two separate joins on the same table want two separate +indexes, and a composite serves only the leading one. + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_workload_rules.py -k adv007 -q` + +Expected: FAIL — `NameError: name 'propose_join_keys' is not defined`. + +- [ ] **Step 3: Implement `propose_join_keys`** + +Place it directly after `propose_indexes` in `postgres.py`: + +```python +def propose_join_keys( + usage: Sequence[ColumnUsage], + facts: Mapping[Relation, TableFacts], + existing: Mapping[Relation, Sequence[PgIndex]], + *, + min_cost_share: float, + min_rows: int = MIN_ROWS_FOR_INDEX, + have_index_data: bool = True, +) -> list[Proposal]: + """ADV007 — a hot join key with no index leading with it. + + Deliberately one proposal per join column rather than a composite: two joins against the + same table want two indexes, and a composite `(a, b)` serves only probes on `a`. + + Not folded into ADV001. That rule's rationale is the B-tree ordering argument — + "equality columns first so the range column can be scanned last" — and a join key is not + a filter predicate: it is probed once per outer row. Adding JOIN to ADV001's candidate + list would have left that sentence in the report while making it false of the index it + describes. Postgres does not index the referencing side of a foreign key either, so this + gap is both common and expensive. + """ + proposals: list[Proposal] = [] + for relation, items in sorted(_by_relation(usage).items()): + table_facts = facts.get(relation) + rows = table_facts.row_estimate if table_facts else None + if rows is not None and rows < min_rows: + continue + ndv = table_facts.ndv if table_facts else {} + joins = sorted( + (i for i in items if i.role is ColumnRole.JOIN), + key=lambda i: (-i.cost_ms, i.column), + ) + for item in joins: + if item.cost_share < min_cost_share: + continue + if _covered((item.column,), existing.get(relation, ())) is not None: + continue + column_ndv = ndv.get(item.column) + if rows is None or not have_index_data: + confidence = Confidence.LOW + elif column_ndv is None: + confidence = Confidence.MEDIUM + elif column_ndv >= SELECTIVE_NDV: + confidence = Confidence.HIGH + else: + confidence = Confidence.LOW + + rationale = ( + "This column carries the table's hottest join predicate. A join key is " + "probed once per outer row, so without an index leading with it every probe " + "is a scan." + ) + if have_index_data: + rationale += " No existing index leads with it." + else: + rationale += ( + " The existing-index list could not be read, so whether an index " + "already leads with it is unknown — check before applying." + ) + if rows is None: + rationale += _UNKNOWN_ROWS_NOTE + if column_ndv is not None and column_ndv < SELECTIVE_NDV: + rationale += ( + f" Only about {column_ndv:.0f} distinct values, so the index may not be " + "selective enough to be worth its write cost." + ) + + proposals.append( + Proposal( + code="ADV007", + title=f"Add index on join key {relation}({item.column})", + rationale=rationale, + evidence={ + "schema": relation.schema, + "table": relation.table, + "columns": (item.column,), + "roles": (item.role.value,), + "cost_share": item.cost_share, + "calls": item.calls, + "fingerprints": item.fingerprints, + "row_estimate": rows, + "leading_ndv": column_ndv, + }, + confidence=confidence, + ddl=( + f"CREATE INDEX ON {_qualified(relation.schema, relation.table)} " + f"({_quote_ident(item.column)});" + ), + ) + ) + return proposals +``` + +- [ ] **Step 4: Wire it into `propose`** + +Add `*propose_join_keys(aggregation.usage, facts, existing, min_cost_share=min_cost_share, +have_index_data=have_index_data),` to the `proposals` list in +`PostgresWorkloadAdapter.propose`, directly after the `propose_indexes(...)` entry. + +- [ ] **Step 5: Run the tests** + +Run: `uv run pytest tests/test_workload_rules.py -q` then `uv run pytest -q` + +Expected: PASS. + +- [ ] **Step 6: Prove the coverage test discriminates** + +Delete the `if _covered(...) is not None: continue` guard. Run +`tests/test_workload_rules.py -k adv007`. Expected: +`test_adv007_is_silent_when_an_index_already_leads_with_the_join_key` FAILS with a +one-proposal list where `[]` was expected. Restore. Report the mutation. + +- [ ] **Step 7: Update `--min-cost-share` help and README** + +The `--min-cost-share` help enumerates the cost-weighted rules by code +(`ADV001, ADV004, ADV005, ADV006`). ADV007 is cost-weighted, so add it. Add ADV007 to the +README's rule table. + +- [ ] **Step 8: Commit** + +```bash +git add src/sqlquality/workload/postgres.py src/sqlquality/cli.py README.md \ + tests/test_workload_rules.py +git commit -m "feat(advise): ADV007 -- index the hot join key" +``` + +--- + +### Task 7: ADV008 — an index to serve a hot GROUP BY + +**Files:** +- Modify: `src/sqlquality/workload/postgres.py` +- Test: `tests/test_workload_rules.py` + +**Interfaces:** +- Consumes: `_by_relation`, `_covered`, `_first_co_occurring`-style fingerprint overlap, + `MIN_ROWS_FOR_INDEX`, `_UNKNOWN_ROWS_NOTE`. +- Produces: + `propose_grouping_indexes(usage, facts, existing, *, min_cost_share, min_rows=MIN_ROWS_FOR_INDEX, max_arity=MAX_INDEX_ARITY, have_index_data=True) -> list[Proposal]` + emitting code `"ADV008"`, wired into `propose`. + +**Confidence is capped at MEDIUM, always.** Whether Postgres uses an index for grouping +depends on the choice between `GroupAggregate` (needs sorted input, which the index +provides) and `HashAggregate` (does not), and that choice depends on `work_mem`, the number +of groups and the aggregate functions used — none of which `advise` can see. HIGH would be +a claim about the planner's decision, not about the catalog. So: MEDIUM when the row count +is known, LOW when it is not or the index list could not be read. Never HIGH. State this in +the docstring so a later reader does not "fix" the missing HIGH branch. + +**Multiple grouping columns are proposed as one composite,** in the grouping order the +queries use — unlike ADV007. A `GROUP BY a, b` needs input sorted by `(a, b)`; two +single-column indexes serve it no better than one. Because a redacted fingerprint does not +preserve which position each column held, order the composite by cost descending with the +column name as tiebreak, and say in the rationale that the order is inferred from cost +rather than read from the query. + +- [ ] **Step 1: Write the failing tests** + +```python +def test_adv008_proposes_a_composite_index_for_a_hot_group_by(): + relation = Relation("public", "events") + usage = ( + _usage(relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, + fingerprint_ids=frozenset({"fp1"})), + _usage(relation, "day", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, + fingerprint_ids=frozenset({"fp1"})), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert [p.code for p in proposals] == ["ADV008"] + assert proposals[0].evidence["columns"] == ("tenant_id", "day") + assert proposals[0].ddl == 'CREATE INDEX ON "public"."events" ("tenant_id", "day");' + + +def test_adv008_never_reaches_high_confidence(): + """Whether the planner picks GroupAggregate over HashAggregate is not visible to us.""" + relation = Relation("public", "events") + usage = ( + _usage(relation, "tenant_id", ColumnRole.GROUP, cost_share=0.9, cost_ms=900.0, + fingerprint_ids=frozenset({"fp1"})), + ) + facts = {relation: _facts(relation, rows=5_000_000, ndv={"tenant_id": 100_000.0})} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.MEDIUM + + +def test_adv008_groups_only_columns_that_co_occur_in_one_query(): + """Two GROUP BYs in two different queries are not one composite index.""" + relation = Relation("public", "events") + usage = ( + _usage(relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, + fingerprint_ids=frozenset({"fp1"})), + _usage(relation, "day", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, + fingerprint_ids=frozenset({"fp2"})), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["columns"] == ("tenant_id",) + + +def test_adv008_is_silent_when_an_index_already_leads_with_the_grouping_columns(): + relation = Relation("public", "events") + usage = ( + _usage(relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, + fingerprint_ids=frozenset({"fp1"})), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + existing = { + relation: ( + PgIndex(name="idx_events_tenant", columns=("tenant_id", "day"), is_unique=False, + is_primary=False, scans=3, size_bytes=1), + ) + } + assert propose_grouping_indexes(usage, facts, existing, min_cost_share=0.01) == [] + + +def test_adv008_respects_max_arity(): + relation = Relation("public", "events") + usage = tuple( + _usage(relation, name, ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0 - i, + fingerprint_ids=frozenset({"fp1"})) + for i, name in enumerate(["a", "b", "c", "d"]) + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["columns"] == ("a", "b", "c") + + +def test_adv008_discloses_that_the_column_order_is_inferred(): + relation = Relation("public", "events") + usage = ( + _usage(relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, + fingerprint_ids=frozenset({"fp1"})), + _usage(relation, "day", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, + fingerprint_ids=frozenset({"fp1"})), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert "inferred" in proposals[0].rationale.lower() + + +def test_adv008_ignores_non_group_roles(): + relation = Relation("public", "orders") + usage = (_usage(relation, "status", ColumnRole.EQUALITY, cost_share=0.9),) + facts = {relation: _facts(relation, rows=5_000_000)} + assert propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) == [] +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_workload_rules.py -k adv008 -q` + +Expected: FAIL — `NameError: name 'propose_grouping_indexes' is not defined`. + +- [ ] **Step 3: Implement `propose_grouping_indexes`** + +Place it after `propose_join_keys`. The co-occurrence rule: take the hottest GROUP column as +the seed, then extend the composite only with GROUP columns sharing at least one fingerprint +with the seed, up to `max_arity`. + +```python +def propose_grouping_indexes( + usage: Sequence[ColumnUsage], + facts: Mapping[Relation, TableFacts], + existing: Mapping[Relation, Sequence[PgIndex]], + *, + min_cost_share: float, + min_rows: int = MIN_ROWS_FOR_INDEX, + max_arity: int = MAX_INDEX_ARITY, + have_index_data: bool = True, +) -> list[Proposal]: + """ADV008 — an index that can feed a hot GROUP BY already sorted. + + Confidence is capped at MEDIUM and there is deliberately no HIGH branch. Whether + Postgres uses such an index depends on its choice between `GroupAggregate` (which wants + sorted input, and is what the index provides) and `HashAggregate` (which does not) — a + decision driven by `work_mem`, the number of groups and the aggregates involved, none of + which this tool can see. Claiming HIGH would be asserting something about the planner + rather than about the catalog. Do not add a HIGH branch here for symmetry with ADV001. + + One composite rather than several single-column indexes, unlike ADV007: `GROUP BY a, b` + wants input ordered by `(a, b)`, which two separate indexes cannot provide. The column + *order* is inferred from cost, not read from the query — redaction and fingerprinting do + not preserve each column's position in the GROUP BY clause — and the rationale says so, + because getting the order wrong makes the index serve only its leading column. + """ + proposals: list[Proposal] = [] + for relation, items in sorted(_by_relation(usage).items()): + table_facts = facts.get(relation) + rows = table_facts.row_estimate if table_facts else None + if rows is not None and rows < min_rows: + continue + grouping = sorted( + (i for i in items if i.role is ColumnRole.GROUP), + key=lambda i: (-i.cost_ms, i.column), + ) + if not grouping: + continue + seed = grouping[0] + # Extend the composite only with columns some single query groups by *alongside* the + # seed. Without this, two unrelated GROUP BYs on the same table are welded into one + # composite index that serves neither beyond its leading column. + chosen = [seed] + for candidate in grouping[1:]: + if len(chosen) >= max_arity: + break + if candidate.fingerprint_ids & seed.fingerprint_ids: + chosen.append(candidate) + + cost_share = max(i.cost_share for i in chosen) + if cost_share < min_cost_share: + continue + columns = tuple(i.column for i in chosen) + if _covered(columns, existing.get(relation, ())) is not None: + continue + + rationale = ( + "This grouping carries a hot share of workload cost. An index on these columns " + "lets the planner read the rows already ordered and group them without a sort. " + "The column order here is inferred from cost, not read from the query — " + "redaction does not preserve each column's position in the GROUP BY — so check " + "it against the actual grouping before applying, since a composite index only " + "serves the grouping it leads with." + ) + if not have_index_data: + rationale += ( + " The existing-index list could not be read, so whether an index already " + "leads with these columns is unknown." + ) + if rows is None: + rationale += _UNKNOWN_ROWS_NOTE + + proposals.append( + Proposal( + code="ADV008", + title=f"Add index for GROUP BY on {relation}({', '.join(columns)})", + rationale=rationale, + evidence={ + "schema": relation.schema, + "table": relation.table, + "columns": columns, + "roles": tuple(i.role.value for i in chosen), + "cost_share": cost_share, + "calls": max(i.calls for i in chosen), + "fingerprints": max(i.fingerprints for i in chosen), + "row_estimate": rows, + }, + confidence=( + Confidence.LOW if rows is None or not have_index_data else Confidence.MEDIUM + ), + ddl=( + f"CREATE INDEX ON {_qualified(relation.schema, relation.table)} " + f"({', '.join(_quote_ident(c) for c in columns)});" + ), + ) + ) + return proposals +``` + +- [ ] **Step 4: Wire it into `propose`** + +Add `*propose_grouping_indexes(aggregation.usage, facts, existing, +min_cost_share=min_cost_share, have_index_data=have_index_data),` after the +`propose_join_keys(...)` entry. + +- [ ] **Step 5: Run the tests** + +Run: `uv run pytest tests/test_workload_rules.py -q` then `uv run pytest -q` + +Expected: PASS. + +- [ ] **Step 6: Prove the co-occurrence test discriminates** + +Change the extension condition to unconditional (`chosen.append(candidate)` with no +fingerprint check). Run `tests/test_workload_rules.py -k adv008`. Expected: +`test_adv008_groups_only_columns_that_co_occur_in_one_query` FAILS with +`("tenant_id", "day") != ("tenant_id",)`. Restore. Report the mutation. + +- [ ] **Step 7: Update help text, README** + +Add ADV008 to `--min-cost-share`'s enumerated cost-weighted rules and to the README rule +table. + +- [ ] **Step 8: Commit** + +```bash +git add src/sqlquality/workload/postgres.py src/sqlquality/cli.py README.md \ + tests/test_workload_rules.py +git commit -m "feat(advise): ADV008 -- index to serve a hot GROUP BY" +``` + +--- + +### Task 8: restore the `_dedupe_by_ddl` tie-break the new rules make reachable + +**Files:** +- Modify: `src/sqlquality/workload/postgres.py` (`_dedupe_by_ddl` and its docstring) +- Test: `tests/test_workload_postgres.py` + +**Interfaces:** +- Consumes: ADV007 and ADV008 (Tasks 6-7). +- Produces: `_dedupe_by_ddl` with a deterministic tie-break; `_CODE_PREFERENCE` mapping. + +**Why this task exists.** `_dedupe_by_ddl`'s docstring currently argues at length that a +tie-break is unnecessary and was deliberately deleted: + +> That preference needs no tie-break rule to state it: the two codes cannot tie. ADV002 is +> hardcoded MEDIUM ... and ADV003 is hardcoded HIGH ... A tie-break that cannot be reached +> is worse than none. + +That reasoning was sound when the only colliding pair was ADV002/ADV003. Tasks 6-7 break +it: ADV001, ADV007 and ADV008 all emit `CREATE INDEX ON ();`, and their +confidences overlap — ADV001 MEDIUM (NDV unknown) and ADV008 MEDIUM (rows known) can +produce byte-identical DDL at the same confidence. `best[proposal.ddl] is p` then keeps +whichever the list order happened to put first, which is stable today only because +`propose` hardcodes the call order. That is a coincidence, not a rule, and it decides which +rationale the operator reads. + +- [ ] **Step 1: Write the failing test** + +```python +def test_identical_ddl_at_equal_confidence_resolves_by_code_preference(): + """ADV001 and ADV008 can emit byte-identical DDL at the same confidence.""" + ddl = 'CREATE INDEX ON "public"."events" ("tenant_id");' + adv008 = Proposal(code="ADV008", title="group", rationale="g", + evidence={"cost_share": 0.5}, confidence=Confidence.MEDIUM, ddl=ddl) + adv001 = Proposal(code="ADV001", title="filter", rationale="f", + evidence={"cost_share": 0.5}, confidence=Confidence.MEDIUM, ddl=ddl) + # Both orderings must pick the same winner, or list order is deciding. + assert [p.code for p in PostgresWorkloadAdapter._dedupe_by_ddl([adv008, adv001])] == ["ADV001"] + assert [p.code for p in PostgresWorkloadAdapter._dedupe_by_ddl([adv001, adv008])] == ["ADV001"] + + +def test_confidence_still_beats_code_preference(): + ddl = 'CREATE INDEX ON "public"."events" ("tenant_id");' + adv001_low = Proposal(code="ADV001", title="filter", rationale="f", + evidence={"cost_share": 0.5}, confidence=Confidence.LOW, ddl=ddl) + adv008_med = Proposal(code="ADV008", title="group", rationale="g", + evidence={"cost_share": 0.5}, confidence=Confidence.MEDIUM, ddl=ddl) + assert [p.code for p in PostgresWorkloadAdapter._dedupe_by_ddl([adv001_low, adv008_med])] == [ + "ADV008" + ] + + +def test_every_ddl_emitting_code_has_a_preference_rank(): + """A code missing from the map would raise KeyError mid-run, after all the analysis.""" + ddl_codes = {"ADV001", "ADV002", "ADV003", "ADV004", "ADV007", "ADV008"} + assert ddl_codes <= set(PostgresWorkloadAdapter._CODE_PREFERENCE) +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `uv run pytest tests/test_workload_postgres.py -k dedupe -q` plus the new names. + +Expected: FAIL — `AttributeError: _CODE_PREFERENCE`, and the both-orderings assertion fails +because list order decides today. + +- [ ] **Step 3: Add the tie-break** + +```python + #: Which rule's rationale to keep when two rules propose byte-identical DDL at equal + #: confidence. Lower wins. The order is by how directly the evidence supports *this* + #: index: a filter predicate (ADV001) is the most direct reason to build a B-tree, a + #: join key (ADV007) next, and a grouping (ADV008) last, since whether the planner uses + #: an index for grouping depends on choices this tool cannot see. The DROP rules are + #: ranked below them so a CREATE never loses to a DROP that happens to render the same + #: text — which it cannot today, but this map is the place that would have to change. + _CODE_PREFERENCE = { + "ADV001": 0, + "ADV007": 1, + "ADV004": 2, + "ADV008": 3, + "ADV003": 4, + "ADV002": 5, + } +``` + +and use it as the second element of the comparison, replacing the `is`-identity dance with +an explicit key so both the winner and the filter agree: + +```python + @classmethod + def _dedupe_by_ddl(cls, proposals: list[Proposal]) -> list[Proposal]: + """Collapse proposals that would run identical DDL, keeping the strongest evidence. + + Two rules can genuinely reach the same index from different evidence — a filter + predicate, a join key and a grouping on the same column all render the same + `CREATE INDEX` — and an unused index that is also a prefix of a wider one is flagged + by both ADV002 and ADV003 as the same `DROP INDEX`. They do not contradict each + other, but a reader should not have to notice they are the same object twice. + + Confidence decides first. When it ties, `_CODE_PREFERENCE` decides, because + something has to and list order must not: the losing proposal's rationale is + discarded, so "whichever `propose()` happened to append first" is not an acceptable + answer to which explanation the operator reads. + + There was a window where no tie was reachable — ADV002 is hardcoded MEDIUM and + ADV003 HIGH, the only colliding pair at the time — and the tie-break was removed as + unreachable code. ADV007 and ADV008 made it reachable again: ADV001 at MEDIUM (NDV + unknown) and ADV008 at MEDIUM (row count known) produce byte-identical DDL at equal + confidence. + """ + def rank(proposal: Proposal) -> tuple[int, int]: + return ( + cls._CONFIDENCE_ORDER[proposal.confidence], + cls._CODE_PREFERENCE.get(proposal.code, len(cls._CODE_PREFERENCE)), + ) + + best: dict[str, Proposal] = {} + for proposal in proposals: + if not proposal.ddl: + continue + incumbent = best.get(proposal.ddl) + if incumbent is None or rank(proposal) < rank(incumbent): + best[proposal.ddl] = proposal + return [p for p in proposals if not p.ddl or best[p.ddl] is p] +``` + +`.get(..., len(...))` rather than `[...]`: an unranked future code sorts last instead of +raising `KeyError` after the whole analysis has run. The test above is what keeps the map +complete for the codes that exist. + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest tests/test_workload_postgres.py -q` then `uv run pytest -q` + +Expected: PASS. + +- [ ] **Step 5: Prove the both-orderings assertion discriminates** + +Remove the `_CODE_PREFERENCE` element from `rank`, leaving only confidence. Run the test. +Expected: FAIL on the second assertion (`["ADV008"] != ["ADV001"]`) while the first still +passes — which is precisely the list-order dependence this task removes. Restore. Report +the mutation and note that a test asserting only one ordering would have passed against the +broken code. + +- [ ] **Step 6: Commit** + +```bash +git add src/sqlquality/workload/postgres.py tests/test_workload_postgres.py +git commit -m "fix(advise): break identical-DDL ties by rule, not by list order" +``` + +--- + +## Phase B — stop discarding wrapped reads + +### Task 9: unwrap `DECLARE ... CURSOR FOR` and `COPY (...) TO` + +**Files:** +- Modify: `src/sqlquality/workload/fingerprint.py` +- Modify: `src/sqlquality/cli.py` (the `_coverage_line` docstring, which documents this as a + known wart) +- Test: `tests/test_workload_fingerprint.py` + +**Interfaces:** +- Consumes: nothing from Phases C/A — this task is independent. +- Produces: `unwrap(sql: str) -> str` in `fingerprint.py`, applied in `ingest` **before** + `is_noise`. + +**What the engine actually hands us.** Verified against sqlglot 30.12: + +| statement | parses as | notes | +|---|---|---| +| `DECLARE c CURSOR FOR SELECT ...` | `exp.Command` (`this='DECLARE'`) | "unsupported syntax", falls back to `Command`; the tail is a **string literal**, so the inner query is not in the AST | +| `COPY (SELECT ...) TO STDOUT` | `exp.Copy` (`this=Subquery`, `kind=False`) | inner SELECT **is** in the AST | +| `COPY orders TO STDOUT` | `exp.Copy` (`this=Table`, `kind=False`) | whole-table dump, no predicates | +| `COPY orders (id) FROM STDIN` | `exp.Copy` (`this=Schema`, `kind=True`) | a write | +| `FETCH 100 FROM c` | `exp.Command` | no query text at all | + +So `DECLARE` needs text extraction and `COPY` can be done on the AST — but doing both on +the raw string, before parsing, keeps one code path and lets `is_noise` run on the unwrapped +text. `FETCH` and `CLOSE` stay noise: they carry no query. + +Every psycopg2 server-side cursor (`cursor(name=...)`) emits `DECLARE`, so on a Django or +SQLAlchemy workload this is not an edge case — it can be the majority of the read traffic, +and today all of it is counted as "filtered" and thrown away. + +- [ ] **Step 1: Write the failing tests** + +```python +import pytest + +from sqlquality.models import RawQueryRow, WorkloadFetch +from sqlquality.workload.fingerprint import ingest, is_noise, unwrap + + +@pytest.mark.parametrize( + "sql,expected", + [ + ( + "DECLARE c CURSOR FOR SELECT id FROM orders WHERE status = 'x'", + "SELECT id FROM orders WHERE status = 'x'", + ), + ( + "DECLARE c CURSOR WITH HOLD FOR SELECT id FROM orders", + "SELECT id FROM orders", + ), + ( + "DECLARE c NO SCROLL CURSOR FOR SELECT id FROM orders", + "SELECT id FROM orders", + ), + ( + "DECLARE c BINARY INSENSITIVE SCROLL CURSOR WITH HOLD FOR SELECT a FROM t", + "SELECT a FROM t", + ), + ( + 'DECLARE "my cursor" CURSOR FOR SELECT a FROM t', + "SELECT a FROM t", + ), + ( + "COPY (SELECT id FROM orders WHERE status = 'x') TO STDOUT", + "SELECT id FROM orders WHERE status = 'x'", + ), + ("copy (select 1) to stdout", "select 1"), + ], +) +def test_unwrap_recovers_the_inner_query(sql, expected): + assert unwrap(sql) == expected + + +@pytest.mark.parametrize( + "sql", + [ + "SELECT id FROM orders", + "COPY orders TO STDOUT", + "COPY orders (id, status) FROM STDIN", + "FETCH 100 FROM c", + "CLOSE c", + "DECLARE c CURSOR FOR", + "DECLARE", + ], +) +def test_unwrap_leaves_everything_else_alone(sql): + """Anything without a recoverable inner query is returned unchanged, not mangled.""" + assert unwrap(sql) == sql + + +def test_a_declared_cursor_is_analyzed_not_filtered(): + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="DECLARE c CURSOR FOR SELECT id FROM orders WHERE status = 'x'", + calls=3, + total_time_ms=300.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 0 + assert len(workload.stats) == 1 + assert "DECLARE" not in workload.stats[0].sql.upper() + assert workload.stats[0].calls == 3 + + +def test_a_copy_subquery_is_analyzed_not_filtered(): + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="COPY (SELECT id FROM orders WHERE status = 'x') TO STDOUT", + calls=1, + total_time_ms=10.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 0 + assert len(workload.stats) == 1 + + +def test_a_declared_cursor_over_introspection_is_still_filtered(): + """Unwrapping must not become a way to smuggle our own catalog reads into the workload.""" + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="DECLARE c CURSOR FOR SELECT * FROM pg_stat_statements", + calls=1, + total_time_ms=1.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 1 + assert workload.stats == () + + +def test_a_whole_table_copy_is_still_filtered(): + fetch = WorkloadFetch( + rows=(RawQueryRow(sql="COPY orders TO STDOUT", calls=1, total_time_ms=1.0),), + window_description="w", + ) + assert ingest(fetch, "postgres").skipped_noise == 1 + + +def test_the_unwrapped_query_is_still_redacted(): + """Redaction runs after unwrapping, so the inner literal must not survive.""" + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="DECLARE c CURSOR FOR SELECT id FROM orders WHERE email = 'a@b.test'", + calls=1, + total_time_ms=1.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert "a@b.test" not in workload.stats[0].sql + assert "a@b.test" not in workload.stats[0].fingerprint +``` + +That last test is the one that matters most: unwrapping introduces a **new path by which +raw user SQL reaches the pipeline**, and the redaction guarantee has to hold on it. + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_workload_fingerprint.py -x -q` + +Expected: FAIL — `ImportError: cannot import name 'unwrap'`. + +- [ ] **Step 3: Implement `unwrap`** + +Add to `fingerprint.py`, above `is_noise`: + +```python +#: `DECLARE [BINARY] [ASENSITIVE|INSENSITIVE] [[NO] SCROLL] CURSOR +#: [WITH|WITHOUT HOLD] FOR ` — the full PostgreSQL grammar for the statement every +#: psycopg2 server-side cursor emits. +#: +#: Anchored on `CURSOR ... FOR` rather than on the first `FOR`, because a cursor name is an +#: identifier and a quoted one may contain the word: `DECLARE "for sale" CURSOR FOR ...` +#: would otherwise be cut at the wrong place and yield unparseable text. The name alternative +#: matches a quoted identifier (with doubled quotes escaped) before an unquoted one for the +#: same reason. +_DECLARE_CURSOR = re.compile( + r"^\s*DECLARE\s+" + r'(?:"(?:[^"]|"")*"|[A-Za-z_]\w*)\s+' + r"(?:BINARY\s+)?" + r"(?:ASENSITIVE\s+|INSENSITIVE\s+)?" + r"(?:NO\s+SCROLL\s+|SCROLL\s+)?" + r"CURSOR\s+" + r"(?:WITH\s+HOLD\s+|WITHOUT\s+HOLD\s+)?" + r"FOR\s+(?P\S.*)$", + re.IGNORECASE | re.DOTALL, +) + +#: `COPY ( ) TO ...` — the only COPY form carrying predicates worth analysing. +#: `COPY TO` is a whole-relation dump with no predicates, and `COPY ... FROM` is a +#: write; both stay noise. The capture is greedy to the last `)` so a query containing +#: parentheses survives; the result is validated by the caller's parse, so a mis-cut yields +#: an unparseable count rather than a wrong analysis. +_COPY_QUERY = re.compile( + r"^\s*COPY\s*\(\s*(?P.*)\s*\)\s*TO\b", + re.IGNORECASE | re.DOTALL, +) + + +def unwrap(sql: str) -> str: + """The inner query of a cursor declaration or `COPY (...) TO`, else ``sql`` unchanged. + + `DECLARE ... CURSOR FOR SELECT ...` and `COPY (SELECT ...) TO ...` are ordinary reads + with real predicates, but both begin with a keyword the noise filter drops — so on any + workload using server-side cursors (every psycopg2 `cursor(name=...)`, which is what + Django and SQLAlchemy emit for large result sets) the hottest reads were counted as + "filtered" and thrown away. + + Text surgery rather than AST surgery, for a reason that is not a preference: sqlglot + cannot parse `DECLARE` at all — it falls back to `exp.Command` and leaves the entire + tail as a single string literal, so there is no inner tree to lift. `COPY` *does* parse + (to `exp.Copy` with a `Subquery`), but doing both here keeps one code path and, more + importantly, lets `is_noise` run on the *unwrapped* text — which is what stops a + `DECLARE c CURSOR FOR SELECT * FROM pg_stat_statements` from smuggling our own + introspection into the analysed workload. + + Returns the input unchanged when nothing matches. The caller parses the result, so a + partial or malformed wrapper degrades to the pre-existing behaviour — counted + unparseable or filtered — rather than producing a wrong analysis. + """ + for pattern in (_DECLARE_CURSOR, _COPY_QUERY): + match = pattern.match(sql) + if match is not None: + return match.group("query").strip() + return sql +``` + +- [ ] **Step 4: Apply it in `ingest`** + +Change the loop head so unwrapping happens before the noise test: + +```python + for row in fetch.rows: + # Unwrap *before* the noise test, so a cursor declaration is judged on the query it + # declares. Judging the wrapper drops the read; judging the inner query keeps a real + # read and still filters an inner introspection query. + sql = unwrap(row.sql) + if is_noise(sql): + skipped_noise += 1 + continue + try: + tree = parse(sql, dialect) + except SqlParseError: + skipped_unparseable += 1 + continue +``` + +- [ ] **Step 5: Run the tests** + +Run: `uv run pytest tests/test_workload_fingerprint.py -q` then +`uv run pytest tests/test_workload_redaction.py -q` + +Expected: PASS. The redaction suite is called out separately because this task widens what +reaches the redactor. + +- [ ] **Step 6: Prove the ordering of unwrap and is_noise matters** + +Move the `sql = unwrap(row.sql)` line to *after* the `is_noise(row.sql)` check (testing the +raw string, unwrapping only what survives). Run `tests/test_workload_fingerprint.py`. +Expected: `test_a_declared_cursor_is_analyzed_not_filtered` FAILS with +`assert 1 == 0` on `skipped_noise`. Restore. + +Then run the opposite mutation: delete the `_INTROSPECTION` half of `is_noise`'s return +(`return bool(_LEADING_NOISE.match(sql))`). Expected: +`test_a_declared_cursor_over_introspection_is_still_filtered` FAILS. Restore. Report both. + +- [ ] **Step 7: Correct the `_coverage_line` docstring** + +`src/sqlquality/cli.py`'s `_coverage_line` docstring currently documents this exact wart as +a live limitation: + +> The filter is a statement-prefix match, so it also swallows `DECLARE cur CURSOR FOR +> SELECT ...` and `COPY (SELECT ...) TO STDOUT` — ordinary reads with real predicates ... + +Rewrite that paragraph: those two forms are now unwrapped and analysed, and "filtered" +counts session control, DDL, maintenance, introspection, whole-table `COPY`, and cursor +statements that carry no query (`FETCH`, `CLOSE`). Leave the "N of M" reasoning intact. + +- [ ] **Step 8: Commit** + +```bash +git add src/sqlquality/workload/fingerprint.py src/sqlquality/cli.py \ + tests/test_workload_fingerprint.py +git commit -m "fix(advise): analyse cursor and COPY-subquery reads instead of filtering them" +``` + +--- + +### Task 10: prove all three phases against a live Postgres + +**Files:** +- Modify: `tests/integration/conftest.py` (seed a second schema) +- Modify: `tests/integration/test_advise_live.py` +- Modify: `tests/integration/test_introspection_live.py` +- Modify: `README.md` +- Modify: `docs/superpowers/specs/2026-07-26-advise-workload-analysis-design.md` + +**Interfaces:** +- Consumes: everything above. +- Produces: integration coverage for multi-schema keying, ADV007/ADV008 and wrapper + unwrapping; documentation matching shipped behaviour. + +Batch 1's live suite found two production bugs the 400-test unit suite structurally could +not see (`reltuples = -1` suppressing every proposal; `redact_tree` dismembering `$N` and +silently dropping a whole query group). Both were shape-of-real-data problems. This task +exists because Batch 2 changes the shape of every catalog row it reads. + +- [ ] **Step 1: Seed a second schema and a wrapped read** + +In `tests/integration/conftest.py`'s `seeded` fixture, add alongside the existing setup — +read the fixture first and match its style: + +```sql +CREATE SCHEMA IF NOT EXISTS staging; +CREATE TABLE IF NOT EXISTS staging.orders ( + id bigint, status text, tenant_id bigint, day date +); +INSERT INTO staging.orders +SELECT g, 'draft', g % 7, current_date FROM generate_series(1, 50000) g; +ANALYZE staging.orders; +``` + +so that `orders` exists in **both** `public` and `staging` with different statistics — the +exact collision the old `_validate_schemas` refused to allow. + +Then drive workload that exercises the new paths, and `ANALYZE` both tables so +`reltuples` is not the `-1` sentinel: + +```sql +-- a schema-qualified filter on each side, so both relations get their own usage +SELECT id FROM public.orders WHERE status = 'shipped'; +SELECT id FROM staging.orders WHERE status = 'draft'; +-- a join key with no index (ADV007) +SELECT o.id FROM public.orders o JOIN public.order_items i ON i.order_id = o.id; +-- a hot GROUP BY with no index (ADV008) +SELECT tenant_id, day, count(*) FROM staging.orders GROUP BY tenant_id, day; +-- a server-side cursor: filtered before Task 9, analysed after +DECLARE live_cur CURSOR FOR SELECT id FROM public.orders WHERE status = 'pending'; +FETCH 10 FROM live_cur; +CLOSE live_cur; +``` + +- [ ] **Step 2: Write the failing live tests** + +```python +@pytest.mark.integration +def test_two_same_named_tables_keep_their_own_row_estimates(seeded): + """The aliasing bug, against real catalog rows rather than canned ones.""" + adapter = PostgresWorkloadAdapter() + adapter.connect(seeded, timeout_s=30) + adapter.schemas = ("public", "staging") + facts = adapter.fetch_table_facts( + ("public", "staging"), + frozenset({Relation("public", "orders"), Relation("staging", "orders")}), + ) + public_rows = facts[Relation("public", "orders")].row_estimate + staging_rows = facts[Relation("staging", "orders")].row_estimate + assert public_rows is not None and public_rows > 0 + assert staging_rows is not None and staging_rows > 0 + assert public_rows != staging_rows, "both relations reported the same estimate" + + +@pytest.mark.integration +def test_multi_schema_advise_run_produces_qualified_proposals(seeded, tmp_path): + result = runner.invoke( + app, + ["advise", "--dsn", seeded_dsn(seeded), "--schema", "public", "--schema", "staging", + "--json", "--min-cost-share", "0.0"], + ) + assert result.exit_code == 0 + payload = json.loads(result.stdout) + schemas = {p["evidence"].get("schema") for p in payload["proposals"]} + assert "staging" in schemas + # Every DDL statement names the schema it belongs to. + for proposal in payload["proposals"]: + if proposal["ddl"]: + assert f'"{proposal["evidence"]["schema"]}".' in proposal["ddl"] + + +@pytest.mark.integration +def test_a_declared_cursor_reaches_the_analysis(seeded): + """DECLARE is what psycopg2 server-side cursors emit; before Task 9 it was discarded.""" + adapter = PostgresWorkloadAdapter() + adapter.connect(seeded, timeout_s=30) + fetch = adapter.fetch_workload(None, 500) + assert any(row.sql.upper().startswith("DECLARE") for row in fetch.rows), ( + "the seeded cursor never reached pg_stat_statements — fixture problem, not a bug" + ) + workload = ingest(fetch, "postgres") + assert not any(s.sql.upper().startswith("DECLARE") for s in workload.stats) + assert any("pending" not in s.sql and "orders" in s.sql for s in workload.stats) + + +@pytest.mark.integration +def test_the_new_rules_fire_on_a_real_workload(seeded): + codes = {p.code for p in _run_advise(seeded, schemas=("public", "staging"))} + assert "ADV007" in codes or "ADV008" in codes, ( + f"neither join-key nor grouping rule fired; got {sorted(codes)}" + ) +``` + +The non-vacuity guard in the third test is deliberate: if the fixture's `DECLARE` never +lands in `pg_stat_statements` the test would otherwise pass while proving nothing, which is +how a hollow test survives. Follow the same pattern for any test you add here. + +- [ ] **Step 3: Run the integration suite** + +```bash +docker compose -f tests/integration/docker-compose.yml up -d +sleep 8 +uv run pytest -m integration -q +docker compose -f tests/integration/docker-compose.yml down +``` + +Expected: PASS. If any test fails, the live behaviour is the source of truth — fix the +production code, not the assertion, and report what real Postgres did differently. + +- [ ] **Step 4: Confirm the default suite still needs neither Docker nor extras** + +```bash +docker compose -f tests/integration/docker-compose.yml down +uv run pytest -q +``` + +Expected: `N passed, M deselected` — **no skips**. If anything skips, the marker or the +guard is wrong. + +- [ ] **Step 5: Update the docs** + +- `README.md`: multi-schema `--schema` usage with the ambiguity caveat; ADV007 and ADV008 in + the rule table; remove the "wrapped reads are filtered" limitation if it is stated there. +- The design spec's deviations section: record that bare-name-plus-schema-map resolution was + chosen over reading `Table.db`, and why (`qualify()` leaves `db` empty for bare + references, so `Table.db` alone keys everything under `schema=""`). +- Check the spec and README for any surviving claim that only one schema is supported. + +- [ ] **Step 6: Run all four gates** + +```bash +uv run ruff check . && uv run ruff format --check . && \ + uv run mypy src/sqlquality && uv run pytest -q +``` + +- [ ] **Step 7: Commit** + +```bash +git add tests/integration README.md docs/superpowers/specs/2026-07-26-advise-workload-analysis-design.md +git commit -m "test(integration): multi-schema, join/group rules and cursor reads against real postgres" +``` + +--- + +## Self-Review + +**Spec coverage.** The three items the user approved for Batch 2: + +| Item | Tasks | +|---|---| +| join-key and grouping-column proposals | 6 (ADV007), 7 (ADV008), 8 (the tie-break they make reachable) | +| `DECLARE`/`COPY` unwrapping | 9 | +| multi-schema `(schema, table)` keying | 1, 2, 3, 4, 5 | +| proof against real data + docs | 10 | + +**Known ripple this plan accepts deliberately.** Task 1 leaves the suite red for three test +modules that Tasks 2-4 then fix. The alternative — one giant task threading `Relation` +through every layer at once — is not reviewable. Each task's own tests pass at its own +commit, and Task 1's step 8 says so explicitly so an implementer does not try to fix +`test_workload_rules.py` out of turn. + +**Ordering rationale.** Phase C is first because Phases A and B would otherwise be written +against the bare-name model and rewritten immediately. Phase B is genuinely independent and +could run at any point; it is last because it is the smallest. + +**Type consistency.** `Relation` is introduced in Task 1 and used with the same field names +(`schema`, `table`) and the same `__str__` contract in Tasks 2-10. `TableFacts.name` → +`TableFacts.relation` happens once, in Task 3, which is also where every `fetch_*` return +type changes. `propose_join_keys` and `propose_grouping_indexes` take the parameter names +Task 4 establishes for the existing rules (`usage`, `facts`, `existing`, `min_cost_share`, +`min_rows`, `have_index_data`), minus the `schema=` keyword Task 4 removes. + +**Carried forward from Batch 1** (surfaced to the user, not implemented here): `ci.yml` +runs `uv sync --all-extras`, so no CI job can catch a regression of the psycopg guard that +keeps the default suite Docker-free and extra-free. Global Constraints restate the +invariant; a permanent guard remains a separate follow-up. diff --git a/docs/superpowers/specs/2026-07-26-advise-workload-analysis-design.md b/docs/superpowers/specs/2026-07-26-advise-workload-analysis-design.md index 5b62737..72e3654 100644 --- a/docs/superpowers/specs/2026-07-26-advise-workload-analysis-design.md +++ b/docs/superpowers/specs/2026-07-26-advise-workload-analysis-design.md @@ -1,10 +1,13 @@ # Design: `sqlquality advise` — workload-driven database optimization Date: 2026-07-26 -Status: Postgres (steps 1–4 below) shipped in `sqlquality advise`; Redshift, Snowflake and -dbt enrichment (steps 5–7) remain design-only, not yet implemented. See +Status: Postgres (steps 1–4 below) shipped in `sqlquality advise`, now including ADV007 +(join keys), ADV008 (`GROUP BY`), multi-schema `(schema, table)` keying, and +`DECLARE`/`COPY` unwrapping (Batch 2, 2026-07-27); Redshift, Snowflake and dbt enrichment +(steps 5–7) remain design-only, not yet implemented. See `docs/superpowers/plans/2026-07-26-advise-postgres.md` for the implementation plan and its -"Deviations from the spec" section, reconciled into this document below. +"Deviations from the spec" section, reconciled into this document below, and "Deviations +from the spec (Batch 2)" further down for what changed after the initial ship. ## Summary @@ -104,16 +107,21 @@ class WorkloadAdapter(ABC): redacted — redaction happens once, in the engine-agnostic `ingest()`, so there is exactly one place to audit for literal leakage instead of one per adapter.""" - def fetch_schema(self, tables: set[str]) -> dict: - """Schema mapping for sqlglot qualify().""" + def fetch_schema(self, schemas: tuple[str, ...]) -> dict: + """Nested schema mapping for sqlglot qualify(): {schema: {table: {column: type}}}. + Nested, not flat, so qualify() can tell two same-named tables in different + schemas apart — see "Deviations from the spec (Batch 2)" below.""" - def fetch_table_facts(self, tables: set[str]) -> dict[str, TableFacts]: - """Row estimates, sizes, per-column NDV, current physical design.""" + def fetch_table_facts( + self, schemas: tuple[str, ...], relations: frozenset[Relation] + ) -> dict[Relation, TableFacts]: + """Row estimates, sizes, per-column NDV, current physical design, keyed by the + schema-qualified relation each row belongs to.""" def propose( self, aggregation: Aggregation, - facts: dict[str, TableFacts], + facts: dict[Relation, TableFacts], workload: Workload, *, min_cost_share: float, @@ -136,6 +144,18 @@ class WorkloadAdapter(ABC): Added to `models.py` alongside the existing `Finding` / `ComplexityScore`: ```python +@dataclass(frozen=True, order=True) +class Relation: + """A schema-qualified relation — the key every catalog fact is stored under. + Bare table names aliased: two schemas each holding an `orders` merged into one entry, + so the last catalog row read won the row estimate. `order=True` so rules can sort + their output for stable report ordering.""" + schema: str + table: str + + def __str__(self) -> str: + return f"{self.schema}.{self.table}" + @dataclass(frozen=True) class ConnectionParams: engine: str # postgres | redshift | snowflake @@ -192,17 +212,20 @@ class ColumnRole(str, Enum): @dataclass(frozen=True) class ColumnUsage: - table: str + relation: Relation column: str role: ColumnRole calls: int cost_ms: float cost_share: float # fraction of total analyzed workload cost — NOT a partition, # see the "cost_share is not a partition" note in the README - fingerprints: int fingerprint_ids: frozenset[str] = frozenset() # which query groups contributed this # usage, so rules can test co-occurrence + @property + def fingerprints(self) -> int: + return len(self.fingerprint_ids) + @dataclass(frozen=True) class Aggregation: """Produced by `aggregate()`: the rolled-up usage index plus what could not be used.""" @@ -211,11 +234,16 @@ class Aggregation: skipped_unqualifiable: int # queries that failed qualify() — lives here, not on # Workload, because qualification happens during # aggregation, not ingest - tables: frozenset[str] + tables: frozenset[Relation] + skipped_ambiguous: int = 0 # a bare table name held by 2+ introspected schemas, + # named without qualification — attributing it would be a + # coin flip, so it is counted and dropped instead @dataclass(frozen=True) class TableFacts: - name: str + relation: Relation # the schema-qualified key this table is stored under — not + # a display name; two same-named tables in different + # schemas each get their own TableFacts row_estimate: int | None size_bytes: int | None columns: tuple[str, ...] @@ -356,6 +384,8 @@ Workload from `pg_stat_statements` (`queryid`, `query`, `calls`, `total_exec_tim | ADV004 | Partial index for a hot fingerprint carrying a constant structural predicate | fingerprint count, cost share | | ADV005 | Non-sargable hot predicate (`lower(col) =`, casts, leading wildcard) → rewrite or expression index | cost share | | ADV006 | Hot `SELECT *` on a wide table | column count, cost share | +| ADV007 | Add index on a hot join key with no existing index leading with it | cost share, NDV, row estimate, absence of a covering index | +| ADV008 | Composite index for a hot `GROUP BY`, column order inferred from cost, capped at MEDIUM | cost share, row estimate, absence of a covering index | Proposals are suppressed entirely for tables below a row-count floor, where a sequential scan is the correct plan and an index would be pure overhead. @@ -365,6 +395,70 @@ per-statement timestamps before PostgreSQL 17 added `stats_since`. On earlier ve `--since` cannot be honored: the report states the window as "since stats reset at ``" rather than implying the requested window was applied. +## Deviations from the spec (Batch 2) + +Found and agreed while implementing ADV007/ADV008, multi-schema support and wrapped-read +handling, after the initial ship this document otherwise describes. The dataclass shapes +in "Core dataclasses" and "Interface" above already reflect what shipped as a result — +`ColumnUsage.relation: Relation`, `Aggregation.tables: frozenset[Relation]`, +`TableFacts.relation`, `fetch_schema`/`fetch_table_facts`/`fetch_indexes` keyed and +parameterized by `Relation` — not the bare-string shapes (`ColumnUsage.table: str`, +`Aggregation.tables: frozenset[str]`, `TableFacts.name`) that shipped first. This section +records why they changed. + +1. **Every catalog fact is keyed by `Relation(schema, table)`, not a bare table name.** A + bare-name key aliased two same-named tables in different schemas: whichever catalog row + was read last won the row estimate, while `qualify()` resolved columns against the + union of both tables' columns. `ColumnUsage.table` becomes `ColumnUsage.relation`, + `Aggregation.tables` becomes `frozenset[Relation]`, `TableFacts.name` becomes + `TableFacts.relation`, and `fetch_schema`/`fetch_table_facts`/`fetch_indexes` are all + keyed and parameterized by `Relation`. `fetch_schema` also changes shape, from a flat + `{table: {column: type}}` to a nested `{schema: {table: {column: type}}}` — the nesting + is what lets `qualify()` tell two same-named tables apart at all; a flat map resolves a + column against the union of both column sets. +2. **Schema resolution reads the introspected schema map, not `Table.db`.** The obvious + implementation attributes a query's table to `table.db`, the schema sqlglot's `qualify()` + already resolved. That is wrong for the common case: `qualify()` leaves `db` **empty** + for a bare table reference — `SELECT * FROM orders`, not `SELECT * FROM public.orders` + — because production SQL relies on `search_path`, not full qualification, and + `qualify()` has no schema to fill `db` in with. A `Table.db`-only implementation would + therefore key every search_path-reliant workload under `Relation(schema="", ...)`, + silently suppressing every proposal for it — exactly the shape-of-real-data failure + this batch's live suite exists to catch. The shipped resolver (`resolve_relation` in + `workload/extract.py`) instead trusts `table.db` only once it is checked against the + introspected schema map, and falls back to looking the bare table name up in that map: + exactly one introspected schema holding the name resolves unambiguously; more than one + is genuinely ambiguous and is counted (`Aggregation.skipped_ambiguous`) rather than + guessed at; none means the table lives outside the introspected schemas and the column + is dropped. +3. **`advise` accepts repeated `--schema`.** Multiple schemas used to be rejected outright + because the bare-name key could not tell two schemas' same-named tables apart; the + `Relation` keying above is what makes accepting more than one safe. +4. **`DECLARE ... CURSOR FOR` and `COPY (...) TO` reads are unwrapped to their inner query** + before the noise filter runs, rather than filtered as maintenance statements. Both are + ordinary reads with real predicates — `DECLARE` is what every psycopg2 server-side + cursor emits — but both begin with a keyword the noise filter otherwise drops. +5. **The rules are evaluated independently but not *reported* independently.** The spec above + describes eight rules that each report their own findings; in the shipped code a + reconciliation pass runs over the assembled proposal list before the report is written. + Proposals with identical DDL collapse into one, and a proposed index whose columns are a + leading prefix of another proposed index for the same table collapses into the wider one — + without this, ADV001 and ADV007 shipped a `CREATE INDEX` pair on the same table that + ADV003 would flag as redundant on the following run, i.e. the tool contradicting itself + across runs. The absorbed proposal's rationale and confidence are folded into the + survivor's, attributed by code; its `evidence` is discarded. Same column *set* in a + different order is not a prefix relationship and both are kept, each disclosing the other. + Consequence a consumer must know: a rule can fire and contribute no entry to `proposals`. +6. **A composite index proposal requires *joint* support.** ADV001, ADV004 and ADV008 only + combine columns that some single query group uses together, tracked as a running + intersection of contributing fingerprints, and report that joint count as + `co_occurring_fingerprints` instead of a per-column `fingerprints`. Cost weighting alone + is not enough: a proposal's `cost_share` is the *max* over its columns, so a column + carrying ~0% of workload cost could not be filtered out by it, and a near-free query + contributing one column to the middle of a composite produced an index no query could use. + ADV002 and ADV003, the two `DROP INDEX` rules, are likewise both scoped to the relations + the workload was observed using rather than to every relation the catalog query returned. + ### Redshift Workload from `SYS_QUERY_HISTORY` when available, falling back to `STL_QUERY` plus diff --git a/src/sqlquality/cli.py b/src/sqlquality/cli.py index 1ca3a7e..f12b3ca 100644 --- a/src/sqlquality/cli.py +++ b/src/sqlquality/cli.py @@ -26,7 +26,13 @@ from sqlquality.gate import evaluate_gate from sqlquality.linter import fix_sql, lint_sql from sqlquality.llm import Suggestion, enrich_findings, resolve_provider -from sqlquality.models import Aggregation, Severity, Workload, cost_share_of +from sqlquality.models import ( + Aggregation, + Severity, + Workload, + analyzed_query_groups, + cost_share_of, +) from sqlquality.report import ( advise_payload, gate_payload, @@ -607,15 +613,6 @@ def _validate_timeout(value: int) -> int: return value -def _analyzed_count(workload: Workload, aggregation: Aggregation) -> int: - """Query groups whose usage was actually extracted. - - Unresolvable groups are a *subset* of ``workload.stats``, not a separate pool, so - ``len(stats)`` overstates what was understood. - """ - return max(0, len(workload.stats) - aggregation.skipped_unqualifiable) - - def _coverage_line(workload: Workload, aggregation: Aggregation) -> str: """One-line coverage disclosure, printed on every run. @@ -623,17 +620,21 @@ def _coverage_line(workload: Workload, aggregation: Aggregation) -> str: "2 unresolvable" contradicts itself on a single line — and does so least accurately exactly when coverage is worst, which is the situation the line exists to reveal. - ``skipped_noise`` is reported as "filtered", not "introspection/DDL". The filter is a - statement-prefix match, so it also swallows `DECLARE cur CURSOR FOR SELECT ...` and - `COPY (SELECT ...) TO STDOUT` — ordinary reads with real predicates, and what every - psycopg2 server-side cursor emits. Calling those introspection or DDL told the user - their hot reads were maintenance traffic. "filtered" claims only what is true. + ``skipped_noise`` is reported as "filtered", not "introspection/DDL": it covers session + control, DDL, maintenance, introspection, a whole-table `COPY ... TO`, and a cursor + statement that carries no query at all (`FETCH`, `CLOSE`). `DECLARE cur CURSOR FOR + SELECT ...` and `COPY (SELECT ...) TO STDOUT` — what every psycopg2 server-side cursor + emits, and ordinary reads with real predicates — are unwrapped to their inner query + before this count is taken, so they are analysed rather than filtered. Calling + `skipped_noise` "introspection or DDL" would tell the user their hot reads were + maintenance traffic; "filtered" claims only what is true. """ return ( - f"analyzed {_analyzed_count(workload, aggregation)} of {len(workload.stats)} " + f"analyzed {analyzed_query_groups(workload, aggregation)} of {len(workload.stats)} " f"query group(s); skipped {workload.skipped_unparseable} unparseable, " f"{workload.skipped_noise} filtered, " - f"{aggregation.skipped_unqualifiable} unresolvable" + f"{aggregation.skipped_unqualifiable} unresolvable, " + f"{aggregation.skipped_ambiguous} ambiguous" ) @@ -642,10 +643,15 @@ def _coverage_warning(workload: Workload, aggregation: Aggregation) -> str | Non Noise (introspection, DDL, session control) is excluded from the denominator: those are deliberately filtered, not failures to understand. Only statements we tried and failed - to use count against coverage. + to use count against coverage — an ambiguous statement belongs in that sum exactly like + an unresolvable one: both are statements `aggregate()` tried and failed to attribute. """ - analyzed = _analyzed_count(workload, aggregation) - unexplained = workload.skipped_unparseable + aggregation.skipped_unqualifiable + analyzed = analyzed_query_groups(workload, aggregation) + unexplained = ( + workload.skipped_unparseable + + aggregation.skipped_unqualifiable + + aggregation.skipped_ambiguous + ) considered = analyzed + unexplained if not considered: return None @@ -655,34 +661,42 @@ def _coverage_warning(workload: Workload, aggregation: Aggregation) -> str | Non return ( f"low coverage: {share:.0%} of candidate statements could not be analyzed " f"({workload.skipped_unparseable} unparseable, " - f"{aggregation.skipped_unqualifiable} unresolvable against the schema). " + f"{aggregation.skipped_unqualifiable} unresolvable against the schema, " + f"{aggregation.skipped_ambiguous} ambiguous across the introspected schemas). " "Cost shares are computed against the whole window, so they are diluted and " "--min-cost-share is effectively stricter — few or no proposals may reflect " "coverage rather than a healthy workload." ) -def _validate_schemas(values: list[str]) -> tuple[str, ...]: - """Accept exactly one distinct schema, or exit 2 explaining why. +def _ambiguity_warning(aggregation: Aggregation) -> str | None: + """A warning naming the remedy for schema-ambiguous statements, or None. - Every catalog fact `advise` collects is keyed on the bare relation name: table sizes, - NDV maps, index lists and the qualify() schema all merge across schemas, so two - schemas each holding an `orders` alias into one another silently — the last catalog - row wins the row estimate, and `qualify()` resolves columns against a union of the - two column sets. Rejecting is the honest minimum until the keys are schema-qualified. + Separate from `_coverage_warning`, which fires on a *fraction* and says "coverage is + low". This fires on any occurrence at all, because the remedy is specific and + actionable — and because a handful of ambiguous statements can be the hottest ones in + the workload without moving the coverage fraction enough to trip a threshold. """ - distinct = tuple(dict.fromkeys(values)) - if len(distinct) > 1: - typer.echo( - f"--schema accepts one schema at a time (got {', '.join(distinct)}). " - "Multi-schema introspection is not supported yet: table facts, index lists " - "and NDV statistics are keyed on the bare table name, not schema-qualified, " - "so same-named tables in two schemas would silently alias. Run advise once " - "per schema.", - err=True, - ) - raise typer.Exit(code=2) - return distinct + if not aggregation.skipped_ambiguous: + return None + return ( + f"{aggregation.skipped_ambiguous} statement(s) named a table held by more than one " + "of the introspected schemas without qualifying it, so they could not be attributed " + "and were dropped. Qualify the table in the query, or run advise once per --schema." + ) + + +def _validate_schemas(values: list[str]) -> tuple[str, ...]: + """Deduplicate `--schema` values, preserving the order they were given in. + + Multiple schemas used to be rejected because every catalog fact was keyed on the bare + relation name, so two schemas each holding an `orders` aliased into one another. Facts, + NDV maps, index lists and the `qualify()` schema are all keyed by `Relation` now, so the + rejection is gone. What survives is a narrower caveat, surfaced by + `_ambiguity_warning`: a query that says `from orders` when two introspected schemas both + hold `orders` is genuinely ambiguous, and is counted and reported rather than guessed at. + """ + return tuple(dict.fromkeys(values)) def _parse_since(value: str | None) -> timedelta | None: @@ -713,7 +727,7 @@ def advise( schema: list[str] = typer.Option( ["public"], "--schema", - help="Schema to introspect. One at a time — passing two is rejected.", + help="Schema to introspect. Repeat for several: --schema public --schema sales.", ), since: str | None = typer.Option( None, "--since", help="Window, e.g. 7d. Not supported by pg_stat_statements." @@ -730,8 +744,9 @@ def advise( # share they do not have would be inventing evidence. help=( "Suppress proposals below this share of workload cost. Applies to the " - "cost-weighted rules (ADV001, ADV004, ADV005, ADV006); the index-hygiene rules " - "ADV002 and ADV003 carry no cost evidence and are always reported." + "cost-weighted rules (ADV001, ADV004, ADV005, ADV006, ADV007, ADV008); the " + "index-hygiene rules ADV002 and ADV003 carry no cost evidence and are always " + "reported." ), ), keep_literals: bool = typer.Option( @@ -885,6 +900,9 @@ def advise( coverage = _coverage_warning(workload, aggregation) if coverage is not None: typer.echo(coverage, err=True) + ambiguity = _ambiguity_warning(aggregation) + if ambiguity is not None: + typer.echo(ambiguity, err=True) table = Table( title=( diff --git a/src/sqlquality/models.py b/src/sqlquality/models.py index 5a69a47..23c7545 100644 --- a/src/sqlquality/models.py +++ b/src/sqlquality/models.py @@ -104,6 +104,27 @@ def total_cost_ms(self) -> float: return sum(s.total_time_ms for s in self.stats) +@dataclass(frozen=True, order=True) +class Relation: + """A schema-qualified relation — the key every catalog fact is stored under. + + Bare table names were the key until multi-schema support landed, and they aliased: two + schemas each holding an `orders` merged into one entry, so the last catalog row won the + row estimate while `qualify()` resolved columns against the union of both column sets. + + ``order=True`` because the rules sort their output for canonical, run-to-run stable + report ordering, and a bare `sorted()` over relation keys has to work. Field order is + (schema, table) so that ordering groups a schema's tables together. + """ + + schema: str + table: str + + def __str__(self) -> str: + """`schema.table` — how the relation appears in a proposal title or JSON key.""" + return f"{self.schema}.{self.table}" + + class ColumnRole(str, Enum): EQUALITY = "equality" RANGE = "range" @@ -117,7 +138,7 @@ class ColumnRole(str, Enum): @dataclass(frozen=True) class ColumnUsage: - table: str + relation: Relation column: str role: ColumnRole calls: int @@ -152,14 +173,30 @@ class Aggregation: usage: tuple[ColumnUsage, ...] total_cost_ms: float skipped_unqualifiable: int - tables: frozenset[str] + tables: frozenset[Relation] + #: Statements dropped because a bare table name is held by two introspected schemas. + #: Separate from `skipped_unqualifiable` because the remedy differs: qualify the query + #: or run once per schema, rather than widen the schema. + #: + #: Counts two distinct discovery situations, both the same underlying fact. Most + #: statements reference a column by name, so `qualify()` (or the DML sole-target check) + #: raises trying to resolve it and `aggregate()` counts the exception directly. A + #: statement that names the ambiguous table but references none of its columns by name + #: — `select * from orders`, `select count(*) from orders`, `select 1 from orders` — + #: gives `qualify()` nothing to validate, so it raises nothing either; `aggregate()` + #: instead recognizes this case directly (zero usage extracted, plus a bare table name + #: two introspected schemas both hold) and counts it the same way. + skipped_ambiguous: int = 0 @dataclass(frozen=True) class TableFacts: """Engine-neutral catalog facts. Engine-specific physical design stays in the adapter.""" - name: str + #: The schema-qualified key this table is stored under — not a display name. Two + #: same-named tables in different schemas each get their own `TableFacts`, keyed by + #: their own `Relation`; a bare name here would alias them back together. + relation: Relation row_estimate: int | None size_bytes: int | None columns: tuple[str, ...] @@ -183,6 +220,31 @@ class Proposal: ddl: str | None = None +def analyzed_query_groups(workload: Workload, aggregation: Aggregation) -> int: + """Query groups whose usage was actually extracted. + + Unresolvable *and* ambiguous groups are both a *subset* of ``workload.stats``, not a + separate pool, so ``len(stats)`` overstates what was understood unless both are + subtracted. Omitting `skipped_ambiguous` used to let an ambiguous statement count as + "analyzed" in the terminal's coverage line while the *same* statement counted as + "unexplained" in the low-coverage share — a statement cannot honestly be both, and the + share was the one that mattered: it silently deflated toward "coverage is fine", + suppressing the low-coverage warning exactly when ambiguity was the reason coverage was + bad. + + Lives here, beside `cost_share_of`, for the same reason: the terminal, the markdown + report and the JSON payload each present this number, and when the terminal alone + subtracted the skips, markdown printed "**Query groups analyzed:** 8" directly above + "2 ambiguous" while the terminal said "analyzed 6 of 8" — the self-contradiction this + function exists to prevent, reintroduced on two surfaces out of three. One helper, three + call sites, no way to omit it again. + """ + return max( + 0, + len(workload.stats) - aggregation.skipped_unqualifiable - aggregation.skipped_ambiguous, + ) + + def cost_share_of(evidence: Mapping[str, object]) -> float | None: """A proposal's cost share as a number, or None when it is absent or not one. diff --git a/src/sqlquality/report.py b/src/sqlquality/report.py index c085fd5..3cc8c05 100644 --- a/src/sqlquality/report.py +++ b/src/sqlquality/report.py @@ -5,7 +5,13 @@ import html as _html from sqlquality.gate import GateReport -from sqlquality.models import Aggregation, Proposal, Workload, cost_share_of +from sqlquality.models import ( + Aggregation, + Proposal, + Workload, + analyzed_query_groups, + cost_share_of, +) def _md_escape(value: object) -> str: @@ -144,14 +150,21 @@ def advise_payload( "redacted": redacted, "window": workload.window_description, "analyzed": { - "query_groups": len(workload.stats), + # The count of groups whose usage was actually extracted — not `len(stats)`, + # which includes the unresolvable and ambiguous groups reported under "skipped" + # below and so contradicted them under a key named "analyzed". The window total + # is kept beside it rather than dropped, since a consumer computing "how much of + # the window did this run understand" needs both numbers. + "query_groups": analyzed_query_groups(workload, aggregation), + "query_groups_in_window": len(workload.stats), "total_cost_ms": workload.total_cost_ms, - "tables": sorted(aggregation.tables), + "tables": sorted(str(relation) for relation in aggregation.tables), }, "skipped": { "unparseable": workload.skipped_unparseable, "noise": workload.skipped_noise, "unqualifiable": aggregation.skipped_unqualifiable, + "ambiguous": aggregation.skipped_ambiguous, }, "degraded": [{"capability": cap, "reason": reason} for cap, reason in degraded], "proposals": [ @@ -192,16 +205,26 @@ def render_advise_markdown( f"# sqlquality advise — {_md_escape(engine)}", "", f"**Window:** {_md_escape(workload.window_description)}", - f"**Query groups analyzed:** {len(workload.stats)} ", + # "N of M", the same form and the same numbers as the terminal's coverage line. + # Printing `len(workload.stats)` alone as *analyzed* contradicted the "Skipped:" + # line directly beneath it — "8 analyzed" above "2 ambiguous" out of 8 groups. + f"**Query groups analyzed:** {analyzed_query_groups(workload, aggregation)} of " + f"{len(workload.stats)} ", f"**Literals:** {'redacted' if redacted else 'retained (--keep-literals)'}", "", ( - # "filtered", not "introspection/DDL": the noise filter matches on the leading - # keyword, so it also discards DECLARE ... CURSOR FOR SELECT and - # COPY (SELECT ...) — ordinary reads. See cli._coverage_line. + # "filtered", not "introspection/DDL": it covers session control, DDL, + # maintenance, introspection, a whole-table COPY, and a cursor statement + # carrying no query (FETCH, CLOSE). DECLARE ... CURSOR FOR SELECT and + # COPY (SELECT ...) TO — ordinary reads — are unwrapped to their inner query + # before this count is taken, so they land in "analyzed" instead. + # See cli._coverage_line and workload.fingerprint.unwrap. f"Skipped: {workload.skipped_unparseable} unparseable, " - f"{workload.skipped_noise} filtered as non-workload by statement prefix, " - f"{aggregation.skipped_unqualifiable} unresolvable against the schema." + f"{workload.skipped_noise} filtered as non-workload (session control, DDL, " + f"maintenance, introspection, whole-table COPY, or a cursor statement " + f"carrying no query), " + f"{aggregation.skipped_unqualifiable} unresolvable against the schema, " + f"{aggregation.skipped_ambiguous} ambiguous across the introspected schemas." ), "", ] diff --git a/src/sqlquality/workload/aggregate.py b/src/sqlquality/workload/aggregate.py index 256f9c6..73c05fa 100644 --- a/src/sqlquality/workload/aggregate.py +++ b/src/sqlquality/workload/aggregate.py @@ -6,22 +6,30 @@ from collections import defaultdict from functools import lru_cache -from sqlquality.models import Aggregation, ColumnRole, ColumnUsage, Workload +from sqlglot import exp + +from sqlquality.models import Aggregation, ColumnRole, ColumnUsage, Relation, Workload from sqlquality.sqlast import SqlParseError, parse -from sqlquality.workload.extract import UnqualifiableQuery, extract_usage +from sqlquality.workload.extract import ( + AmbiguousRelation, + UnqualifiableQuery, + extract_usage, + resolve_relation, +) from sqlquality.workload.fingerprint import FLAG_SELECT_STAR -_Key = tuple[str, str, ColumnRole] +_Key = tuple[Relation, str, ColumnRole] @lru_cache(maxsize=4096) def _identifier_pattern(name: str) -> re.Pattern[str]: """Compiled whole-identifier matcher for one name, compiled once per name. - ``star_tables`` tests every (star-stat, table) pair, and a schema with many tables was - recompiling the same handful of table-name patterns over and over, thrashing `re`'s own - pattern cache. Caching by name here means each identifier is compiled once regardless of - how many stats or tables it is checked against. + Callers such as ADV006's wide-table detection and the expression-index disclosure in + `postgres.py` test one name against many statements (or vice versa), and a schema with + many tables was recompiling the same handful of name patterns over and over, thrashing + `re`'s own pattern cache. Caching by name here means each identifier is compiled once + regardless of how many times it is checked. """ return re.compile(rf"\b{re.escape(name)}\b") @@ -39,49 +47,112 @@ def mentions_identifier(name: str, text: str) -> bool: return _identifier_pattern(name).search(text) is not None -def mentions_table(name: str, sql: str) -> bool: - """True if a query mentions this table. See :func:`mentions_identifier`.""" - return mentions_identifier(name, sql) - - -def star_tables(workload: Workload, schema: dict) -> frozenset[str]: - """Tables a `SELECT *` query group merely *mentions*, matched against ``schema``. +def star_tables(workload: Workload, schema: dict, dialect: str = "postgres") -> frozenset[Relation]: + """Relations a `SELECT *` query group merely *mentions*, matched against ``schema``. A bare `select * from wide_t` filters nothing, so it contributes no column usage and - the table never appears in ``Aggregation.tables``. Introspecting only the tables that - produced usage therefore left the star rule with no column counts to test — inert for - precisely the workload it exists to catch. These names are unioned in before catalog - facts are fetched. - - Deliberately *not* added to ``Aggregation.tables``: that set means "tables with + the relation never appears in ``Aggregation.tables``. Introspecting only the relations + that produced usage therefore left the star rule with no column counts to test — inert + for precisely the workload it exists to catch. These relations are unioned in before + catalog facts are fetched. + + Resolved by parsing each starred statement and running its actual `exp.Table` nodes + through `resolve_relation` — not by text-matching the schema's table names against the + raw SQL. Text matching cannot see a schema qualifier at all, and that blindness cuts + both ways: `select * from nosuch.items` would resolve through a bare-name collision + with an unrelated schema (a phantom `resolve_relation`'s `table.db` guard exists + specifically to refuse), while `select * from sales.orders` naming one side of a + same-table-name collision would be dropped even though it is not actually ambiguous. + Resolving through the same function `extract_usage` uses makes the two agree by + construction; a second, hand-rolled ambiguity policy here previously did not. + + A parse failure is not counted here: the same statement was already counted + unparseable at ingest (`Workload.skipped_unparseable`), so re-counting it under a + different name would make the two counters disagree about what "unparseable" means. + + Deliberately *not* added to ``Aggregation.tables``: that set means "relations with recorded column usage" and feeds the unused-index rule's notion of a hot table. + + ``dialect`` defaults to `"postgres"`, the only workload adapter registered today + (see `sqlquality.workload.get_workload_adapter`); a caller wiring in a second engine + must pass its dialect explicitly rather than rely on the default. """ - return frozenset( - name - for stat in workload.stats - if FLAG_SELECT_STAR in stat.flags - for name in schema - if mentions_table(name, stat.sql) - ) + found: set[Relation] = set() + for stat in workload.stats: + if FLAG_SELECT_STAR not in stat.flags: + continue + try: + tree = parse(stat.sql, dialect) + except SqlParseError: + continue + for table in tree.find_all(exp.Table): + relation = resolve_relation(table, schema) + if relation is not None: + found.add(relation) + return frozenset(found) + + +def _references_an_ambiguous_bare_table(tree: exp.Expression, schema: dict) -> bool: + """True if `tree` names a bare table held by more than one introspected schema. + + Mirrors `resolve_relation`'s own bare-name branch exactly: a table reference with no + `.db` qualifier is ambiguous precisely when more than one introspected schema defines a + table of that name. A schema-qualified reference is never ambiguous by this test — a + qualifier that names a schema we did not introspect is a *different* case + (`resolve_relation` returns `None` for it, not an ambiguity), and is deliberately not + reported here. + """ + for table in tree.find_all(exp.Table): + if table.db: + continue + owners = [name for name, tables in schema.items() if table.name in tables] + if len(owners) > 1: + return True + return False def aggregate(workload: Workload, schema: dict, dialect: str) -> Aggregation: - """Weight every (table, column, role) by the cost of the queries that use it.""" + """Weight every (relation, column, role) by the cost of the queries that use it.""" calls: dict[_Key, int] = defaultdict(int) cost: dict[_Key, float] = defaultdict(float) #: Which query groups contributed each usage, so downstream rules can ask whether two #: usages co-occur in a single query rather than merely both being hot on the table. contributors: dict[_Key, set[str]] = defaultdict(set) - tables: set[str] = set() + tables: set[Relation] = set() skipped_unqualifiable = 0 + skipped_ambiguous = 0 for stat in workload.stats: try: tree = parse(stat.sql, dialect) triples = extract_usage(tree, dialect, schema) + except AmbiguousRelation: + # Counted before the broader handler below, because AmbiguousRelation *is* an + # UnqualifiableQuery — ordering these the other way round makes the specific + # counter unreachable and the specific remedy unreportable. + skipped_ambiguous += 1 + continue except (SqlParseError, UnqualifiableQuery): skipped_unqualifiable += 1 continue + # Any statement that names a table but references none of its columns by name + # (`select * from orders`, `select count(*) from orders`, `select 1 from orders`, + # `select now() from orders`) contributes no usage either way, so `qualify()` above + # has nothing to validate and neither raises nor records anything for it — even when + # "orders" is a name two introspected schemas both hold. Not gated on + # `FLAG_SELECT_STAR`: that flag only marks a literal `SELECT *`, so gating on it let + # `select count(*) from orders` and `select 1 from orders` over the same colliding + # schema escape *both* counters — parsed fine, zero usage, never raised, never + # counted, reported as if fully understood. Left uncounted, any of these reads as + # "understood and irrelevant" when it is really the same unattributable-bare-name + # fact `AmbiguousRelation` reports elsewhere (and the reason ADV006's own + # `_wide_relations_touched` later declines to guess at it too); it just surfaces + # without an exception because there is no column reference for `qualify()` to trip + # over. Gated on `not triples`: a statement that already produced usage from some + # *other*, unambiguous table was not silently dropped, so it does not belong here. + if not triples and _references_an_ambiguous_bare_table(tree, schema): + skipped_ambiguous += 1 + continue for key in triples: calls[key] += stat.calls cost[key] += stat.total_time_ms @@ -99,20 +170,21 @@ def aggregate(workload: Workload, schema: dict, dialect: str) -> Aggregation: sorted( ( ColumnUsage( - table=table, + relation=relation, column=column, role=role, - calls=calls[(table, column, role)], - cost_ms=cost[(table, column, role)], - cost_share=(cost[(table, column, role)] / total) if total else 0.0, - fingerprint_ids=frozenset(contributors[(table, column, role)]), + calls=calls[(relation, column, role)], + cost_ms=cost[(relation, column, role)], + cost_share=(cost[(relation, column, role)] / total) if total else 0.0, + fingerprint_ids=frozenset(contributors[(relation, column, role)]), ) - for (table, column, role) in calls + for (relation, column, role) in calls ), # Descending cost with a canonical tiebreak. Without the trailing keys, two # logically identical workloads that happened to arrive in a different order # produce different output order, and downstream tasks' tests depend on it. - key=lambda u: (-u.cost_ms, u.table, u.column, u.role.value), + # `Relation` is `order=True`, so it sorts directly with no key function. + key=lambda u: (-u.cost_ms, u.relation, u.column, u.role.value), ) ) return Aggregation( @@ -120,4 +192,5 @@ def aggregate(workload: Workload, schema: dict, dialect: str) -> Aggregation: total_cost_ms=total, skipped_unqualifiable=skipped_unqualifiable, tables=frozenset(tables), + skipped_ambiguous=skipped_ambiguous, ) diff --git a/src/sqlquality/workload/base.py b/src/sqlquality/workload/base.py index b9e1ef4..09f4c1c 100644 --- a/src/sqlquality/workload/base.py +++ b/src/sqlquality/workload/base.py @@ -19,6 +19,7 @@ Aggregation, ConnectionParams, Proposal, + Relation, TableFacts, Workload, WorkloadFetch, @@ -74,19 +75,25 @@ def fetch_workload(self, since: timedelta | None, limit: int) -> WorkloadFetch: @abstractmethod def fetch_schema(self, schemas: tuple[str, ...]) -> dict: - """Schema mapping for sqlglot qualify(): {table: {column: type}}.""" + """Schema mapping for sqlglot qualify(): {schema: {table: {column: type}}}. + + Nested rather than flat: a flat `{table: {column: type}}` map cannot tell two + same-named tables in different schemas apart, so a column that exists in only one + of them resolves against the union of both — the exact aliasing this task exists + to remove. + """ @abstractmethod def fetch_table_facts( - self, schemas: tuple[str, ...], tables: frozenset[str] - ) -> dict[str, TableFacts]: - """Row estimates, sizes, columns and per-column NDV for the given tables.""" + self, schemas: tuple[str, ...], relations: frozenset[Relation] + ) -> dict[Relation, TableFacts]: + """Row estimates, sizes, columns and per-column NDV for the given relations.""" @abstractmethod def propose( self, aggregation: Aggregation, - facts: dict[str, TableFacts], + facts: dict[Relation, TableFacts], workload: Workload, *, min_cost_share: float, diff --git a/src/sqlquality/workload/extract.py b/src/sqlquality/workload/extract.py index 71b2b07..f4b2486 100644 --- a/src/sqlquality/workload/extract.py +++ b/src/sqlquality/workload/extract.py @@ -3,11 +3,11 @@ from __future__ import annotations from sqlglot import exp -from sqlglot.errors import OptimizeError +from sqlglot.errors import OptimizeError, SchemaError from sqlglot.optimizer.qualify import qualify from sqlglot.optimizer.scope import Scope, build_scope -from sqlquality.models import ColumnRole +from sqlquality.models import ColumnRole, Relation #: Comparison nodes that an index can satisfy with an equality probe. _EQUALITY_NODES = (exp.EQ, exp.In) @@ -21,6 +21,16 @@ class UnqualifiableQuery(ValueError): """Raised when a query's columns cannot be resolved against the supplied schema.""" +class AmbiguousRelation(UnqualifiableQuery): + """A table name that two introspected schemas both hold, in a query that did not qualify it. + + A subclass, not a sibling: every caller that wants to treat all resolution failures + alike keeps working with one `except UnqualifiableQuery`, while `aggregate` can count + this case separately because its remedy is different — qualify the query or run once per + schema, rather than widen the schema. + """ + + def _within(node: exp.Expression, *types: type[exp.Expression]) -> bool: """True if any ancestor of ``node`` is one of ``types``. Mirrors antipatterns._within_exists.""" parent = node.parent @@ -81,69 +91,152 @@ def _role(column: exp.Column) -> ColumnRole | None: return comparison -def _scope_tables(scope: Scope) -> dict[str, str]: - """Alias (or bare name) -> real table name, for one scope only. +def resolve_relation(table: exp.Table, schema: dict) -> Relation | None: + """The schema-qualified relation for one `exp.Table`, or None if it is not attributable. + + `table.db` is authoritative when present, but only once it is checked against the + introspected schema map. `qualify()` leaves it EMPTY for a bare table reference even when + the nested schema resolves the name unambiguously — and bare references are the normal + case, because production SQL relies on `search_path`. So the fallback is a lookup in the + schema map we actually introspected: + + * exactly one introspected schema holds the name -> that is the schema, no guess involved + * more than one -> ambiguous, and attributing it would be a coin flip. `qualify()` will + normally have raised `SchemaError` before we get here, but a table whose columns are + never referenced by name reaches this line, so the guard is real. + * none -> the table lives outside the introspected schemas; the caller drops the column. + + The `table.db` branch is guarded the same way, and not just defensively: probing sqlglot + 30.12 confirms `qualify()` validates SELECT-scope column references against the schema + (an explicitly-qualified table `qualify()` never introspected raises `OptimizeError` + before this function runs), but it does **not** validate UPDATE/DELETE targets or their + bare columns — `UPDATE other.orders SET status = 'x' WHERE id = 1` against a schema + without an `other` key passes `qualify()` untouched. Trusting `table.db` unconditionally + there would manufacture `Relation("other", "orders")`, a phantom that matches no catalog + fact, from a schema-qualified DML statement we never introspected. + """ + if table.db: + if table.name in schema.get(table.db, {}): + return Relation(schema=table.db, table=table.name) + return None + owners = [name for name, tables in schema.items() if table.name in tables] + if len(owners) == 1: + return Relation(schema=owners[0], table=table.name) + return None + + +def _scope_relations(scope: Scope, schema: dict) -> dict[str, Relation]: + """Alias (or bare name) -> schema-qualified relation, for one scope only. A sub-scope source (CTE, derived table) maps to a ``Scope``, not an ``exp.Table``. Columns resolving to one of those reference a projection rather than a base-table column, so they are omitted here and skipped — the sub-scope contributes its own base - tables when ``traverse()`` reaches it. + tables when ``traverse()`` reaches it. A source we cannot attribute to a schema is + omitted for the same reason: no key, no usage. """ - return { - name: source.name for name, source in scope.sources.items() if isinstance(source, exp.Table) - } - - -def _record(seen: set[tuple[str, str, ColumnRole]], table: str | None, column: exp.Column) -> None: - """Add one (table, column, role) triple, skipping unattributable or unused columns.""" - if not table or not column.name: + resolved: dict[str, Relation] = {} + for name, source in scope.sources.items(): + if isinstance(source, exp.Table): + relation = resolve_relation(source, schema) + if relation is not None: + resolved[name] = relation + return resolved + + +def _record( + seen: set[tuple[Relation, str, ColumnRole]], + relation: Relation | None, + column: exp.Column, +) -> None: + """Add one (relation, column, role) triple, skipping unattributable or unused columns.""" + if relation is None or not column.name: return role = _role(column) if role is None: return - seen.add((table, column.name, role)) + seen.add((relation, column.name, role)) -def _collect_dml(qualified: exp.Expression, seen: set[tuple[str, str, ColumnRole]]) -> None: +def _collect_dml( + qualified: exp.Expression, seen: set[tuple[Relation, str, ColumnRole]], schema: dict +) -> None: """Attribute the columns of an UPDATE/DELETE to its sole target table. ``qualify()`` leaves DML columns bare (``column.table == ''``) rather than raising. With exactly one table in the statement the target is unambiguous; with more than one (``UPDATE ... FROM``) attribution would be a guess, so bare columns are dropped instead of misattributed. + + A `len(tables) == 1` target is the one case that needs its own ambiguity check rather + than deferring to `resolve_relation`'s plain `None`: `qualify()` does not validate + UPDATE/DELETE targets (see `resolve_relation`'s docstring), so a bare name held by two + introspected schemas reaches here with no `SchemaError` ever raised. Left as a silent + `None`, the statement would vanish with no usage recorded and *neither* counter + incremented — reported as analysed when it was not. Raising here routes it into the + same `skipped_ambiguous` counter as the SELECT-path ambiguity sqlglot itself detects. """ tables = tuple(qualified.find_all(exp.Table)) - aliases = {t.alias_or_name: t.name for t in tables} - sole_table = tables[0].name if len(tables) == 1 else None + aliases: dict[str, Relation] = {} + for table in tables: + relation = resolve_relation(table, schema) + if relation is not None: + aliases[table.alias_or_name] = relation + sole: Relation | None = None + if len(tables) == 1: + target = tables[0] + sole = resolve_relation(target, schema) + if sole is None and not target.db: + owners = [name for name, tbls in schema.items() if target.name in tbls] + if len(owners) > 1: + raise AmbiguousRelation( + f"Ambiguous mapping for DML target '{target.name}': " + f"held by {', '.join(sorted(owners))}." + ) for column in qualified.find_all(exp.Column): - _record(seen, aliases.get(column.table) if column.table else sole_table, column) + _record(seen, aliases.get(column.table) if column.table else sole, column) def extract_usage( tree: exp.Expression, dialect: str, schema: dict -) -> tuple[tuple[str, str, ColumnRole], ...]: - """(table, column, role) triples for one query, deduplicated. - - Stars are not expanded: a projected star tells us nothing about which columns are - filtered, and expanding it would drown the rollup in projection noise. +) -> tuple[tuple[Relation, str, ColumnRole], ...]: + """(relation, column, role) triples for one query, deduplicated. + + ``schema`` is nested — ``{schema_name: {table: {column: type}}}`` — because relations + are keyed by schema. Stars are not expanded: a projected star tells us nothing about + which columns are filtered, and expanding it would drown the rollup in projection noise. + + ``SchemaError`` is *not* an ``OptimizeError`` subclass — its bases are ``SqlglotError``, + ``Exception`` — so catching only ``OptimizeError`` let an ambiguous bare table name + (`Ambiguous mapping for orders: sales, staging.`) escape `aggregate()` and abort the whole + run with a traceback. It is now caught on its own and, when the message signals ambiguity + specifically, re-raised as `AmbiguousRelation` rather than the plain `UnqualifiableQuery` + every other resolution failure gets. """ try: qualified = qualify(tree.copy(), dialect=dialect, schema=schema, expand_stars=False) + except SchemaError as exc: + # sqlglot has exactly one ambiguity message and no error code to match on, so the + # text is the only signal available. Matching it loosely (lowercased substring) + # rather than exactly, because a wording change upstream should degrade this to + # "counted as unqualifiable" — the pre-existing behaviour — not crash. + if "ambiguous mapping" in str(exc).lower(): + raise AmbiguousRelation(str(exc)) from exc + raise UnqualifiableQuery(str(exc)) from exc except OptimizeError as exc: raise UnqualifiableQuery(str(exc)) from exc - seen: set[tuple[str, str, ColumnRole]] = set() + seen: set[tuple[Relation, str, ColumnRole]] = set() root = build_scope(qualified) if root is None: # build_scope() returns None for UPDATE/DELETE — they are not SELECT-rooted. - _collect_dml(qualified, seen) + _collect_dml(qualified, seen, schema) else: # Resolve aliases per scope, never with one flat map over the whole tree. Two # different tables in different scopes can share an alias, and a flat map keeps # whichever `find_all` visited last — silently attributing an outer filter to an # inner table and losing the outer one entirely. for scope in root.traverse(): - aliases = _scope_tables(scope) + aliases = _scope_relations(scope, schema) for column in scope.columns: _record(seen, aliases.get(column.table), column) return tuple(sorted(seen, key=lambda triple: (triple[0], triple[1], triple[2].value))) diff --git a/src/sqlquality/workload/fingerprint.py b/src/sqlquality/workload/fingerprint.py index 5c762d0..a84439d 100644 --- a/src/sqlquality/workload/fingerprint.py +++ b/src/sqlquality/workload/fingerprint.py @@ -42,6 +42,81 @@ ) +#: `DECLARE [BINARY] [ASENSITIVE|INSENSITIVE] [[NO] SCROLL] CURSOR +#: [WITH|WITHOUT HOLD] FOR ` — the full PostgreSQL grammar for the statement every +#: psycopg2 server-side cursor emits. +#: +#: Anchored on `CURSOR ... FOR` rather than on the first `FOR`, because a cursor name is an +#: identifier and a quoted one may contain the word: `DECLARE "for sale" CURSOR FOR ...` +#: would otherwise be cut at the wrong place and yield unparseable text. The name alternative +#: matches a quoted identifier (with doubled quotes escaped) before an unquoted one for the +#: same reason. +_DECLARE_CURSOR = re.compile( + r"^\s*DECLARE\s+" + r'(?:"(?:[^"]|"")*"|[A-Za-z_]\w*)\s+' + r"(?:BINARY\s+)?" + r"(?:ASENSITIVE\s+|INSENSITIVE\s+)?" + r"(?:NO\s+SCROLL\s+|SCROLL\s+)?" + r"CURSOR\s+" + r"(?:WITH\s+HOLD\s+|WITHOUT\s+HOLD\s+)?" + r"FOR\s+(?P\S.*)$", + re.IGNORECASE | re.DOTALL, +) + +#: `COPY ( ) TO ...` — the only COPY form carrying predicates worth analysing. +#: `COPY
TO` is a whole-relation dump with no predicates, and `COPY ... FROM` is a +#: write; both stay noise. The capture is greedy to the last `)` so a query containing +#: parentheses survives; the result is validated by the caller's parse, so a mis-cut yields +#: an unparseable count rather than a wrong analysis. +_COPY_QUERY = re.compile( + r"^\s*COPY\s*\(\s*(?P.*)\s*\)\s*TO\b", + re.IGNORECASE | re.DOTALL, +) + + +def unwrap(sql: str) -> str: + """The inner query of a cursor declaration or `COPY (...) TO`, else ``sql`` unchanged. + + `DECLARE ... CURSOR FOR SELECT ...` and `COPY (SELECT ...) TO ...` are ordinary reads + with real predicates, but both begin with a keyword the noise filter drops — so on any + workload using server-side cursors (every psycopg2 `cursor(name=...)`, which is what + Django and SQLAlchemy emit for large result sets) the hottest reads were counted as + "filtered" and thrown away. + + What this actually restores differs by form, measured on PostgreSQL 16. `COPY + (SELECT ...) TO ...` attributes correctly: `pg_stat_statements` charges the whole + execution's time and row count to the `COPY` statement itself, so cost-weighted rules + see the real number. + + `DECLARE ... CURSOR FOR` does **not**: Postgres attributes the work of actually reading + rows to the `FETCH` statements that follow, which `is_noise` still filters (a `FETCH` + carries no query text, so it has no predicates to attribute and unfiltering it would + only inflate the denominator with uncounted cost). The `DECLARE` itself is recorded + accurately by *call count* — one call per cursor opened — but with near-zero time and + rows, since opening a cursor does no scanning. So a cursor + read recovered here contributes its predicate columns to `aggregate` and can still join + an index candidate, but it cannot earn a proposal on cost alone: the default + `--min-cost-share` can suppress it outright, and `WITH HOLD` does not change this. + + Text surgery rather than AST surgery, for a reason that is not a preference: sqlglot + cannot parse `DECLARE` at all — it falls back to `exp.Command` and leaves the entire + tail as a single string literal, so there is no inner tree to lift. `COPY` *does* parse + (to `exp.Copy` with a `Subquery`), but doing both here keeps one code path and, more + importantly, lets `is_noise` run on the *unwrapped* text — which is what stops a + `DECLARE c CURSOR FOR SELECT * FROM pg_stat_statements` from smuggling our own + introspection into the analysed workload. + + Returns the input unchanged when nothing matches. The caller parses the result, so a + partial or malformed wrapper degrades to the pre-existing behaviour — counted + unparseable or filtered — rather than producing a wrong analysis. + """ + for pattern in (_DECLARE_CURSOR, _COPY_QUERY): + match = pattern.match(sql) + if match is not None: + return match.group("query").strip() + return sql + + def is_noise(sql: str) -> bool: """True for session control, DDL, maintenance, and introspection statements. @@ -105,11 +180,15 @@ def ingest(fetch: WorkloadFetch, dialect: str, *, keep_literals: bool = False) - skipped_noise = 0 for row in fetch.rows: - if is_noise(row.sql): + # Unwrap *before* the noise test, so a cursor declaration is judged on the query it + # declares. Judging the wrapper drops the read; judging the inner query keeps a real + # read and still filters an inner introspection query. + sql = unwrap(row.sql) + if is_noise(sql): skipped_noise += 1 continue try: - tree = parse(row.sql, dialect) + tree = parse(sql, dialect) except SqlParseError: skipped_unparseable += 1 continue diff --git a/src/sqlquality/workload/postgres.py b/src/sqlquality/workload/postgres.py index 75af292..f2a7ff9 100644 --- a/src/sqlquality/workload/postgres.py +++ b/src/sqlquality/workload/postgres.py @@ -3,11 +3,14 @@ from __future__ import annotations import hashlib +import re import sys from collections.abc import Mapping, Sequence -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from datetime import timedelta +from sqlglot import exp + from sqlquality.models import ( Aggregation, ColumnRole, @@ -16,12 +19,14 @@ Confidence, Proposal, RawQueryRow, + Relation, TableFacts, Workload, WorkloadFetch, cost_share_of, ) -from sqlquality.workload.aggregate import mentions_identifier, mentions_table +from sqlquality.sqlast import parse +from sqlquality.workload.aggregate import mentions_identifier from sqlquality.workload.base import ( MAX_TIMEOUT_S, MIN_TIMEOUT_S, @@ -211,10 +216,10 @@ class PgIndex: ) -def _by_table(usage: Sequence[ColumnUsage]) -> dict[str, list[ColumnUsage]]: - grouped: dict[str, list[ColumnUsage]] = {} +def _by_relation(usage: Sequence[ColumnUsage]) -> dict[Relation, list[ColumnUsage]]: + grouped: dict[Relation, list[ColumnUsage]] = {} for item in usage: - grouped.setdefault(item.table, []).append(item) + grouped.setdefault(item.relation, []).append(item) return grouped @@ -246,10 +251,20 @@ def _covered(candidate: tuple[str, ...], existing: Sequence[PgIndex]) -> str | N return None -#: Schema every rule qualifies its DDL with unless told otherwise. It matches the CLI's -#: `--schema` default, so the default is truthful rather than merely convenient; the CLI -#: always passes the resolved schema explicitly (see PostgresWorkloadAdapter.propose). -DEFAULT_SCHEMA = "public" +#: A period followed by whitespace, i.e. a sentence boundary in this module's own generated +#: rationale prose — never an abbreviation, since none of the rationale text this module +#: writes uses one. +_SENTENCE_BOUNDARY = re.compile(r"(?<=\.)\s+") + + +def _sentences(text: str) -> list[str]: + """Split rationale prose into its component sentences. + + Used to detect whole-sentence duplication when folding one proposal's rationale into + another's — not a general-purpose sentence splitter, just good enough for text this + same module generates and controls the wording of. + """ + return [s for s in (chunk.strip() for chunk in _SENTENCE_BOUNDARY.split(text.strip())) if s] def _qualified(schema: str, name: str) -> str: @@ -264,13 +279,12 @@ def _qualified(schema: str, name: str) -> str: def propose_indexes( usage: Sequence[ColumnUsage], - facts: Mapping[str, TableFacts], - existing: Mapping[str, Sequence[PgIndex]], + facts: Mapping[Relation, TableFacts], + existing: Mapping[Relation, Sequence[PgIndex]], *, min_cost_share: float, min_rows: int = MIN_ROWS_FOR_INDEX, max_arity: int = MAX_INDEX_ARITY, - schema: str = DEFAULT_SCHEMA, have_index_data: bool = True, ) -> list[Proposal]: """ADV001 — composite index candidates: equality columns first, one range column last. @@ -283,10 +297,29 @@ def propose_indexes( unknowable, so it is not made and confidence is capped at LOW. ``existing`` being empty cannot distinguish "no such index" from "could not look", which is exactly the conflation the row-estimate branch below exists to prevent. + + Extending the composite requires *joint* support — every added column sharing a query + group with every column already chosen, tracked as a running intersection of + ``fingerprint_ids`` — exactly as ADV008 does. Without it this rule welded together the + hottest equality columns and the hottest range column from a relation whether or not any + single query used them together, and `cost_share` could not filter that out because it is + the *max* over the chosen columns, not the min. Measured case: a `DECLARE ... CURSOR FOR + SELECT ... WHERE tenant_id = $1` read costing 0.003% of the window put `tenant_id` into + position 2 of an otherwise correct `(customer_id, created_at)`, and + `(customer_id, tenant_id, created_at)` cannot satisfy the hot query's `ORDER BY + created_at` that `(customer_id, created_at)` serves — a strictly worse index, emitted at + HIGH, for the query carrying most of the workload's cost. + + Evidence carries `co_occurring_fingerprints` (the size of that intersection) and + deliberately no plain `fingerprints` count, matching ADV004 and ADV008 — see + `propose_grouping_indexes` for the principle. A rule whose claim is about columns + appearing *together* must not also report a per-column count: reports render evidence as + bare `k=v` pairs with no per-rule text, so `fingerprints: 1` beside a three-column index + read as "one query group uses all three" when zero did. """ proposals: list[Proposal] = [] - for table, items in sorted(_by_table(usage).items()): - table_facts = facts.get(table) + for relation, items in sorted(_by_relation(usage).items()): + table_facts = facts.get(relation) rows = table_facts.row_estimate if table_facts else None if rows is not None and rows < min_rows: continue @@ -312,13 +345,30 @@ def propose_indexes( # even suppress: `_is_prefix(("id","id"), ("id",))` is False, so the table's own # primary key did not match. Equality wins because equality-first is the B-tree # ordering the whole rule is built on. + # + # `shared` is the running intersection of the chosen columns' query groups, so a + # column only joins the composite when some single query group filters on it + # *together with* everything already chosen — the same guard, for the same reason, + # as `propose_grouping_indexes`. A candidate rejected for lack of joint support does + # not narrow `shared`, so a later candidate that does co-occur with the chosen set + # can still join it. chosen: list[ColumnUsage] = [] picked: set[str] = set() + shared: frozenset[str] = frozenset() for item in candidate: if item.column in picked: continue - picked.add(item.column) + if not chosen: + chosen.append(item) + picked.add(item.column) + shared = item.fingerprint_ids + continue + overlap = shared & item.fingerprint_ids + if not overlap: + continue chosen.append(item) + picked.add(item.column) + shared = overlap if not chosen: continue @@ -327,11 +377,11 @@ def propose_indexes( continue columns = tuple(i.column for i in chosen) - covered_by = _covered(columns, existing.get(table, ())) + covered_by = _covered(columns, existing.get(relation, ())) if covered_by is not None: continue - table_indexes = existing.get(table, ()) + table_indexes = existing.get(relation, ()) partial_skipped = tuple( index.name for index in table_indexes @@ -386,6 +436,15 @@ def propose_indexes( ) if rows is None: rationale += _UNKNOWN_ROWS_NOTE + # Same caveat as ADV007's, word for word: a low leading NDV is why this proposal is + # LOW rather than HIGH, and until this line existed ADV001 downgraded silently while + # ADV007 explained itself for the identical reason — an asymmetry an operator should + # not have to notice depends on which rule happened to propose the index. + if leading_ndv is not None and leading_ndv < SELECTIVE_NDV: + rationale += ( + f" Only about {leading_ndv:.0f} distinct values, so the index may not be " + "selective enough to be worth its write cost." + ) if partial_skipped: rationale += ( f" A partial index ({', '.join(partial_skipped)}) leads with these columns " @@ -402,15 +461,23 @@ def propose_indexes( proposals.append( Proposal( code="ADV001", - title=f"Add index on {table}({', '.join(columns)})", + title=f"Add index on {relation}({', '.join(columns)})", rationale=rationale, evidence={ - "table": table, + "schema": relation.schema, + "table": relation.table, "columns": columns, "roles": tuple(i.role.value for i in chosen), "cost_share": cost_share, "calls": max(i.calls for i in chosen), - "fingerprints": max(i.fingerprints for i in chosen), + #: How many query groups filter on *every* chosen column together — the + #: running intersection, not a per-column count. Same name and same + #: meaning as ADV004's and ADV008's identical field, and deliberately no + #: plain `fingerprints` key beside it: this proposal is justified by the + #: columns appearing together, so a per-column count rendered as a bare + #: `k=v` pair next to the joint one can only read as support that is not + #: there. + "co_occurring_fingerprints": len(shared), "row_estimate": rows, "leading_ndv": leading_ndv, "partial_indexes_skipped": partial_skipped, @@ -418,7 +485,296 @@ def propose_indexes( }, confidence=confidence, ddl=( - f"CREATE INDEX ON {_qualified(schema, table)} " + f"CREATE INDEX ON {_qualified(relation.schema, relation.table)} " + f"({', '.join(_quote_ident(c) for c in columns)});" + ), + ) + ) + return proposals + + +def propose_join_keys( + usage: Sequence[ColumnUsage], + facts: Mapping[Relation, TableFacts], + existing: Mapping[Relation, Sequence[PgIndex]], + *, + min_cost_share: float, + min_rows: int = MIN_ROWS_FOR_INDEX, + have_index_data: bool = True, +) -> list[Proposal]: + """ADV007 — a hot join key with no index leading with it. + + Deliberately one proposal per join column rather than a composite: two joins against the + same table want two indexes, and a composite `(a, b)` serves only probes on `a`. + + Not folded into ADV001. That rule's rationale is the B-tree ordering argument — + "equality columns first so the range column can be scanned last" — and a join key is not + a filter predicate: it is probed once per outer row. Adding JOIN to ADV001's candidate + list would have left that sentence in the report while making it false of the index it + describes. Postgres does not index the referencing side of a foreign key either, so this + gap is both common and expensive. + """ + proposals: list[Proposal] = [] + for relation, items in sorted(_by_relation(usage).items()): + table_facts = facts.get(relation) + rows = table_facts.row_estimate if table_facts else None + if rows is not None and rows < min_rows: + continue + ndv = table_facts.ndv if table_facts else {} + joins = sorted( + (i for i in items if i.role is ColumnRole.JOIN), + key=lambda i: (-i.cost_ms, i.column), + ) + table_indexes = existing.get(relation, ()) + for item in joins: + if item.cost_share < min_cost_share: + continue + if _covered((item.column,), table_indexes) is not None: + continue + # Same guards as ADV001, same reasons: a partial index leading with this column + # does not serve an unfiltered lookup, and an expression index's `columns` tuple + # understates it, so `_covered` already treats neither as coverage. But silence + # on both would let this rule say "No existing index leads with it" at HIGH next + # to an index that, in plain English, does lead with it — just partially, or + # under an expression. Naming them is what ADV001 does for the same reason. + partial_skipped = tuple( + index.name + for index in table_indexes + if index.is_partial and _is_prefix((item.column,), index.columns) + ) + expression_indexes = tuple( + index.name + for index in table_indexes + if index.has_expressions + and mentions_identifier(item.column, index.definition or "") + ) + column_ndv = ndv.get(item.column) + if rows is None or not have_index_data: + confidence = Confidence.LOW + elif column_ndv is None: + confidence = Confidence.MEDIUM + elif column_ndv >= SELECTIVE_NDV: + confidence = Confidence.HIGH + else: + confidence = Confidence.LOW + + rationale = ( + "This column carries the table's hottest join predicate. A join key is " + "probed once per outer row, so without an index leading with it every probe " + "is a scan." + ) + if have_index_data: + rationale += " No existing index leads with it." + else: + rationale += ( + " The existing-index list could not be read, so whether an index " + "already leads with it is unknown — check before applying." + ) + if rows is None: + rationale += _UNKNOWN_ROWS_NOTE + if column_ndv is not None and column_ndv < SELECTIVE_NDV: + rationale += ( + f" Only about {column_ndv:.0f} distinct values, so the index may not be " + "selective enough to be worth its write cost." + ) + if partial_skipped: + rationale += ( + f" A partial index ({', '.join(partial_skipped)}) leads with these " + "columns but carries a WHERE predicate, so it does not serve an " + "unfiltered lookup — it is not treated as covering this proposal." + ) + if expression_indexes: + rationale += ( + f" An expression index ({', '.join(expression_indexes)}) mentions " + f"{item.column}; sqlquality cannot tell whether it already serves this " + "lookup, so confirm before applying." + ) + + proposals.append( + Proposal( + code="ADV007", + title=f"Add index on join key {relation}({item.column})", + rationale=rationale, + evidence={ + "schema": relation.schema, + "table": relation.table, + "columns": (item.column,), + "roles": (item.role.value,), + "cost_share": item.cost_share, + "calls": item.calls, + "fingerprints": item.fingerprints, + "row_estimate": rows, + "leading_ndv": column_ndv, + "partial_indexes_skipped": partial_skipped, + "expression_indexes": expression_indexes, + }, + confidence=confidence, + ddl=( + f"CREATE INDEX ON {_qualified(relation.schema, relation.table)} " + f"({_quote_ident(item.column)});" + ), + ) + ) + return proposals + + +def propose_grouping_indexes( + usage: Sequence[ColumnUsage], + facts: Mapping[Relation, TableFacts], + existing: Mapping[Relation, Sequence[PgIndex]], + *, + min_cost_share: float, + min_rows: int = MIN_ROWS_FOR_INDEX, + max_arity: int = MAX_INDEX_ARITY, + have_index_data: bool = True, +) -> list[Proposal]: + """ADV008 — an index that can feed a hot GROUP BY already sorted. + + Confidence is capped at MEDIUM and there is deliberately no HIGH branch. Whether + Postgres uses such an index depends on its choice between `GroupAggregate` (which wants + sorted input, and is what the index provides) and `HashAggregate` (which does not) — a + decision driven by `work_mem`, the number of groups and the aggregates involved, none of + which this tool can see. Claiming HIGH would be asserting something about the planner + rather than about the catalog. Do not add a HIGH branch here for symmetry with ADV001. + + One composite rather than several single-column indexes, unlike ADV007: `GROUP BY a, b` + wants input ordered by `(a, b)`, which two separate indexes cannot provide. The column + *order* is inferred from cost, not read from the query — redaction and fingerprinting do + not preserve each column's position in the GROUP BY clause — and the rationale says so, + because getting the order wrong makes the index serve only its leading column. + + Extension requires *joint* support — every chosen column sharing a fingerprint with every + other chosen column, via a running intersection — not merely pairwise support with the + seed. Checking only against the seed lets a transitive chain through: `a` grouped with `b` + in one query and with `c` in another welds `(a, b, c)` into one composite that no query + groups by, even though `a` alone passes both pairwise checks. That composite would still + report cost and fingerprint evidence that reads as support, which is worse than proposing + nothing. + + Evidence carries `co_occurring_fingerprints` (the joint overlap size) rather than a plain + `fingerprints` count — unlike ADV007 and ADV005, which each speak for a single column, so a + per-column count is the whole truth about them. This rule, ADV004 and ADV001 all propose an + index justified by columns appearing *together*, where the joint overlap is the only number + that actually supports the proposal; a per-column count sitting beside it in a report that + renders evidence as bare `k=v` pairs would read as corroborating support that is not there. + The split is by what the rule claims, not by which rule came first — ADV001 joined this side + of it once it too required joint co-occurrence. + """ + proposals: list[Proposal] = [] + for relation, items in sorted(_by_relation(usage).items()): + table_facts = facts.get(relation) + rows = table_facts.row_estimate if table_facts else None + if rows is not None and rows < min_rows: + continue + grouping = sorted( + (i for i in items if i.role is ColumnRole.GROUP), + key=lambda i: (-i.cost_ms, i.column), + ) + if not grouping: + continue + seed = grouping[0] + # Extend the composite only with columns that share a fingerprint with *every* + # column already chosen — not merely with the seed. Pairwise-with-seed is not + # enough: given a in {fp1, fp2}, b in {fp1} and c in {fp2}, checking each candidate + # against the seed alone welds (a, b, c) into one composite even though no query + # groups by all three — fp1 groups by (a, b), fp2 groups by (a, c). Tracking the + # running intersection catches this: the moment a candidate's overlap does not + # include every fingerprint the chosen set already agrees on, its joint support for + # the *whole* composite is a query that does not exist. + chosen = [seed] + shared = seed.fingerprint_ids + for candidate in grouping[1:]: + if len(chosen) >= max_arity: + break + overlap = shared & candidate.fingerprint_ids + if overlap: + chosen.append(candidate) + shared = overlap + + cost_share = max(i.cost_share for i in chosen) + if cost_share < min_cost_share: + continue + columns = tuple(i.column for i in chosen) + if _covered(columns, existing.get(relation, ())) is not None: + continue + + # Same guards as ADV001 and ADV007, same reasons: a partial index leading with + # these columns does not serve an unfiltered GROUP BY, and an expression index's + # `columns` tuple understates it, so `_covered` already treats neither as coverage. + # But silence on both would let this rule say nothing next to an index that, in + # plain English, does lead with these columns — just partially, or under an + # expression. Naming them is what ADV001 and ADV007 do for the same gap. + table_indexes = existing.get(relation, ()) + partial_skipped = tuple( + index.name + for index in table_indexes + if index.is_partial and _is_prefix(columns, index.columns) + ) + expression_indexes = tuple( + index.name + for index in table_indexes + if index.has_expressions and mentions_identifier(columns[0], index.definition or "") + ) + + rationale = ( + "This grouping carries a hot share of workload cost. An index on these columns " + "lets the planner read the rows already ordered and group them without a sort. " + "The column order here is inferred from cost, not read from the query — " + "redaction does not preserve each column's position in the GROUP BY — so check " + "it against the actual grouping before applying, since a composite index only " + "serves the grouping it leads with." + ) + if not have_index_data: + rationale += ( + " The existing-index list could not be read, so whether an index already " + "leads with these columns is unknown." + ) + if rows is None: + rationale += _UNKNOWN_ROWS_NOTE + if partial_skipped: + rationale += ( + f" A partial index ({', '.join(partial_skipped)}) leads with these columns " + "but carries a WHERE predicate, so it does not serve an unfiltered lookup — " + "it is not treated as covering this proposal." + ) + if expression_indexes: + rationale += ( + f" An expression index ({', '.join(expression_indexes)}) mentions " + f"{columns[0]}; sqlquality cannot tell whether it already serves this " + "lookup, so confirm before applying." + ) + + proposals.append( + Proposal( + code="ADV008", + title=f"Add index for GROUP BY on {relation}({', '.join(columns)})", + rationale=rationale, + evidence={ + "schema": relation.schema, + "table": relation.table, + "columns": columns, + "roles": tuple(i.role.value for i in chosen), + "cost_share": cost_share, + "calls": max(i.calls for i in chosen), + #: How many query groups actually group by *every* chosen column + #: together — the running intersection, not a per-column count. Same + #: name and same meaning as ADV004's and ADV001's identical field. + #: Deliberately no plain `fingerprints` key here (unlike ADV007, which + #: speaks for a single column and so has only a per-column count to give): + #: this proposal is justified by columns appearing *together*, so a + #: per-column count sitting beside the joint one in a report that + #: renders evidence as bare `k=v` pairs, with no per-rule text, would + #: read as more support than actually exists. + "co_occurring_fingerprints": len(shared), + "row_estimate": rows, + "partial_indexes_skipped": partial_skipped, + "expression_indexes": expression_indexes, + }, + confidence=( + Confidence.LOW if rows is None or not have_index_data else Confidence.MEDIUM + ), + ddl=( + f"CREATE INDEX ON {_qualified(relation.schema, relation.table)} " f"({', '.join(_quote_ident(c) for c in columns)});" ), ) @@ -427,10 +783,9 @@ def propose_indexes( def propose_unused_indexes( - existing: Mapping[str, Sequence[PgIndex]], + existing: Mapping[Relation, Sequence[PgIndex]], *, - hot_tables: frozenset[str], - schema: str = DEFAULT_SCHEMA, + hot_tables: frozenset[Relation], ) -> list[Proposal]: """ADV002 — indexes with zero recorded scans, excluding constraint-backing indexes. @@ -438,36 +793,37 @@ def propose_unused_indexes( reset, so zero scans cannot prove an index is unused across a full business cycle. """ proposals: list[Proposal] = [] - for table in sorted(hot_tables): - for index in existing.get(table, ()): + for relation in sorted(hot_tables): + for index in existing.get(relation, ()): if index.scans != 0 or index.is_unique or index.is_primary: continue proposals.append( Proposal( code="ADV002", - title=f"Drop unused index {index.name} on {table}", + title=f"Drop unused index {index.name} on {relation}", rationale=( "No recorded scans since the last statistics reset. Verify the " "reset time covers a full business cycle before dropping." ), evidence={ - "table": table, + "schema": relation.schema, + "table": relation.table, "index": index.name, "columns": index.columns, "scans": index.scans, "size_bytes": index.size_bytes, }, confidence=Confidence.MEDIUM, - ddl=f"DROP INDEX {_qualified(schema, index.name)};", + ddl=f"DROP INDEX {_qualified(relation.schema, index.name)};", ) ) return proposals def propose_redundant_indexes( - existing: Mapping[str, Sequence[PgIndex]], + existing: Mapping[Relation, Sequence[PgIndex]], *, - schema: str = DEFAULT_SCHEMA, + hot_tables: frozenset[Relation], ) -> list[Proposal]: """ADV003 — an index whose column list is a strict prefix of another's is redundant. @@ -478,9 +834,20 @@ def propose_redundant_indexes( "serves the same lookups" is simply false, since the partial index exists precisely to serve a subset the wider index serves differently. Emitting that at MEDIUM would still be advising a `DROP INDEX` with no basis: "probably wrong" is not a confidence level. + + Scoped to ``hot_tables`` — the relations the workload was actually observed using — + exactly as ADV002 is, and not to every key in ``existing``. ``fetch_indexes`` filters + tables by *bare* name, so with two requested schemas holding a same-named table it + returns rows for relations the run never analysed; iterating all of ``existing`` made + whether a schema got `DROP INDEX` hygiene depend on whether one of its tables happened to + collide by name with a hot table in another requested schema. Prefix redundancy is + provable from the catalog alone, so the advice was not *wrong* — but arbitrary scope for + a rule that emits `DROP` is not a scope, and this rule should be able to say which + workload its recommendation came from. """ proposals: list[Proposal] = [] - for table, indexes in sorted(existing.items()): + for relation in sorted(hot_tables): + indexes = existing.get(relation, ()) for narrow in indexes: # A partial or expression index is not comparable on column lists alone: the # predicate or the expression is the whole point of it. Skipping the pair is @@ -506,7 +873,7 @@ def propose_redundant_indexes( proposals.append( Proposal( code="ADV003", - title=f"Drop redundant index {narrow.name} on {table}", + title=f"Drop redundant index {narrow.name} on {relation}", rationale=( f"Its columns are a leading prefix of {wider.name}, which can serve " "the same lookups. Both indexes are plain — neither carries a WHERE " @@ -514,7 +881,8 @@ def propose_redundant_indexes( "whole comparison." ), evidence={ - "table": table, + "schema": relation.schema, + "table": relation.table, "index": narrow.name, "columns": narrow.columns, "superseded_by": wider.name, @@ -522,7 +890,7 @@ def propose_redundant_indexes( "size_bytes": narrow.size_bytes, }, confidence=Confidence.HIGH, - ddl=f"DROP INDEX {_qualified(schema, narrow.name)};", + ddl=f"DROP INDEX {_qualified(relation.schema, narrow.name)};", ) ) return proposals @@ -556,11 +924,10 @@ def _first_co_occurring( def propose_partial_indexes( usage: Sequence[ColumnUsage], - facts: Mapping[str, TableFacts], + facts: Mapping[Relation, TableFacts], *, min_cost_share: float, min_rows: int = MIN_ROWS_FOR_INDEX, - schema: str = DEFAULT_SCHEMA, ) -> list[Proposal]: """ADV004 — index the hot equality column, restricted by a hot null-check predicate. @@ -572,8 +939,8 @@ def propose_partial_indexes( count caps confidence at LOW rather than being assumed large. """ proposals: list[Proposal] = [] - for table, items in sorted(_by_table(usage).items()): - table_facts = facts.get(table) + for relation, items in sorted(_by_relation(usage).items()): + table_facts = facts.get(relation) rows = table_facts.row_estimate if table_facts else None if rows is not None and rows < min_rows: continue @@ -612,11 +979,13 @@ def propose_partial_indexes( Proposal( code="ADV004", title=( - f"Partial index on {table}({leading.column}) WHERE {guard.column} {predicate}" + f"Partial index on {relation}({leading.column}) " + f"WHERE {guard.column} {predicate}" ), rationale=rationale, evidence={ - "table": table, + "schema": relation.schema, + "table": relation.table, "columns": (leading.column,), "guard_column": guard.column, "guard_predicate": predicate, @@ -629,7 +998,7 @@ def propose_partial_indexes( }, confidence=Confidence.LOW if rows is None else Confidence.MEDIUM, ddl=( - f"CREATE INDEX ON {_qualified(schema, table)} " + f"CREATE INDEX ON {_qualified(relation.schema, relation.table)} " f"({_quote_ident(leading.column)}) " f"WHERE {_quote_ident(guard.column)} {predicate};" ), @@ -652,14 +1021,15 @@ def propose_sargability( proposals.append( Proposal( code="ADV005", - title=f"Non-sargable predicate on {item.table}.{item.column}", + title=f"Non-sargable predicate on {item.relation}.{item.column}", rationale=( "The column is wrapped in a cast or function inside a predicate, so a " "plain B-tree index cannot be used. Rewrite the predicate to leave the " "column bare, or add a matching expression index." ), evidence={ - "table": item.table, + "schema": item.relation.schema, + "table": item.relation.table, "column": item.column, "cost_share": item.cost_share, "calls": item.calls, @@ -699,15 +1069,73 @@ def propose_sargability( return proposals +def _wide_relations_touched( + sql: str, dialect: str, wide: Mapping[Relation, TableFacts] +) -> tuple[Relation, ...]: + """Which of the *wide* relations this one statement provably references. + + Bare-name text matching was the original approach — a `mentions_identifier` test of + `relation.table` against the statement text, since deleted along with its `mentions_table` + alias — and it is exactly right for an *unqualified* reference: real SQL says `from + orders`, not `from public.orders`. Its failure mode is the same one Task 2 already fixed + once on this branch for `star_tables`: text matching cannot see a schema qualifier, so + `select * from public.orders` matched *both* `public.orders` and `staging.orders` + whenever both were wide — an evidence block naming a table the statement never + referenced. + + So this parses the statement (with the adapter's own dialect — the same one `aggregate` + used to build `facts` in the first place) and resolves each `exp.Table` node against + `wide` specifically, not the full introspected schema: a schema-qualified reference + (`table.db` set) is matched only against that same relation; a bare reference is + attributed only when exactly one wide relation shares its name. An ambiguous bare name + — two wide relations sharing it — is dropped rather than guessed at, the same + cannot-prove-it policy `resolve_relation`/`star_tables` already apply, just restricted + here to the (usually much smaller) wide set, since that is all this rule can ever report + on. + + No parse-failure fallback. The premise is *not* "the same text parsed twice": `sql` is + `stat.sql`, which is sqlglot's own re-serialisation of the **redacted** tree, not the + `row.sql` that `ingest()` parsed — different text, so "it already parsed once" would not + be an argument. The real premise is that sqlglot re-parses its own generated SQL under + the dialect that generated it, which was measured rather than assumed: 0 reparse failures + across the 14-statement adversarial corpus, redaction included. `ingest()` also + guarantees the *original* row parsed (an unparseable row is counted as + `skipped_unparseable` and never becomes a `QueryStat`), so the input to redaction was + always a real tree. A bare-name text-match fallback used to sit here for "just in case", + but it reintroduced the exact over-attribution this function exists to fix — + `select * from public.orders` matching both `public.orders` and `staging.orders` — for a + branch nothing can reach. Same reasoning as `_dedupe_by_ddl`'s deleted tie-break: a + fallback that cannot be reached is worse than none, since it reads as evidence the case + is handled when the real handling is "it cannot happen". If a round-trip ever did fail, + it raises here instead of silently mis-attributing evidence — the direction to fail in. + """ + by_bare_name: dict[str, list[Relation]] = {} + for relation in wide: + by_bare_name.setdefault(relation.table, []).append(relation) + tree = parse(sql, dialect) + touched: set[Relation] = set() + for table in tree.find_all(exp.Table): + if table.db: + candidate = Relation(schema=table.db, table=table.name) + if candidate in wide: + touched.add(candidate) + continue + owners = by_bare_name.get(table.name, ()) + if len(owners) == 1: + touched.add(owners[0]) + return tuple(sorted(touched)) + + def propose_select_star( workload: Workload, - facts: Mapping[str, TableFacts], + facts: Mapping[Relation, TableFacts], *, min_cost_share: float, min_columns: int = WIDE_TABLE_COLUMNS, + dialect: str = "postgres", ) -> list[Proposal]: """ADV006 — hot query groups projecting a star from a wide table.""" - wide = {name for name, fact in facts.items() if len(fact.columns) >= min_columns} + wide = {relation: fact for relation, fact in facts.items() if len(fact.columns) >= min_columns} if not wide: return [] total = workload.total_cost_ms @@ -715,7 +1143,7 @@ def propose_select_star( for stat in workload.stats: if FLAG_SELECT_STAR not in stat.flags: continue - touched = sorted(name for name in wide if mentions_table(name, stat.sql)) + touched = sorted(_wide_relations_touched(stat.sql, dialect, wide)) if not touched: continue share = (stat.total_time_ms / total) if total else 0.0 @@ -724,14 +1152,19 @@ def propose_select_star( proposals.append( Proposal( code="ADV006", - title=f"Hot SELECT * over wide table(s): {', '.join(touched)}", + title=( + f"Hot SELECT * over wide table(s): " + f"{', '.join(str(relation) for relation in touched)}" + ), rationale=( "Projecting every column of a wide table moves data no consumer asked " "for. List the columns the query actually needs." ), evidence={ - "tables": tuple(touched), - "column_counts": {name: len(facts[name].columns) for name in touched}, + "tables": tuple(str(relation) for relation in touched), + "column_counts": { + str(relation): len(facts[relation].columns) for relation in touched + }, "cost_share": share, "calls": stat.calls, "fingerprint": _fingerprint_id(stat.fingerprint), @@ -777,6 +1210,40 @@ class PostgresWorkloadAdapter(WorkloadAdapter): # nothing. Task 8 unpacks it as `_rows`. # `total_exec_time` requires PostgreSQL 13+; it was `total_time` on 12 and older, # both long past end-of-life. The privilege hint states the floor. + # + # Deliberately NOT filtered on `s.toplevel`, and the reason is a *cost*, not an + # impossibility. Under `pg_stat_statements.track = all` a `COPY (SELECT ...) TO ...` + # produces two rows for one execution — the verbatim top-level utility statement and + # its normalised nested query — which `unwrap`/redaction give different fingerprints, + # so the same execution is counted as two query groups at roughly twice its true + # cost (documented in the README's "Prerequisites and limits"). + # + # A blanket `AND s.toplevel` is not the answer: `toplevel = false` is the *only* way + # Postgres ever exposes the SQL inside a PL/pgSQL function body, and verified live + # the blanket filter made a genuinely hot, function-wrapped query (3x the cost of the + # next candidate) vanish from evidence entirely while the surrounding + # `SELECT my_function()` call sites stayed counted as zero-column-usage cost with no + # signal that anything was dropped — a confidently wrong proposal, worse than an + # inflated cost_share. + # + # A *narrow* predicate, however, does exist and does work. Measured on PostgreSQL 16 + # under `track = all`, the two nested forms are textually distinguishable: a COPY's + # nested row KEEPS its wrapper (`COPY (SELECT ... $1) TO STDOUT`) while a PL/pgSQL + # body is recorded bare (`SELECT count(*) FROM ... WHERE status = $1`), so + # `NOT (s.toplevel = false AND s.query ~* '^\\s*COPY\\s*\\(')` removed exactly the + # duplicate (4 rows -> 3) and left the function body untouched. It is declined for a + # stated price rather than because nothing could work: *naming* `s.toplevel` at all + # requires PostgreSQL 14 (the column does not exist on 13), and the documented floor + # is 13+, so a PG13 user would lose the entire workload capability — one missing + # column costing the whole run — to remove a 2x over-count of one statement form + # under a non-default setting. That trade is why the filter is absent; if the floor + # ever rises to 14, this is the predicate to add. + # + # Not fixable by any predicate, and documented alongside the COPY case: under + # `track = all` every PL/pgSQL call is counted twice — measured, `SELECT lc.hot()` at + # 68.21 ms plus its body at 67.67 ms for one execution — which roughly halves every + # `cost_share` in the run. The call carries the cost while the body carries the + # predicates, so excluding either row loses something real. CAP_WORKLOAD: """ SELECT s.query, s.calls, s.total_exec_time, s.rows FROM pg_stat_statements s @@ -791,18 +1258,18 @@ class PostgresWorkloadAdapter(WorkloadAdapter): WHERE datname = current_database() """, CAP_SCHEMA: """ - SELECT c.table_name, c.column_name, c.data_type + SELECT c.table_schema, c.table_name, c.column_name, c.data_type FROM information_schema.columns c WHERE c.table_schema = ANY(%s) """, CAP_TABLE_FACTS: """ - SELECT c.relname, c.reltuples::bigint, pg_total_relation_size(c.oid) + SELECT n.nspname, c.relname, c.reltuples::bigint, pg_total_relation_size(c.oid) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind = 'r' AND n.nspname = ANY(%s) AND c.relname = ANY(%s) """, CAP_NDV: """ - SELECT s.tablename, s.attname, s.n_distinct + SELECT s.schemaname, s.tablename, s.attname, s.n_distinct FROM pg_stats s WHERE s.schemaname = ANY(%s) AND s.tablename = ANY(%s) """, @@ -816,7 +1283,7 @@ class PostgresWorkloadAdapter(WorkloadAdapter): # serve its bare column — both of which the coverage and redundancy rules previously # had to guess at. CAP_INDEXES: """ - SELECT t.relname, i.relname, a.attname, k.ordinality, + SELECT n.nspname, t.relname, i.relname, a.attname, k.ordinality, ix.indisunique, ix.indisprimary, COALESCE(psui.idx_scan, 0), pg_relation_size(i.oid), ix.indpred IS NOT NULL, @@ -831,7 +1298,7 @@ class PostgresWorkloadAdapter(WorkloadAdapter): LEFT JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum LEFT JOIN pg_stat_user_indexes psui ON psui.indexrelid = i.oid WHERE n.nspname = ANY(%s) AND t.relname = ANY(%s) - ORDER BY t.relname, i.relname, k.ordinality + ORDER BY n.nspname, t.relname, i.relname, k.ordinality """, } @@ -955,32 +1422,52 @@ def _schema_rows(self, schemas: tuple[str, ...]) -> list[tuple[object, ...]]: return self._schema_cache[schemas] def fetch_schema(self, schemas: tuple[str, ...]) -> dict: - schema: dict[str, dict[str, str]] = {} - for table, column, data_type in self._schema_rows(schemas): - schema.setdefault(str(table), {})[str(column)] = str(data_type) + """Nested schema mapping for sqlglot qualify(): {schema: {table: {column: type}}}. + + Nested rather than flat because `qualify()` needs to be able to *tell* two + same-named tables apart — a flat map resolves a column against the union of both + column sets, which is how a filter on a column that exists in only one of them was + silently accepted. + """ + schema: dict[str, dict[str, dict[str, str]]] = {} + for schema_name, table, column, data_type in self._schema_rows(schemas): + schema.setdefault(str(schema_name), {}).setdefault(str(table), {})[str(column)] = str( + data_type + ) return schema def fetch_table_facts( - self, schemas: tuple[str, ...], tables: frozenset[str] - ) -> dict[str, TableFacts]: - wanted = sorted(tables) + self, schemas: tuple[str, ...], relations: frozenset[Relation] + ) -> dict[Relation, TableFacts]: + # The `= ANY(%s)` table parameter stays a list of bare names: Postgres filters on + # `relname`/`tablename`, and narrowing per-schema would need one statement per + # schema. `n.nspname = ANY(%s)` still restricts rows to `schemas`, so a table in a + # schema we were not asked to introspect at all never comes back. What can + # over-fetch is a same-named table in a *different requested* schema that is not + # itself in `relations` — `schemas=("sales", "staging")` with `relations` naming + # only `sales.orders` still returns `staging.orders`, because the bare-name filter + # cannot distinguish the two. That row's relation key then simply has no consumer, + # since only the relations in `relations` are ever assembled into the result below. + wanted = sorted({relation.table for relation in relations}) sizes = { - str(name): ( + Relation(schema=str(schema_name), table=str(name)): ( _row_estimate(rows), _as_int(size) if size is not None else None, ) - for name, rows, size in self._run(CAP_TABLE_FACTS, (list(schemas), wanted)) + for schema_name, name, rows, size in self._run(CAP_TABLE_FACTS, (list(schemas), wanted)) } - columns: dict[str, list[str]] = {} - for table, column, _type in self._schema_rows(schemas): - if str(table) in tables: - columns.setdefault(str(table), []).append(str(column)) - - ndv: dict[str, dict[str, float]] = {} - for table, column, n_distinct in self._run(CAP_NDV, (list(schemas), wanted)): + columns: dict[Relation, list[str]] = {} + for schema_name, table, column, _type in self._schema_rows(schemas): + relation = Relation(schema=str(schema_name), table=str(table)) + if relation in relations: + columns.setdefault(relation, []).append(str(column)) + + ndv: dict[Relation, dict[str, float]] = {} + for schema_name, table, column, n_distinct in self._run(CAP_NDV, (list(schemas), wanted)): if n_distinct is None: continue value = _as_float(n_distinct) + relation = Relation(schema=str(schema_name), table=str(table)) if value < 0: # Postgres encodes "distinct as a fraction of row count" as a negative # value, which is meaningless without the row count. If the row-count query @@ -988,33 +1475,47 @@ def fetch_table_facts( # privileges can hide it — omit the column so it reads as *unknown*. # Defaulting the row estimate to 0 would fabricate "zero distinct values" # and hand every proposal on this table a false LOW-confidence rating. - row_estimate = sizes.get(str(table), (None, None))[0] + row_estimate = sizes.get(relation, (None, None))[0] if row_estimate is None: continue resolved = -value * row_estimate else: resolved = value - ndv.setdefault(str(table), {})[str(column)] = resolved + ndv.setdefault(relation, {})[str(column)] = resolved - facts: dict[str, TableFacts] = {} - for table in wanted: - rows, size = sizes.get(table, (None, None)) - facts[table] = TableFacts( - name=table, + facts: dict[Relation, TableFacts] = {} + for relation in sorted(relations): + rows, size = sizes.get(relation, (None, None)) + facts[relation] = TableFacts( + relation=relation, row_estimate=rows, size_bytes=size, - columns=tuple(columns.get(table, ())), - ndv=ndv.get(table, {}), + columns=tuple(columns.get(relation, ())), + ndv=ndv.get(relation, {}), ) return facts def fetch_indexes( - self, schemas: tuple[str, ...], tables: frozenset[str] - ) -> dict[str, tuple[PgIndex, ...]]: - """Existing indexes per table, columns in ordinal order.""" - grouped: dict[tuple[str, str], _IndexRows] = {} - for row in self._run(CAP_INDEXES, (list(schemas), sorted(tables))): + self, schemas: tuple[str, ...], relations: frozenset[Relation] + ) -> dict[Relation, tuple[PgIndex, ...]]: + """Existing indexes per relation, columns in ordinal order.""" + # See the identical note in fetch_table_facts: the table parameter is bare names, + # so a same-named table in a *different requested* schema not itself in + # `relations` can come back too — `n.nspname = ANY(%s)` still excludes a schema we + # were not asked to introspect at all. + # + # Unlike fetch_table_facts, this method does NOT drop those rows: `_covered` needs + # only the relations it is asked about, but the returned mapping is also handed + # whole to ADV002 and ADV003, which iterate it. Both are therefore scoped to + # `aggregation.tables` by their callers rather than to `existing`'s key set — an + # earlier version of this comment claimed the over-fetched rows had "no consumer", + # and ADV003 was that consumer, emitting `DROP INDEX` for relations the workload + # never touched whenever a bare name collided across two requested schemas. + wanted = sorted({relation.table for relation in relations}) + grouped: dict[tuple[Relation, str], _IndexRows] = {} + for row in self._run(CAP_INDEXES, (list(schemas), wanted)): ( + schema_name, table, index, column, @@ -1028,8 +1529,9 @@ def fetch_indexes( has_expressions, definition, ) = row + relation = Relation(schema=str(schema_name), table=str(table)) entry = grouped.setdefault( - (str(table), str(index)), + (relation, str(index)), _IndexRows( is_unique=bool(unique), is_primary=bool(primary), @@ -1050,9 +1552,9 @@ def fetch_indexes( if column is not None: entry.columns.append((_as_int(ordinality), str(column))) - result: dict[str, list[PgIndex]] = {} - for (table, index), entry in grouped.items(): - result.setdefault(table, []).append( + result: dict[Relation, list[PgIndex]] = {} + for (relation, index), entry in grouped.items(): + result.setdefault(relation, []).append( PgIndex( name=index, columns=tuple(column for _ordinality, column in sorted(entry.columns)), @@ -1066,61 +1568,349 @@ def fetch_indexes( definition=entry.definition, ) ) - return {table: tuple(indexes) for table, indexes in result.items()} + return {relation: tuple(indexes) for relation, indexes in result.items()} #: Highest confidence first, then largest cost share — the reading order a human wants. _CONFIDENCE_ORDER = {Confidence.HIGH: 0, Confidence.MEDIUM: 1, Confidence.LOW: 2} + #: Which rule's rationale to keep when two rules propose byte-identical DDL at equal + #: confidence. Lower wins. The order is by how directly the evidence supports *this* + #: index: a filter predicate (ADV001) is the most direct reason to build a B-tree, a + #: join key (ADV007) next, a partial index (ADV004) next since its own `WHERE` clause is + #: already a stronger claim than a plain composite's, and a grouping (ADV008) last, since + #: whether the planner uses an index for grouping depends on choices this tool cannot + #: see. The DROP rules are ranked below them so a CREATE never loses to a DROP that + #: happens to render the same text — which it cannot today, but this map is the place + #: that would have to change. `.get(code, len(...))` at every call site rather than + #: `[code]`, so a code missing from this map sorts last instead of raising `KeyError` + #: after the whole analysis has run. + _CODE_PREFERENCE = { + "ADV001": 0, + "ADV007": 1, + "ADV004": 2, + "ADV008": 3, + "ADV003": 4, + "ADV002": 5, + } + + @classmethod + def _attribution(cls, discarded: Proposal, *, same_index: bool) -> str: + """How the folded text introduces a discarded proposal — one phrasing per collapse + kind, because the two collapses did different things to it. + + `_dedupe_by_ddl` collapses byte-identical DDL, so "reached the same index" is true by + construction. `_collapse_index_prefixes` collapses a *narrower* proposal into a wider + one, where it is never true: the operator was told "ADV007 reached the same index at + high confidence" under `Add index on sales.orders(customer_id, tenant_id, + created_at)` when ADV007 proposed `(customer_id)` — an endorsement of a three-column + index that no rule ever made, in the paragraph someone reads before running DDL. The + second symptom is the same sentence: an ADV008 survivor carried ADV001's "Equality + columns come first so the range column can be scanned last" with no equality columns + anywhere in it. Naming the narrower column list gives the borrowed sentences the + subject they are actually about. + """ + if same_index: + return f"{discarded.code} reached the same index" + key = cls._index_creation_columns(discarded) + # `None` is unreachable from `_collapse_index_prefixes`, which only ever absorbs + # proposals this same function accepted — but the fallback keeps the sentence + # grammatical rather than raising in a report renderer if a future caller differs. + narrower = f"({', '.join(key[1])})" if key else "a narrower index" + return ( + f"{discarded.code} proposed the narrower {narrower} on the same table, which this " + f"index's leading columns already serve, and said of it" + ) + + @classmethod + def _fold_discarded( + cls, survivor: Proposal, discarded: Sequence[Proposal], *, same_index: bool + ) -> Proposal: + """Attach every discarded proposal's distinguishing rationale, attributed, to the + survivor's. + + ``same_index`` distinguishes the two callers — see `_attribution`. It is required + rather than defaulted: a new collapse rule that forgets to say which kind it is would + otherwise silently claim an endorsement it did not get. + + `_dedupe_by_ddl` and `_collapse_index_prefixes` each throw a whole `Proposal` away + and keep only one rationale where two, or more, existed. Diffing the texts to keep + "only the part that's new" at the paragraph level would need to guess which + sentence is the caveat worth keeping — fragile, and silently wrong the moment a + rule's wording changes elsewhere. Instead, every discarded rationale is split into + whole sentences and folded in verbatim, in order, *skipping a sentence only if that + exact sentence already appears* — in the survivor's rationale or in an + already-folded discarded one. ADV001, ADV007 and ADV008 share verbatim wording for + the partial-index and expression-index disclosures, so a real three-way collision + would otherwise repeat the same sentence up to three times in one paragraph — a + report an operator reads to decide whether to run the DDL should not look broken + that way. A sentence unique to one discarded proposal always survives: only exact + repeats are dropped, never trimmed or summarised. The discarded proposal's + confidence is stated regardless of whether any of its sentences are new, since a + reader comparing two proposals for the same index needs to know they disagreed on + how sure to be, not just what each said. + """ + if not discarded: + return survivor + seen = set(_sentences(survivor.rationale)) + notes: list[str] = [] + for p in discarded: + fresh = [s for s in _sentences(p.rationale) if s not in seen] + seen.update(fresh) + attribution = cls._attribution(p, same_index=same_index) + if fresh: + notes.append( + f" {attribution} at {p.confidence.value} confidence: {' '.join(fresh)}" + ) + else: + notes.append( + f" {attribution} at {p.confidence.value} confidence, stating nothing " + "beyond what is already covered above." + ) + return replace(survivor, rationale=survivor.rationale + "".join(notes)) + @classmethod def _dedupe_by_ddl(cls, proposals: list[Proposal]) -> list[Proposal]: """Collapse proposals that would run identical DDL, keeping the strongest evidence. - An index with no recorded scans that is *also* a prefix of a wider index gets - flagged by both ADV002 and ADV003, producing two entries with the same - `DROP INDEX`. They do not contradict each other, but a reader should not have to - notice they are the same object twice. ADV003 is the one kept, because prefix - redundancy is structural — provable from the column lists alone — whereas ADV002 - rests on a scan counter that only covers the window since the last statistics reset. - - That preference needs no tie-break rule to state it: the two codes cannot tie. - ADV002 is hardcoded MEDIUM — `idx_scan` accumulates only since the last statistics - reset, so zero scans can never prove disuse across a business cycle — and ADV003 is - hardcoded HIGH, because it can now read `indpred` and so declines to call a partial - index redundant rather than guessing. Confidence alone decides, and it decides the - way this docstring says it should. - - There was a window where they *did* tie, and it is why a second tuple element used - to be here: ADV003 was briefly capped at MEDIUM on the grounds that it could not see - partial-index predicates. It can, so the cap is gone and so is the tie. A tie-break - that cannot be reached is worse than none — it reads as evidence the collision is - handled where the confidence values are what actually handle it. + Two rules can genuinely reach the same index from different evidence — a filter + predicate, a join key and a grouping on the same column all render the same + `CREATE INDEX` — and an unused index that is also a prefix of a wider one is flagged + by both ADV002 and ADV003 as the same `DROP INDEX`. They do not contradict each + other, but a reader should not have to notice they are the same object twice. + + Confidence decides first. When it ties, `_CODE_PREFERENCE` decides, because + something has to and list order must not: which rationale leads should not be + "whichever `propose()` happened to append first". The proposal that does not lead + is not thrown away, though — its rationale is folded into the survivor's via + `_fold_discarded`, so a caveat the winner never states does not vanish with it. + + There was a window where no tie was reachable — ADV002 is hardcoded MEDIUM and + ADV003 HIGH, the only colliding pair at the time — and the tie-break was removed as + unreachable code, on the reasoning that a tie-break nothing can reach is worse than + none. That reasoning stopped holding the moment two more index-creating rules + existed: ADV001 at MEDIUM (NDV unknown) and ADV008 at MEDIUM (row count known, which + is all ADV008 ever checks) can produce byte-identical DDL at the same confidence, and + list order is not a rule — it is a coincidence of which call happens to come first in + `propose()`, and deciding which rationale the operator reads is too important to + leave to that. """ - best: dict[str, Proposal] = {} + + def rank(proposal: Proposal) -> tuple[int, int]: + return ( + cls._CONFIDENCE_ORDER[proposal.confidence], + cls._CODE_PREFERENCE.get(proposal.code, len(cls._CODE_PREFERENCE)), + ) + + groups: dict[str, list[Proposal]] = {} + for proposal in proposals: + if proposal.ddl: + groups.setdefault(proposal.ddl, []).append(proposal) + + merged: dict[str, Proposal] = {} + for ddl, group in groups.items(): + if len(group) == 1: + merged[ddl] = group[0] + continue + # `sorted` is stable, so a true tie in `rank` (both confidence and code + # preference equal — only possible today if the same code proposes the same + # DDL twice) keeps whichever proposal `propose()` happened to append first. + # That residual is accepted rather than papered over with a further tie-break + # key: two proposals with the same code, the same confidence and the same DDL + # carry no information that distinguishes them, so which one is kept cannot + # matter to a reader the way which *code* is kept does. + ranked = sorted(group, key=rank) + winner, *losers = ranked + merged[ddl] = cls._fold_discarded(winner, losers, same_index=True) + + result: list[Proposal] = [] + emitted: set[str] = set() for proposal in proposals: if not proposal.ddl: + result.append(proposal) continue - incumbent = best.get(proposal.ddl) - if ( - incumbent is None - or cls._CONFIDENCE_ORDER[proposal.confidence] - < cls._CONFIDENCE_ORDER[incumbent.confidence] - ): - best[proposal.ddl] = proposal - return [p for p in proposals if not p.ddl or best[p.ddl] is p] + if proposal.ddl in emitted: + continue + emitted.add(proposal.ddl) + result.append(merged[proposal.ddl]) + return result + + @classmethod + def _index_creation_columns(cls, proposal: Proposal) -> tuple[Relation, tuple[str, ...]] | None: + """`(relation, columns)` for a plain `CREATE INDEX` proposal, or `None` if the + proposal cannot participate in prefix collapsing or overlap disclosure. + + Restricted to plain composite/single-column indexes: a `WHERE` predicate (ADV004's + partial indexes) makes the index a different object even when its column list is a + prefix of a plain index's — the same reasoning `_covered` and + `propose_redundant_indexes` already apply to *existing* indexes, applied here to + proposals that do not exist as catalog rows yet. `DROP INDEX` proposals (ADV002, + ADV003) are excluded by the `CREATE INDEX` check: a prefix relationship between + something being created and something being dropped is not a meaningful comparison. + """ + if ( + not proposal.ddl + or not proposal.ddl.startswith("CREATE INDEX") + or "WHERE" in proposal.ddl + ): + return None + schema = proposal.evidence.get("schema") + table = proposal.evidence.get("table") + columns = proposal.evidence.get("columns") + if not isinstance(schema, str) or not isinstance(table, str): + return None + if not isinstance(columns, tuple) or not columns: + return None + return Relation(schema=schema, table=table), columns + + @classmethod + def _collapse_index_prefixes(cls, proposals: list[Proposal]) -> list[Proposal]: + """Collapse a `CREATE INDEX` proposal whose columns are a strict prefix of + another's, within the same relation. + + ADV001, ADV007 and ADV008 can each reach a plain index from different evidence, and + nothing stopped one proposing `(customer_id, created_at)` while another proposed + `(customer_id)` in the same report — confirmed end-to-end through `propose()` from a + single ordinary query, both at HIGH. An operator who creates both then holds a pair + where the narrower is a strict prefix of the wider: exactly what + `propose_redundant_indexes` (ADV003) flags as redundant on the *next* run. Shipping + both here would be advising a CREATE today and a DROP tomorrow for the same index. + + The wider proposal always survives — it serves every lookup the narrower one does — + and the narrower one is folded into it via `_fold_discarded`, so its rationale is + never silently dropped. When a narrower proposal is a prefix of more than one + *incomparable* wider proposal (say `(a)` under both `(a, b)` and `(a, c)`, neither a + prefix of the other), it is folded into all of them: there is no principled way to + prefer one over the other, and folding into both costs nothing but a repeated + sentence. + + Two proposals that cover the same column *set* in a different order are not a + prefix pair — same length, unequal tuples, so `_is_prefix` is false in both + directions — and are deliberately left alone here; `_disclose_column_set_overlaps` + handles that case by disclosure instead of collapse, since neither is redundant with + the other. + + Only within one relation: a prefix relationship across two different tables is + meaningless. Only plain proposals participate — see `_index_creation_columns`. + """ + eligible: dict[int, tuple[Relation, tuple[str, ...]]] = {} + for i, proposal in enumerate(proposals): + key = cls._index_creation_columns(proposal) + if key is not None: + eligible[i] = key + + # `covers[i]` collects every j whose columns strictly contain i's as a leading + # prefix — direct parents and transitive ancestors alike, since the prefix relation + # on tuples is transitive: checking every pair once already finds them all. + covers: dict[int, list[int]] = {} + for i, (relation_i, columns_i) in eligible.items(): + for j, (relation_j, columns_j) in eligible.items(): + if ( + i != j + and relation_i == relation_j + and len(columns_i) < len(columns_j) + and _is_prefix(columns_i, columns_j) + ): + covers.setdefault(i, []).append(j) + + # Maximal: nothing wider exists for it, so it is never removed. + maximal = {i for i in eligible if i not in covers} + + # Group each absorbed proposal under every maximal proposal it is a prefix of, then + # sort each group by a key with no dependency on `proposals`' incoming order — the + # collapse must not depend on which order `propose()` happened to append rules in. + absorbed_into: dict[int, list[Proposal]] = {} + for i, targets in covers.items(): + for j in targets: + if j in maximal: + absorbed_into.setdefault(j, []).append(proposals[i]) + + result = list(proposals) + for j, absorbed in absorbed_into.items(): + ordered = sorted(absorbed, key=lambda p: (p.code, p.title, p.ddl or "")) + result[j] = cls._fold_discarded(proposals[j], ordered, same_index=False) + + drop = set(covers) + return [p for idx, p in enumerate(result) if idx not in drop] + + @classmethod + def _disclose_column_set_overlaps(cls, proposals: list[Proposal]) -> list[Proposal]: + """When two surviving `CREATE INDEX` proposals cover the same column *set* for the + same relation in a different order, say so in both, naming the other. + + `(status, region)` and `(region, status)` are not redundant — different leading + columns genuinely serve different probes — so `_collapse_index_prefixes` correctly + leaves both standing, and no future ADV003 pass will ever reconcile them either: + prefix redundancy is the only structural overlap it can prove, and neither is a + prefix of the other. Silence here would recommend two overlapping indexes with no + acknowledgement that they overlap, leaving the operator to notice on their own that + creating both means indexing the same columns twice. + + With only two proposals sharing a column set this is symmetric and order cannot + matter. With three or more (today latent: ADV001 and ADV008 each propose at most one + composite per relation and ADV007 is single-column only, so three same-set + proposals cannot occur yet), a given proposal names *every* other member sharing its + set, and those names are sorted by the same canonical key + `_collapse_index_prefixes` uses — rather than by the order pairs happened to be + discovered in — so which proposal appended first cannot change the resulting text. + """ + pairs: list[tuple[int, tuple[Relation, tuple[str, ...]]]] = [] + for i, proposal in enumerate(proposals): + key = cls._index_creation_columns(proposal) + if key is not None: + pairs.append((i, key)) + + def sort_key(proposal: Proposal) -> tuple[str, str, str]: + return (proposal.code, proposal.title, proposal.ddl or "") + + notes: dict[int, list[tuple[tuple[str, str, str], str]]] = {} + for a in range(len(pairs)): + i, (relation_i, columns_i) = pairs[a] + for b in range(a + 1, len(pairs)): + j, (relation_j, columns_j) = pairs[b] + if relation_i != relation_j or columns_i == columns_j: + continue + if set(columns_i) != set(columns_j): + continue + notes.setdefault(i, []).append( + ( + sort_key(proposals[j]), + f"{proposals[j].code} proposes an index on the same columns in a " + f"different order ({', '.join(columns_j)}) for the same table. " + "Neither is redundant — different leading columns serve different " + "probes — but creating both means two overlapping indexes; confirm " + "the workload needs both orderings before applying both.", + ) + ) + notes.setdefault(j, []).append( + ( + sort_key(proposals[i]), + f"{proposals[i].code} proposes an index on the same columns in a " + f"different order ({', '.join(columns_i)}) for the same table. " + "Neither is redundant — different leading columns serve different " + "probes — but creating both means two overlapping indexes; confirm " + "the workload needs both orderings before applying both.", + ) + ) + + if not notes: + return proposals + result = list(proposals) + for idx, entries in notes.items(): + messages = [message for _key, message in sorted(entries, key=lambda e: e[0])] + result[idx] = replace( + result[idx], rationale=result[idx].rationale + " " + " ".join(messages) + ) + return result def propose( self, aggregation: Aggregation, - facts: dict[str, TableFacts], + facts: dict[Relation, TableFacts], workload: Workload, *, min_cost_share: float, ) -> list[Proposal]: existing = self.fetch_indexes(self.schemas, aggregation.tables) - # The rules are module-level and emit DDL, so they need the schema they are talking - # about. Only one schema is ever introspected (the CLI rejects more than one), so - # this is unambiguous rather than a guess about which one a proposal belongs to. - schema = self.schemas[0] if self.schemas else DEFAULT_SCHEMA # An empty `existing` means one of two very different things: this table genuinely # has no indexes, or the catalog query was denied. Only the adapter can tell, so it # is the adapter's job to say — ADV001 must not claim "no existing index leads with @@ -1132,18 +1922,34 @@ def propose( facts, existing, min_cost_share=min_cost_share, - schema=schema, have_index_data=have_index_data, ), - *propose_partial_indexes( - aggregation.usage, facts, min_cost_share=min_cost_share, schema=schema + *propose_join_keys( + aggregation.usage, + facts, + existing, + min_cost_share=min_cost_share, + have_index_data=have_index_data, + ), + *propose_grouping_indexes( + aggregation.usage, + facts, + existing, + min_cost_share=min_cost_share, + have_index_data=have_index_data, ), + *propose_partial_indexes(aggregation.usage, facts, min_cost_share=min_cost_share), *propose_sargability(aggregation.usage, workload, min_cost_share=min_cost_share), - *propose_select_star(workload, facts, min_cost_share=min_cost_share), - *propose_unused_indexes(existing, hot_tables=aggregation.tables, schema=schema), - *propose_redundant_indexes(existing, schema=schema), + *propose_select_star( + workload, facts, min_cost_share=min_cost_share, dialect=self.engine + ), + *propose_unused_indexes(existing, hot_tables=aggregation.tables), + *propose_redundant_indexes(existing, hot_tables=aggregation.tables), ] - return sorted(self._dedupe_by_ddl(proposals), key=self._ranking_key) + proposals = self._dedupe_by_ddl(proposals) + proposals = self._collapse_index_prefixes(proposals) + proposals = self._disclose_column_set_overlaps(proposals) + return sorted(proposals, key=self._ranking_key) @classmethod def _ranking_key(cls, proposal: Proposal) -> tuple[int, float, str, str]: diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 75f6892..7452f61 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -99,6 +99,61 @@ def seeded(live_dsn: str) -> tuple[str, str]: # fixture's caller queries it. ANALYZE makes the row estimate and NDV # deterministic instead of racing autovacuum. cur.execute(f"ANALYZE {schema}.orders") + + # Multi-schema keying: `orders` in both `public` and `staging`, with + # deliberately different row counts. This is the exact collision the old + # bare-name-keyed `_validate_schemas` refused to allow, and the only way an + # aliasing regression in `Relation`-keyed catalog facts can be caught. + cur.execute("DROP TABLE IF EXISTS public.orders CASCADE") + cur.execute( + "CREATE TABLE public.orders (id bigint, status text, tenant_id bigint, day date)" + ) + cur.execute( + "INSERT INTO public.orders " + "SELECT g, 'shipped', g % 7, current_date FROM generate_series(1, 20000) g" + ) + cur.execute("DROP SCHEMA IF EXISTS staging CASCADE") + cur.execute("CREATE SCHEMA staging") + cur.execute( + "CREATE TABLE staging.orders (id bigint, status text, tenant_id bigint, day date)" + ) + cur.execute( + "INSERT INTO staging.orders " + "SELECT g, 'draft', g % 7, current_date FROM generate_series(1, 50000) g" + ) + cur.execute("ANALYZE public.orders") + cur.execute("ANALYZE staging.orders") + + # An unindexed join key (ADV007) and, on the other side of it, a table left + # deliberately un-analyzed: `public.order_items` therefore carries + # `reltuples = -1` (Postgres's never-analyzed sentinel) for the whole run, + # proving the `row_estimate is None` path still proposes (at LOW confidence) + # rather than the pre-fix behaviour of reading -1 as "tiny table" and + # suppressing the proposal outright. + # + # `autovacuum_enabled = false` is load-bearing, not decoration: a bare "don't + # ANALYZE it" was measured to be non-deterministic — autovacuum picked up this + # table and analyzed it mid-run, about 2.5 seconds after seeding, well before + # any test's assertions ran, which silently turned this into a *never* case + # rather than a "not yet" case. Disabling autovacuum on this table, set before + # its INSERT, is what actually keeps `reltuples = -1` for the fixture's whole + # lifetime. Do not add an ANALYZE (or remove this setting) here. + cur.execute("DROP TABLE IF EXISTS public.order_items CASCADE") + cur.execute( + "CREATE TABLE public.order_items (id bigint, order_id bigint, sku text) " + "WITH (autovacuum_enabled = false)" + ) + cur.execute( + "INSERT INTO public.order_items " + "SELECT g, (g % 20000) + 1, 'sku' || g FROM generate_series(1, 20000) g" + ) + + # An index nothing in the workload below ever touches: not on `status` (the + # only equality predicate), not part of the GROUP BY -- so its scan count stays + # genuinely zero, giving ADV002 a real DROP INDEX candidate in `staging` to pair + # against the CREATE INDEX candidate `advise` proposes in `public`. + cur.execute("CREATE INDEX idx_unused_staging_id ON staging.orders (id)") + cur.execute("SELECT pg_stat_statements_reset()") # Real workload for the history statement to find. for _ in range(3): @@ -108,4 +163,45 @@ def seeded(live_dsn: str) -> tuple[str, str]: ("paid",), ) cur.fetchall() + + # A schema-qualified filter on each side of the public/staging collision, so + # both relations get their own usage and their own cost share. + for _ in range(5): + cur.execute("SELECT id FROM public.orders WHERE status = 'shipped'") + cur.fetchall() + cur.execute("SELECT id FROM staging.orders WHERE status = 'draft'") + cur.fetchall() + # A join key with no index leading with it (ADV007). + for _ in range(5): + cur.execute( + "SELECT o.id FROM public.orders o " + "JOIN public.order_items i ON i.order_id = o.id" + ) + cur.fetchall() + # A hot GROUP BY with no covering index (ADV008). + for _ in range(5): + cur.execute( + "SELECT tenant_id, day, count(*) FROM staging.orders GROUP BY tenant_id, day" + ) + cur.fetchall() + # A server-side cursor read. `WITH HOLD` is what keeps the cursor alive past + # this connection's per-statement autocommit boundary -- without it the cursor + # is dropped the instant the DECLARE's own implicit transaction commits, and + # FETCH fails with "cursor does not exist", not merely a filtered read. + # + # The predicate is `tenant_id = 3` deliberately, not `status = 'shipped'`: + # every other seeded statement filters on `status`, and once literals are + # redacted `status = 'shipped'` and `status = 'pending'` fingerprint + # identically. A test asserting unwrapping worked would pass just as well if + # unwrapping were reverted and this row were dropped as noise, because the + # *other* `status` query already produces the exact same query group. Filtering + # on a column no other statement filters on makes this query group exist if + # and only if the DECLARE was actually unwrapped. + cur.execute( + "DECLARE live_cur CURSOR WITH HOLD FOR " + "SELECT id FROM public.orders WHERE tenant_id = 3" + ) + cur.execute("FETCH 10 FROM live_cur") + cur.fetchall() + cur.execute("CLOSE live_cur") return live_dsn, schema diff --git a/tests/integration/test_advise_live.py b/tests/integration/test_advise_live.py index eccc4c7..4dddb69 100644 --- a/tests/integration/test_advise_live.py +++ b/tests/integration/test_advise_live.py @@ -12,11 +12,32 @@ from typer.testing import CliRunner from sqlquality.cli import app +from sqlquality.models import ConnectionParams, Relation +from sqlquality.workload.fingerprint import ingest +from sqlquality.workload.postgres import PostgresWorkloadAdapter pytestmark = pytest.mark.integration runner = CliRunner() +def _run_advise(seeded: tuple[str, str], *, schemas: tuple[str, ...]) -> dict: + """Invoke `advise --json` against the seeded database and return the whole payload. + + `--min-cost-share 0.0` so a rule firing is never masked by an unrelated cost-share + threshold — the tests using this helper are checking *whether a rule fires at all*, + not how it ranks against `--min-cost-share`'s default. The whole payload, not just + `proposals`, so callers can check `analyzed.tables` as a non-vacuity guard: a rule that + "fires" only because the relation it needed was never actually analyzed proves nothing. + """ + dsn, _schema = seeded + args = ["advise", "--dsn", dsn, "--json", "--min-cost-share", "0.0"] + for name in schemas: + args += ["--schema", name] + result = runner.invoke(app, args) + assert result.exit_code == 0, result.output + return json.loads(result.stdout) + + def test_advise_end_to_end(seeded, tmp_path): dsn, schema = seeded md = tmp_path / "report.md" @@ -41,7 +62,16 @@ def test_advise_end_to_end(seeded, tmp_path): assert payload["engine"] == "postgres" assert payload["redacted"] is True - assert payload["analyzed"]["query_groups"] > 0 + # Both numbers, and the arithmetic between them. `query_groups` is what the run + # *understood*; `query_groups_in_window` is what `pg_stat_statements` offered. A bare + # `> 0` on the former passed equally well when it silently carried the raw window count. + analyzed = payload["analyzed"] + assert analyzed["query_groups"] > 0 + assert analyzed["query_groups"] == ( + analyzed["query_groups_in_window"] + - payload["skipped"]["unqualifiable"] + - payload["skipped"]["ambiguous"] + ) assert payload["degraded"] == [] assert md.read_text(encoding="utf-8").startswith("# sqlquality advise") assert "REVIEW BEFORE RUNNING" in ddl.read_text(encoding="utf-8") @@ -89,3 +119,131 @@ def test_advise_dry_run_needs_no_server(tmp_path): result = runner.invoke(app, ["advise", "--engine", "postgres", "--dry-run"]) assert result.exit_code == 0 assert "pg_stat_statements" in result.stdout + + +def test_multi_schema_advise_run_produces_qualified_proposals(seeded): + """A multi-schema run must attribute proposals to the schema they came from. + + `staging` holds an index (`idx_unused_staging_id`) the workload never touches, and + `public` holds a hot, unindexed equality predicate — so a real run against both + schemas must produce at least one DROP INDEX for `staging` and at least one CREATE + INDEX for `public`, coexisting in the same proposal list. + """ + dsn, _schema = seeded + raw = PostgresWorkloadAdapter() + raw.connect(ConnectionParams(engine="postgres", dsn=dsn, fields={}, source="--dsn"), 30) + fetch = raw.fetch_workload(None, 500) + assert any("staging" in row.sql for row in fetch.rows), ( + "no staging-schema statement reached pg_stat_statements — fixture problem, not a bug" + ) + + payload = _run_advise(seeded, schemas=("public", "staging")) + assert "public.orders" in payload["analyzed"]["tables"], payload["analyzed"] + assert "staging.orders" in payload["analyzed"]["tables"], payload["analyzed"] + + proposals = payload["proposals"] + schemas = {p["evidence"].get("schema") for p in proposals} + assert "staging" in schemas, f"no proposal attributed to staging; got {sorted(schemas)}" + assert "public" in schemas, f"no proposal attributed to public; got {sorted(schemas)}" + + # Every DDL statement names the schema of the relation it belongs to. + ddl_actions = set() + for proposal in proposals: + ddl = proposal["ddl"] + if not ddl: + continue + assert f'"{proposal["evidence"]["schema"]}".' in ddl, proposal + ddl_actions.add((proposal["evidence"]["schema"], ddl.split()[0])) + + assert ("public", "CREATE") in ddl_actions, ( + f"no CREATE INDEX proposal for public; got {sorted(ddl_actions)}" + ) + assert ("staging", "DROP") in ddl_actions, ( + f"no DROP INDEX proposal for staging; got {sorted(ddl_actions)}" + ) + + +def test_a_declared_cursor_reaches_the_analysis(seeded): + """DECLARE is what psycopg2 server-side cursors emit; before Task 9 it was discarded. + + The cursor's inner query filters `tenant_id = 3`, a predicate no other seeded + statement uses (see the comment in conftest.py) — its redacted query group can exist + in `workload.stats` only if the DECLARE was actually unwrapped rather than dropped as + noise, so asserting that group's presence (and its call count of exactly 1, since the + cursor is opened once) pins unwrapping directly instead of merely pinning the absence + of a literal `DECLARE` prefix, which would pass whether or not the row survived. + """ + dsn, _schema = seeded + adapter = PostgresWorkloadAdapter() + adapter.connect(ConnectionParams(engine="postgres", dsn=dsn, fields={}, source="--dsn"), 30) + fetch = adapter.fetch_workload(None, 500) + assert any(row.sql.upper().startswith("DECLARE") for row in fetch.rows), ( + "the seeded cursor never reached pg_stat_statements — fixture problem, not a bug" + ) + workload = ingest(fetch, "postgres") + assert not any(s.sql.upper().startswith("DECLARE") for s in workload.stats) + + cursor_groups = [ + s + for s in workload.stats + if "tenant_id" in s.sql and "orders" in s.sql and "GROUP BY" not in s.sql.upper() + ] + assert cursor_groups, ( + "no query group carries the cursor's tenant_id = 3 predicate — the DECLARE was " + "dropped as noise instead of unwrapped" + ) + assert cursor_groups[0].calls == 1, ( + f"expected exactly one call (the cursor is opened once); got {cursor_groups[0].calls}" + ) + + +def test_the_new_rules_fire_on_a_real_workload(seeded): + """ADV007 (join key) and ADV008 (GROUP BY) must each fire on the seeded workload. + + Asserted individually, not as a disjunction: `"ADV007" in codes or "ADV008" in codes` + stays green if either rule's whole proposal block is deleted, so it pins neither rule. + """ + dsn, _schema = seeded + raw = PostgresWorkloadAdapter() + raw.connect(ConnectionParams(engine="postgres", dsn=dsn, fields={}, source="--dsn"), 30) + fetch = raw.fetch_workload(None, 500) + assert any("order_items" in row.sql for row in fetch.rows), ( + "the seeded join query never reached pg_stat_statements — fixture problem, not a bug" + ) + assert any("GROUP BY" in row.sql.upper() for row in fetch.rows), ( + "the seeded GROUP BY query never reached pg_stat_statements — fixture problem, not a bug" + ) + + payload = _run_advise(seeded, schemas=("public", "staging")) + codes = {p["code"] for p in payload["proposals"]} + assert "ADV007" in codes, f"join-key rule did not fire; got {sorted(codes)}" + assert "ADV008" in codes, f"grouping rule did not fire; got {sorted(codes)}" + + +def test_never_analysed_join_key_still_proposes_at_low_confidence(seeded): + """Batch 1's `reltuples = -1` bug, downstream of the catalog read: a join-key proposal + on a never-analysed table must still fire, at LOW confidence with an unknown row + estimate — not be silently suppressed by reading -1 as "this table is tiny". + + `public.order_items` (see conftest.py) has `autovacuum_enabled = false` and is never + explicitly ANALYZEd, so its `reltuples` stays -1 for the whole run. + """ + dsn, _schema = seeded + order_items = Relation("public", "order_items") + raw = PostgresWorkloadAdapter() + raw.connect(ConnectionParams(engine="postgres", dsn=dsn, fields={}, source="--dsn"), 30) + facts = raw.fetch_table_facts(("public",), frozenset({order_items})) + assert facts[order_items].row_estimate is None, ( + "public.order_items reports a real row count — it was analysed before this " + "assertion ran, so the None-path assertion below would be vacuous" + ) + + payload = _run_advise(seeded, schemas=("public", "staging")) + order_items_proposals = [ + p + for p in payload["proposals"] + if p["code"] == "ADV007" and p["evidence"].get("table") == "order_items" + ] + assert order_items_proposals, "ADV007 did not fire for public.order_items" + assert order_items_proposals[0]["evidence"]["row_estimate"] is None + assert order_items_proposals[0]["confidence"] == "low", order_items_proposals[0] diff --git a/tests/integration/test_introspection_live.py b/tests/integration/test_introspection_live.py index b432d37..0eb1f6a 100644 --- a/tests/integration/test_introspection_live.py +++ b/tests/integration/test_introspection_live.py @@ -8,7 +8,7 @@ import pytest -from sqlquality.models import ConnectionParams +from sqlquality.models import ConnectionParams, Relation from sqlquality.workload.postgres import PostgresWorkloadAdapter @@ -24,10 +24,11 @@ def adapter(seeded: tuple[str, str]) -> PostgresWorkloadAdapter: def test_every_introspection_statement_executes(adapter, seeded): """No statement may raise, and none may report a degraded capability.""" _dsn, schema = seeded + orders = Relation(schema, "orders") adapter.fetch_workload(None, 500) adapter.fetch_schema((schema,)) - adapter.fetch_table_facts((schema,), frozenset({"orders"})) - adapter.fetch_indexes((schema,), frozenset({"orders"})) + adapter.fetch_table_facts((schema,), frozenset({orders})) + adapter.fetch_indexes((schema,), frozenset({orders})) assert adapter.degraded == [], f"a statement failed against a real server: {adapter.degraded}" @@ -48,7 +49,8 @@ def test_workload_statement_returns_our_own_queries(adapter): def test_table_facts_reports_a_real_row_estimate_and_ndv(adapter, seeded): _dsn, schema = seeded - facts = adapter.fetch_table_facts((schema,), frozenset({"orders"}))["orders"] + orders = Relation(schema, "orders") + facts = adapter.fetch_table_facts((schema,), frozenset({orders}))[orders] assert facts.row_estimate is not None and facts.row_estimate > 0 assert "status" in facts.columns assert facts.ndv, "pg_stats returned no distinct-value estimates" @@ -57,7 +59,8 @@ def test_table_facts_reports_a_real_row_estimate_and_ndv(adapter, seeded): def test_indexes_statement_reads_partial_and_expression_metadata(adapter, seeded): """The reason Task 2 exists, verified against a real catalog rather than a fixture.""" _dsn, schema = seeded - by_name = {i.name: i for i in adapter.fetch_indexes((schema,), frozenset({"orders"}))["orders"]} + orders = Relation(schema, "orders") + by_name = {i.name: i for i in adapter.fetch_indexes((schema,), frozenset({orders}))[orders]} assert by_name["idx_plain"].columns == ("status", "created_at") assert by_name["idx_plain"].is_partial is False @@ -80,3 +83,50 @@ def test_the_session_really_is_read_only(adapter, seeded): _dsn, schema = seeded with pytest.raises(psycopg.errors.ReadOnlySqlTransaction): adapter._query(f"CREATE TABLE {schema}.should_not_exist (x int)", ()) + + +def test_fetch_schema_nests_columns_under_schema_then_table(seeded): + """`fetch_schema`'s nested shape, asserted directly rather than called for side effect. + + A flat `{table: {column: type}}` map cannot tell `public.orders` and `staging.orders` + apart — a column lookup for one resolves against the union of both column sets. This + is the specific shape guarantee `qualify()` depends on to keep them separate, and + nothing in this suite checked it directly before now. + """ + dsn, _schema = seeded + adapter = PostgresWorkloadAdapter() + adapter.schemas = ("public", "staging") + adapter.connect(ConnectionParams(engine="postgres", dsn=dsn, fields={}, source="--dsn"), 30) + db_schema = adapter.fetch_schema(("public", "staging")) + + assert "public" in db_schema and "staging" in db_schema, db_schema.keys() + assert "orders" in db_schema["public"], db_schema["public"].keys() + assert "orders" in db_schema["staging"], db_schema["staging"].keys() + assert db_schema["public"]["orders"] is not db_schema["staging"]["orders"], ( + "both schemas' orders table resolved to the same column dict — flat, aliased shape" + ) + assert "status" in db_schema["public"]["orders"] + assert "status" in db_schema["staging"]["orders"] + + +def test_two_same_named_tables_keep_their_own_row_estimates(seeded): + """The aliasing bug, against real catalog rows rather than canned ones. + + Bare-name keying used to merge `public.orders` and `staging.orders` into one entry, so + the last catalog row read won the row estimate for both. `seeded` loads them with + deliberately different row counts (20,000 vs 50,000) specifically so an aliasing + regression cannot pass this assertion by coincidence. + """ + dsn, _schema = seeded + adapter = PostgresWorkloadAdapter() + adapter.schemas = ("public", "staging") + adapter.connect(ConnectionParams(engine="postgres", dsn=dsn, fields={}, source="--dsn"), 30) + facts = adapter.fetch_table_facts( + ("public", "staging"), + frozenset({Relation("public", "orders"), Relation("staging", "orders")}), + ) + public_rows = facts[Relation("public", "orders")].row_estimate + staging_rows = facts[Relation("staging", "orders")].row_estimate + assert public_rows is not None and public_rows > 0 + assert staging_rows is not None and staging_rows > 0 + assert public_rows != staging_rows, "both relations reported the same estimate" diff --git a/tests/test_advise_cli.py b/tests/test_advise_cli.py index 2ee1a2d..35115da 100644 --- a/tests/test_advise_cli.py +++ b/tests/test_advise_cli.py @@ -3,7 +3,15 @@ from typer.testing import CliRunner -from sqlquality.cli import app +from sqlquality.cli import ( + _ambiguity_warning, + _coverage_line, + _coverage_warning, + _validate_schemas, + app, +) +from sqlquality.models import Aggregation, QueryStat, Relation, Workload +from sqlquality.report import advise_payload runner = CliRunner() @@ -57,24 +65,204 @@ def explode(*args, **kwargs): assert "between 1 and 3600" in result.output -def test_multiple_schemas_are_rejected_before_connecting(monkeypatch): - """Table facts are keyed on relname alone, so two schemas holding `orders` alias. +def test_two_schemas_are_accepted_and_both_reach_every_catalog_query(monkeypatch): + """A second `--schema` is accepted *and* forwarded to every schema-scoped statement. - Rejecting is the honest minimum: the last row of whichever schema the catalog returned - last would otherwise decide the row estimate, silently. + The docstring used to claim this test proved `Relation` keying prevented cross-schema + aliasing; all it actually asserted was `exit_code == 0`, and `_stub_adapter` overwrote + `adapter.schemas` with `("public",)` so it could not have proved anything about + `--schema` at all. It now asserts the bind parameter of each schema-scoped query, and + names both members rather than checking that the list is non-empty. """ + recorded = _stub_adapter( + monkeypatch, {"pg_stat_statements": [], "pg_stat_database": [("2026-07-01",)]} + ) + result = runner.invoke( + app, + [ + "advise", + "--dsn", + "postgresql://u@h/db", + "--schema", + "sales", + "--schema", + "staging", + "--json", + ], + ) + assert result.exit_code == 0 + # Every statement that takes a schema list: the qualify() schema map, table facts, NDV + # and the existing-index catalog. Each is checked separately — one of the four carrying + # both schemas while another silently narrowed to one is the failure being excluded. + # One marker per statement, each unique to it: `pg_class` would have matched both the + # table-facts and the index statement and so could not tell which one narrowed. + for marker in ("information_schema.columns", "pg_total_relation_size", "pg_stats", "pg_index"): + binds = [bind for sql, bind in recorded if marker in sql] + assert binds, f"no query ran against {marker}" + for bind in binds: + assert bind[0] == ["sales", "staging"], f"{marker} received {bind[0]!r}" + + +def test_the_resolved_schemas_reach_the_existing_index_query(monkeypatch): + """`adapter.schemas` is the *only* route `--schema` takes into `fetch_indexes`. + + `fetch_schema`, `fetch_table_facts` and `fetch_ndv` are handed the resolved tuple + directly by the CLI; `propose()` reads `self.schemas` instead. Drop the CLI's + `adapter.schemas = schemas` assignment and `fetch_indexes` silently queries `("public",)` + — which returns zero rows *without raising*, so nothing lands in `degraded`, + `have_index_data` stays True, and ADV001/ADV007 then claim "no existing index leads with + them" at HIGH for tables that are fully indexed while ADV002/ADV003 go silent. That is a + check that could not run, reported as a check that ran and passed. + """ + recorded = _stub_adapter( + monkeypatch, {"pg_stat_statements": [], "pg_stat_database": [("2026-07-01",)]} + ) + result = runner.invoke( + app, + ["advise", "--dsn", "postgresql://u@h/db", "--schema", "sales", "--schema", "staging"], + ) + assert result.exit_code == 0 + index_binds = [bind for sql, bind in recorded if "pg_index" in sql] + assert index_binds, "the existing-index catalog query never ran" + assert all(bind[0] == ["sales", "staging"] for bind in index_binds), ( + f"fetch_indexes was called with {[bind[0] for bind in index_binds]!r} — the CLI's " + "resolved --schema tuple never reached the adapter" + ) - def explode(*args, **kwargs): - raise AssertionError("must not connect with more than one --schema") - monkeypatch.setattr("sqlquality.workload.postgres.PostgresWorkloadAdapter.connect", explode) +def test_duplicate_schemas_are_deduplicated(): + assert _validate_schemas(["public", "public"]) == ("public",) + + +def test_schema_order_is_preserved(): + assert _validate_schemas(["b", "a"]) == ("b", "a") + + +def test_coverage_line_reports_ambiguous_separately(): + workload = _workload_with(stats=3, unparseable=1, noise=0) + aggregation = _aggregation_with(skipped_unqualifiable=1, skipped_ambiguous=2) + line = _coverage_line(workload, aggregation) + assert "2 ambiguous" in line + + +def test_analyzed_count_excludes_ambiguous_statements(): + """`analyzed N of M` must not double-book an ambiguous statement as both analysed here + and unexplained in `_coverage_warning`'s share — it cannot honestly be both. Of 4 query + groups, 2 were dropped as ambiguous and 0 as otherwise unresolvable, so only 2 were + actually analyzed.""" + workload = _workload_with(stats=4, unparseable=0, noise=0) + aggregation = _aggregation_with(skipped_unqualifiable=0, skipped_ambiguous=2) + line = _coverage_line(workload, aggregation) + assert "analyzed 2 of 4" in line + + +def test_coverage_warning_fires_when_ambiguity_alone_crosses_the_threshold(): + """100 stats, 25 ambiguous, nothing else unexplained: the true unexplained share is + 25/100 = 25%, above the 20% low-coverage threshold. Before `analyzed_query_groups` subtracted + `skipped_ambiguous`, the 25 ambiguous statements were counted as both analyzed (inflating + `considered`) and unexplained, diluting the share to exactly 20% — at the threshold, not + above it — so the warning never fired precisely when ambiguity was the whole reason + coverage was bad.""" + workload = _workload_with(stats=100, unparseable=0, noise=0) + aggregation = _aggregation_with(skipped_unqualifiable=0, skipped_ambiguous=25) + assert _coverage_warning(workload, aggregation) is not None + + +def test_ambiguity_warning_names_the_remedy(): + aggregation = _aggregation_with(skipped_unqualifiable=0, skipped_ambiguous=4) + warning = _ambiguity_warning(aggregation) + assert warning is not None + assert "--schema" in warning + + +def test_no_ambiguity_means_no_warning(): + """The warning must not fire on the single-schema path, which is every existing run.""" + aggregation = _aggregation_with(skipped_unqualifiable=3, skipped_ambiguous=0) + assert _ambiguity_warning(aggregation) is None + + +def test_ambiguity_warning_reaches_the_user_on_a_real_run(monkeypatch): + """`_ambiguity_warning` is unit-tested above in isolation, but nothing else in the + suite exercises the wiring that actually echoes it from the `advise` command body — + deleting that echo leaves every other test green. Two introspected schemas both hold + `orders`; the query names it bare, so it cannot be attributed and must surface here.""" + from sqlquality.workload.postgres import PostgresWorkloadAdapter + + rows = { + "pg_stat_statements": [ + ("select id from orders where status = $1", 5, 100.0, 5), + ], + "pg_stat_database": [("2026-07-01",)], + "information_schema.columns": [ + ("sales", "orders", "id", "integer"), + ("sales", "orders", "status", "text"), + ("staging", "orders", "id", "integer"), + ("staging", "orders", "status", "text"), + ], + "pg_total_relation_size": [], + "pg_stats": [], + "pg_index": [], + } + + def fake_connect(self, params, timeout_s): + # Unlike `_stub_adapter`, this does not overwrite `self.schemas`: the CLI already + # set it from `--schema` before calling `connect()`, and this scenario needs both. + def query(sql, bind): + for marker, result in rows.items(): + if marker in sql: + return result + return [] + + self._query = query + + monkeypatch.setattr(PostgresWorkloadAdapter, "connect", fake_connect) result = runner.invoke( app, - ["advise", "--dsn", "postgresql://u@h/db", "--schema", "public", "--schema", "app"], + ["advise", "--dsn", "postgresql://u@h/db", "--schema", "sales", "--schema", "staging"], + ) + assert result.exit_code == 0 + assert "could not be attributed" in result.output + assert "--schema" in result.output + + +def test_payload_tables_are_qualified_strings(): + payload = advise_payload( + [], + _workload_with(stats=0, unparseable=0, noise=0), + _aggregation_with(tables=frozenset({Relation("sales", "orders")})), + engine="postgres", + redacted=True, + degraded=[], + ) + assert payload["analyzed"]["tables"] == ["sales.orders"] + json.dumps(payload) # must not raise + + +def _workload_with(*, stats: int, unparseable: int, noise: int) -> Workload: + return Workload( + stats=tuple( + QueryStat(fingerprint=f"fp{i}", sql="select 1", calls=1, total_time_ms=1.0) + for i in range(stats) + ), + window_description="w", + skipped_unparseable=unparseable, + skipped_noise=noise, + ) + + +def _aggregation_with( + *, + skipped_unqualifiable: int = 0, + skipped_ambiguous: int = 0, + tables: frozenset[Relation] = frozenset(), +) -> Aggregation: + return Aggregation( + usage=(), + total_cost_ms=0.0, + skipped_unqualifiable=skipped_unqualifiable, + tables=tables, + skipped_ambiguous=skipped_ambiguous, ) - assert result.exit_code == 2 - assert "schema-qualified" in result.output - assert "app" in result.output and "public" in result.output def test_a_single_schema_is_still_accepted(monkeypatch): @@ -97,8 +285,8 @@ def test_low_coverage_warns_on_stderr(monkeypatch): ], "pg_stat_database": [("2026-07-01",)], "information_schema.columns": WIDE_COLUMNS, - "pg_total_relation_size": [("orders", 5_000_000, 10**8)], - "pg_stats": [("orders", "status", 5000.0)], + "pg_total_relation_size": [("public", "orders", 5_000_000, 10**8)], + "pg_stats": [("public", "orders", "status", 5000.0)], "pg_index": [], }, ) @@ -118,8 +306,8 @@ def test_good_coverage_does_not_warn(monkeypatch): ], "pg_stat_database": [("2026-07-01",)], "information_schema.columns": WIDE_COLUMNS, - "pg_total_relation_size": [("orders", 5_000_000, 10**8)], - "pg_stats": [("orders", "status", 5000.0)], + "pg_total_relation_size": [("public", "orders", 5_000_000, 10**8)], + "pg_stats": [("public", "orders", "status", 5000.0)], "pg_index": [], }, ) @@ -129,13 +317,22 @@ def test_good_coverage_does_not_warn(monkeypatch): def _stub_adapter(monkeypatch, rows): - """Replace connect() with an injected fake querier.""" + """Replace connect() with an injected fake querier. Returns the recorded `(sql, bind)`. + + Deliberately does **not** assign `self.schemas`. It used to hard-code `("public",)`, + *after* the CLI had already resolved `--schema` onto the adapter — so every test in this + module ran `fetch_indexes(("public",), ...)` no matter what schemas it passed, and + deleting the CLI's `adapter.schemas = schemas` line left the whole default suite green. + The adapter's own `__init__` default is `("public",)` already, so single-schema tests are + unaffected; multi-schema ones now exercise the real wiring. + """ from sqlquality.workload.postgres import PostgresWorkloadAdapter - def fake_connect(self, params, timeout_s): - self.schemas = ("public",) + recorded: list[tuple[str, object]] = [] + def fake_connect(self, params, timeout_s): def query(sql, bind): + recorded.append((sql, bind)) for marker, result in rows.items(): if marker in sql: return result @@ -144,12 +341,13 @@ def query(sql, bind): self._query = query monkeypatch.setattr(PostgresWorkloadAdapter, "connect", fake_connect) + return recorded WIDE_COLUMNS = [ - ("orders", "id", "integer"), - ("orders", "status", "text"), - ("orders", "created_at", "timestamp"), + ("public", "orders", "id", "integer"), + ("public", "orders", "status", "text"), + ("public", "orders", "created_at", "timestamp"), ] @@ -157,34 +355,42 @@ def query(sql, bind): STAR_ONLY_ROWS = { "pg_stat_statements": [("select * from wide_t", 100, 5000.0, 10)], "pg_stat_database": [("2026-07-01",)], - "information_schema.columns": [("wide_t", f"c{i}", "text") for i in range(20)], - "pg_total_relation_size": [("wide_t", 5_000_000, 10**8)], + "information_schema.columns": [("public", "wide_t", f"c{i}", "text") for i in range(20)], + "pg_total_relation_size": [("public", "wide_t", 5_000_000, 10**8)], "pg_stats": [], "pg_index": [], } -def test_the_filtered_counter_does_not_claim_introspection_or_ddl(monkeypatch): +def test_declared_cursors_and_copy_subqueries_are_analyzed_not_filtered(monkeypatch): """`DECLARE cur CURSOR FOR SELECT ...` and `COPY (SELECT ...) TO STDOUT` are reads. Django's `QuerySet.iterator()` and every psycopg2 server-side cursor emit exactly the - first form, so a Django shop's hot reads land in this counter — and were then reported - as "introspection/DDL", i.e. as maintenance traffic nobody needed to care about, on the - one line that exists to disclose what was lost. + first form, so a Django shop's hot reads used to land in the "filtered" counter and be + thrown away entirely — and reported as "introspection/DDL" on the one line that exists + to disclose what was lost. `unwrap()` (`sqlquality.workload.fingerprint`) now recovers + the inner query from both statements before the noise test runs, so each is analyzed + as its own query group instead. """ _stub_adapter( monkeypatch, { "pg_stat_statements": [ ("declare cur cursor for select id from orders where status = $1", 9, 900.0, 9), - ("copy (select id from orders where status = $1) to stdout", 5, 500.0, 5), + ( + "copy (select id, status from orders where status = $1) to stdout", + 5, + 500.0, + 5, + ), ], "pg_stat_database": [("2026-07-01",)], }, ) result = runner.invoke(app, ["advise", "--dsn", "postgresql://u@h/db"]) assert result.exit_code == 0 - assert "2 filtered" in result.output + assert "analyzed 2 of 2" in result.output + assert "0 filtered" in result.output assert "introspection/DDL" not in result.output @@ -285,8 +491,8 @@ def test_successful_run_exits_0_and_emits_json(monkeypatch): ], "pg_stat_database": [("2026-07-01",)], "information_schema.columns": WIDE_COLUMNS, - "pg_total_relation_size": [("orders", 5_000_000, 10**8)], - "pg_stats": [("orders", "status", 5000.0)], + "pg_total_relation_size": [("public", "orders", 5_000_000, 10**8)], + "pg_stats": [("public", "orders", "status", 5000.0)], "pg_index": [], }, ) @@ -314,8 +520,8 @@ def test_ddl_and_markdown_files_are_written(monkeypatch, tmp_path): ], "pg_stat_database": [("2026-07-01",)], "information_schema.columns": WIDE_COLUMNS, - "pg_total_relation_size": [("orders", 5_000_000, 10**8)], - "pg_stats": [("orders", "status", 5000.0)], + "pg_total_relation_size": [("public", "orders", 5_000_000, 10**8)], + "pg_stats": [("public", "orders", "status", 5000.0)], "pg_index": [], }, ) @@ -390,8 +596,8 @@ def test_coverage_is_disclosed_even_on_a_clean_run(monkeypatch): ], "pg_stat_database": [("2026-07-01",)], "information_schema.columns": WIDE_COLUMNS, - "pg_total_relation_size": [("orders", 5_000_000, 10**8)], - "pg_stats": [("orders", "status", 5000.0)], + "pg_total_relation_size": [("public", "orders", 5_000_000, 10**8)], + "pg_stats": [("public", "orders", "status", 5000.0)], "pg_index": [], }, ) diff --git a/tests/test_models.py b/tests/test_models.py index 313b72a..42560b3 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -11,6 +11,7 @@ Proposal, QueryStat, RawQueryRow, + Relation, TableFacts, Workload, ) @@ -68,13 +69,15 @@ def test_workload_cost_totals_only_its_own_stats(): def test_table_facts_ndv_defaults_empty(): - facts = TableFacts(name="orders", row_estimate=100, size_bytes=None, columns=("id",)) + facts = TableFacts( + relation=Relation("public", "orders"), row_estimate=100, size_bytes=None, columns=("id",) + ) assert facts.ndv == {} def test_proposal_and_aggregation_construct(): usage = ColumnUsage( - table="orders", + relation=Relation("public", "orders"), column="status", role=ColumnRole.EQUALITY, calls=5, @@ -83,7 +86,10 @@ def test_proposal_and_aggregation_construct(): fingerprint_ids=frozenset({"fp1", "fp2"}), ) agg = Aggregation( - usage=(usage,), total_cost_ms=100.0, skipped_unqualifiable=0, tables=frozenset({"orders"}) + usage=(usage,), + total_cost_ms=100.0, + skipped_unqualifiable=0, + tables=frozenset({Relation("public", "orders")}), ) assert agg.usage[0].role is ColumnRole.EQUALITY proposal = Proposal( @@ -106,7 +112,7 @@ def test_fingerprints_is_derived_from_the_id_set(): """One source of truth. The two used to be separate fields kept in step by convention, with nothing stopping a caller setting one and not the other.""" usage = ColumnUsage( - table="orders", + relation=Relation("public", "orders"), column="status", role=ColumnRole.EQUALITY, calls=5, @@ -118,7 +124,7 @@ def test_fingerprints_is_derived_from_the_id_set(): with pytest.raises(TypeError): ColumnUsage( # type: ignore[call-arg] - table="orders", + relation=Relation("public", "orders"), column="status", role=ColumnRole.EQUALITY, calls=5, diff --git a/tests/test_report_markdown.py b/tests/test_report_markdown.py index 1f81827..b3da15a 100644 --- a/tests/test_report_markdown.py +++ b/tests/test_report_markdown.py @@ -1,6 +1,6 @@ from sqlquality.delta import ModelDelta from sqlquality.gate import GateReport -from sqlquality.models import Aggregation, Confidence, Proposal, QueryStat, Workload +from sqlquality.models import Aggregation, Confidence, Proposal, QueryStat, Relation, Workload from sqlquality.report import advise_payload, render_advise_markdown, render_markdown PASS = GateReport( @@ -103,7 +103,11 @@ def test_markdown_injection_is_inert(): skipped_noise=7, ) AGGREGATION = Aggregation( - usage=(), total_cost_ms=500.0, skipped_unqualifiable=3, tables=frozenset({"orders"}) + usage=(), + total_cost_ms=500.0, + skipped_unqualifiable=3, + tables=frozenset({Relation("public", "orders")}), + skipped_ambiguous=4, ) @@ -124,7 +128,12 @@ def test_payload_reports_proposals_window_and_skips(): assert payload["redacted"] is True assert payload["window"] == "since stats reset at 2026-07-01" assert payload["proposals"][0]["code"] == "ADV001" - assert payload["skipped"] == {"unparseable": 2, "noise": 7, "unqualifiable": 3} + assert payload["skipped"] == { + "unparseable": 2, + "noise": 7, + "unqualifiable": 3, + "ambiguous": 4, + } assert payload["degraded"] == [{"capability": "ndv", "reason": "permission denied"}] @@ -159,6 +168,54 @@ def test_markdown_discloses_the_window_and_the_skips(): assert "7 filtered" in md assert "introspection/DDL" not in md assert "3 unresolvable" in md + assert "4 ambiguous" in md + + +def _eight_groups_two_ambiguous(): + """Eight query groups of which two were dropped as ambiguous — six were understood.""" + workload = Workload( + stats=tuple( + QueryStat(fingerprint=f"fp{i}", sql="select 1", calls=1, total_time_ms=1.0) + for i in range(8) + ), + window_description="since stats reset at 2026-07-01", + skipped_unparseable=1, + skipped_noise=2, + ) + aggregation = Aggregation( + usage=(), + total_cost_ms=8.0, + skipped_unqualifiable=0, + tables=frozenset(), + skipped_ambiguous=2, + ) + return workload, aggregation + + +def test_all_three_surfaces_report_the_same_analyzed_count(): + """The terminal said "analyzed 6 of 8" while markdown said "analyzed: 8" and the JSON + payload carried 8 under a key named `analyzed` — directly above its own "2 ambiguous". + + The README promises the terminal, markdown *and* JSON paths all print how many query + groups were actually understood, and nothing pinned the number on two of the three: it + was the sole mutation to survive a 47-mutation whole-branch sweep. All three are asserted + here together, so fixing one surface and leaving another cannot pass. + """ + from sqlquality.cli import _coverage_line + + workload, aggregation = _eight_groups_two_ambiguous() + terminal = _coverage_line(workload, aggregation) + md = render_advise_markdown( + PROPOSALS, workload, aggregation, engine="postgres", redacted=True, degraded=[] + ) + payload = advise_payload( + PROPOSALS, workload, aggregation, engine="postgres", redacted=True, degraded=[] + ) + assert "analyzed 6 of 8 query group(s)" in terminal + assert "**Query groups analyzed:** 6 of 8" in md + assert payload["analyzed"]["query_groups"] == 6 + # The window total is still available, just no longer labelled "analyzed". + assert payload["analyzed"]["query_groups_in_window"] == 8 def test_markdown_escapes_an_evidence_key_as_well_as_its_value(): diff --git a/tests/test_workload_aggregate.py b/tests/test_workload_aggregate.py index 476d697..c69d7bc 100644 --- a/tests/test_workload_aggregate.py +++ b/tests/test_workload_aggregate.py @@ -1,8 +1,8 @@ -from sqlquality.models import ColumnRole, QueryStat, Workload -from sqlquality.workload.aggregate import aggregate +from sqlquality.models import ColumnRole, QueryStat, Relation, Workload +from sqlquality.workload.aggregate import aggregate, star_tables from sqlquality.workload.fingerprint import FLAG_SELECT_STAR -SCHEMA = {"orders": {"id": "INT", "status": "TEXT", "created_at": "TIMESTAMP"}} +SCHEMA = {"public": {"orders": {"id": "INT", "status": "TEXT", "created_at": "TIMESTAMP"}}} def _workload(*pairs): @@ -58,7 +58,7 @@ def test_unqualifiable_queries_are_counted_not_raised(): "postgres", ) assert agg.skipped_unqualifiable == 1 - assert agg.tables == frozenset({"orders"}) + assert agg.tables == frozenset({Relation("public", "orders")}) def test_usage_is_sorted_by_cost_descending(): @@ -96,8 +96,8 @@ def test_equal_cost_entries_are_ordered_canonically_not_by_arrival(): SCHEMA, "postgres", ) - assert [(u.table, u.column, u.role) for u in forward.usage] == [ - (u.table, u.column, u.role) for u in reverse.usage + assert [(u.relation, u.column, u.role) for u in forward.usage] == [ + (u.relation, u.column, u.role) for u in reverse.usage ] @@ -163,8 +163,16 @@ def test_empty_workload_yields_empty_aggregation_and_no_division_error(): assert agg.tables == frozenset() -def test_star_tables_compiles_each_table_pattern_once(monkeypatch): - """A fresh regex per (stat x table) pair thrashes re's pattern cache on a wide schema.""" +def test_identifier_pattern_is_compiled_once_per_name(monkeypatch): + """A fresh regex per identifier check thrashes re's own pattern cache. + + ``star_tables`` no longer text-matches (it parses and resolves through + ``resolve_relation`` instead — see the ``star_tables`` tests below for why), so the + caching this pins is exercised directly through ``mentions_identifier``, which is what + the expression-index disclosures in ADV001/ADV007/ADV008 actually call many times over. + It used to go through a `mentions_table` alias, which by then had no production caller at + all — a test exercising a wrapper nobody used, of a cache everybody used. + """ import re as _re from sqlquality.workload import aggregate as agg @@ -177,21 +185,313 @@ def counting_compile(pattern, *args, **kwargs): return real_compile(pattern, *args, **kwargs) monkeypatch.setattr(agg._re if hasattr(agg, "_re") else _re, "compile", counting_compile) - workload = Workload( + names = [f"t{i}" for i in range(20)] + ["orders"] + for _ in range(5): + for name in names: + agg.mentions_identifier(name, "select * from orders") + assert len(compiles) <= len(names), ( + f"compiled {len(compiles)} patterns for {len(names)} distinct identifiers across 5 passes" + ) + + +ONE_SCHEMA = {"public": {"orders": {"id": "int", "status": "text"}}} +TWO_SCHEMAS = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"items": {"sku": "text", "qty": "int"}}, +} +COLLIDING = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"orders": {"id": "int", "status": "text"}}, +} + + +def _mixed_workload(*sql: str) -> Workload: + return Workload( stats=tuple( + QueryStat(fingerprint=f"fp{i}", sql=s, calls=1, total_time_ms=100.0) + for i, s in enumerate(sql) + ), + window_description="test", + ) + + +def test_usage_is_keyed_by_relation(): + result = aggregate( + _mixed_workload("select id from orders where status = 'x'"), ONE_SCHEMA, "postgres" + ) + assert {u.relation for u in result.usage} == {Relation("public", "orders")} + assert result.tables == frozenset({Relation("public", "orders")}) + + +def test_same_table_name_in_two_schemas_does_not_alias(): + """The bug multi-schema keying exists to fix: two relations, not one merged entry.""" + result = aggregate( + _mixed_workload( + "select id from sales.orders where status = 'x'", + "select id from staging.orders where status = 'y'", + ), + COLLIDING, + "postgres", + ) + assert result.tables == frozenset({Relation("sales", "orders"), Relation("staging", "orders")}) + + +def test_ambiguous_bare_name_is_counted_not_crashed(): + result = aggregate( + _mixed_workload("select id from orders where status = 'x'"), COLLIDING, "postgres" + ) + assert result.skipped_ambiguous == 1 + assert result.usage == () + + +def test_bare_select_star_over_a_colliding_name_is_counted_ambiguous(): + """A bare `select * from orders` has no predicate, so `qualify()` has no column + reference to validate and never raises for it — it just silently produces zero usage, + the same as it would for an *unambiguous* bare star. Left uncounted, that reads as + "nothing to see here" when what actually happened is the same unattributable-bare-name + fact `AmbiguousRelation` reports for a predicated statement (see + `test_ambiguous_bare_name_is_counted_not_crashed` above) — and the same fact ADV006's + `_wide_relations_touched` later declines to guess at for exactly this statement shape. + """ + workload = Workload( + stats=( QueryStat( - fingerprint=f"fp{i}", + fingerprint="fp", sql="select * from orders", calls=1, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), + ), + ), + window_description="test", + ) + result = aggregate(workload, COLLIDING, "postgres") + assert result.skipped_ambiguous == 1 + assert result.usage == () + assert result.tables == frozenset() + + +def test_bare_select_star_over_an_unambiguous_name_is_not_counted(): + """The new check must not fire just because a statement is a bare star — only when the + bare name it references is genuinely held by more than one introspected schema.""" + workload = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from items", + calls=1, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), + ), + ), + window_description="test", + ) + result = aggregate(workload, TWO_SCHEMAS, "postgres") + assert result.skipped_ambiguous == 0 + assert result.usage == () + + +def test_ambiguous_bare_reference_is_counted_even_without_a_literal_star(): + """`select count(*) from orders` and `select 1 from orders` are not flagged + `FLAG_SELECT_STAR` — that flag only marks a literal `SELECT *` — but neither references + any column by name either, so `qualify()` never raises for either of them, exactly like + the bare-star case above. Gating the check on the star flag let these two escape *both* + counters: parsed fine, zero usage, never raised, never counted. + """ + result = aggregate( + _mixed_workload("select count(*) from orders", "select 1 from orders"), + COLLIDING, + "postgres", + ) + assert result.skipped_ambiguous == 2 + assert result.usage == () + + +def test_a_qualified_reference_to_a_colliding_name_is_not_counted_ambiguous(): + """`select * from sales.orders` produces no usage — a star has no predicate to attribute + — but it is not *ambiguous*: the statement says which schema it means. + + `_references_an_ambiguous_bare_table` skips any reference carrying a `.db` qualifier, and + nothing pinned that skip: removing it left the whole suite green while this statement + started counting toward `skipped_ambiguous`, which drives both the low-coverage share and + a warning whose remedy is "qualify the table in the query" — advice already followed. The + unqualified twin below is asserted in the same test so a guard that silently swallowed + *both* cases could not pass either. + """ + qualified = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from sales.orders", + calls=1, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), + ), + ), + window_description="test", + ) + bare = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from orders", + calls=1, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), + ), + ), + window_description="test", + ) + assert aggregate(qualified, COLLIDING, "postgres").skipped_ambiguous == 0 + assert aggregate(bare, COLLIDING, "postgres").skipped_ambiguous == 1 + + +def test_a_plain_parse_failure_is_not_counted_as_ambiguous(): + """The two counters must not both fire for the same statement.""" + result = aggregate(_mixed_workload("this is not sql at all"), ONE_SCHEMA, "postgres") + assert result.skipped_ambiguous == 0 + assert result.skipped_unqualifiable == 1 + + +def test_star_tables_returns_qualified_relations(): + workload = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from items", + calls=1, total_time_ms=1.0, flags=frozenset({FLAG_SELECT_STAR}), - ) - for i in range(5) + ), ), - window_description="w", + window_description="test", + ) + assert star_tables(workload, TWO_SCHEMAS) == frozenset({Relation("staging", "items")}) + + +def test_star_tables_skips_an_ambiguous_name(): + """Attributing a bare `select *` to one of two same-named tables would be a guess.""" + workload = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from orders", + calls=1, + total_time_ms=1.0, + flags=frozenset({FLAG_SELECT_STAR}), + ), + ), + window_description="test", + ) + assert star_tables(workload, COLLIDING) == frozenset() + + +def test_star_tables_does_not_attribute_an_unintrospected_schema_qualifier(): + """`star_tables` must decline exactly what `resolve_relation` declines. + + Text-matching `nosuch.items` against the schema's table names cannot see the + qualifier at all, so it would previously resolve through a bare-name collision with + `staging.items` — the phantom `resolve_relation`'s `table.db` guard exists to refuse. + """ + workload = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from nosuch.items", + calls=1, + total_time_ms=1.0, + flags=frozenset({FLAG_SELECT_STAR}), + ), + ), + window_description="test", + ) + assert star_tables(workload, TWO_SCHEMAS) == frozenset() + + +def test_star_tables_resolves_an_explicitly_qualified_colliding_name(): + """`orders` collides across two schemas, but an explicit qualifier is not ambiguous. + + `resolve_relation` resolves `sales.orders` outright; `star_tables`'s old text-match + path could not see the qualifier and dropped it as if the query had said `orders` + bare. The two must agree. + """ + workload = Workload( + stats=( + QueryStat( + fingerprint="fp", + sql="select * from sales.orders", + calls=1, + total_time_ms=1.0, + flags=frozenset({FLAG_SELECT_STAR}), + ), + ), + window_description="test", + ) + assert star_tables(workload, COLLIDING) == frozenset({Relation("sales", "orders")}) + + +def test_relation_breaks_ties_when_cost_column_and_role_all_match(): + """The fifth sort key. Two relations, same column/role/cost — order must be + canonical (by `Relation`), not the order the statements happened to arrive in. + + ``alpha`` sorts before ``zeta``, but the ``zeta`` statement is listed — and therefore + processed — first, so this only holds if the sort key actually includes `u.relation`. + """ + schema = { + "zeta": {"orders": {"id": "int", "status": "text"}}, + "alpha": {"orders": {"id": "int", "status": "text"}}, + } + agg = aggregate( + _mixed_workload( + "select id from zeta.orders where status = 'x'", + "select id from alpha.orders where status = 'y'", + ), + schema, + "postgres", ) - schema = {f"t{i}": {"c": "int"} for i in range(20)} | {"orders": {"c": "int"}} - assert agg.star_tables(workload, schema) == frozenset({"orders"}) - assert len(compiles) <= len(schema), ( - f"compiled {len(compiles)} patterns for {len(schema)} tables across 5 stats" + matches = [u for u in agg.usage if u.column == "status"] + assert [u.relation for u in matches] == [ + Relation("alpha", "orders"), + Relation("zeta", "orders"), + ] + + +def test_ambiguous_dml_target_is_counted_not_silently_dropped(): + """`qualify()` does not validate UPDATE/DELETE targets, so an ambiguous bare DML + target used to vanish with no usage recorded and neither counter incremented — + reported as analysed by the coverage line when it was not. + """ + result = aggregate( + _mixed_workload("update orders set status = 'x' where id = 1"), COLLIDING, "postgres" + ) + assert result.skipped_ambiguous == 1 + assert result.usage == () + + +def test_ambiguous_statement_cost_stays_in_the_denominator(): + """Same 'not a partition' semantics as an unqualifiable statement: an ambiguous + statement's cost is not excluded from the denominator merely because it produced no + usage (see test_skipped_stats_still_count_toward_the_denominator). + """ + workload = Workload( + stats=( + QueryStat( + fingerprint="fp0", + sql="select id from orders where status = 'x'", + calls=1, + total_time_ms=90.0, + ), + QueryStat( + fingerprint="fp1", + sql="select id from sales.orders where status = 'y'", + calls=1, + total_time_ms=10.0, + ), + ), + window_description="test", ) + result = aggregate(workload, COLLIDING, "postgres") + assert result.skipped_ambiguous == 1 + assert result.total_cost_ms == 100.0 + usage = next(u for u in result.usage if u.column == "status") + assert usage.cost_share == 0.1 diff --git a/tests/test_workload_extract.py b/tests/test_workload_extract.py index 4d35147..bb4574a 100644 --- a/tests/test_workload_extract.py +++ b/tests/test_workload_extract.py @@ -1,20 +1,31 @@ import pytest import sqlglot +from sqlglot import exp -from sqlquality.models import ColumnRole -from sqlquality.workload.extract import UnqualifiableQuery, extract_usage +from sqlquality.models import ColumnRole, Relation +from sqlquality.sqlast import parse +from sqlquality.workload.extract import ( + AmbiguousRelation, + UnqualifiableQuery, + extract_usage, + resolve_relation, +) SCHEMA = { - "orders": { - "id": "INT", - "customer_id": "INT", - "status": "TEXT", - "created_at": "TIMESTAMP", - "note": "TEXT", - "shipped_at": "TIMESTAMP", - }, - "customers": {"id": "INT", "email": "TEXT", "status": "TEXT"}, + "public": { + "orders": { + "id": "INT", + "customer_id": "INT", + "status": "TEXT", + "created_at": "TIMESTAMP", + "note": "TEXT", + "shipped_at": "TIMESTAMP", + }, + "customers": {"id": "INT", "email": "TEXT", "status": "TEXT"}, + } } +ORDERS = Relation("public", "orders") +CUSTOMERS = Relation("public", "customers") def _usage(sql): @@ -23,44 +34,44 @@ def _usage(sql): def test_where_equality_is_equality_role(): - assert ("orders", "status", ColumnRole.EQUALITY) in _usage( + assert (ORDERS, "status", ColumnRole.EQUALITY) in _usage( "select id from orders where status = $1" ) def test_in_predicate_is_equality_role(): - assert ("orders", "status", ColumnRole.EQUALITY) in _usage( + assert (ORDERS, "status", ColumnRole.EQUALITY) in _usage( "select id from orders where status in ($1, $2)" ) def test_comparison_is_range_role(): - assert ("orders", "created_at", ColumnRole.RANGE) in _usage( + assert (ORDERS, "created_at", ColumnRole.RANGE) in _usage( "select id from orders where created_at > $1" ) def test_between_is_range_role(): - assert ("orders", "created_at", ColumnRole.RANGE) in _usage( + assert (ORDERS, "created_at", ColumnRole.RANGE) in _usage( "select id from orders where created_at between $1 and $2" ) def test_join_key_is_join_role_not_equality(): usage = _usage("select o.id from orders o join customers c on c.id = o.customer_id") - assert ("orders", "customer_id", ColumnRole.JOIN) in usage - assert ("customers", "id", ColumnRole.JOIN) in usage - assert ("orders", "customer_id", ColumnRole.EQUALITY) not in usage + assert (ORDERS, "customer_id", ColumnRole.JOIN) in usage + assert (CUSTOMERS, "id", ColumnRole.JOIN) in usage + assert (ORDERS, "customer_id", ColumnRole.EQUALITY) not in usage def test_order_by_is_sort_role(): - assert ("orders", "created_at", ColumnRole.SORT) in _usage( + assert (ORDERS, "created_at", ColumnRole.SORT) in _usage( "select id from orders order by created_at desc" ) def test_group_by_is_group_role(): - assert ("orders", "status", ColumnRole.GROUP) in _usage( + assert (ORDERS, "status", ColumnRole.GROUP) in _usage( "select status, count(*) from orders group by status" ) @@ -69,26 +80,26 @@ def test_window_order_by_is_not_a_query_sort_key(): usage = _usage( "select id, row_number() over (partition by status order by created_at) from orders" ) - assert ("orders", "created_at", ColumnRole.SORT) not in usage + assert (ORDERS, "created_at", ColumnRole.SORT) not in usage def test_null_checks_carry_polarity(): - assert ("orders", "shipped_at", ColumnRole.NULL_CHECK) in _usage( + assert (ORDERS, "shipped_at", ColumnRole.NULL_CHECK) in _usage( "select id from orders where shipped_at is null" ) - assert ("orders", "shipped_at", ColumnRole.NOT_NULL_CHECK) in _usage( + assert (ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK) in _usage( "select id from orders where shipped_at is not null" ) def test_function_wrapped_predicate_is_non_sargable(): usage = _usage("select id from orders where lower(status) = $1") - assert ("orders", "status", ColumnRole.NON_SARGABLE) in usage - assert ("orders", "status", ColumnRole.EQUALITY) not in usage + assert (ORDERS, "status", ColumnRole.NON_SARGABLE) in usage + assert (ORDERS, "status", ColumnRole.EQUALITY) not in usage def test_cast_wrapped_predicate_is_non_sargable(): - assert ("orders", "id", ColumnRole.NON_SARGABLE) in _usage( + assert (ORDERS, "id", ColumnRole.NON_SARGABLE) in _usage( "select id from orders where id::text = $1" ) @@ -99,25 +110,35 @@ def test_projected_columns_produce_no_usage(): def test_update_where_predicate_is_attributed_to_the_target_table(): """qualify() leaves DML columns bare (table == ''), so this needs the sole-table path.""" - assert ("orders", "created_at", ColumnRole.RANGE) in _usage( + assert (ORDERS, "created_at", ColumnRole.RANGE) in _usage( "update orders set status = $1 where created_at < $2" ) +def test_update_from_second_table_bare_column_is_dropped_not_guessed(): + """`UPDATE ... FROM` puts two tables in scope; a bare column has no unambiguous target. + + ``customer_id`` is unqualified here (qualify() leaves DML columns bare), so without the + ``len(tables) == 1`` guard in ``_collect_dml`` it would be attributed to ``tables[0]`` + (``orders``) purely because that table happened to be found first — a guess dressed up + as a fact, not something the query actually told us. + """ + usage = _usage("update orders set status = 'x' from customers where customer_id = customers.id") + assert not any(column == "customer_id" for _relation, column, _role in usage) + + def test_update_set_clause_column_is_not_a_predicate(): """`SET status = $1` parses to EQ(status, $1) with no WHERE ancestor. Recording it as EQUALITY would make us propose indexing the column being written. """ usage = _usage("update orders set status = $1 where created_at < $2") - assert ("orders", "status", ColumnRole.EQUALITY) not in usage - assert not any(column == "status" for _table, column, _role in usage) + assert (ORDERS, "status", ColumnRole.EQUALITY) not in usage + assert not any(column == "status" for _relation, column, _role in usage) def test_delete_where_predicate_is_attributed(): - assert ("orders", "status", ColumnRole.EQUALITY) in _usage( - "delete from orders where status = $1" - ) + assert (ORDERS, "status", ColumnRole.EQUALITY) in _usage("delete from orders where status = $1") def test_insert_values_produces_no_usage(): @@ -125,7 +146,7 @@ def test_insert_values_produces_no_usage(): def test_insert_from_select_attributes_the_source_table(): - assert ("orders", "status", ColumnRole.EQUALITY) in _usage( + assert (ORDERS, "status", ColumnRole.EQUALITY) in _usage( "insert into customers (id) select customer_id from orders where status = $1" ) @@ -145,8 +166,8 @@ def test_reused_alias_across_scopes_does_not_corrupt_attribution(): "select o.id from orders o where o.status = $1 " "and o.id in (select o.id from customers o where o.status = $2)" ) - assert ("orders", "status", ColumnRole.EQUALITY) in usage - assert ("customers", "status", ColumnRole.EQUALITY) in usage + assert (ORDERS, "status", ColumnRole.EQUALITY) in usage + assert (CUSTOMERS, "status", ColumnRole.EQUALITY) in usage def test_distinct_aliases_across_scopes_both_resolve(): @@ -154,16 +175,16 @@ def test_distinct_aliases_across_scopes_both_resolve(): "select o.id from orders o where o.status = $1 " "and o.id in (select c.id from customers c where c.status = $2)" ) - assert ("orders", "status", ColumnRole.EQUALITY) in usage - assert ("customers", "status", ColumnRole.EQUALITY) in usage + assert (ORDERS, "status", ColumnRole.EQUALITY) in usage + assert (CUSTOMERS, "status", ColumnRole.EQUALITY) in usage def test_self_join_aliases_both_resolve_to_the_same_table(): usage = _usage( "select a.id from orders a join orders b on b.customer_id = a.id where a.status = $1" ) - assert ("orders", "customer_id", ColumnRole.JOIN) in usage - assert ("orders", "status", ColumnRole.EQUALITY) in usage + assert (ORDERS, "customer_id", ColumnRole.JOIN) in usage + assert (ORDERS, "status", ColumnRole.EQUALITY) in usage def test_cte_predicate_resolves_to_the_underlying_base_table(): @@ -173,11 +194,122 @@ def test_cte_predicate_resolves_to_the_underlying_base_table(): "with recent as (select id, status from orders where created_at > $1) " "select id from recent where status = $2" ) - assert ("orders", "created_at", ColumnRole.RANGE) in usage - assert not any(table == "recent" for table, _column, _role in usage) + assert (ORDERS, "created_at", ColumnRole.RANGE) in usage + assert not any(relation.table == "recent" for relation, _column, _role in usage) def test_unresolvable_column_raises_unqualifiable(): tree = sqlglot.parse_one("select nope from mystery_table", dialect="postgres") with pytest.raises(UnqualifiableQuery): extract_usage(tree, "postgres", SCHEMA) + + +ONE_SCHEMA = {"public": {"orders": {"id": "int", "status": "text", "shipped_at": "timestamp"}}} +TWO_SCHEMAS = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"items": {"sku": "text", "qty": "int"}}, +} +COLLIDING = { + "sales": {"orders": {"id": "int", "status": "text"}}, + "staging": {"orders": {"id": "int", "status": "text"}}, +} + + +def test_bare_table_resolves_to_its_only_owning_schema(): + """The common case: production SQL relies on search_path and says `from orders`. + + qualify() leaves Table.db empty here, so a `table.db`-only implementation keys this + under Relation("", "orders") and every catalog lookup misses. + """ + tree = parse("select id from orders where status = 'x'", "postgres") + usage = extract_usage(tree, "postgres", ONE_SCHEMA) + assert {relation for relation, _c, _r in usage} == {Relation("public", "orders")} + + +def test_explicitly_qualified_table_uses_the_schema_it_names(): + """`orders` exists in both schemas of COLLIDING, so the schema-map fallback alone would + find two owners and refuse to guess. Only the explicit `staging.` qualifier can produce + the expected answer — deleting `resolve_relation`'s `if table.db:` branch collapses this + to `set()` because the fallback returns None for an ambiguous bare name. + """ + tree = parse("select status from staging.orders where id > 1", "postgres") + usage = extract_usage(tree, "postgres", COLLIDING) + assert {relation for relation, _c, _r in usage} == {Relation("staging", "orders")} + + +def test_two_schemas_distinct_names_attribute_to_the_right_one(): + """A join across schemas must not collapse both sides onto one relation.""" + tree = parse("select o.id, i.sku from orders o join items i on i.sku = o.status", "postgres") + usage = extract_usage(tree, "postgres", TWO_SCHEMAS) + assert {relation for relation, _c, _r in usage} == { + Relation("sales", "orders"), + Relation("staging", "items"), + } + + +def test_ambiguous_bare_name_is_unqualifiable_not_a_crash(): + """sqlglot raises SchemaError, which is NOT an OptimizeError subclass.""" + tree = parse("select id from orders where status = 'x'", "postgres") + with pytest.raises(UnqualifiableQuery): + extract_usage(tree, "postgres", COLLIDING) + + +def test_resolve_relation_prefers_an_explicit_db_over_the_map(): + table = exp.Table(this=exp.to_identifier("orders"), db=exp.to_identifier("sales")) + assert resolve_relation(table, COLLIDING) == Relation("sales", "orders") + + +def test_resolve_relation_returns_none_when_ambiguous(): + """Two owners is not a guess we are entitled to make.""" + table = exp.Table(this=exp.to_identifier("orders")) + assert resolve_relation(table, COLLIDING) is None + + +def test_resolve_relation_returns_none_for_a_table_outside_the_map(): + table = exp.Table(this=exp.to_identifier("nowhere")) + assert resolve_relation(table, ONE_SCHEMA) is None + + +def test_resolve_relation_returns_none_for_an_explicit_schema_never_introspected(): + """An explicit qualifier naming a schema we never introspected must not be trusted + blindly. `qualify()` does not validate this for UPDATE/DELETE targets (see + resolve_relation's docstring) — an ungated `table.db` branch would manufacture + `Relation("other", "orders")`, a phantom that matches no catalog fact. + """ + table = exp.Table(this=exp.to_identifier("orders"), db=exp.to_identifier("other")) + assert resolve_relation(table, ONE_SCHEMA) is None + + +def test_dml_columns_attribute_to_the_qualified_target(): + tree = parse("update orders set status = 'y' where id = 1", "postgres") + usage = extract_usage(tree, "postgres", ONE_SCHEMA) + assert (Relation("public", "orders"), "id", ColumnRole.EQUALITY) in usage + + +def test_dml_with_an_unintrospected_explicit_schema_does_not_leak_a_phantom_relation(): + """The reachable case: `qualify()` leaves UPDATE/DELETE columns and their target table + unvalidated against the schema, so an explicitly-qualified schema qualify() never saw + sails through untouched. Confirmed by probing sqlglot 30.12 directly — this statement + raises nothing. Without the guard in resolve_relation, this attributes `id` to a + phantom `Relation("other", "orders")` instead of dropping it. + """ + tree = parse("update other.orders set status = 'x' where id = 1", "postgres") + usage = extract_usage(tree, "postgres", ONE_SCHEMA) + assert usage == () + + +def test_ambiguous_bare_dml_target_raises_ambiguous_relation(): + """`qualify()` does not validate UPDATE/DELETE targets, so a bare name held by two + introspected schemas reaches `_collect_dml` with no `SchemaError` ever raised. Left + silent, the statement would vanish with no usage and no counter moved — reported as + analysed by `aggregate` when it was not. It must raise the same way the SELECT-path + ambiguity does. + """ + tree = parse("update orders set status = 'x' where id = 1", "postgres") + with pytest.raises(AmbiguousRelation): + extract_usage(tree, "postgres", COLLIDING) + + +def test_relation_str_is_schema_dot_table(): + """Proposal titles and JSON keys render a Relation through this — pin the contract.""" + assert str(Relation("public", "orders")) == "public.orders" diff --git a/tests/test_workload_fingerprint.py b/tests/test_workload_fingerprint.py index d8b2f82..fcbf5ff 100644 --- a/tests/test_workload_fingerprint.py +++ b/tests/test_workload_fingerprint.py @@ -1,3 +1,4 @@ +import pytest import sqlglot from sqlglot import exp @@ -10,6 +11,7 @@ is_noise, literal_flags, redact_tree, + unwrap, ) @@ -50,14 +52,14 @@ def test_is_noise_filters_our_own_introspection_and_ddl(): assert not is_noise("select id from orders where status = $1") -def test_is_noise_also_discards_predicate_bearing_declare_and_copy(): - """A documented loss, pinned so it cannot become an undocumented one. +def test_is_noise_still_discards_the_raw_wrapper_text(): + """`is_noise` itself is a statement-prefix filter and stays that way. - `_LEADING_NOISE` is a statement-prefix filter, so a cursor declaration or a COPY that - wraps a real SELECT — with real predicates — is discarded whole. Django's - `QuerySet.iterator()` emits the first form. Unwrapping to the inner SELECT is a - follow-up; until then this is a README limitation and the skip counter says only - "filtered", never "introspection/DDL". + A cursor declaration or a `COPY (...) TO` still starts with a keyword `_LEADING_NOISE` + matches, so calling `is_noise` on the *raw* row still discards it whole. That is no + longer a loss: `ingest` calls `unwrap()` first and tests `is_noise` on the inner query, + so the real read survives — see `test_a_declared_cursor_is_analyzed_not_filtered` and + `test_a_copy_subquery_is_analyzed_not_filtered` below. """ assert is_noise("DECLARE cur CURSOR FOR SELECT id FROM orders WHERE status = $1") assert is_noise("COPY (SELECT id FROM orders WHERE status = $1) TO STDOUT") @@ -98,6 +100,30 @@ def test_ingest_groups_by_fingerprint_and_sums_cost(): assert "999" not in workload.stats[0].sql +def test_query_stat_sql_is_the_reserialised_redacted_tree_not_the_row_text(): + """Pins the premise `_wide_relations_touched` relies on, which its docstring once got + backwards. + + That function re-parses `stat.sql` with no fallback, and justified the missing fallback + with "it is the same text `ingest()` already parsed". It is not: `stat.sql` is sqlglot's + re-serialisation of the *redacted* tree, so the real guarantee is that sqlglot re-parses + its own generated SQL — measured, not inherited from the row having parsed. A comment + that justifies deleting code with the wrong reason is how the code comes back, so the + two properties it actually depends on are asserted here: the text is not the row's, and + it round-trips. + """ + raw = "select id from t where email = 'a@b.de' and n > 42" + fetch = WorkloadFetch( + rows=(RawQueryRow(sql=raw, calls=1, total_time_ms=1.0),), window_description="w" + ) + stat = ingest(fetch, "postgres").stats[0] + assert stat.sql != raw, "stat.sql is the row's own text — the false premise, made true" + assert "a@b.de" not in stat.sql + # Round-trip: parsing sqlglot's own output under the same dialect must succeed, since + # ADV006 does exactly this and has no fallback if it raises. + assert parse(stat.sql, "postgres") is not None + + def test_ingest_counts_unparseable_and_noise_without_raising(): fetch = WorkloadFetch( rows=( @@ -113,6 +139,35 @@ def test_ingest_counts_unparseable_and_noise_without_raising(): assert len(workload.stats) == 1 +def test_ingest_filters_fetch_as_noise(): + """`FETCH` carries no query text, so it must stay filtered rather than analysed. + + Left unfiltered, sqlglot parses `FETCH 100 FROM c` as `exp.Command` — it would become + an analysed query group with zero column usage but the cursor's full cost attached + (see `unwrap`'s docstring on where a cursor's cost actually lands), which is worse than + the noise it should have been. + """ + fetch = WorkloadFetch( + rows=(RawQueryRow(sql="FETCH 100 FROM c", calls=2, total_time_ms=72.9),), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 1 + assert workload.stats == () + + +def test_ingest_filters_close_as_noise(): + """`CLOSE` carries no query text either; left unfiltered, sqlglot parses `CLOSE bigcur` + as `exp.Alias`, which would also become an analysed, zero-column-usage query group.""" + fetch = WorkloadFetch( + rows=(RawQueryRow(sql="CLOSE bigcur", calls=1, total_time_ms=0.1),), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 1 + assert workload.stats == () + + def test_ingest_captures_literal_flags_before_redaction(): """The load-bearing ordering guard. @@ -198,3 +253,154 @@ def test_redaction_still_erases_a_real_literal_beside_a_placeholder(): redacted = redact_tree(parse(sql, "postgres")).sql("postgres") assert "secret-value" not in redacted assert "$1" in redacted + + +@pytest.mark.parametrize( + "sql,expected", + [ + ( + "DECLARE c CURSOR FOR SELECT id FROM orders WHERE status = 'x'", + "SELECT id FROM orders WHERE status = 'x'", + ), + ( + "DECLARE c CURSOR WITH HOLD FOR SELECT id FROM orders", + "SELECT id FROM orders", + ), + ( + "DECLARE c NO SCROLL CURSOR FOR SELECT id FROM orders", + "SELECT id FROM orders", + ), + ( + "DECLARE c BINARY INSENSITIVE SCROLL CURSOR WITH HOLD FOR SELECT a FROM t", + "SELECT a FROM t", + ), + ( + 'DECLARE "my cursor" CURSOR FOR SELECT a FROM t', + "SELECT a FROM t", + ), + ( + "COPY (SELECT id FROM orders WHERE status = 'x') TO STDOUT", + "SELECT id FROM orders WHERE status = 'x'", + ), + ("copy (select 1) to stdout", "select 1"), + # Not cosmetic: greedy `.*` under DOTALL lets the query group absorb the space + # before the closing paren, so without `.strip()` this would return + # "SELECT a FROM t " (trailing space) — a distinct fingerprint-grouping key from + # the space-free forms above, even though it is the same query. + ( + "COPY ( SELECT a FROM t ) TO STDOUT", + "SELECT a FROM t", + ), + ], +) +def test_unwrap_recovers_the_inner_query(sql, expected): + assert unwrap(sql) == expected + + +@pytest.mark.parametrize( + "sql", + [ + "SELECT id FROM orders", + "COPY orders TO STDOUT", + # A later `(` must not be enough: guards a loosened prefix anchor (e.g. + # `^\s*COPY\s*[^(]*\(`) that skips ahead to the first paren instead of requiring + # one immediately after `COPY`. + "COPY orders (id, status) TO STDOUT", + # Guards the `\bTO\b` anchor that excludes writes: a `COPY (...) FROM STDIN` wraps + # a query but is loading it, not reading it, and must stay noise. + "COPY (SELECT 1) FROM STDIN", + # Guards the capture group's `\S` requirement: a query after `FOR` is mandatory, + # not optional. + "DECLARE c CURSOR FOR", + ], +) +def test_unwrap_leaves_everything_else_alone(sql): + """Anything without a recoverable inner query is returned unchanged, not mangled. + + `FETCH 100 FROM c`, `CLOSE c` and a bare `DECLARE` used to be members here too, but none + of them discriminated anything: none starts with a keyword either pattern's grammar + depends on past its first literal token, so no plausible single mutation of + `_DECLARE_CURSOR` or `_COPY_QUERY`'s internals would make any of them match — they + passed before the patterns existed and would pass against almost any broken version of + them just the same. `FETCH`/`CLOSE` staying noise is `is_noise`'s contract, not + `unwrap`'s — see `test_ingest_filters_fetch_as_noise` and + `test_ingest_filters_close_as_noise`, which pin that contract directly and go red if + either keyword is removed from `_LEADING_NOISE`. + """ + assert unwrap(sql) == sql + + +def test_a_declared_cursor_is_analyzed_not_filtered(): + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="DECLARE c CURSOR FOR SELECT id FROM orders WHERE status = 'x'", + calls=3, + total_time_ms=300.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 0 + assert len(workload.stats) == 1 + assert "DECLARE" not in workload.stats[0].sql.upper() + assert workload.stats[0].calls == 3 + + +def test_a_copy_subquery_is_analyzed_not_filtered(): + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="COPY (SELECT id FROM orders WHERE status = 'x') TO STDOUT", + calls=1, + total_time_ms=10.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 0 + assert len(workload.stats) == 1 + + +def test_a_declared_cursor_over_introspection_is_still_filtered(): + """Unwrapping must not become a way to smuggle our own catalog reads into the workload.""" + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="DECLARE c CURSOR FOR SELECT * FROM pg_stat_statements", + calls=1, + total_time_ms=1.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert workload.skipped_noise == 1 + assert workload.stats == () + + +def test_a_whole_table_copy_is_still_filtered(): + fetch = WorkloadFetch( + rows=(RawQueryRow(sql="COPY orders TO STDOUT", calls=1, total_time_ms=1.0),), + window_description="w", + ) + assert ingest(fetch, "postgres").skipped_noise == 1 + + +def test_the_unwrapped_query_is_still_redacted(): + """Redaction runs after unwrapping, so the inner literal must not survive.""" + fetch = WorkloadFetch( + rows=( + RawQueryRow( + sql="DECLARE c CURSOR FOR SELECT id FROM orders WHERE email = 'a@b.test'", + calls=1, + total_time_ms=1.0, + ), + ), + window_description="w", + ) + workload = ingest(fetch, "postgres") + assert "a@b.test" not in workload.stats[0].sql + assert "a@b.test" not in workload.stats[0].fingerprint diff --git a/tests/test_workload_postgres.py b/tests/test_workload_postgres.py index 345afd1..3902d0a 100644 --- a/tests/test_workload_postgres.py +++ b/tests/test_workload_postgres.py @@ -1,10 +1,13 @@ +import inspect import re from datetime import timedelta +from pathlib import Path import pytest -from sqlquality.models import ConnectionParams +from sqlquality.models import ConnectionParams, Relation from sqlquality.workload import get_workload_adapter +from sqlquality.workload import postgres as postgres_module from sqlquality.workload.base import MAX_TIMEOUT_S from sqlquality.workload.postgres import ( CAP_INDEXES, @@ -114,6 +117,84 @@ def test_workload_statement_is_scoped_to_the_current_database(): assert "order by" in sql and "limit" in sql +def test_workload_statement_does_not_filter_on_toplevel(): + """Deliberately not filtered — a documented trade-off, not an oversight. + + A blanket `AND s.toplevel` would deduplicate a `COPY (...) TO` execution under + `pg_stat_statements.track = all` (see the README's "Prerequisites and limits"), but + `toplevel = false` is also the *only* way Postgres exposes the SQL executed inside a + PL/pgSQL function body. Tried and reverted: verified live that the filter made a + genuinely hot, function-wrapped query disappear from evidence entirely while a colder + query took its place as a `high`-confidence proposal — confidently wrong, which is worse + than the double-count it would have fixed. A *narrow* predicate does work — the two + nested forms are textually distinguishable, see + `test_the_toplevel_tradeoff_is_documented_as_a_price_not_an_impossibility` — and is + declined only because naming `s.toplevel` at all raises the floor to PostgreSQL 14. This + test exists so a future attempt to reintroduce the blanket filter fails here first, + rather than silently reopening that regression. + """ + sql = PostgresWorkloadAdapter().SQL[CAP_WORKLOAD].lower() + assert "toplevel" not in sql + + +def _source_of(module) -> str: + return Path(inspect.getsourcefile(module)).read_text(encoding="utf-8") + + +def test_the_toplevel_tradeoff_is_documented_as_a_price_not_an_impossibility(): + """The reason the filter is absent must be the reason it is actually absent. + + Both the source comment and the README claimed no `s.query` text pattern could separate + a COPY's nested duplicate from a PL/pgSQL function's nested statement. Measured on + PostgreSQL 16 under `track = all` that is false: the COPY's nested row keeps its wrapper + while a function body is recorded bare, and + `NOT (s.toplevel = false AND s.query ~* '^\\s*COPY\\s*\\(')` removed exactly the + duplicate (4 rows -> 3). The filter is declined because *naming* `s.toplevel` requires + PostgreSQL 14 while the supported floor is 13 — a price, not an impossibility. A comment + that justifies an absence with a false premise is how the wrong decision gets made next + time, so the claim is pinned here. + """ + # Scoped to the CAP_WORKLOAD comment block rather than the whole module: searching all of + # `postgres.py` for "postgresql 14" is satisfied by `_row_estimate`'s unrelated docstring + # about the `reltuples` sentinel, so the assertion passed with the real sentence deleted. + source = _source_of(postgres_module) + marker = "Deliberately NOT filtered on `s.toplevel`" + assert marker in source, "the CAP_WORKLOAD comment explaining the absence is gone" + start = source.index(marker) + comment_block = source[start : source.index('"""', start)] + readme = (Path(__file__).resolve().parents[1] / "README.md").read_text(encoding="utf-8") + for text, where in ((comment_block, "the CAP_WORKLOAD comment"), (readme, "the README")): + lowered = text.lower() + # One absence check, not an enumeration of every phrasing the false claim once had: + # whack-a-mole substrings go stale and give false confidence. The positive assertion + # below is what actually pins the reasoning. + assert "text pattern tells the two apart" not in lowered, ( + f"{where} still claims no predicate can separate the two" + ) + assert "postgresql 14" in lowered or "postgres 14" in lowered, ( + f"{where} does not state the version cost that is the actual reason" + ) + + +def test_the_plpgsql_double_count_is_documented_as_a_limitation(): + """The larger, unfixable half of the `track = all` inaccuracy. + + Every PL/pgSQL call is counted twice under `track = all` — the call and its body are + recorded as separate rows with nearly identical durations — which roughly halves every + `cost_share`. Unlike the `COPY` duplicate no predicate can fix it (the call carries the + cost, the body carries the predicates), so disclosure is the only honest treatment, and + the README documented only the smaller `COPY` half. + + Deliberately does *not* assert the specific milliseconds the README quotes. Two runs of + the same fixture measured 68.21/67.67 and 46.44/46.37 — the shape reproduces, the numbers + are that machine's. Pinning them would make an honest re-measurement look like a + regression and train the next person to edit the test rather than read it. + """ + readme = (Path(__file__).resolve().parents[1] / "README.md").read_text(encoding="utf-8") + assert "PL/pgSQL function call is counted twice" in readme + assert "no predicate can fix it" in readme + + class FakeQuerier: """Returns canned rows per capability, keyed by a distinctive SQL substring.""" @@ -133,6 +214,21 @@ def __call__(self, sql, params): return [] +def _canned(rows_by_capability): + """A FakeQuerier addressed by capability constant rather than a raw SQL substring. + + Same dispatch as FakeQuerier — a capability's own statement text is already a unique + substring of itself — just keyed by the name a test actually cares about instead of a + fragile fragment of SQL. + """ + return FakeQuerier( + { + PostgresWorkloadAdapter.SQL[capability]: rows + for capability, rows in rows_by_capability.items() + } + ) + + def test_fetch_workload_maps_rows_and_reports_the_window(): querier = FakeQuerier( { @@ -147,6 +243,32 @@ def test_fetch_workload_maps_rows_and_reports_the_window(): assert "2026-07-01" in fetch.window_description +def test_a_copy_executions_two_rows_both_pass_through_under_track_all(): + """Pins the accepted, documented limitation — not a filter that no longer exists. + + Under `pg_stat_statements.track = all`, one `COPY (SELECT ...) TO ...` execution + produces two rows: the verbatim top-level statement and its normalised nested query. + `fetch_workload` deliberately does not filter either out (see + `test_workload_statement_does_not_filter_on_toplevel`), so both reach `ingest`, which + fingerprints them differently (a real literal survives redaction in one, `$1` sits in + the other) and counts the one execution as two query groups at roughly twice its true + cost. If this assertion ever fails, either the double-count was fixed some other way + (update the README's "Prerequisites and limits") or the row pass-through broke by + accident. + """ + querier = FakeQuerier( + { + "pg_stat_statements": [ + ("copy (select id from orders where status = 'x') to stdout", 1, 20.1, 40000), + ("select id from orders where status = $1", 1, 19.8, 40000), + ], + "pg_stat_database": [("2026-07-01",)], + } + ) + fetch = PostgresWorkloadAdapter(querier=querier).fetch_workload(None, 500) + assert len(fetch.rows) == 2 + + def test_fetch_workload_window_is_honest_that_since_is_not_supported(): querier = FakeQuerier( { @@ -198,34 +320,147 @@ def test_fetch_schema_builds_a_sqlglot_schema_mapping(): querier = FakeQuerier( { "information_schema.columns": [ - ("orders", "id", "integer"), - ("orders", "status", "text"), - ("customers", "id", "integer"), + ("public", "orders", "id", "integer"), + ("public", "orders", "status", "text"), + ("public", "customers", "id", "integer"), ] } ) schema = PostgresWorkloadAdapter(querier=querier).fetch_schema(("public",)) assert schema == { - "orders": {"id": "integer", "status": "text"}, - "customers": {"id": "integer"}, + "public": { + "orders": {"id": "integer", "status": "text"}, + "customers": {"id": "integer"}, + }, } +def test_fetch_schema_is_nested_by_schema(): + rows = { + CAP_SCHEMA: [ + ("sales", "orders", "id", "integer"), + ("sales", "orders", "status", "text"), + ("staging", "orders", "id", "integer"), + ] + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + assert adapter.fetch_schema(("sales", "staging")) == { + "sales": {"orders": {"id": "integer", "status": "text"}}, + "staging": {"orders": {"id": "integer"}}, + } + + +def test_table_facts_do_not_alias_across_schemas(): + """Two same-named tables must keep their own row estimates.""" + rows = { + CAP_SCHEMA: [("sales", "orders", "id", "integer"), ("staging", "orders", "id", "integer")], + CAP_TABLE_FACTS: [("sales", "orders", 50_000, 1024), ("staging", "orders", 7, 64)], + CAP_NDV: [], + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + facts = adapter.fetch_table_facts( + ("sales", "staging"), + frozenset({Relation("sales", "orders"), Relation("staging", "orders")}), + ) + assert facts[Relation("sales", "orders")].row_estimate == 50_000 + assert facts[Relation("staging", "orders")].row_estimate == 7 + + +def test_ndv_does_not_leak_between_same_named_tables(): + rows = { + CAP_SCHEMA: [("sales", "orders", "id", "integer"), ("staging", "orders", "id", "integer")], + CAP_TABLE_FACTS: [("sales", "orders", 50_000, 1024), ("staging", "orders", 50_000, 1024)], + CAP_NDV: [("sales", "orders", "id", 5000.0), ("staging", "orders", "id", 3.0)], + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + facts = adapter.fetch_table_facts( + ("sales", "staging"), + frozenset({Relation("sales", "orders"), Relation("staging", "orders")}), + ) + assert facts[Relation("sales", "orders")].ndv["id"] == 5000.0 + assert facts[Relation("staging", "orders")].ndv["id"] == 3.0 + + +def test_indexes_do_not_alias_across_schemas(): + rows = { + CAP_INDEXES: [ + ("sales", "orders", "idx_a", "id", 1, False, False, 0, 100, False, None, False, "..."), + ( + "staging", + "orders", + "idx_b", + "id", + 1, + False, + False, + 9, + 200, + False, + None, + False, + "...", + ), + ] + } + adapter = PostgresWorkloadAdapter(querier=_canned(rows)) + indexes = adapter.fetch_indexes( + ("sales", "staging"), + frozenset({Relation("sales", "orders"), Relation("staging", "orders")}), + ) + assert [i.name for i in indexes[Relation("sales", "orders")]] == ["idx_a"] + assert [i.name for i in indexes[Relation("staging", "orders")]] == ["idx_b"] + assert indexes[Relation("staging", "orders")][0].scans == 9 + + +def _select_list(sql: str) -> str: + """The text between `SELECT` and the first `FROM` — the columns actually returned. + + Grepping the whole statement cannot tell a `SELECT` list from a `WHERE` clause, and + every one of these statements already filtered on the schema before this task — the + substring the naive version of this check looked for was there from the start, in the + WHERE clause, regardless of what the SELECT list returned. + """ + match = re.search(r"select\s+(.*?)\s+from\b", sql, re.IGNORECASE | re.DOTALL) + assert match, f"no SELECT ... FROM found in statement: {sql!r}" + return match.group(1) + + +def test_every_relation_returning_statement_selects_its_schema(): + """A statement that filters on schema but does not return it cannot be keyed by it. + + This is the whole defect class of this task: the rows come back indistinguishable and + the last one silently wins. An earlier version of this test grepped the *entire* + statement for `nspname`/`schemaname`/`table_schema` and passed even with the schema + column stripped from the SELECT list — those substrings were already present in every + WHERE clause at d0421d0, since each statement already filtered on schema without + returning it. Restricting the search to the select list (see `_select_list`) is what + actually pins the defect this task exists to close. + """ + for capability in (CAP_SCHEMA, CAP_TABLE_FACTS, CAP_NDV, CAP_INDEXES): + select_list = _select_list(PostgresWorkloadAdapter.SQL[capability]) + assert ( + "nspname" in select_list or "schemaname" in select_list or "table_schema" in select_list + ), capability + + def test_fetch_table_facts_resolves_negative_n_distinct_as_a_row_fraction(): querier = FakeQuerier( { - "pg_total_relation_size": [("orders", 1000, 8192)], - "information_schema.columns": [("orders", "id", "integer"), ("orders", "s", "text")], - "pg_stats": [("orders", "id", 500.0), ("orders", "s", -0.25)], + "pg_total_relation_size": [("public", "orders", 1000, 8192)], + "information_schema.columns": [ + ("public", "orders", "id", "integer"), + ("public", "orders", "s", "text"), + ], + "pg_stats": [("public", "orders", "id", 500.0), ("public", "orders", "s", -0.25)], } ) facts = PostgresWorkloadAdapter(querier=querier).fetch_table_facts( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - assert facts["orders"].row_estimate == 1000 - assert facts["orders"].ndv["id"] == 500.0 + assert facts[Relation("public", "orders")].row_estimate == 1000 + assert facts[Relation("public", "orders")].ndv["id"] == 500.0 # -0.25 means "a quarter of the rows are distinct" - assert facts["orders"].ndv["s"] == 250.0 + assert facts[Relation("public", "orders")].ndv["s"] == 250.0 def test_negative_n_distinct_without_a_row_count_is_omitted_not_zeroed(): @@ -237,30 +472,30 @@ def test_negative_n_distinct_without_a_row_count_is_omitted_not_zeroed(): """ querier = FakeQuerier( { - "information_schema.columns": [("orders", "id", "integer")], - "pg_stats": [("orders", "id", -0.25)], + "information_schema.columns": [("public", "orders", "id", "integer")], + "pg_stats": [("public", "orders", "id", -0.25)], # No pg_total_relation_size rows: the row count is unknown. } ) facts = PostgresWorkloadAdapter(querier=querier).fetch_table_facts( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - assert facts["orders"].row_estimate is None - assert "id" not in facts["orders"].ndv + assert facts[Relation("public", "orders")].row_estimate is None + assert "id" not in facts[Relation("public", "orders")].ndv def test_absolute_n_distinct_survives_a_missing_row_count(): """A positive n_distinct is an absolute count and needs no row estimate.""" querier = FakeQuerier( { - "information_schema.columns": [("orders", "id", "integer")], - "pg_stats": [("orders", "id", 500.0)], + "information_schema.columns": [("public", "orders", "id", "integer")], + "pg_stats": [("public", "orders", "id", 500.0)], } ) facts = PostgresWorkloadAdapter(querier=querier).fetch_table_facts( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - assert facts["orders"].ndv["id"] == 500.0 + assert facts[Relation("public", "orders")].ndv["id"] == 500.0 def test_a_never_analyzed_table_reports_an_unknown_row_count(): @@ -272,28 +507,28 @@ def test_a_never_analyzed_table_reports_an_unknown_row_count(): """ querier = FakeQuerier( { - "information_schema.columns": [("orders", "id", "integer")], - "pg_total_relation_size": [("orders", -1, 10**9)], + "information_schema.columns": [("public", "orders", "id", "integer")], + "pg_total_relation_size": [("public", "orders", -1, 10**9)], } ) facts = PostgresWorkloadAdapter(querier=querier).fetch_table_facts( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - assert facts["orders"].row_estimate is None + assert facts[Relation("public", "orders")].row_estimate is None def test_an_analyzed_empty_table_still_reports_zero(): """0 is a real answer — analyzed and empty — and must not be conflated with unknown.""" querier = FakeQuerier( { - "information_schema.columns": [("orders", "id", "integer")], - "pg_total_relation_size": [("orders", 0, 8192)], + "information_schema.columns": [("public", "orders", "id", "integer")], + "pg_total_relation_size": [("public", "orders", 0, 8192)], } ) facts = PostgresWorkloadAdapter(querier=querier).fetch_table_facts( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - assert facts["orders"].row_estimate == 0 + assert facts[Relation("public", "orders")].row_estimate == 0 def test_fetch_indexes_restores_column_order_from_ordinality(): @@ -307,6 +542,7 @@ def test_fetch_indexes_restores_column_order_from_ordinality(): { "pg_index": [ ( + "public", "orders", "idx_status_created", "created_at", @@ -321,6 +557,7 @@ def test_fetch_indexes_restores_column_order_from_ordinality(): "CREATE INDEX idx_status_created ON orders (status, created_at)", ), ( + "public", "orders", "idx_status_created", "status", @@ -338,9 +575,9 @@ def test_fetch_indexes_restores_column_order_from_ordinality(): } ) indexes = PostgresWorkloadAdapter(querier=querier).fetch_indexes( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - assert indexes["orders"][0].columns == ("status", "created_at") + assert indexes[Relation("public", "orders")][0].columns == ("status", "created_at") def test_connect_scrubs_a_password_from_a_driver_failure(monkeypatch): @@ -383,6 +620,7 @@ def test_fetch_indexes_groups_columns_in_ordinal_order(): { "pg_index": [ ( + "public", "orders", "orders_pkey", "id", @@ -397,6 +635,7 @@ def test_fetch_indexes_groups_columns_in_ordinal_order(): "CREATE UNIQUE INDEX orders_pkey ON orders (id)", ), ( + "public", "orders", "idx_status_created", "status", @@ -411,6 +650,7 @@ def test_fetch_indexes_groups_columns_in_ordinal_order(): "CREATE INDEX idx_status_created ON orders (status, created_at)", ), ( + "public", "orders", "idx_status_created", "created_at", @@ -428,9 +668,9 @@ def test_fetch_indexes_groups_columns_in_ordinal_order(): } ) indexes = PostgresWorkloadAdapter(querier=querier).fetch_indexes( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - by_name = {i.name: i for i in indexes["orders"]} + by_name = {i.name: i for i in indexes[Relation("public", "orders")]} assert by_name["idx_status_created"].columns == ("status", "created_at") assert by_name["orders_pkey"].is_primary is True assert by_name["idx_status_created"].scans == 0 @@ -447,14 +687,14 @@ def test_a_denied_statement_degrades_and_names_the_privilege(): """ querier = FakeQuerier( { - "information_schema.columns": [("orders", "id", "integer")], - "pg_stats": [("orders", "id", 500.0)], + "information_schema.columns": [("public", "orders", "id", "integer")], + "pg_stats": [("public", "orders", "id", 500.0)], }, fail_markers=("pg_stats",), ) adapter = PostgresWorkloadAdapter(querier=querier) - facts = adapter.fetch_table_facts(("public",), frozenset({"orders"})) - assert facts["orders"].ndv == {} + facts = adapter.fetch_table_facts(("public",), frozenset({Relation("public", "orders")})) + assert facts[Relation("public", "orders")].ndv == {} assert any(cap == CAP_NDV for cap, _ in adapter.degraded) assert any("pg_stats" in reason for _, reason in adapter.degraded) @@ -464,14 +704,14 @@ def test_the_denial_fixture_would_otherwise_have_returned_statistics(): so the emptiness there is attributable to the denial rather than to an empty fixture.""" querier = FakeQuerier( { - "information_schema.columns": [("orders", "id", "integer")], - "pg_stats": [("orders", "id", 500.0)], + "information_schema.columns": [("public", "orders", "id", "integer")], + "pg_stats": [("public", "orders", "id", 500.0)], } ) facts = PostgresWorkloadAdapter(querier=querier).fetch_table_facts( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - assert facts["orders"].ndv == {"id": 500.0} + assert facts[Relation("public", "orders")].ndv == {"id": 500.0} class _FakeCursor: @@ -738,19 +978,53 @@ def test_the_schema_statement_runs_once_per_run(): Twice the catalog work, and — worse — two identical `degraded` entries when it is denied, so the user is told the same thing twice. """ - querier = FakeQuerier({"information_schema.columns": [("orders", "id", "integer")]}) + querier = FakeQuerier({"information_schema.columns": [("public", "orders", "id", "integer")]}) adapter = PostgresWorkloadAdapter(querier=querier) adapter.fetch_schema(("public",)) - adapter.fetch_table_facts(("public",), frozenset({"orders"})) + adapter.fetch_table_facts(("public",), frozenset({Relation("public", "orders")})) schema_calls = [sql for sql, _ in querier.calls if "information_schema.columns" in sql] assert len(schema_calls) == 1 +def test_the_catalog_statements_are_given_the_relations_they_filter_on(): + """Each `= ANY(%s)` gets the schema list *and* the table list, not just the schema list. + + `FakeQuerier` keys its canned rows on a SQL substring and ignores the bind parameters, so + nothing read `params[1]` anywhere in the suite: replacing the table list with a bogus value + in all three relation-scoped statements left every test green. A real server would then be + asked about the wrong relations — returning nothing, which reads exactly like a table with + no statistics and no indexes, and suppresses proposals with no message. + + Asserted per statement rather than in aggregate: one shared assertion would pass while two + of the three passed nothing, which is the shape of six separate findings on this branch. + """ + querier = FakeQuerier( + { + "information_schema.columns": [("public", "orders", "id", "integer")], + "pg_total_relation_size": [("public", "orders", 50_000, 1024)], + "pg_stats": [], + "pg_get_indexdef": [], + } + ) + adapter = PostgresWorkloadAdapter(querier=querier) + relations = frozenset({Relation("public", "orders"), Relation("public", "order_items")}) + adapter.fetch_table_facts(("public",), relations) + adapter.fetch_indexes(("public",), relations) + + expected_tables = ["order_items", "orders"] + for marker in ("pg_total_relation_size", "pg_stats", "pg_get_indexdef"): + binds = [params for sql, params in querier.calls if marker in sql] + assert binds, f"{marker} statement never ran" + for params in binds: + assert params[0] == ["public"], f"{marker} lost its schema list" + assert params[1] == expected_tables, f"{marker} lost its table list: {params[1]!r}" + + def test_a_denied_schema_statement_is_reported_once_not_twice(): querier = FakeQuerier({}, fail_markers=("information_schema.columns",)) adapter = PostgresWorkloadAdapter(querier=querier) adapter.fetch_schema(("public",)) - adapter.fetch_table_facts(("public",), frozenset({"orders"})) + adapter.fetch_table_facts(("public",), frozenset({Relation("public", "orders")})) assert [cap for cap, _ in adapter.degraded].count(CAP_SCHEMA) == 1 @@ -787,6 +1061,7 @@ def test_fetch_indexes_records_an_expression_index_rather_than_dropping_it(): "pg_index": [ # attname is NULL for the expression column, as a LEFT JOIN yields. ( + "public", "orders", "idx_lower_status", None, @@ -804,9 +1079,9 @@ def test_fetch_indexes_records_an_expression_index_rather_than_dropping_it(): } ) indexes = PostgresWorkloadAdapter(querier=querier).fetch_indexes( - ("public",), frozenset({"orders"}) + ("public",), frozenset({Relation("public", "orders")}) ) - index = indexes["orders"][0] + index = indexes[Relation("public", "orders")][0] assert index.has_expressions is True assert index.columns == () assert "lower(status)" in (index.definition or "") @@ -817,6 +1092,7 @@ def test_fetch_indexes_records_a_partial_index_predicate(): { "pg_index": [ ( + "public", "orders", "idx_open", "status", @@ -834,8 +1110,8 @@ def test_fetch_indexes_records_a_partial_index_predicate(): } ) index = PostgresWorkloadAdapter(querier=querier).fetch_indexes( - ("public",), frozenset({"orders"}) - )["orders"][0] + ("public",), frozenset({Relation("public", "orders")}) + )[Relation("public", "orders")][0] assert index.is_partial is True assert index.predicate == "(shipped_at IS NULL)" assert index.columns == ("status",) @@ -846,6 +1122,7 @@ def test_fetch_indexes_leaves_a_plain_index_unmarked(): { "pg_index": [ ( + "public", "orders", "idx_status", "status", @@ -863,8 +1140,8 @@ def test_fetch_indexes_leaves_a_plain_index_unmarked(): } ) index = PostgresWorkloadAdapter(querier=querier).fetch_indexes( - ("public",), frozenset({"orders"}) - )["orders"][0] + ("public",), frozenset({Relation("public", "orders")}) + )[Relation("public", "orders")][0] assert (index.is_partial, index.predicate, index.has_expressions) == (False, None, False) diff --git a/tests/test_workload_redaction.py b/tests/test_workload_redaction.py index e5a6cd7..0d104f9 100644 --- a/tests/test_workload_redaction.py +++ b/tests/test_workload_redaction.py @@ -61,16 +61,16 @@ STAR_QUERY = "select * from wide_events where actor_email = 'hans@betterdoc.de'" COLUMNS = [ - ("orders", "id", "integer"), - ("orders", "note", "text"), - ("orders", "status", "text"), - ("orders", "customer_email", "text"), - ("orders", "reference", "text"), - ("orders", "iban", "text"), - ("orders", "created_at", "timestamp"), - ("wide_events", "actor_email", "text"), + ("public", "orders", "id", "integer"), + ("public", "orders", "note", "text"), + ("public", "orders", "status", "text"), + ("public", "orders", "customer_email", "text"), + ("public", "orders", "reference", "text"), + ("public", "orders", "iban", "text"), + ("public", "orders", "created_at", "timestamp"), + ("public", "wide_events", "actor_email", "text"), # Wide enough to trip ADV006's ≥15-column floor. - *[("wide_events", f"c{i}", "text") for i in range(20)], + *[("public", "wide_events", f"c{i}", "text") for i in range(20)], ] ROWS = { @@ -82,10 +82,13 @@ "pg_stat_database": [("2026-07-01",)], "information_schema.columns": COLUMNS, "pg_total_relation_size": [ - ("orders", 8_000_000, 10**9), - ("wide_events", 4_000_000, 10**9), + ("public", "orders", 8_000_000, 10**9), + ("public", "wide_events", 4_000_000, 10**9), + ], + "pg_stats": [ + ("public", "orders", "status", 4.0), + ("public", "orders", "customer_email", 900_000.0), ], - "pg_stats": [("orders", "status", 4.0), ("orders", "customer_email", 900_000.0)], "pg_index": [], } diff --git a/tests/test_workload_rules.py b/tests/test_workload_rules.py index 55f14b5..96bf39c 100644 --- a/tests/test_workload_rules.py +++ b/tests/test_workload_rules.py @@ -1,3 +1,5 @@ +from pathlib import Path + from sqlquality.models import ( Aggregation, ColumnRole, @@ -5,6 +7,7 @@ Confidence, Proposal, QueryStat, + Relation, TableFacts, Workload, ) @@ -13,7 +16,9 @@ PgIndex, PostgresWorkloadAdapter, _quote_ident, + propose_grouping_indexes, propose_indexes, + propose_join_keys, propose_partial_indexes, propose_redundant_indexes, propose_sargability, @@ -21,12 +26,16 @@ propose_unused_indexes, ) +#: The relation nearly every test in this file talks about, so each test only has to name +#: a different relation when the point of the test *is* the relation. +_ORDERS = Relation("public", "orders") + -def usage(column, role, cost_share=0.5, cost_ms=50.0, table="orders", fps=("fp1",)): +def _usage(relation, column, role, cost_share=0.5, cost_ms=50.0, fps=("fp1",)): """`fps` defaults to a single shared fingerprint, so usages co-occur unless a test deliberately gives them disjoint sets.""" return ColumnUsage( - table=table, + relation=relation, column=column, role=role, calls=10, @@ -36,12 +45,17 @@ def usage(column, role, cost_share=0.5, cost_ms=50.0, table="orders", fps=("fp1" ) -def facts(rows=1_000_000, ndv=None, columns=("id", "status", "created_at", "customer_id")): - return { - "orders": TableFacts( - name="orders", row_estimate=rows, size_bytes=10**8, columns=columns, ndv=ndv or {} - ) - } +def _facts( + relation, rows=1_000_000, ndv=None, columns=("id", "status", "created_at", "customer_id") +): + return TableFacts( + relation=relation, row_estimate=rows, size_bytes=10**8, columns=columns, ndv=ndv or {} + ) + + +def _facts_map(relation=_ORDERS, **kwargs): + """The common case: one relation's facts, as the `facts` mapping every rule expects.""" + return {relation: _facts(relation, **kwargs)} def codes(proposals): @@ -51,10 +65,10 @@ def codes(proposals): def test_equality_then_range_ordering_in_the_candidate_index(): proposals = propose_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("created_at", ColumnRole.RANGE, cost_ms=80.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_ms=80.0), ], - facts(), + _facts_map(), {}, min_cost_share=0.01, ) @@ -65,11 +79,11 @@ def test_equality_then_range_ordering_in_the_candidate_index(): def test_only_one_range_column_is_included(): proposals = propose_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("created_at", ColumnRole.RANGE, cost_ms=80.0), - usage("shipped_at", ColumnRole.RANGE, cost_ms=70.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_ms=80.0), + _usage(_ORDERS, "shipped_at", ColumnRole.RANGE, cost_ms=70.0), ], - facts(columns=("status", "created_at", "shipped_at")), + _facts_map(columns=("status", "created_at", "shipped_at")), {}, min_cost_share=0.01, ) @@ -79,12 +93,12 @@ def test_only_one_range_column_is_included(): def test_arity_is_capped(): proposals = propose_indexes( [ - usage("a", ColumnRole.EQUALITY, cost_ms=99.0), - usage("b", ColumnRole.EQUALITY, cost_ms=98.0), - usage("c", ColumnRole.EQUALITY, cost_ms=97.0), - usage("d", ColumnRole.EQUALITY, cost_ms=96.0), + _usage(_ORDERS, "a", ColumnRole.EQUALITY, cost_ms=99.0), + _usage(_ORDERS, "b", ColumnRole.EQUALITY, cost_ms=98.0), + _usage(_ORDERS, "c", ColumnRole.EQUALITY, cost_ms=97.0), + _usage(_ORDERS, "d", ColumnRole.EQUALITY, cost_ms=96.0), ], - facts(columns=("a", "b", "c", "d")), + _facts_map(columns=("a", "b", "c", "d")), {}, min_cost_share=0.01, ) @@ -93,8 +107,8 @@ def test_arity_is_capped(): def test_small_tables_are_suppressed_entirely(): proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(rows=500), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(rows=500), {}, min_cost_share=0.01, ) @@ -103,8 +117,8 @@ def test_small_tables_are_suppressed_entirely(): def test_below_min_cost_share_is_suppressed(): proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY, cost_share=0.001)], - facts(), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_share=0.001)], + _facts_map(), {}, min_cost_share=0.01, ) @@ -112,13 +126,13 @@ def test_below_min_cost_share_is_suppressed(): def test_existing_index_with_the_same_leading_prefix_is_not_reproposed(): - existing = {"orders": (PgIndex("idx", ("status", "created_at"), False, False, 10, 8192),)} + existing = {_ORDERS: (PgIndex("idx", ("status", "created_at"), False, False, 10, 8192),)} proposals = propose_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("created_at", ColumnRole.RANGE, cost_ms=80.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_ms=80.0), ], - facts(), + _facts_map(), existing, min_cost_share=0.01, ) @@ -126,10 +140,10 @@ def test_existing_index_with_the_same_leading_prefix_is_not_reproposed(): def test_a_wider_existing_index_still_covers_a_narrower_candidate(): - existing = {"orders": (PgIndex("idx", ("status", "created_at", "id"), False, False, 5, 1),)} + existing = {_ORDERS: (PgIndex("idx", ("status", "created_at", "id"), False, False, 5, 1),)} proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY, cost_ms=90.0)], - facts(), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0)], + _facts_map(), existing, min_cost_share=0.01, ) @@ -143,13 +157,13 @@ def test_a_narrower_existing_index_does_not_cover_a_wider_candidate(): wider-covers-narrower test existed, so `candidate[:len(existing)] == existing` would have shipped silently. """ - existing = {"orders": (PgIndex("idx_status", ("status",), False, False, 5, 1),)} + existing = {_ORDERS: (PgIndex("idx_status", ("status",), False, False, 5, 1),)} proposals = propose_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("created_at", ColumnRole.RANGE, cost_ms=80.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_ms=80.0), ], - facts(), + _facts_map(), existing, min_cost_share=0.01, ) @@ -164,7 +178,7 @@ def test_a_partial_index_does_not_suppress_a_candidate(): confidently-wrong failures, and just as invisible. """ existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_open", ("status",), @@ -178,7 +192,10 @@ def test_a_partial_index_does_not_suppress_a_candidate(): ) } proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], facts(), existing, min_cost_share=0.01 + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(), + existing, + min_cost_share=0.01, ) assert codes(proposals) == ["ADV001"] assert proposals[0].evidence["partial_indexes_skipped"] == ("idx_open",) @@ -187,9 +204,12 @@ def test_a_partial_index_does_not_suppress_a_candidate(): def test_a_plain_index_still_suppresses_a_candidate(): """The control. Task 2's new fields default to False, so this must not have changed.""" - existing = {"orders": (PgIndex("idx_status", ("status",), False, False, 5, 4096),)} + existing = {_ORDERS: (PgIndex("idx_status", ("status",), False, False, 5, 4096),)} proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], facts(), existing, min_cost_share=0.01 + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(), + existing, + min_cost_share=0.01, ) assert proposals == [] @@ -198,7 +218,7 @@ def test_an_expression_index_is_disclosed_not_silently_ignored(): """We cannot prove `lower(status)` makes an index on `status` redundant — or that it doesn't. Saying so beats both suppressing and pretending it isn't there.""" existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_lower_status", (), @@ -212,7 +232,10 @@ def test_an_expression_index_is_disclosed_not_silently_ignored(): ) } proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], facts(), existing, min_cost_share=0.01 + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(), + existing, + min_cost_share=0.01, ) assert codes(proposals) == ["ADV001"] assert proposals[0].evidence["expression_indexes"] == ("idx_lower_status",) @@ -235,7 +258,7 @@ def test_a_mixed_expression_index_sharing_a_prefix_does_not_count_as_coverage(): with `lower(note)` and cannot serve a bare `status` lookup. """ existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_mixed", # Non-empty on purpose: the reconstructed tuple from a mixed index, with the @@ -251,8 +274,8 @@ def test_a_mixed_expression_index_sharing_a_prefix_does_not_count_as_coverage(): ) } proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(ndv={"status": 500.0}), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={"status": 500.0}), existing, min_cost_share=0.01, ) @@ -265,7 +288,7 @@ def test_a_mixed_expression_index_sharing_a_prefix_does_not_count_as_coverage(): def test_an_expression_index_not_mentioning_the_column_is_not_disclosed(): """Only expression indexes that plausibly relate to the candidate are worth naming.""" existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_lower_note", (), @@ -279,7 +302,10 @@ def test_an_expression_index_not_mentioning_the_column_is_not_disclosed(): ) } proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], facts(), existing, min_cost_share=0.01 + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(), + existing, + min_cost_share=0.01, ) assert proposals[0].evidence["expression_indexes"] == () assert "expression index" not in proposals[0].rationale.lower() @@ -292,7 +318,7 @@ def test_a_column_name_inside_a_longer_identifier_is_not_disclosed(): `lower(guid)` — a false claim in the text someone reads before running DDL. """ existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_lower_guid", (), @@ -306,8 +332,8 @@ def test_a_column_name_inside_a_longer_identifier_is_not_disclosed(): ) } proposals = propose_indexes( - [usage("id", ColumnRole.EQUALITY)], - facts(columns=("id", "guid")), + [_usage(_ORDERS, "id", ColumnRole.EQUALITY)], + _facts_map(columns=("id", "guid")), existing, min_cost_share=0.01, ) @@ -317,7 +343,7 @@ def test_a_column_name_inside_a_longer_identifier_is_not_disclosed(): def test_an_expression_index_on_a_cast_of_the_column_is_still_disclosed(): """The control for the fix: word boundaries must not cost a true positive.""" existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_status_cast", (), @@ -331,7 +357,10 @@ def test_an_expression_index_on_a_cast_of_the_column_is_still_disclosed(): ) } proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], facts(), existing, min_cost_share=0.01 + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(), + existing, + min_cost_share=0.01, ) assert proposals[0].evidence["expression_indexes"] == ("idx_status_cast",) @@ -345,13 +374,13 @@ def test_arity_cap_keeps_the_range_column_last_when_it_bites(): """ proposals = propose_indexes( [ - usage("a", ColumnRole.EQUALITY, cost_ms=99.0), - usage("b", ColumnRole.EQUALITY, cost_ms=98.0), - usage("c", ColumnRole.EQUALITY, cost_ms=97.0), - usage("d", ColumnRole.EQUALITY, cost_ms=96.0), - usage("e", ColumnRole.RANGE, cost_ms=50.0), + _usage(_ORDERS, "a", ColumnRole.EQUALITY, cost_ms=99.0), + _usage(_ORDERS, "b", ColumnRole.EQUALITY, cost_ms=98.0), + _usage(_ORDERS, "c", ColumnRole.EQUALITY, cost_ms=97.0), + _usage(_ORDERS, "d", ColumnRole.EQUALITY, cost_ms=96.0), + _usage(_ORDERS, "e", ColumnRole.RANGE, cost_ms=50.0), ], - facts(columns=("a", "b", "c", "d", "e")), + _facts_map(columns=("a", "b", "c", "d", "e")), {}, min_cost_share=0.01, ) @@ -369,10 +398,10 @@ def test_a_column_used_in_two_roles_is_not_proposed_twice(): """ proposals = propose_indexes( [ - usage("id", ColumnRole.EQUALITY, cost_ms=90.0), - usage("id", ColumnRole.SORT, cost_ms=80.0), + _usage(_ORDERS, "id", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "id", ColumnRole.SORT, cost_ms=80.0), ], - facts(ndv={"id": 5000.0}), + _facts_map(ndv={"id": 5000.0}), {}, min_cost_share=0.01, ) @@ -388,13 +417,13 @@ def test_the_existing_primary_key_suppresses_the_deduplicated_candidate(): Once `(id, id)` collapses to `(id,)`, `orders_pkey(id)` covers it and there is no proposal at all — which is the correct answer for this workload. """ - existing = {"orders": (PgIndex("orders_pkey", ("id",), True, True, 900, 4096),)} + existing = {_ORDERS: (PgIndex("orders_pkey", ("id",), True, True, 900, 4096),)} proposals = propose_indexes( [ - usage("id", ColumnRole.EQUALITY, cost_ms=90.0), - usage("id", ColumnRole.SORT, cost_ms=80.0), + _usage(_ORDERS, "id", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "id", ColumnRole.SORT, cost_ms=80.0), ], - facts(ndv={"id": 5000.0}), + _facts_map(ndv={"id": 5000.0}), existing, min_cost_share=0.01, ) @@ -409,11 +438,11 @@ def test_dedupe_prefers_the_equality_occurrence_of_a_two_role_column(): """ proposals = propose_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=99.0), - usage("created_at", ColumnRole.EQUALITY, cost_ms=90.0), - usage("created_at", ColumnRole.RANGE, cost_ms=95.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=99.0), + _usage(_ORDERS, "created_at", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_ms=95.0), ], - facts(ndv={"status": 5000.0}), + _facts_map(ndv={"status": 5000.0}), {}, min_cost_share=0.01, ) @@ -429,8 +458,8 @@ def test_unknown_row_count_is_low_confidence_and_says_why(): ordinarily-trustworthy, so it is LOW and the rationale states the gap. """ proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(rows=None, ndv={"status": 9999.0}), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(rows=None, ndv={"status": 9999.0}), {}, min_cost_share=0.01, ) @@ -447,8 +476,8 @@ def test_a_denied_index_list_caps_confidence_at_low_and_stops_claiming_coverage( exists to prevent, three lines away. """ proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(ndv={"status": 5000.0}), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={"status": 5000.0}), {}, min_cost_share=0.01, have_index_data=False, @@ -463,8 +492,8 @@ def test_a_denied_index_list_caps_confidence_at_low_and_stops_claiming_coverage( def test_a_readable_index_list_still_reaches_high(): """The other half of the branch: the cap must not fire when the evidence is there.""" proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(ndv={"status": 5000.0}), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={"status": 5000.0}), {}, min_cost_share=0.01, have_index_data=True, @@ -482,14 +511,14 @@ def denied(sql, params): return [] aggregation = Aggregation( - usage=(usage("status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=60.0),), + usage=(_usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=60.0),), total_cost_ms=100.0, skipped_unqualifiable=0, - tables=frozenset({"orders"}), + tables=frozenset({_ORDERS}), ) adapter = PostgresWorkloadAdapter(querier=denied) proposals = adapter.propose( - aggregation, facts(ndv={"status": 9999.0}), _workload(), min_cost_share=0.01 + aggregation, _facts_map(ndv={"status": 9999.0}), _workload(), min_cost_share=0.01 ) adv001 = [p for p in proposals if p.code == "ADV001"] assert adv001, "the proposal must survive a missing grant, just at lower confidence" @@ -500,36 +529,109 @@ def denied(sql, params): def test_cost_share_is_the_max_never_the_sum(): proposals = propose_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=90.0), - usage("created_at", ColumnRole.RANGE, cost_share=0.6, cost_ms=80.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=90.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_share=0.6, cost_ms=80.0), ], - facts(), + _facts_map(), {}, min_cost_share=0.01, ) assert proposals[0].evidence["cost_share"] == 0.6 +def test_adv001_does_not_weld_in_a_column_no_query_uses_with_the_others(): + """The measured cross-task regression: a near-free cursor read degrading the hot index. + + `DECLARE ... CURSOR FOR SELECT ... WHERE tenant_id = $1` became analysable, and at 0.003% + of window cost it put `tenant_id` into position 2 of the hot query's + `(customer_id, created_at)`. `cost_share` cannot filter that out — it is the *max* over + the chosen columns — and `(customer_id, tenant_id, created_at)` no longer satisfies the + hot query's `ORDER BY created_at`, so the tool proposed a strictly worse index at HIGH. + """ + proposals = propose_indexes( + [ + _usage(_ORDERS, "customer_id", ColumnRole.EQUALITY, cost_ms=5000.0, fps=("hot",)), + _usage(_ORDERS, "tenant_id", ColumnRole.EQUALITY, cost_ms=0.4, fps=("cursor",)), + _usage(_ORDERS, "created_at", ColumnRole.SORT, cost_ms=5000.0, fps=("hot",)), + ], + _facts_map(columns=("customer_id", "tenant_id", "created_at")), + {}, + min_cost_share=0.01, + ) + assert proposals[0].evidence["columns"] == ("customer_id", "created_at") + + +def test_adv001_requires_joint_support_not_merely_pairwise_with_the_seed(): + """Transitive support is not support: `a` with `b` in one query and with `c` in another + is no query filtering on all three. Same guard, same reason, as ADV008's.""" + proposals = propose_indexes( + [ + _usage(_ORDERS, "a", ColumnRole.EQUALITY, cost_ms=99.0, fps=("fp1", "fp2")), + _usage(_ORDERS, "b", ColumnRole.EQUALITY, cost_ms=98.0, fps=("fp1",)), + _usage(_ORDERS, "c", ColumnRole.EQUALITY, cost_ms=97.0, fps=("fp2",)), + ], + _facts_map(columns=("a", "b", "c")), + {}, + min_cost_share=0.01, + ) + assert proposals[0].evidence["columns"] == ("a", "b") + + +def test_adv001_rejecting_one_column_does_not_block_a_later_co_occurring_one(): + """A rejected candidate must not narrow the running intersection, or the rule would + silently drop a column that a query really does use alongside the seed.""" + proposals = propose_indexes( + [ + _usage(_ORDERS, "a", ColumnRole.EQUALITY, cost_ms=99.0, fps=("fp1",)), + _usage(_ORDERS, "b", ColumnRole.EQUALITY, cost_ms=98.0, fps=("fp2",)), + _usage(_ORDERS, "c", ColumnRole.EQUALITY, cost_ms=97.0, fps=("fp1",)), + ], + _facts_map(columns=("a", "b", "c")), + {}, + min_cost_share=0.01, + ) + assert proposals[0].evidence["columns"] == ("a", "c") + + +def test_adv001_reports_the_honest_joint_support_count_and_omits_the_plain_one(): + """`fingerprints` was `max(per-column)`, so a three-column composite that no single + query group supports reported `fingerprints: 1` — a number that reads as corroboration. + Only the joint count is reported now, under ADV004's and ADV008's existing key.""" + proposals = propose_indexes( + [ + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0, fps=("fp1", "fp2")), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_ms=80.0, fps=("fp1",)), + ], + _facts_map(), + {}, + min_cost_share=0.01, + ) + evidence = proposals[0].evidence + assert evidence["columns"] == ("status", "created_at") + assert evidence["co_occurring_fingerprints"] == 1 + assert "fingerprints" not in evidence + + def test_confidence_is_high_only_with_stats_and_a_selective_leading_column(): high = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(ndv={"status": 5000.0}), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={"status": 5000.0}), {}, min_cost_share=0.01, ) assert high[0].confidence is Confidence.HIGH no_stats = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(ndv={}), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={}), {}, min_cost_share=0.01, ) assert no_stats[0].confidence is Confidence.MEDIUM unselective = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(ndv={"status": 3.0}), + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={"status": 3.0}), {}, min_cost_share=0.01, ) @@ -538,32 +640,88 @@ def test_confidence_is_high_only_with_stats_and_a_selective_leading_column(): def test_unused_index_proposed_for_drop_but_never_a_constraint_index(): existing = { - "orders": ( + _ORDERS: ( PgIndex("idx_cold", ("note",), False, False, 0, 4096), PgIndex("orders_pkey", ("id",), True, True, 0, 4096), PgIndex("uq_email", ("email",), True, False, 0, 4096), PgIndex("idx_warm", ("status",), False, False, 42, 4096), ) } - proposals = propose_unused_indexes(existing, hot_tables=frozenset({"orders"})) + proposals = propose_unused_indexes(existing, hot_tables=frozenset({_ORDERS})) assert codes(proposals) == ["ADV002"] assert proposals[0].evidence["index"] == "idx_cold" assert proposals[0].confidence is Confidence.MEDIUM def test_unused_index_rule_ignores_tables_outside_the_workload(): - existing = {"archive": (PgIndex("idx_cold", ("a",), False, False, 0, 1),)} - assert propose_unused_indexes(existing, hot_tables=frozenset({"orders"})) == [] + existing = {Relation("public", "archive"): (PgIndex("idx_cold", ("a",), False, False, 0, 1),)} + assert propose_unused_indexes(existing, hot_tables=frozenset({_ORDERS})) == [] + + +def test_adv002_drop_ddl_qualifies_the_index_with_its_relations_schema(): + existing = { + Relation("staging", "orders"): ( + PgIndex( + name="idx_cold", + columns=("note",), + is_unique=False, + is_primary=False, + scans=0, + size_bytes=1, + ), + ) + } + proposals = propose_unused_indexes( + existing, hot_tables=frozenset({Relation("staging", "orders")}) + ) + assert proposals[0].ddl == 'DROP INDEX "staging"."idx_cold";' + + +def test_unused_index_ddl_uses_its_own_relations_schema_not_the_others(): + """`orders_pkey`/any index name can exist identically in two schemas at once (proven + live: `orders_pkey` exists under both `public` and `staging`). Grouping by relation must + not let one schema's index bleed into the other's DROP statement.""" + existing = { + Relation("sales", "orders"): ( + PgIndex( + name="idx_cold", + columns=("note",), + is_unique=False, + is_primary=False, + scans=0, + size_bytes=1, + ), + ), + Relation("staging", "orders"): ( + PgIndex( + name="idx_cold", + columns=("note",), + is_unique=False, + is_primary=False, + scans=0, + size_bytes=1, + ), + ), + } + proposals = propose_unused_indexes( + existing, + hot_tables=frozenset({Relation("sales", "orders"), Relation("staging", "orders")}), + ) + ddls = {p.ddl for p in proposals} + assert ddls == { + 'DROP INDEX "sales"."idx_cold";', + 'DROP INDEX "staging"."idx_cold";', + } def test_a_plain_redundant_pair_is_high_confidence(): existing = { - "orders": ( + _ORDERS: ( PgIndex("idx_narrow", ("status",), False, False, 5, 1), PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), ) } - proposals = propose_redundant_indexes(existing) + proposals = propose_redundant_indexes(existing, hot_tables=frozenset(existing)) assert codes(proposals) == ["ADV003"] assert proposals[0].confidence is Confidence.HIGH assert proposals[0].evidence["index"] == "idx_narrow" @@ -574,11 +732,34 @@ def test_a_plain_redundant_pair_is_high_confidence(): assert "partial" not in proposals[0].rationale +def test_adv003_ignores_relations_the_workload_never_touched(): + """Scope, for a rule whose output is `DROP INDEX`, must not be an accident. + + `fetch_indexes` filters tables by *bare* name, so `schemas=("sales", "staging")` with + only `sales.orders` in the workload still returns `staging.orders`'s indexes. Iterating + every key of `existing` therefore emitted `DROP INDEX "staging"."idx_narrow"` for a + relation with zero recorded column usage — and whether it did depended on whether a + bare name happened to collide across two requested schemas. ADV002 was already scoped to + `hot_tables`; this asserts both members, so scoping that silently dropped the hot + relation as well would fail here rather than look like a pass. + """ + sales, staging = Relation("sales", "orders"), Relation("staging", "orders") + redundant_pair = ( + PgIndex("idx_narrow", ("status",), False, False, 5, 1), + PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), + ) + proposals = propose_redundant_indexes( + {sales: redundant_pair, staging: redundant_pair}, + hot_tables=frozenset({sales}), + ) + assert {p.ddl for p in proposals} == {'DROP INDEX "sales"."idx_narrow";'} + + def test_a_partial_narrow_index_is_never_called_redundant(): """The partial index exists to serve a subset; the wider full index serves it differently. Dropping it is not less certain, it is probably wrong.""" existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_open", ("status",), @@ -592,12 +773,12 @@ def test_a_partial_narrow_index_is_never_called_redundant(): PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), ) } - assert propose_redundant_indexes(existing) == [] + assert propose_redundant_indexes(existing, hot_tables=frozenset(existing)) == [] def test_a_partial_wider_index_does_not_supersede_a_plain_one(): existing = { - "orders": ( + _ORDERS: ( PgIndex("idx_narrow", ("status",), False, False, 5, 1), PgIndex( "idx_wide_open", @@ -611,7 +792,7 @@ def test_a_partial_wider_index_does_not_supersede_a_plain_one(): ), ) } - assert propose_redundant_indexes(existing) == [] + assert propose_redundant_indexes(existing, hot_tables=frozenset(existing)) == [] def test_a_wider_expression_index_does_not_supersede_a_plain_one(): @@ -622,7 +803,7 @@ def test_a_wider_expression_index_does_not_supersede_a_plain_one(): not the has_expressions filter existed at all. """ existing = { - "orders": ( + _ORDERS: ( PgIndex("idx_narrow", ("status",), False, False, 5, 1), PgIndex( "idx_expr", @@ -636,7 +817,7 @@ def test_a_wider_expression_index_does_not_supersede_a_plain_one(): ), ) } - assert propose_redundant_indexes(existing) == [] + assert propose_redundant_indexes(existing, hot_tables=frozenset(existing)) == [] def test_a_narrow_expression_index_is_never_called_redundant(): @@ -647,7 +828,7 @@ def test_a_narrow_expression_index_is_never_called_redundant(): column-list comparison would discard an index nothing else provides. """ existing = { - "orders": ( + _ORDERS: ( PgIndex( "idx_narrow_expr", ("status",), @@ -661,17 +842,38 @@ def test_a_narrow_expression_index_is_never_called_redundant(): PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), ) } - assert propose_redundant_indexes(existing) == [] + assert propose_redundant_indexes(existing, hot_tables=frozenset(existing)) == [] def test_a_unique_prefix_index_is_never_called_redundant(): existing = { - "orders": ( + _ORDERS: ( PgIndex("uq_status", ("status",), True, False, 5, 1), PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), ) } - assert propose_redundant_indexes(existing) == [] + assert propose_redundant_indexes(existing, hot_tables=frozenset(existing)) == [] + + +def test_redundant_index_ddl_uses_its_own_relations_schema_not_the_others(): + """Same collision as ADV002's, for ADV003: `idx_narrow`/`idx_wide` named identically in + two schemas must each produce a DROP scoped to their own schema.""" + existing = { + Relation("sales", "orders"): ( + PgIndex("idx_narrow", ("status",), False, False, 5, 1), + PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), + ), + Relation("staging", "orders"): ( + PgIndex("idx_narrow", ("status",), False, False, 5, 1), + PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), + ), + } + proposals = propose_redundant_indexes(existing, hot_tables=frozenset(existing)) + ddls = {p.ddl for p in proposals} + assert ddls == { + 'DROP INDEX "sales"."idx_narrow";', + 'DROP INDEX "staging"."idx_narrow";', + } def _workload(*stats): @@ -681,10 +883,10 @@ def _workload(*stats): def test_partial_index_proposed_for_a_hot_not_null_check(): proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), ], - facts(), + _facts_map(), min_cost_share=0.01, ) assert codes(proposals) == ["ADV004"] @@ -694,10 +896,10 @@ def test_partial_index_proposed_for_a_hot_not_null_check(): def test_partial_index_polarity_follows_the_predicate(): proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("shipped_at", ColumnRole.NULL_CHECK, cost_share=0.4, cost_ms=40.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "shipped_at", ColumnRole.NULL_CHECK, cost_share=0.4, cost_ms=40.0), ], - facts(), + _facts_map(), min_cost_share=0.01, ) assert "IS NULL" in proposals[0].ddl @@ -713,8 +915,9 @@ def test_no_partial_index_when_the_columns_never_co_occur(): """ proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0, fps=("fp_a",)), - usage( + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0, fps=("fp_a",)), + _usage( + _ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, @@ -722,7 +925,7 @@ def test_no_partial_index_when_the_columns_never_co_occur(): fps=("fp_b",), ), ], - facts(), + _facts_map(), min_cost_share=0.01, ) assert proposals == [] @@ -731,8 +934,9 @@ def test_no_partial_index_when_the_columns_never_co_occur(): def test_partial_index_reports_the_co_occurrence_that_justifies_it(): proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0, fps=("fp_a", "fp_b")), - usage( + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0, fps=("fp_a", "fp_b")), + _usage( + _ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, @@ -740,7 +944,7 @@ def test_partial_index_reports_the_co_occurrence_that_justifies_it(): fps=("fp_b", "fp_c"), ), ], - facts(), + _facts_map(), min_cost_share=0.01, ) assert codes(proposals) == ["ADV004"] @@ -751,9 +955,10 @@ def test_partial_index_picks_the_costliest_pair_that_actually_co_occurs(): """A cheaper pair that co-occurs beats a hotter pair that does not.""" proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=99.0, fps=("fp_lonely",)), - usage("region", ColumnRole.EQUALITY, cost_ms=50.0, fps=("fp_shared",)), - usage( + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=99.0, fps=("fp_lonely",)), + _usage(_ORDERS, "region", ColumnRole.EQUALITY, cost_ms=50.0, fps=("fp_shared",)), + _usage( + _ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, @@ -761,7 +966,7 @@ def test_partial_index_picks_the_costliest_pair_that_actually_co_occurs(): fps=("fp_shared",), ), ], - facts(columns=("status", "region", "shipped_at")), + _facts_map(columns=("status", "region", "shipped_at")), min_cost_share=0.01, ) assert proposals[0].evidence["columns"] == ("region",) @@ -776,10 +981,10 @@ def test_partial_index_is_suppressed_on_a_small_table(): """ proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), ], - facts(rows=50), + _facts_map(rows=50), min_cost_share=0.01, ) assert proposals == [] @@ -789,10 +994,10 @@ def test_partial_index_with_an_unknown_row_count_is_low_and_says_why(): """Same treatment ADV001 gives an unknown row count: keep the advice, lower the claim.""" proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(_ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), ], - facts(rows=None), + _facts_map(rows=None), min_cost_share=0.01, ) assert codes(proposals) == ["ADV004"] @@ -803,8 +1008,8 @@ def test_partial_index_with_an_unknown_row_count_is_low_and_says_why(): def test_no_partial_index_without_an_equality_column_to_index(): proposals = propose_partial_indexes( - [usage("shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4)], - facts(), + [_usage(_ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4)], + _facts_map(), min_cost_share=0.01, ) assert proposals == [] @@ -812,12 +1017,14 @@ def test_no_partial_index_without_an_equality_column_to_index(): def test_non_sargable_column_gets_an_attributed_proposal(): proposals = propose_sargability( - [usage("status", ColumnRole.NON_SARGABLE, cost_share=0.3)], + [_usage(_ORDERS, "status", ColumnRole.NON_SARGABLE, cost_share=0.3)], _workload(), min_cost_share=0.01, ) assert codes(proposals) == ["ADV005"] assert proposals[0].evidence["column"] == "status" + assert proposals[0].evidence["schema"] == "public" + assert proposals[0].evidence["table"] == "orders" assert proposals[0].confidence is Confidence.HIGH @@ -845,8 +1052,8 @@ def test_hot_select_star_on_a_wide_table(): flags=frozenset({FLAG_SELECT_STAR}), ) wide = { - "orders": TableFacts( - name="orders", + _ORDERS: TableFacts( + relation=_ORDERS, row_estimate=10**6, size_bytes=10**8, columns=tuple(f"c{i}" for i in range(30)), @@ -865,7 +1072,7 @@ def test_select_star_ignored_on_a_narrow_table(): flags=frozenset({FLAG_SELECT_STAR}), ) narrow = { - "orders": TableFacts(name="orders", row_estimate=10**6, size_bytes=1, columns=("a", "b")) + _ORDERS: TableFacts(relation=_ORDERS, row_estimate=10**6, size_bytes=1, columns=("a", "b")) } assert propose_select_star(_workload(stat), narrow, min_cost_share=0.01) == [] @@ -885,26 +1092,532 @@ def test_select_star_table_matching_ignores_substring_false_positives(): total_time_ms=100.0, flags=frozenset({FLAG_SELECT_STAR}), ) + order = Relation("public", "order") + orders = Relation("public", "orders") + cart = Relation("public", "cart") wide = { - "order": TableFacts( - name="order", + order: TableFacts( + relation=order, row_estimate=10**6, size_bytes=1, columns=tuple(f"c{i}" for i in range(30)), ), - "orders": TableFacts( - name="orders", + orders: TableFacts( + relation=orders, row_estimate=10**6, size_bytes=1, columns=tuple(f"c{i}" for i in range(30)), ), - "cart": TableFacts( - name="cart", row_estimate=10**6, size_bytes=1, columns=tuple(f"c{i}" for i in range(30)) + cart: TableFacts( + relation=cart, + row_estimate=10**6, + size_bytes=1, + columns=tuple(f"c{i}" for i in range(30)), ), } assert propose_select_star(_workload(stat), wide, min_cost_share=0.01) == [] +def test_select_star_evidence_reports_the_qualified_relation(): + """`touched` is matched against the bare name in the SQL text, but the evidence must + still surface the schema-qualified name — the bare match is an implementation detail of + text-matching, not what should be shown to an operator.""" + stat = QueryStat( + fingerprint="fp", + sql="select * from orders", + calls=5, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), + ) + staging_orders = Relation("staging", "orders") + wide = { + staging_orders: TableFacts( + relation=staging_orders, + row_estimate=10**6, + size_bytes=10**8, + columns=tuple(f"c{i}" for i in range(30)), + ) + } + proposals = propose_select_star(_workload(stat), wide, min_cost_share=0.01) + assert proposals[0].evidence["tables"] == ("staging.orders",) + assert "staging.orders" in proposals[0].title + + +def test_select_star_does_not_attribute_a_schema_qualified_statement_to_the_wrong_schema(): + """`select * from public.orders` must report only `public.orders`, even when + `staging.orders` is also wide and shares the bare name `orders`. + + Bare-name text matching cannot see the schema qualifier the statement itself carries — + the same defect class Task 2 already fixed once on this branch for `star_tables`. Before + the fix, a bare-name text match of `orders` against `select * from public.orders` hit *both* + wide relations, so the evidence named a table (`staging.orders`) the statement never + referenced at all. + """ + stat = QueryStat( + fingerprint="fp", + sql="select * from public.orders", + calls=5, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), + ) + public_orders = Relation("public", "orders") + staging_orders = Relation("staging", "orders") + wide = { + public_orders: TableFacts( + relation=public_orders, + row_estimate=10**6, + size_bytes=10**8, + columns=tuple(f"c{i}" for i in range(30)), + ), + staging_orders: TableFacts( + relation=staging_orders, + row_estimate=10**6, + size_bytes=10**8, + columns=tuple(f"c{i}" for i in range(30)), + ), + } + proposals = propose_select_star(_workload(stat), wide, min_cost_share=0.01) + assert proposals[0].evidence["tables"] == ("public.orders",) + + +def test_select_star_attributes_an_ambiguous_bare_reference_to_neither_wide_relation(): + """A *bare* `select * from orders` with both `public.orders` and `staging.orders` wide + cannot be attributed to either — the statement itself does not say which. Reporting + either would be a guess; reporting both repeats the original defect. Dropping it entirely + is the same cannot-prove-it-so-drop-it policy `resolve_relation`/`star_tables` follow.""" + stat = QueryStat( + fingerprint="fp", + sql="select * from orders", + calls=5, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), + ) + public_orders = Relation("public", "orders") + staging_orders = Relation("staging", "orders") + wide = { + public_orders: TableFacts( + relation=public_orders, + row_estimate=10**6, + size_bytes=10**8, + columns=tuple(f"c{i}" for i in range(30)), + ), + staging_orders: TableFacts( + relation=staging_orders, + row_estimate=10**6, + size_bytes=10**8, + columns=tuple(f"c{i}" for i in range(30)), + ), + } + assert propose_select_star(_workload(stat), wide, min_cost_share=0.01) == [] + + +def test_identical_ddl_at_equal_confidence_resolves_by_code_preference(): + """ADV001 and ADV008 can emit byte-identical DDL at the same confidence.""" + ddl = 'CREATE INDEX ON "public"."events" ("tenant_id");' + adv008 = Proposal( + code="ADV008", + title="group", + rationale="g", + evidence={"cost_share": 0.5}, + confidence=Confidence.MEDIUM, + ddl=ddl, + ) + adv001 = Proposal( + code="ADV001", + title="filter", + rationale="f", + evidence={"cost_share": 0.5}, + confidence=Confidence.MEDIUM, + ddl=ddl, + ) + # Both orderings must pick the same winner, or list order is deciding. + assert [p.code for p in PostgresWorkloadAdapter._dedupe_by_ddl([adv008, adv001])] == ["ADV001"] + assert [p.code for p in PostgresWorkloadAdapter._dedupe_by_ddl([adv001, adv008])] == ["ADV001"] + + +def test_confidence_still_beats_code_preference(): + ddl = 'CREATE INDEX ON "public"."events" ("tenant_id");' + adv001_low = Proposal( + code="ADV001", + title="filter", + rationale="f", + evidence={"cost_share": 0.5}, + confidence=Confidence.LOW, + ddl=ddl, + ) + adv008_med = Proposal( + code="ADV008", + title="group", + rationale="g", + evidence={"cost_share": 0.5}, + confidence=Confidence.MEDIUM, + ddl=ddl, + ) + assert [p.code for p in PostgresWorkloadAdapter._dedupe_by_ddl([adv001_low, adv008_med])] == [ + "ADV008" + ] + + +def test_every_ddl_emitting_code_has_a_preference_rank(): + """A code missing from the map would raise KeyError mid-run, after all the analysis.""" + ddl_codes = {"ADV001", "ADV002", "ADV003", "ADV004", "ADV007", "ADV008"} + assert ddl_codes <= set(PostgresWorkloadAdapter._CODE_PREFERENCE) + + +def test_dedupe_folds_the_discarded_proposals_rationale_into_the_survivor(): + """Collapsing on identical DDL must not silently drop the loser's rationale. + + ADV008 does not read NDV at all, so it has no selectivity caveat to lose — but ADV007 + does, and simply keeping "the more confident" text on identical DDL would drop it. + """ + aggregation = Aggregation( + usage=( + _usage(_ORDERS, "tenant_id", ColumnRole.JOIN, cost_share=0.6, cost_ms=60.0), + _usage(_ORDERS, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=50.0), + ), + total_cost_ms=100.0, + skipped_unqualifiable=0, + tables=frozenset({_ORDERS}), + ) + adapter = PostgresWorkloadAdapter(querier=lambda sql, params: []) + proposals = adapter.propose( + aggregation, _facts_map(ndv={"tenant_id": 3.0}), _workload(), min_cost_share=0.01 + ) + ddl = 'CREATE INDEX ON "public"."orders" ("tenant_id");' + creates = [p for p in proposals if p.ddl == ddl] + assert len(creates) == 1 + survivor = creates[0] + # ADV008 has no NDV to read, so it wins on confidence (MEDIUM beats LOW) — but ADV007's + # rationale, including the caveat ADV008 could never have stated, must survive with it. + assert survivor.code == "ADV008" + assert survivor.confidence is Confidence.MEDIUM + assert "ADV007" in survivor.rationale + assert "distinct values" in survivor.rationale + + +def test_adv001_and_adv007_prefix_collision_collapses_instead_of_shipping_a_pair_adv003_would_flag(): + """ADV001 proposing (customer_id, created_at) and ADV007 proposing (customer_id), both + HIGH, in the same report would advise creating a pair where the second is a strict + prefix of the first — exactly what ADV003 flags as redundant on the next run. The + narrower proposal must collapse into the wider one, not ship alongside it. + """ + aggregation = Aggregation( + usage=( + _usage(_ORDERS, "customer_id", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=60.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_share=0.5, cost_ms=50.0), + _usage(_ORDERS, "customer_id", ColumnRole.JOIN, cost_share=0.55, cost_ms=55.0), + ), + total_cost_ms=100.0, + skipped_unqualifiable=0, + tables=frozenset({_ORDERS}), + ) + adapter = PostgresWorkloadAdapter(querier=lambda sql, params: []) + proposals = adapter.propose( + aggregation, _facts_map(ndv={"customer_id": 9999.0}), _workload(), min_cost_share=0.01 + ) + creates = [p for p in proposals if p.ddl and p.ddl.startswith("CREATE INDEX")] + assert codes(creates) == ["ADV001"] + assert creates[0].evidence["columns"] == ("customer_id", "created_at") + assert creates[0].confidence is Confidence.HIGH + assert "ADV007" in creates[0].rationale + + +def _plain_index_proposal(code, columns, confidence=Confidence.HIGH, relation=_ORDERS): + """A minimal `CREATE INDEX` proposal eligible for `_collapse_index_prefixes` / + `_disclose_column_set_overlaps` — built by hand so prefix-collision tests can control + exactly which proposals collide, instead of tuning real rule inputs to get there.""" + quoted = ", ".join(f'"{c}"' for c in columns) + return Proposal( + code=code, + title=f"{code} on {relation}({', '.join(columns)})", + rationale=f"{code} rationale for {columns}.", + evidence={"schema": relation.schema, "table": relation.table, "columns": columns}, + confidence=confidence, + ddl=f'CREATE INDEX ON "{relation.schema}"."{relation.table}" ({quoted});', + ) + + +def test_prefix_collision_collapses_regardless_of_which_rule_appended_first(): + """Determinism: `propose()` hardcodes a call order today, but the collapse must not + depend on it. Two proposals must be absorbed into the same wider one — with only one + absorbed proposal, `sorted(absorbed, ...)` inside `_collapse_index_prefixes` is a + no-op and this test cannot tell a real sort from none at all. + """ + wide = _plain_index_proposal("ADV001", ("customer_id", "created_at", "region")) + narrow_one = _plain_index_proposal("ADV007", ("customer_id",)) + narrow_two = _plain_index_proposal("ADV008", ("customer_id", "created_at"), Confidence.MEDIUM) + + forward = PostgresWorkloadAdapter._collapse_index_prefixes([wide, narrow_one, narrow_two]) + backward = PostgresWorkloadAdapter._collapse_index_prefixes([narrow_two, narrow_one, wide]) + shuffled = PostgresWorkloadAdapter._collapse_index_prefixes([narrow_one, wide, narrow_two]) + + assert codes(forward) == codes(backward) == codes(shuffled) == ["ADV001"] + assert forward[0].rationale == backward[0].rationale == shuffled[0].rationale + # Both absorbed proposals' rationale must actually be present — this is what makes the + # sort's job non-trivial: two notes, and their concatenation order must not vary. + assert "ADV007" in forward[0].rationale + assert "ADV008" in forward[0].rationale + + +#: Repository root, for the documentation claims the collapse layer is pinned against. +_ROOT = Path(__file__).resolve().parents[1] + + +def test_the_proposal_collapse_layer_is_documented_for_users(): + """The collapse layer changes what `proposals` contains, so it cannot be internal-only. + + It shipped documented nowhere user-facing, including the two facts an operator or a + `--json` consumer actually has to know: a rule can fire and contribute no proposal at + all, and the absorbed proposal's `evidence` is discarded while its rationale is kept. + Each claim is asserted separately, so documenting one and omitting another cannot pass. + """ + readme = (_ROOT / "README.md").read_text(encoding="utf-8") + changelog = (_ROOT / "CHANGELOG.md").read_text(encoding="utf-8") + assert "How overlapping proposals are reconciled" in readme + assert "Identical DDL collapses to one proposal" in readme + assert "A narrower index collapses into a wider one" in readme + assert "Same columns in a different order are both kept" in readme + assert "A rule can fire and still contribute no proposal" in readme + assert "discarded, not merged" in readme, "the evidence loss is not disclosed" + assert "Overlapping proposals are reconciled" in changelog + + +def test_a_prefix_collapse_does_not_claim_the_absorbed_rule_endorsed_this_index(): + """The absorbed proposal reached a *narrower* index, and the text must say which one. + + "ADV007 reached the same index at high confidence" under `(customer_id, tenant_id, + created_at)` told the operator ADV007 endorsed a three-column index when ADV007 proposed + `(customer_id)` — and the borrowed sentences that follow ("Equality columns come first so + the range column can be scanned last", from an ADV001 absorbed into an ADV008 survivor + with no equality columns at all) then have no subject they are true of. + """ + wide = _plain_index_proposal("ADV001", ("customer_id", "tenant_id", "created_at")) + narrow = _plain_index_proposal("ADV007", ("customer_id",)) + + survivor = PostgresWorkloadAdapter._collapse_index_prefixes([wide, narrow])[0] + + assert "reached the same index" not in survivor.rationale + assert "ADV007 proposed the narrower (customer_id) on the same table" in survivor.rationale + # The absorbed rationale itself is still carried over, attributed — the collapse must + # change the attribution, not start discarding text. + assert "ADV007 rationale" in survivor.rationale + + +def test_identical_ddl_dedupe_still_says_the_two_rules_reached_the_same_index(): + """The other half of the split: for byte-identical DDL "the same index" is true by + construction, and must not be reworded into the prefix form, which would tell the + operator a narrower index was proposed when none was.""" + survivor = _plain_index_proposal("ADV001", ("tenant_id",)) + loser = _plain_index_proposal("ADV007", ("tenant_id",), Confidence.MEDIUM) + + merged = PostgresWorkloadAdapter._dedupe_by_ddl([survivor, loser]) + + assert len(merged) == 1 + assert "ADV007 reached the same index at medium confidence" in merged[0].rationale + assert "narrower" not in merged[0].rationale + + +def test_adv001_and_adv008_same_column_set_different_order_are_disclosed_not_collapsed(): + """(status, region) and (region, status) are not redundant — different leading columns + serve different probes — so neither `_collapse_index_prefixes` nor a future ADV003 pass + can reconcile them. Both must survive, and each must name the other. + """ + aggregation = Aggregation( + usage=( + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=90.0), + _usage(_ORDERS, "region", ColumnRole.EQUALITY, cost_share=0.55, cost_ms=50.0), + _usage(_ORDERS, "region", ColumnRole.GROUP, cost_share=0.5, cost_ms=90.0), + _usage(_ORDERS, "status", ColumnRole.GROUP, cost_share=0.45, cost_ms=50.0), + ), + total_cost_ms=100.0, + skipped_unqualifiable=0, + tables=frozenset({_ORDERS}), + ) + adapter = PostgresWorkloadAdapter(querier=lambda sql, params: []) + proposals = adapter.propose(aggregation, _facts_map(), _workload(), min_cost_share=0.01) + creates = {p.code: p for p in proposals if p.ddl and p.ddl.startswith("CREATE INDEX")} + assert set(creates) == {"ADV001", "ADV008"} + assert creates["ADV001"].evidence["columns"] == ("status", "region") + assert creates["ADV008"].evidence["columns"] == ("region", "status") + assert "ADV008" in creates["ADV001"].rationale + assert "ADV001" in creates["ADV008"].rationale + + +def test_a_partial_index_is_never_collapsed_against_a_plain_prefix(): + """ADV004's WHERE predicate makes it a different object than a plain index over the + same leading column, even when its single column is a textual prefix of a plain + composite's — the same reasoning `_covered` applies to catalog indexes must hold here + for proposals that do not exist as catalog rows yet. + """ + aggregation = Aggregation( + usage=( + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=90.0), + _usage(_ORDERS, "created_at", ColumnRole.RANGE, cost_share=0.5, cost_ms=50.0), + _usage(_ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), + ), + total_cost_ms=100.0, + skipped_unqualifiable=0, + tables=frozenset({_ORDERS}), + ) + adapter = PostgresWorkloadAdapter(querier=lambda sql, params: []) + proposals = adapter.propose(aggregation, _facts_map(), _workload(), min_cost_share=0.01) + assert "ADV004" in codes(proposals) + adv004 = next(p for p in proposals if p.code == "ADV004") + assert "ADV001" not in adv004.rationale + adv001 = next(p for p in proposals if p.code == "ADV001") + assert "ADV004" not in adv001.rationale + + +def test_disclose_overlaps_orders_notes_deterministically_with_three_same_set_proposals(): + """With only two proposals sharing a column set the note each gets is symmetric and + order cannot matter — the bug needs at least three, so a given proposal names more + than one partner and the join order of those names has something to get wrong. + """ + p1 = _plain_index_proposal("ADV001", ("a", "b", "c")) + p2 = _plain_index_proposal("ADV007", ("b", "a", "c")) + p3 = _plain_index_proposal("ADV008", ("c", "b", "a")) + + def rationale_by_code(order): + result = PostgresWorkloadAdapter._disclose_column_set_overlaps(list(order)) + return {p.code: p.rationale for p in result} + + forward = rationale_by_code([p1, p2, p3]) + backward = rationale_by_code([p3, p2, p1]) + shuffled = rationale_by_code([p2, p3, p1]) + + assert forward == backward == shuffled + # Sanity: each did get both partners named, not merely "no notes at all" everywhere. + assert "ADV007" in forward["ADV001"] and "ADV008" in forward["ADV001"] + + +def test_collapse_index_prefixes_never_crosses_relations(): + """A prefix relationship across two different tables is meaningless, even when the + columns are identical strings.""" + shipments = Relation("public", "shipments") + wide = _plain_index_proposal("ADV001", ("a", "b")) + narrow = _plain_index_proposal("ADV007", ("a",), relation=shipments) + + result = PostgresWorkloadAdapter._collapse_index_prefixes([wide, narrow]) + + assert codes(result) == ["ADV001", "ADV007"] + assert "ADV007" not in result[0].rationale + assert "ADV001" not in result[1].rationale + + +def test_disclose_overlaps_never_crosses_relations(): + """Same column set, different order, but on two different tables: not an overlap.""" + shipments = Relation("public", "shipments") + p1 = _plain_index_proposal("ADV001", ("a", "b")) + p2 = _plain_index_proposal("ADV008", ("b", "a"), relation=shipments) + + result = PostgresWorkloadAdapter._disclose_column_set_overlaps([p1, p2]) + + assert result[0].rationale == p1.rationale + assert result[1].rationale == p2.rationale + + +def test_index_creation_columns_rejects_a_missing_or_empty_columns_tuple(): + """A proposal whose evidence carries no columns, or an empty tuple of them, must never + be treated as eligible for prefix collapsing or overlap disclosure — there is nothing + to compare.""" + missing = Proposal( + code="ADV001", + title="t", + rationale="r", + evidence={"schema": "public", "table": "orders"}, + confidence=Confidence.HIGH, + ddl='CREATE INDEX ON "public"."orders" ("a");', + ) + empty = Proposal( + code="ADV001", + title="t", + rationale="r", + evidence={"schema": "public", "table": "orders", "columns": ()}, + confidence=Confidence.HIGH, + ddl='CREATE INDEX ON "public"."orders" ();', + ) + assert PostgresWorkloadAdapter._index_creation_columns(missing) is None + assert PostgresWorkloadAdapter._index_creation_columns(empty) is None + + +def test_adv001_states_the_same_low_ndv_caveat_as_adv007(): + """Task 6 gave ADV007 a caveat for a low leading NDV; ADV001 emitted LOW for the exact + same reason with no explanation at all. Batch 2's other rules make that asymmetry + visible in one report, so the wording must now match — confidence is unaffected, only + the explanation changes. + """ + low = propose_indexes( + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={"status": 3.0}), + {}, + min_cost_share=0.01, + ) + assert low[0].confidence is Confidence.LOW + assert "Only about 3 distinct values" in low[0].rationale + assert "selective enough to be worth its write cost" in low[0].rationale + + # HIGH and MEDIUM must not gain the sentence — it is specifically the low-NDV caveat. + high = propose_indexes( + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={"status": 5000.0}), + {}, + min_cost_share=0.01, + ) + assert high[0].confidence is Confidence.HIGH + assert "distinct values" not in high[0].rationale + + no_stats = propose_indexes( + [_usage(_ORDERS, "status", ColumnRole.EQUALITY)], + _facts_map(ndv={}), + {}, + min_cost_share=0.01, + ) + assert no_stats[0].confidence is Confidence.MEDIUM + assert "distinct values" not in no_stats[0].rationale + + +def test_fold_discarded_deduplicates_repeated_sentences_across_three_proposals(): + """ADV001, ADV007 and ADV008 share verbatim wording for the partial/expression-index + disclosures, so a real three-way collision would otherwise repeat the same sentence up + to three times. A sentence must be dropped only when it exactly repeats one already + present; a sentence unique to one proposal must always survive. + """ + shared = "This sentence is shared by all three." + survivor = Proposal( + code="ADV001", + title="wide", + rationale=f"{shared} Only ADV001 says this.", + evidence={"cost_share": 0.5}, + confidence=Confidence.HIGH, + ddl="DDL", + ) + loser_one = Proposal( + code="ADV007", + title="n1", + rationale=f"{shared} Only ADV007 says this.", + evidence={"cost_share": 0.5}, + confidence=Confidence.HIGH, + ddl="DDL", + ) + loser_two = Proposal( + code="ADV008", + title="n2", + rationale=f"{shared} Only ADV008 says this.", + evidence={"cost_share": 0.5}, + confidence=Confidence.MEDIUM, + ddl="DDL", + ) + + merged = PostgresWorkloadAdapter._fold_discarded( + survivor, [loser_one, loser_two], same_index=True + ) + + assert merged.rationale.count(shared) == 1 + assert "Only ADV001 says this." in merged.rationale + assert "Only ADV007 says this." in merged.rationale + assert "Only ADV008 says this." in merged.rationale + + def test_propose_collapses_an_index_flagged_both_unused_and_redundant(): """ADV002 and ADV003 can both fire on one index, emitting the same DROP twice. @@ -912,14 +1625,14 @@ def test_propose_collapses_an_index_flagged_both_unused_and_redundant(): ADV002 rests on a scan counter covering only the window since the last stats reset. """ existing = { - "orders": ( + _ORDERS: ( PgIndex("idx_narrow", ("status",), False, False, 0, 4096), PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 8192), ) } adapter = PostgresWorkloadAdapter(querier=lambda sql, params: []) aggregation = Aggregation( - usage=(), total_cost_ms=100.0, skipped_unqualifiable=0, tables=frozenset({"orders"}) + usage=(), total_cost_ms=100.0, skipped_unqualifiable=0, tables=frozenset({_ORDERS}) ) adapter.fetch_indexes = lambda schemas, tables: existing # type: ignore[method-assign] proposals = adapter.propose(aggregation, {}, _workload(), min_cost_share=0.01) @@ -930,23 +1643,76 @@ def test_propose_collapses_an_index_flagged_both_unused_and_redundant(): def test_propose_composes_all_rules_and_ranks_high_confidence_first(): + """Pins that every one of the eight rules `propose()` wires in actually fired. + + Deliberately an equality set, not a superset assertion (`>=`): a set comparison that + only checks two of eight codes would stay green even if a rule's whole block were + deleted from `propose()` — which is exactly what happened to ADV007 before this test + was tightened. Each rule below gets its own trigger, on distinct columns/index names so + none of them suppress or collide with another: + - ADV001: hot equality column `status`. + - ADV002: `idx_unused` on an unrelated column, zero scans. + - ADV003: `idx_narrow_redundant` is a plain prefix of `idx_wide_redundant`. + - ADV004: `status` (equality) and `shipped_at` (not-null check) share fingerprint fp1. + - ADV005: `note` is non-sargable. + - ADV006: a hot `SELECT *` over the wide `orders` table. + - ADV007: hot join key `customer_id`, unrelated to `status` so it cannot collide with + ADV001's candidate. + - ADV008: hot grouping column `region`, unrelated to every other column above so it + cannot collide with ADV001's or ADV007's candidate. + """ aggregation = Aggregation( usage=( - usage("status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=60.0), - usage("note", ColumnRole.NON_SARGABLE, cost_share=0.2, cost_ms=20.0), + _usage(_ORDERS, "status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=60.0), + _usage(_ORDERS, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.3, cost_ms=30.0), + _usage(_ORDERS, "note", ColumnRole.NON_SARGABLE, cost_share=0.2, cost_ms=20.0), + _usage(_ORDERS, "customer_id", ColumnRole.JOIN, cost_share=0.5, cost_ms=50.0), + _usage(_ORDERS, "region", ColumnRole.GROUP, cost_share=0.15, cost_ms=15.0), ), total_cost_ms=100.0, skipped_unqualifiable=0, - tables=frozenset({"orders"}), + tables=frozenset({_ORDERS}), + ) + existing = { + _ORDERS: ( + PgIndex("idx_unused", ("zzz",), False, False, 0, 4096), + PgIndex("idx_narrow_redundant", ("aaa",), False, False, 5, 1), + PgIndex("idx_wide_redundant", ("aaa", "bbb"), False, False, 5, 1), + ) + } + wide_columns = ( + "status", + "shipped_at", + "note", + "customer_id", + "created_at", + *(f"c{i}" for i in range(10)), + ) + select_star_stat = QueryStat( + fingerprint="fp_star", + sql="select * from orders", + calls=5, + total_time_ms=100.0, + flags=frozenset({FLAG_SELECT_STAR}), ) adapter = PostgresWorkloadAdapter(querier=lambda sql, params: []) + adapter.fetch_indexes = lambda schemas, tables: existing # type: ignore[method-assign] proposals = adapter.propose( aggregation, - facts(ndv={"status": 9999.0}), - _workload(), + _facts_map(ndv={"status": 9999.0, "customer_id": 9999.0}, columns=wide_columns), + _workload(select_star_stat), min_cost_share=0.01, ) - assert {p.code for p in proposals} >= {"ADV001", "ADV005"} + assert {p.code for p in proposals} == { + "ADV001", + "ADV002", + "ADV003", + "ADV004", + "ADV005", + "ADV006", + "ADV007", + "ADV008", + } assert proposals[0].confidence is Confidence.HIGH @@ -1045,11 +1811,12 @@ def test_render_ddl_recommends_concurrently_for_index_creation(): def test_generated_ddl_quotes_identifiers(): """Unquoted identifiers break on anything needing quotes — mixed case, reserved words.""" + relation = Relation("public", "Order") proposals = propose_indexes( - [usage("Status", ColumnRole.EQUALITY, table="Order")], + [_usage(relation, "Status", ColumnRole.EQUALITY)], { - "Order": TableFacts( - name="Order", + relation: TableFacts( + relation=relation, row_estimate=10**6, size_bytes=10**8, columns=("Status",), @@ -1073,11 +1840,12 @@ def test_a_newline_in_an_identifier_is_not_rendered_as_a_statement(): DDL against an object that does not exist. So the statement is commented out whole. """ hostile = "orders\nDROP TABLE users; --" + relation = Relation("public", hostile) proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY, table=hostile)], + [_usage(relation, "status", ColumnRole.EQUALITY)], { - hostile: TableFacts( - name=hostile, + relation: TableFacts( + relation=relation, row_estimate=10**6, size_bytes=10**8, columns=("status",), @@ -1104,65 +1872,179 @@ def test_quote_ident_doubles_an_embedded_quote(): assert _quote_ident('we"ird') == '"we""ird"' +def test_adv001_ddl_is_qualified_with_the_relations_own_schema(): + usage = (_usage(Relation("sales", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5),) + facts = {Relation("sales", "orders"): _facts(Relation("sales", "orders"), rows=50_000)} + proposals = propose_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].ddl == 'CREATE INDEX ON "sales"."orders" ("status");' + assert proposals[0].evidence["schema"] == "sales" + assert proposals[0].evidence["table"] == "orders" + assert "sales.orders" in proposals[0].title + + +def test_two_same_named_relations_get_two_independent_proposals(): + """One proposal per relation, each stamped with its own schema.""" + usage = ( + _usage(Relation("sales", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + _usage(Relation("staging", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + ) + facts = { + Relation("sales", "orders"): _facts(Relation("sales", "orders"), rows=50_000), + Relation("staging", "orders"): _facts(Relation("staging", "orders"), rows=50_000), + } + ddls = {p.ddl for p in propose_indexes(usage, facts, {}, min_cost_share=0.01)} + assert ddls == { + 'CREATE INDEX ON "sales"."orders" ("status");', + 'CREATE INDEX ON "staging"."orders" ("status");', + } + + +def test_an_index_in_one_schema_does_not_cover_the_other_schemas_candidate(): + """The coverage check must not reach across schemas.""" + usage = ( + _usage(Relation("sales", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + _usage(Relation("staging", "orders"), "status", ColumnRole.EQUALITY, cost_share=0.5), + ) + facts = { + Relation("sales", "orders"): _facts(Relation("sales", "orders"), rows=50_000), + Relation("staging", "orders"): _facts(Relation("staging", "orders"), rows=50_000), + } + existing = { + Relation("sales", "orders"): ( + PgIndex( + name="idx_status", + columns=("status",), + is_unique=False, + is_primary=False, + scans=1, + size_bytes=1, + ), + ) + } + proposals = propose_indexes(usage, facts, existing, min_cost_share=0.01) + assert [p.evidence["schema"] for p in proposals] == ["staging"] + + def test_created_index_ddl_is_schema_qualified(): """An unqualified name resolves against the *operator's* search_path, not ours. - With `--schema analytics`, `CREATE INDEX ON "orders"` run by someone whose search_path - is `public` targets the wrong table entirely. + `CREATE INDEX ON "orders"` run by someone whose search_path is `public` targets the + wrong table entirely when the relation actually lives in `analytics`. """ + relation = Relation("analytics", "orders") proposals = propose_indexes( - [usage("status", ColumnRole.EQUALITY)], - facts(ndv={"status": 5000.0}), + [_usage(relation, "status", ColumnRole.EQUALITY)], + _facts_map(relation, ndv={"status": 5000.0}), {}, min_cost_share=0.01, - schema="analytics", ) assert proposals[0].ddl == 'CREATE INDEX ON "analytics"."orders" ("status");' def test_partial_index_ddl_is_schema_qualified(): + relation = Relation("analytics", "orders") proposals = propose_partial_indexes( [ - usage("status", ColumnRole.EQUALITY, cost_ms=90.0), - usage("shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), + _usage(relation, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(relation, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), ], - facts(), + _facts_map(relation), min_cost_share=0.01, - schema="analytics", ) assert proposals[0].ddl.startswith('CREATE INDEX ON "analytics"."orders" ("status")') def test_dropped_index_ddl_is_schema_qualified(): """A bare `DROP INDEX idx_cold` drops whichever idx_cold the search_path finds first.""" - unused = {"orders": (PgIndex("idx_cold", ("note",), False, False, 0, 4096),)} - proposals = propose_unused_indexes(unused, hot_tables=frozenset({"orders"}), schema="analytics") + relation = Relation("analytics", "orders") + unused = {relation: (PgIndex("idx_cold", ("note",), False, False, 0, 4096),)} + proposals = propose_unused_indexes(unused, hot_tables=frozenset({relation})) assert proposals[0].ddl == 'DROP INDEX "analytics"."idx_cold";' redundant = { - "orders": ( + relation: ( PgIndex("idx_narrow", ("status",), False, False, 5, 1), PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), ) } - proposals = propose_redundant_indexes(redundant, schema="analytics") + proposals = propose_redundant_indexes(redundant, hot_tables=frozenset(redundant)) assert proposals[0].ddl == 'DROP INDEX "analytics"."idx_narrow";' -def test_propose_passes_the_adapters_schema_into_the_ddl(): - """The rules are module-level, so the adapter is the only thing that knows the schema.""" +def test_adv002_evidence_reports_the_bare_table_name_and_its_own_schema(): + """The brief's evidence contract — `"schema": relation.schema`, `"table": relation.table` + (the *bare* name, so existing JSON consumers keep reading the same value from the same + key) — applies to every rule, not just ADV001. `staging` (not `public`) pins that the + schema is not a hardcoded default, and the bare `"orders"` (not `"staging.orders"`) pins + that `evidence["table"]` was not quietly switched to the qualified string.""" + existing = { + Relation("staging", "orders"): ( + PgIndex( + name="idx_cold", + columns=("note",), + is_unique=False, + is_primary=False, + scans=0, + size_bytes=1, + ), + ) + } + proposals = propose_unused_indexes( + existing, hot_tables=frozenset({Relation("staging", "orders")}) + ) + assert proposals[0].evidence["schema"] == "staging" + assert proposals[0].evidence["table"] == "orders" + + +def test_adv003_evidence_reports_the_bare_table_name_and_its_own_schema(): + existing = { + Relation("staging", "orders"): ( + PgIndex("idx_narrow", ("status",), False, False, 5, 1), + PgIndex("idx_wide", ("status", "created_at"), False, False, 5, 1), + ) + } + proposals = propose_redundant_indexes(existing, hot_tables=frozenset(existing)) + assert proposals[0].evidence["schema"] == "staging" + assert proposals[0].evidence["table"] == "orders" + + +def test_adv004_evidence_reports_the_bare_table_name_and_its_own_schema(): + relation = Relation("staging", "orders") + proposals = propose_partial_indexes( + [ + _usage(relation, "status", ColumnRole.EQUALITY, cost_ms=90.0), + _usage(relation, "shipped_at", ColumnRole.NOT_NULL_CHECK, cost_share=0.4, cost_ms=40.0), + ], + _facts_map(relation), + min_cost_share=0.01, + ) + assert proposals[0].evidence["schema"] == "staging" + assert proposals[0].evidence["table"] == "orders" + + +def test_propose_end_to_end_uses_a_non_public_relations_own_schema(): + """The three adapter-level `propose()` tests elsewhere in this file all use + `public.orders`, so the end-to-end path was only ever exercised with the default + schema. A relation living anywhere else must flow through unchanged.""" + relation = Relation("analytics", "orders") aggregation = Aggregation( - usage=(usage("status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=60.0),), + usage=(_usage(relation, "status", ColumnRole.EQUALITY, cost_share=0.6, cost_ms=60.0),), total_cost_ms=100.0, skipped_unqualifiable=0, - tables=frozenset({"orders"}), + tables=frozenset({relation}), ) adapter = PostgresWorkloadAdapter(querier=lambda sql, params: []) - adapter.schemas = ("analytics",) proposals = adapter.propose( - aggregation, facts(ndv={"status": 9999.0}), _workload(), min_cost_share=0.01 + aggregation, + _facts_map(relation, ndv={"status": 9999.0}), + _workload(), + min_cost_share=0.01, ) - assert any('"analytics"."orders"' in (p.ddl or "") for p in proposals) + adv001 = [p for p in proposals if p.code == "ADV001"] + assert adv001 + assert adv001[0].ddl == 'CREATE INDEX ON "analytics"."orders" ("status");' + assert adv001[0].evidence["schema"] == "analytics" + assert adv001[0].evidence["table"] == "orders" def test_adv005_reports_a_short_fingerprint_id_and_keeps_the_sql_separately(): @@ -1207,8 +2089,8 @@ def test_adv006_also_reports_the_short_id_without_losing_the_query(): flags=frozenset({FLAG_SELECT_STAR}), ) wide = { - "orders": TableFacts( - name="orders", + _ORDERS: TableFacts( + relation=_ORDERS, row_estimate=10**6, size_bytes=10**8, columns=tuple(f"c{i}" for i in range(30)), @@ -1217,3 +2099,455 @@ def test_adv006_also_reports_the_short_id_without_losing_the_query(): proposals = propose_select_star(_workload(stat), wide, min_cost_share=0.01) assert proposals[0].evidence["fingerprint"] != canonical assert proposals[0].evidence["sql"] == stat.sql + + +def test_adv007_proposes_an_index_on_an_unindexed_hot_join_key(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4, cost_ms=400.0),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert [p.code for p in proposals] == ["ADV007"] + assert proposals[0].ddl == 'CREATE INDEX ON "public"."order_items" ("order_id");' + assert proposals[0].confidence is Confidence.HIGH + + +def test_adv007_is_silent_when_an_index_already_leads_with_the_join_key(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + existing = { + relation: ( + PgIndex( + name="idx_oi_order", + columns=("order_id", "sku"), + is_unique=False, + is_primary=False, + scans=5, + size_bytes=1, + ), + ) + } + assert propose_join_keys(usage, facts, existing, min_cost_share=0.01) == [] + + +def test_adv007_respects_the_small_table_floor(): + relation = Relation("public", "tiny") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.9),) + facts = {relation: _facts(relation, rows=10, ndv={"order_id": 5.0})} + assert propose_join_keys(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv007_caps_at_low_when_the_index_list_could_not_be_read(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01, have_index_data=False) + assert proposals[0].confidence is Confidence.LOW + assert "could not be read" in proposals[0].rationale + + +def test_adv007_caps_at_low_and_discloses_an_unknown_row_count(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=None, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.LOW + assert "small-table floor" in proposals[0].rationale + + +def test_adv007_is_low_for_a_low_cardinality_join_key(): + relation = Relation("public", "order_items") + usage = (_usage(relation, "kind", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"kind": 3.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.LOW + + +def test_adv007_is_medium_when_ndv_is_unknown(): + """The middle rung of the confidence ladder: `rows` and `have_index_data` are both + fine, but the NDV catalog has nothing for this column. Changing that branch to return + HIGH instead of MEDIUM would overstate a claim with no selectivity evidence behind it — + exactly the failure mode this rule set exists to avoid — and must fail this test.""" + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.MEDIUM + + +def test_adv007_suppresses_a_join_key_below_the_cost_share_threshold(): + """Pins that `--min-cost-share` actually reaches ADV007, as its own help text now + claims. Replacing the `cost_share < min_cost_share` guard with `if False` must fail + this test.""" + relation = Relation("public", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.005, cost_ms=5.0),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + assert propose_join_keys(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv007_discloses_a_partial_index_leading_with_the_join_key(): + """Mirrors ADV001's `test_a_partial_index_does_not_suppress_a_candidate` exactly. + `_covered` correctly does not treat a partial index as coverage — a WHERE-guarded index + does not serve an unfiltered join probe either — but before this test existed, ADV007 + silently said nothing about `idx_open` at all. Naming it in the evidence and rationale, + the same way ADV001 does for the same gap, is the fix.""" + relation = Relation("public", "order_items") + existing = { + relation: ( + PgIndex( + "idx_open", + ("order_id",), + False, + False, + 5, + 4096, + is_partial=True, + predicate="(shipped_at IS NULL)", + ), + ) + } + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, existing, min_cost_share=0.01) + assert codes(proposals) == ["ADV007"] + assert proposals[0].evidence["partial_indexes_skipped"] == ("idx_open",) + assert "partial" in proposals[0].rationale.lower() + + +def test_adv007_discloses_an_expression_index_mentioning_the_join_key(): + """Mirrors ADV001's `test_an_expression_index_is_disclosed_not_silently_ignored`. The + `columns` tuple of an expression index understates it, so `_covered` cannot see + `lower(order_id)` leads with `order_id` — naming the index is the only honest option.""" + relation = Relation("public", "order_items") + existing = { + relation: ( + PgIndex( + "idx_lower_order_id", + (), + False, + False, + 5, + 4096, + has_expressions=True, + definition="CREATE INDEX idx_lower_order_id ON order_items (lower(order_id))", + ), + ) + } + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, existing, min_cost_share=0.01) + assert codes(proposals) == ["ADV007"] + assert proposals[0].evidence["expression_indexes"] == ("idx_lower_order_id",) + assert "expression" in proposals[0].rationale.lower() + + +def test_adv007_ignores_non_join_roles(): + """The rule must not re-propose what ADV001 already covers.""" + relation = Relation("public", "orders") + usage = (_usage(relation, "status", ColumnRole.EQUALITY, cost_share=0.9),) + facts = {relation: _facts(relation, rows=100_000)} + assert propose_join_keys(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv007_reports_the_hottest_join_key_per_relation(): + relation = Relation("public", "order_items") + usage = ( + _usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4, cost_ms=400.0), + _usage(relation, "sku", ColumnRole.JOIN, cost_share=0.1, cost_ms=100.0), + ) + facts = {relation: _facts(relation, rows=100_000)} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert [p.evidence["columns"] for p in proposals] == [("order_id",), ("sku",)] + + +def test_adv007_evidence_reports_the_bare_table_name_and_its_own_schema(): + relation = Relation("staging", "order_items") + usage = (_usage(relation, "order_id", ColumnRole.JOIN, cost_share=0.4),) + facts = {relation: _facts(relation, rows=100_000, ndv={"order_id": 5000.0})} + proposals = propose_join_keys(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["schema"] == "staging" + assert proposals[0].evidence["table"] == "order_items" + + +def test_adv008_proposes_a_composite_index_for_a_hot_group_by(): + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",) + ), + _usage(relation, "day", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, fps=("fp1",)), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert [p.code for p in proposals] == ["ADV008"] + assert proposals[0].evidence["columns"] == ("tenant_id", "day") + assert proposals[0].ddl == 'CREATE INDEX ON "public"."events" ("tenant_id", "day");' + + +def test_adv008_never_reaches_high_confidence(): + """Whether the planner picks GroupAggregate over HashAggregate is not visible to us.""" + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.9, cost_ms=900.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=5_000_000, ndv={"tenant_id": 100_000.0})} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.MEDIUM + + +def test_adv008_is_low_when_the_row_count_is_unknown(): + """The other rung of the confidence ladder: row count unknown, so the small-table gate + could not run. Changing this branch to MEDIUM would claim a check happened that did not, + exactly the failure mode the ladder exists to avoid.""" + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.9, cost_ms=900.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=None)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].confidence is Confidence.LOW + assert "small-table floor" in proposals[0].rationale + + +def test_adv008_is_low_when_the_index_list_could_not_be_read(): + """The other LOW trigger: the existing-index catalog query was denied, so whether an + index already leads with these columns is unknowable.""" + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.9, cost_ms=900.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes( + usage, facts, {}, min_cost_share=0.01, have_index_data=False + ) + assert proposals[0].confidence is Confidence.LOW + assert "could not be read" in proposals[0].rationale + + +def test_adv008_groups_only_columns_that_co_occur_in_one_query(): + """Two GROUP BYs in two different queries are not one composite index.""" + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",) + ), + _usage(relation, "day", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, fps=("fp2",)), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["columns"] == ("tenant_id",) + + +def test_adv008_requires_joint_support_not_just_pairwise_with_the_seed(): + """A transitive chain must not be welded into one composite. + + `a` co-occurs with `b` in fp1 and with `c` in fp2, but `b` and `c` never co-occur with + each other — no query in the workload groups by `a, b, c` together. Checking each + candidate only against the seed's fingerprints (the old, pairwise rule) let `c` join + once `b` had already been accepted, on the strength of `a`'s membership in fp2 — even + though the *composite so far*, `(a, b)`, is never grouped by alongside `c`. Requiring the + running intersection to stay non-empty catches this: after `b` joins, the shared set + narrows to fp1 alone, and `c` (only in fp2) can no longer extend it. + """ + relation = Relation("public", "events") + usage = ( + _usage(relation, "a", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1", "fp2")), + _usage(relation, "b", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, fps=("fp1",)), + _usage(relation, "c", ColumnRole.GROUP, cost_share=0.5, cost_ms=300.0, fps=("fp2",)), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["columns"] == ("a", "b") + assert proposals[0].evidence["columns"] != ("a", "b", "c") + + +def test_adv008_reports_the_honest_joint_support_count_and_omits_the_plain_one(): + """`co_occurring_fingerprints` must report the running intersection's size, not the + per-column `fingerprints` max, which would (falsely) read as "two query groups back this + three-column composite" when the joint support for `(a, b)` is exactly one query group. + + The plain `fingerprints` key must be *absent*, not merely unused: `report.py` renders + evidence as generic sorted `k=v` pairs with no per-rule text, so a reader sees both + numbers side by side with nothing to say which one actually supports the proposal. + Unlike ADV001/ADV007 (one candidate, so a per-column count is the whole truth), this + rule's claim is about columns appearing *together*, and only the joint count says that. + """ + relation = Relation("public", "events") + usage = ( + _usage(relation, "a", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1", "fp2")), + _usage(relation, "b", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, fps=("fp1",)), + _usage(relation, "c", ColumnRole.GROUP, cost_share=0.5, cost_ms=300.0, fps=("fp2",)), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["co_occurring_fingerprints"] == 1 + assert "fingerprints" not in proposals[0].evidence + + +def test_adv008_composite_is_just_the_seed_when_it_shares_nothing_with_any_other_column(): + relation = Relation("public", "events") + usage = ( + _usage(relation, "a", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",)), + _usage(relation, "b", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, fps=("fp2",)), + _usage(relation, "c", ColumnRole.GROUP, cost_share=0.5, cost_ms=300.0, fps=("fp3",)), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["columns"] == ("a",) + assert proposals[0].evidence["co_occurring_fingerprints"] == 1 + + +def test_adv008_is_silent_when_an_index_already_leads_with_the_grouping_columns(): + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + existing = { + relation: ( + PgIndex( + name="idx_events_tenant", + columns=("tenant_id", "day"), + is_unique=False, + is_primary=False, + scans=3, + size_bytes=1, + ), + ) + } + assert propose_grouping_indexes(usage, facts, existing, min_cost_share=0.01) == [] + + +def test_adv008_respects_max_arity(): + relation = Relation("public", "events") + usage = tuple( + _usage(relation, name, ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0 - i, fps=("fp1",)) + for i, name in enumerate(["a", "b", "c", "d"]) + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["columns"] == ("a", "b", "c") + + +def test_adv008_discloses_that_the_column_order_is_inferred(): + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",) + ), + _usage(relation, "day", ColumnRole.GROUP, cost_share=0.5, cost_ms=400.0, fps=("fp1",)), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert "inferred" in proposals[0].rationale.lower() + + +def test_adv008_ignores_non_group_roles(): + relation = Relation("public", "orders") + usage = (_usage(relation, "status", ColumnRole.EQUALITY, cost_share=0.9),) + facts = {relation: _facts(relation, rows=5_000_000)} + assert propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv008_respects_the_small_table_floor(): + relation = Relation("public", "tiny") + usage = (_usage(relation, "tenant_id", ColumnRole.GROUP, cost_share=0.9),) + facts = {relation: _facts(relation, rows=10)} + assert propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv008_suppresses_a_grouping_below_the_cost_share_threshold(): + """Pins the `cost_share < min_cost_share` guard specifically: rows are well above + MIN_ROWS_FOR_INDEX (10,000) so the small-table floor cannot be why this is suppressed.""" + relation = Relation("public", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.005, cost_ms=5.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + assert propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) == [] + + +def test_adv008_discloses_a_partial_index_leading_with_the_grouping_columns(): + """Mirrors ADV001's/ADV007's identical disclosure. `_covered` correctly does not treat a + partial index as coverage — a WHERE-guarded index does not serve an unfiltered GROUP BY + either — but silence on that gap would let ADV008 say nothing next to an index that, in + plain English, does lead with these columns.""" + relation = Relation("public", "events") + existing = { + relation: ( + PgIndex( + "idx_events_open", + ("tenant_id",), + False, + False, + 5, + 4096, + is_partial=True, + predicate="(closed_at IS NULL)", + ), + ) + } + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, existing, min_cost_share=0.01) + assert codes(proposals) == ["ADV008"] + assert proposals[0].evidence["partial_indexes_skipped"] == ("idx_events_open",) + assert "partial" in proposals[0].rationale.lower() + + +def test_adv008_discloses_an_expression_index_mentioning_the_leading_grouping_column(): + """Mirrors ADV001's/ADV007's identical disclosure. The `columns` tuple of an expression + index understates it, so `_covered` cannot see `lower(tenant_id)` leads with `tenant_id` + — naming the index is the only honest option.""" + relation = Relation("public", "events") + existing = { + relation: ( + PgIndex( + "idx_lower_tenant", + (), + False, + False, + 5, + 4096, + has_expressions=True, + definition="CREATE INDEX idx_lower_tenant ON events (lower(tenant_id))", + ), + ) + } + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, existing, min_cost_share=0.01) + assert codes(proposals) == ["ADV008"] + assert proposals[0].evidence["expression_indexes"] == ("idx_lower_tenant",) + assert "expression" in proposals[0].rationale.lower() + + +def test_adv008_evidence_reports_the_bare_table_name_and_its_own_schema(): + relation = Relation("staging", "events") + usage = ( + _usage( + relation, "tenant_id", ColumnRole.GROUP, cost_share=0.5, cost_ms=500.0, fps=("fp1",) + ), + ) + facts = {relation: _facts(relation, rows=5_000_000)} + proposals = propose_grouping_indexes(usage, facts, {}, min_cost_share=0.01) + assert proposals[0].evidence["schema"] == "staging" + assert proposals[0].evidence["table"] == "events"