docs: 1.9

contents/docs/connecting-to-postgres.mdx

@@ -57,6 +57,20 @@ After your server restarts, show the `wal_level` again to ensure it has changed:
 psql -c 'SHOW wal_level'
 ```
 
+### Socket Inactivity Timeout
+
+`zero-cache` monitors wire activity on its Postgres connections so it can recover when a proxy or network failure leaves a half-open socket. The watchdog samples each connection every 120,000 milliseconds and resets it after one to two intervals without any bytes read or written. In-flight queries on a reset connection are rejected and can recover through their normal retry or restart paths.
+
+Wire activity resets the watchdog, so streaming operations such as `COPY` remain active. A statement that legitimately computes without sending any data for several minutes can be interrupted.
+
+### WAL Sender Timeout
+
+`zero-cache` uses Postgres's `wal_sender_timeout` setting to monitor its replication connection. When the timeout is greater than `0`, Zero sends keepalives and reconnects if the replication stream stops responding. The inbound timeout defaults to twice `wal_sender_timeout`.
+
+A healthy WAL sender can sometimes remain silent longer than this while decoding WAL from unpublished tables or assembling a large transaction. Set [`ZERO_UPSTREAM_PG_STREAM_INBOUND_TIMEOUT_MS`](/docs/zero-cache-config#upstream-pg-stream-inbound-timeout) to widen Zero's inbound threshold without changing the server's timeout. Manual keepalive timing remains derived from `wal_sender_timeout`.
+
+Setting `wal_sender_timeout` to `0` disables the timeout in Postgres and the related keepalive and reconnect checks in Zero, even when an inbound timeout override is configured. Other connection failure detection remains active.
+
 ### Bounding WAL Size
 
 For development databases, you can set a `max_slot_wal_keep_size` value in Postgres. This will help limit the amount of WAL kept around.

contents/docs/connection.mdx

@@ -183,9 +183,9 @@ Reads are allowed while `disconnected`, but writes are rejected and return an of
 
 ### Error
 
-If `zero-cache` itself crashes, or if the [mutate](/docs/mutators) or [query](/docs/queries) endpoints return a network or HTTP error, Zero transitions to the `error` state.
+If `zero-cache` crashes, or [mutate](/docs/mutators) or [query](/docs/queries) endpoints fail, Zero enters the `error` state. If the response code is `5xx`, `zero-cache` will retry up to four times.
 
-This type of error is unlikely to resolve just by retrying, so Zero doesn't try. The app can retry the connection manually by calling `zero.connection.connect()`.
+Zero does not retry from the `error` state. Call `zero.connection.connect()` to retry manually.
 
 Reads are allowed while in the `error` state, but writes are rejected.
 

contents/docs/mutators.mdx

@@ -111,6 +111,8 @@ tx.mutate.user.insert({
 })
 ```
 
+If the Zero primary key already exists, `insert` will succeed without changing the row - use `upsert` to update an existing row.
+
 Optional fields can be set to `null` to explicitly set the new field to `null`. They can also be set to `undefined` to take the default value (which is often `null` but can also be some generated value server-side):
 
 ```tsx
@@ -856,7 +858,7 @@ app.post('/api/zero/mutate', async c => {
 
 </CodeGroup>
 
-If Zero receives any response from the mutate endpoint other than HTTP 200, 401, or 403, it will disconnect and enter the [error state](/docs/connection#error).
+Responses other than 200, 401, or 403 enter the [error state](/docs/connection#error). `zero-cache` will retry on `5xx` up to four times before returning an error.
 
 If Zero receives HTTP 401 or 403, the client will enter the needs auth state and require a manual reconnect. Use `zero.connection.connect()` for cookie auth or `zero.connection.connect({auth: newToken})` for token auth, then Zero will retry all queued mutations.
 

contents/docs/release-notes/1.9.mdx

@@ -0,0 +1,73 @@
+---
+title: Zero 1.9
+description: Stability and Query Correctness
+---
+
+## Installation
+
+```bash
+npm install @rocicorp/zero@1.9
+```
+
+## Overview
+
+Zero 1.9 improves query/mutation correctness and contains numerous reliability improvements.
+
+## Features
+
+- [`zero_sync_e2e_serving_lag`](/docs/otel#zerosync) measures completed replicated work from the upstream transaction commit through view-syncer poke. [`zero_replication_upstream_clock_skew`](/docs/otel#zeroreplication) tries to identify measurements biased by clock differences. ([#6312](https://github.com/rocicorp/mono/pull/6312))
+- Zero's replication-stream inbound timeout can now be [configured separately from PostgreSQL's `wal_sender_timeout`](/docs/connecting-to-postgres#wal-sender-timeout), avoiding unnecessary reconnects during long gaps in WAL output. ([#6351](https://github.com/rocicorp/mono/pull/6351), thanks [@gerardatkonvo](https://github.com/gerardatkonvo)!)
+
+## Performance
+
+### Cold Mutation Latency
+
+Before running mutations, Zero Server fetches and caches PostgreSQL schema metadata. This is now **2.7x faster** in Zero 1.9 (done in [#6292](https://github.com/rocicorp/mono/pull/6292), thanks [@diegopereira99](https://github.com/diegopereira99)!). This is most noticeable with cold-starts in serverless environments like AWS Lambda.
+
+<BenchmarkComparisonChart
+  title="Cold Mutation Latency"
+  description="Measured on M5 Pro. Lower is better."
+  previousLabel="Zero 1.8"
+  currentLabel="Zero 1.9"
+  precision={2}
+  valuePrecision={3}
+  unit="ms"
+  higherIsBetter={false}
+  height={160}
+  data={[
+    {
+      name: 'First mutation',
+      fullName:
+        'First mutation with uncached server-schema metadata',
+      previous: 20.973,
+      current: 7.645
+    }
+  ]}
+/>
+
+## Fixes
+
+- [Restores now use Litestream 0.5.15 for legacy-format compatibility](https://github.com/rocicorp/mono/pull/6260), [retry transient failures](https://github.com/rocicorp/mono/pull/6347), [clean up temporary databases and staged WAL files after failed or interrupted attempts](https://github.com/rocicorp/mono/pull/6355), and [retain the previous snapshot generation during active restores](https://github.com/rocicorp/mono/pull/6267).
+- [`insert` now succeeds without changing the row when its Zero primary key already exists; before 1.9, the server returned an error.](https://github.com/rocicorp/mono/pull/6251)
+- [Ordered queries now return correct results when cursor fields contain `NULL`.](https://github.com/rocicorp/mono/pull/6121) (thanks [@YevheniiKotyrlo](https://github.com/YevheniiKotyrlo)!)
+- [Rebuilt queries now deliver changed rows instead of occasionally leaving clients with stale results.](https://github.com/rocicorp/mono/pull/6196)
+- [Queries no longer drop rows or emit invalid SQL when given an inapplicable scalar hint, and scalar `NOT EXISTS` now handles empty or `NULL` results.](https://github.com/rocicorp/mono/pull/6306)
+- [Schema construction, CRUD mutators, and materialized views now preserve a key named `__proto__` as user data.](https://github.com/rocicorp/mono/pull/6185) (thanks [@tjenkinson](https://github.com/tjenkinson)!)
+- SQLite statement caches now [retain at most 1,000 idle entries each](https://github.com/rocicorp/mono/pull/6202).
+- Terminated client groups now [release custom-query timers and caches](https://github.com/rocicorp/mono/pull/6228).
+- Large replica transactions [can spill dirty pages to WAL instead of retaining the complete write set in native memory](https://github.com/rocicorp/mono/pull/6311).
+- [Missing replication-lag reports are retried and `total_lag` no longer grows when reports stop arriving](https://github.com/rocicorp/mono/pull/6187), while [serving-lag metrics exclude disconnected or not-yet-validated client groups](https://github.com/rocicorp/mono/pull/6219).
+- `zero-cache` now recovers from [half-open PostgreSQL sockets](https://github.com/rocicorp/mono/pull/6220), [including over TLS](https://github.com/rocicorp/mono/pull/6221), and [the official image applies the bundled postgres.js disconnect patch](https://github.com/rocicorp/mono/pull/6310).
+- [With PostgreSQL `wal_sender_timeout=0`, replication no longer enters a continuous reconnect loop.](https://github.com/rocicorp/mono/pull/6244) See [WAL Sender Timeout](/docs/connecting-to-postgres#wal-sender-timeout).
+- [Client connection attempts now time out across setup and the server handshake, abandon late sockets, and retry normally.](https://github.com/rocicorp/mono/pull/6299)
+- [Reconnect confirmations no longer produce false slow-query warnings or inflated materialization metrics.](https://github.com/rocicorp/mono/pull/6308)
+- [Different integration versions in a pnpm workspace no longer create peer-qualified duplicate copies of `@rocicorp/zero`, fixing cross-package type and module-augmentation failures.](https://github.com/rocicorp/mono/pull/6231)
+- [Replicated PostgreSQL type and nullability changes now preserve compound-index column order in SQLite replicas.](https://github.com/rocicorp/mono/pull/6225) To repair an affected replica, resync it from Postgres or recreate the PostgreSQL index.
+- [Expected schema and replica resets now log warnings instead of errors](https://github.com/rocicorp/mono/pull/6248), and [`zero-cache` skips Litestream restore when backups are not configured](https://github.com/rocicorp/mono/pull/6259). (thanks [@asterikx](https://github.com/asterikx)!)
+- [Server CRUD updates and upserts no longer assign primary-key columns, avoiding PostgreSQL locks that could block concurrent foreign-key inserts.](https://github.com/rocicorp/mono/pull/6280) (thanks [@shayonj](https://github.com/shayonj)!)
+- [Mutation and query API calls now retry all `5xx` responses using the existing four-attempt limit and backoff; `4xx` responses still fail without retry.](https://github.com/rocicorp/mono/pull/6315) (thanks [@shayonj](https://github.com/shayonj)!)
+- [SQLite corruption failures now log diagnostics](https://github.com/rocicorp/mono/pull/6215), [delete the corrupted replica before exit](https://github.com/rocicorp/mono/pull/6342), and support [extended corruption errors](https://github.com/rocicorp/mono/pull/6339), with [deeper checks available as an opt-in](https://github.com/rocicorp/mono/pull/6341).
+- [Oversized replication updates now identify the transaction, affected column, and value type without logging the value.](https://github.com/rocicorp/mono/pull/6318)
+- [Fatal replica-writer failures now surface as replication errors and cause `zero-cache` to exit with a failure instead of silently stopping replication.](https://github.com/rocicorp/mono/pull/6326)
+- [Replication now recovers from upstream disconnects](https://github.com/rocicorp/mono/pull/6346) or [stalled PostgreSQL writes](https://github.com/rocicorp/mono/pull/6348) while flow control is blocked.
+- [Mutations from multiple tabs in the same client group are no longer skipped or sent out of order.](https://github.com/rocicorp/mono/pull/6340)

contents/docs/release-notes/index.mdx

@@ -2,6 +2,7 @@
 title: Release Notes
 ---
 
+- [Zero 1.9: Stability and Query Correctness](/docs/release-notes/1.9)
 - [Zero 1.8: Observability and Reliability](/docs/release-notes/1.8)
 - [Zero 1.7: Query Correctness and Performance](/docs/release-notes/1.7)
 - [Zero 1.6: PlanetScale Failover Support](/docs/release-notes/1.6)

contents/docs/zero-cache-config.mdx

@@ -332,6 +332,31 @@ Path to the litestream executable. This must be built from the `rocicorp/litestr
 flag: `--litestream-executable`<br/>
 env: `ZERO_LITESTREAM_EXECUTABLE`<br/>
 
+### Litestream V5 Executable
+
+Path to the official Litestream v0.5.x executable used for restores when `ZERO_LITESTREAM_RESTORE_USING_V5` is enabled. Litestream v0.5.8 and later can restore both legacy WAL backups and LTX backups, choosing the format with the latest data. The official Zero Docker image includes Litestream 0.5.15 at this path.
+
+flag: `--litestream-executable-v5`<br/>
+env: `ZERO_LITESTREAM_EXECUTABLE_V5`<br/>
+
+### Litestream Restore Using V5
+
+Use `ZERO_LITESTREAM_EXECUTABLE_V5` for restores when that executable is configured. If it is unavailable, Zero falls back to the legacy executable. Set this to `false` to force legacy restore behavior.
+
+Litestream v0.5 cannot restore legacy backups encrypted with Age. Keep legacy restore enabled for those backups or migrate them before enabling v5 restore.
+
+flag: `--litestream-restore-using-v5`<br/>
+env: `ZERO_LITESTREAM_RESTORE_USING_V5`<br/>
+default: `true`
+
+### Litestream Backup Using V5
+
+Write LTX backups with Litestream v0.5.x. This is disabled by default to continue writing legacy WAL backups. Enabling it requires v5 restore and makes rollback difficult because older versions cannot restore an LTX-only backup.
+
+flag: `--litestream-backup-using-v5`<br/>
+env: `ZERO_LITESTREAM_BACKUP_USING_V5`<br/>
+default: `false`
+
 ### Litestream Incremental Backup Interval Minutes
 
 The interval between incremental backups of the replica. Shorter intervals reduce the amount of change history that needs to be replayed when catching up a new view-syncer, at the expense of increasing the number of files needed to download for the initial litestream restore.
@@ -407,7 +432,7 @@ default: `48`
 
 ### Litestream Snapshot Backup Interval Hours
 
-The interval between snapshot backups of the replica. Snapshot backups make a full copy of the database to a new litestream generation. This improves restore time at the expense of bandwidth. Applications with a large database and low write rate can increase this interval to reduce network usage for backups (litestream defaults to 24 hours).
+The interval between snapshot backups of the replica. Snapshot backups make a full copy of the database to a new litestream generation. Zero retains the previous generation for six additional hours so an active restore can finish before its snapshot and WAL files are removed. This improves restore time and safety at the expense of bandwidth and temporary backup storage. Applications with a large database and low write rate can increase this interval to reduce network usage for backups (litestream defaults to 24 hours).
 
 flag: `--litestream-snapshot-backup-interval-hours`<br/>
 env: `ZERO_LITESTREAM_SNAPSHOT_BACKUP_INTERVAL_HOURS`<br/>
@@ -545,7 +570,7 @@ default: `60000`
 
 ### PG Replication Slot Failover
 
-For upstream Postgres 17+, creates replication slots with the `failover` flag enabled so they can be synchronized to a standby and survive a failover. This requires additional Postgres-side configuration on your provider; see [High Availability and Failover](/docs/connecting-to-postgres#high-availability-and-failover). Has no effect on Postgres versions before 17.
+For upstream Postgres 17+, creates replication slots with the `failover` flag enabled so they can be synchronized to a standby and survive a failover. This requires additional Postgres-side configuration on your provider; see [High Availability](/docs/connecting-to-postgres#high-availability). Has no effect on Postgres versions before 17.
 
 flag: `--upstream-pg-replication-slot-failover`<br/>
 env: `ZERO_UPSTREAM_PG_REPLICATION_SLOT_FAILOVER`<br/>
@@ -651,7 +676,7 @@ env: `ZERO_REPLICA_VACUUM_INTERVAL_HOURS`<br/>
 
 ### Replication Lag Report Interval (ms)
 
-The minimum interval at which replication lag reports are written upstream and reported via the `zero.replication.total_lag` [OpenTelemetry metric](/docs/otel). Because replication lag reports are only issued after the previous one was received, the actual interval between reports may be longer when there is a backlog in the replication stream.
+The minimum interval at which replication lag reports are written upstream and reported via the `zero.replication.total_lag` [OpenTelemetry metric](/docs/otel). If an expected report is not received before the next interval, Zero emits a new report and increments `zero.replication.lag_report_retries`.
 
 This feature requires write access to upstream Postgres (uses `pg_logical_emit_message()`). For PostgreSQL 17+, lag measurements accurately reflect committed write latency (single-digit milliseconds). For PostgreSQL 16 and earlier, measurements may appear 50-100ms longer due to flush behavior. A negative or 0 value disables lag reporting.
 
@@ -734,6 +759,17 @@ flag: `--upstream-pg-replication-slot-failover`<br/>
 env: `ZERO_UPSTREAM_PG_REPLICATION_SLOT_FAILOVER`<br/>
 default: `false`
 
+### Upstream PG Stream Inbound Timeout
+
+The time, in milliseconds, without an inbound message from the upstream WAL sender after which `zero-cache` tears down the replication stream to force a reconnect. By default, the threshold is twice the server's `wal_sender_timeout`.
+
+Increase this value when a healthy WAL sender can remain silent while decoding unpublished WAL or assembling a large transaction. This changes only Zero's inbound timeout; keepalive timing remains derived from `wal_sender_timeout`. The option has no effect when `wal_sender_timeout` is `0`, which disables inbound liveness detection.
+
+See [WAL Sender Timeout](/docs/connecting-to-postgres#wal-sender-timeout).
+
+flag: `--upstream-pg-stream-inbound-timeout-ms`<br/>
+env: `ZERO_UPSTREAM_PG_STREAM_INBOUND_TIMEOUT_MS`<br/>
+
 ### Websocket Compression
 
 Enable WebSocket per-message deflate compression. Compression can reduce bandwidth usage for sync traffic but increases CPU usage on both client and server. Disabled by default. See: https://github.com/websockets/ws#websocket-compression