fix: renew long-lived subscriptions

[mattisonchao]

Motivation

Oxia keeps shard assignments, notifications, and sequence updates open as long-lived client subscriptions. When traffic passes through Istio/Envoy, the proxy can lose or close its upstream connection while its downstream HTTP/2 connection to the client remains open. In that state, transport keepalive can still succeed on the downstream hop, so the client sees an apparently healthy but silent subscription and never reconnects. The existing request timeout only protects establishment before the first response; it does not detect this failure after a subscription is established.

Kubernetes client-go addresses the same class of hanging watch by giving each Reflector watch a bounded, randomized lifetime and recreating it from the last known state. Oxia can use the same model: shard assignments restart with a complete snapshot, notifications resume after the last received offset, and sequence updates remain monotonic by suppressing equal or older keys replayed during renewal. Randomizing renewal times also avoids a reconnect wave across clients. See the Kubernetes client-go Reflector.

Modifications

• Add subscriptionMaxAge(Duration) for shard assignments, notifications, and sequence updates.
• Default the maximum age to 10 minutes and choose each actual age randomly in [maxAge / 2, maxAge), producing the Kubernetes-style 5–10 minute range by default.
• Add disableSubscriptionMaxAge() for deployments that explicitly require unbounded subscriptions.
• Apply the age as a client-local gRPC deadline, independent of whether Istio propagates an upstream disconnect to the downstream connection.
• Transparently renew expired subscriptions while preserving assignment snapshots and notification offsets; sequence-update callbacks use the fixed-width sequence-key ordering to ignore equal or older replayed keys.
• Handle expected maximum-age expiry consistently across all three operations: log at debug level, reset failure backoff where applicable, and queue immediate renewal on the client scheduler instead of reopening inline on the gRPC callback thread.
• Preserve the existing retry policy and warning logs for ordinary failures.
• Document that “subscription” is the client-level name for these three long-lived operations, not a public transport stream or an inactivity timeout.

Testing

• ./gradlew :client:test --tests '*Test' spotlessCheck
• ./gradlew :client-api:javadoc
• Added coverage for randomized deadlines and opt-out behavior across all three subscription types.
• Added coverage for scheduler-based notification renewal and suppression of equal or older sequence keys after renewal.
• Testcontainers integration tests were not rerun because Docker is unavailable in this environment.