[WIP]: Improve PPCB diagnostics and failback observability

[Abhijeet Mohanty (jeet1995)]

Summary

• include per-region PPCB health state in request diagnostics and validate the data-plane diagnostics shape
• add failback backlog/recovery metrics and rate-limited recovery logging
• cache immutable PPCB diagnostic snapshots so unchanged healthy requests reuse state instead of copying the region map per request
• keep diagnostic snapshots weakly consistent while routing state remains protected by ConcurrentHashMap.compute

Validation

• PPCB manager, holder, and failback unit suites: 47 passed, 0 failed/skipped
• local Gateway V2 HTTP/2 A/B runs showed throughput within approximately 1% after snapshot caching
• JFR map-copy allocation stacks dropped to zero; sampled PPCB allocation weight dropped from 136.5 MB to 10.3 MB
• the final cache commit contains no private static state class and no partition-wide synchronized success/exception handlers

Notes

• opened as draft for review of diagnostics shape, observability, and snapshot consistency semantics
• local benchmark harness and profiling artifacts are not included in this PR

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).
33 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).
33 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR enhances Cosmos DB Java SDK Per-Partition Circuit Breaker (PPCB) diagnostics and failback observability by (1) reshaping diagnostics to include per-region PPCB state, (2) adding failback backlog/recovery telemetry (including a per-collection gauge), and (3) caching immutable PPCB diagnostic snapshots to reduce per-request allocations while keeping routing-state updates concurrency-safe.

Changes:

• Add ppcb.stateByRegion diagnostics shape and update diagnostics serialization to publish cached PPCB snapshots.
• Introduce failback backlog logging (rate-limited) and a new cosmos.client.ppcb.failback.pendingRecoveryCount MultiGauge tagged by collection RID.
• Add/extend unit + E2E tests to validate the new diagnostics shape, snapshot reuse semantics, and meter/logging behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/models/CosmosMetricName.java	Adds a new PPCB failback pending recovery metric name and string mapping.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/PerPartitionCircuitBreakerInfoHolder.java	Implements immutable snapshot storage + snapshot sharing; updates PPCB diagnostics serialization to stateByRegion.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/GlobalPartitionEndpointManagerForPerPartitionCircuitBreaker.java	Publishes cached diagnostics snapshots into request context; adds failback backlog scanning, logging, and MultiGauge publication.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/ClientSideRequestStatistics.java	Records PPCB holder snapshots (not live state) and renames emitted diagnostics field to ppcb.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/CosmosAsyncClient.java	Wires client correlation id into PPCB failback logs and registers/unregisters the pending-recovery MultiGauge based on telemetry config.
sdk/cosmos/azure-cosmos/CHANGELOG.md	Documents the newly added pending-recovery gauge metric.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/PerPartitionCircuitBreakerE2ETests.java	Validates presence and contents of ppcb.stateByRegion in data-plane diagnostics.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/PpcbFailbackLoggingTest.java	Adds unit coverage for failback backlog/failure log sampling and MultiGauge lifecycle behavior.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/PerPartitionCircuitBreakerInfoHolderTest.java	Adds unit coverage for immutability, snapshot semantics, and serialization of initialized empty state.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/GlobalPartitionEndpointManagerForPPCBUnitTests.java	Adds unit coverage ensuring diagnostics snapshot reuse until health state transitions.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Abhijeet Mohanty (jeet1995)]

@sdkReviewAgent

[Abhijeet Mohanty (jeet1995)]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Annie Liang (xinlian12)]

🔴 Correctness: Require a successful probe before declaring failback complete

" has succeeded...");

submitOpenConnectionTasks() does not fail the publisher when connection attempts are exhausted: RntbdOpenConnectionsHandler converts connection exceptions to normal OpenConnectionResponse(isConnected=false) values, and the processor completes the task with that response. This doOnComplete path ignores every emitted response (and also treats an empty Flux as success), so an all-failed probe still calls handleSuccess, moves the region to HealthyTentative, and causes the next pending-recovery gauge snapshot to report zero.

Please aggregate/validate the responses and transition only when the defined success condition is met (at minimum, a non-empty result with a connected/already-connected endpoint). Keep empty or all-failed probes unavailable and route them through the sampled failure path.

⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.

[Annie Liang (xinlian12)]

✅ Review complete (07:31)

Posted 10 inline comment(s).

_{Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage}

[Abhijeet Mohanty (jeet1995)]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Abhijeet Mohanty (jeet1995)]

@sdkReviewAgent

[Annie Liang (xinlian12)]

🟡 Performance: Avoid recopying an already-immutable snapshot for every response

snapshot.setPerPartitionCircuitBreakerInfoHolder(this.perPartitionCircuitBreakerInfoHolder.v);

setPerPartitionCircuitBreakerInfoHolder always creates a new LinkedHashMap and unmodifiable wrapper. The routing path already paid that cost when it published into the request holder, so this call copies the same immutable map again for every direct/gateway statistic (and EMPTY.snapshot() still allocates a holder for PPCB-inapplicable responses). Direct retries and replica responses multiply the cost, which contradicts the PR’s cached-snapshot and zero-map-copy allocation claims.

Because LocationSpecificHealthContext is immutable and the stored map is unmodifiable, capture the existing map reference through a private no-copy snapshot constructor/factory, and use a shared immutable uninitialized snapshot. If caching is intentionally out of scope, the PR’s performance claims should be removed and this new default response-path allocation should be measured explicitly.

⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.

[Annie Liang (xinlian12)]

✅ Review complete (49:03)

Posted 1 inline comment(s).

_{Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage}