Skip to content

feat(sdk): fail over to a surviving node when the current one dies - #3944

Open
numinnex wants to merge 13 commits into
masterfrom
multi_endpoint_failover
Open

feat(sdk): fail over to a surviving node when the current one dies#3944
numinnex wants to merge 13 commits into
masterfrom
multi_endpoint_failover

Conversation

@numinnex

Copy link
Copy Markdown
Contributor

No description provided.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 21, 2026
@numinnex

Copy link
Copy Markdown
Contributor Author

Merge after #3934

@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.85598% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.98%. Comparing base (37d7eee) to head (8570d17).

Files with missing lines Patch % Lines
core/sdk/src/tcp/tcp_client.rs 89.03% 54 Missing and 23 partials ⚠️
...ache/iggy/client/async/tcp/AsyncIggyTcpClient.java 78.65% 11 Missing and 8 partials ⚠️
...SDK/IggyClient/Implementations/TcpMessageStream.cs 84.31% 11 Missing and 5 partials ⚠️
core/sdk/src/leader_aware.rs 89.25% 13 Missing ⚠️
foreign/go/client/tcp/tcp_core.go 88.18% 10 Missing and 3 partials ⚠️
foreign/node/src/client/client.socket.ts 93.06% 12 Missing ⚠️
foreign/node/src/client/client.connection.ts 98.22% 2 Missing and 1 partial ⚠️
foreign/go/client/tcp/tcp_session_management.go 93.33% 1 Missing and 1 partial ⚠️
...figuration/tcp_config/tcp_client_config_builder.rs 95.65% 0 Missing and 1 partial ⚠️
core/sdk/src/clients/mod.rs 92.85% 1 Missing ⚠️
... and 5 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3944      +/-   ##
============================================
- Coverage     84.03%   82.98%   -1.06%     
- Complexity     1358     1395      +37     
============================================
  Files          1215     1217       +2     
  Lines        170421   166509    -3912     
  Branches     138177   133794    -4383     
============================================
- Hits         143218   138173    -5045     
- Misses        23351    24228     +877     
- Partials       3852     4108     +256     
Components Coverage Δ
Rust Core 83.55% <89.29%> (-1.38%) ⬇️
Java SDK 67.09% <82.11%> (+0.41%) ⬆️
C# SDK 75.32% <85.96%> (+0.35%) ⬆️
Python SDK 90.10% <ø> (ø)
PHP SDK 84.68% <ø> (-0.98%) ⬇️
Node SDK 95.92% <95.61%> (+0.11%) ⬆️
Go SDK 68.99% <89.28%> (+0.69%) ⬆️
Files with missing lines Coverage Δ
.../src/traits/binary_impls/personal_access_tokens.rs 100.00% <ø> (ø)
core/common/src/traits/binary_impls/users.rs 100.00% <ø> (ø)
core/common/src/traits/binary_transport.rs 100.00% <100.00%> (ø)
...ypes/configuration/tcp_config/tcp_client_config.rs 100.00% <100.00%> (ø)
core/sdk/src/client_provider.rs 43.62% <100.00%> (-3.48%) ⬇️
...e/sdk/src/clients/binary_personal_access_tokens.rs 100.00% <ø> (ø)
core/sdk/src/clients/binary_users.rs 100.00% <ø> (ø)
core/sdk/src/vsr.rs 93.90% <100.00%> (ø)
...IggyClient/Implementations/TcpMessageStream.Vsr.cs 74.40% <100.00%> (+1.67%) ⬆️
...rg/apache/iggy/client/async/tcp/ReconnectPlan.java 100.00% <ø> (ø)
... and 17 more

... and 165 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cross-sdk:

  • configured seeds (failover_addresses) exist only in rust; go/java/c#/node keep a single seed, so a first connect on a dead configured node has nowhere to go. in scope or follow-up?
  • dial order: rust = current, roster, seeds; the four ports = current, configured seed, roster. every port dials a possibly dead configured node ahead of live survivors on every loss. pick one (rust's). same for reconnection disabled: rust bails on the first failed dial, go still sweeps everything.
  • StaleClient forget: rust/go/c# forget the remembered credentials, java never does, c# misses the outcome-unknown wrapper, go forgets only after the reconnection-disabled return. same contract in all five or in none.
  • empty description for a behaviour change across five sdks (remembered credentials, skipped reestablish_after, the 2s dial bound, replay semantics). write the contract down. the c# readme (InitialDelay, "without credentials a reconnect cannot restore the session") and the python readme (reestablish_after) are now wrong too.

outside the diff:

  • rust: IggyClientBuilder::with_tcp() has no with_failover_addresses passthrough, so seeds are only reachable via TcpClient::create(config). max_retries now counts full sweeps but the builder doc still says retries.
  • go: currentServerAddress is read without c.mtx in exchange (line 533) while dialCandidate writes it under the lock; time.Sleep(remaining) in the reestablish pause (line 926) ignores ctx. both pre-existing.
  • node: the maxRetries doc in client.type.ts says reconnection attempts, it now counts full passes; same for the DefaultReconnectOption comment. the readme has no failover mention.

Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread core/sdk/src/tcp/tcp_client.rs
Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread foreign/node/src/client/client.connection.ts Outdated
Comment thread foreign/node/src/client/client.connection.ts Outdated
Comment thread foreign/node/src/client/client.socket.ts Outdated
Comment thread foreign/node/src/client/client.connection.test.ts
Comment thread foreign/node/src/client/client.socket.test.ts Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 25, 2026
@numinnex

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 25, 2026

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

second pass at 4433f9a. most threads are fixed; what's below and the inline comments are still open.

  • staleclient rule: replay the remembered login everywhere, like rust does now. an eviction is not caller intent. go, c# and java (with and without builder credentials) should match.
  • PR description is still empty. write the contract down: remembered credentials, the 2s dial bound, the reestablish_after rotation, what is replayed after a reconnect and what is not.
  • seeds still exist only in rust, and there only via TcpClient::create(config): no with_failover_addresses on IggyClientBuilder::with_tcp(), no connection string option, cli and python can't set them. in scope or follow-up?
  • dial order still differs: rust = current, roster, seeds; go/java/c#/node = current, configured seed, roster, and the ports now pin their order with tests. pick one.
  • reconnection disabled: rust, go and c# sweep every candidate once; java and node don't dial at all (maxRetries 0 / enabled = false skips the loop). same rule everywhere.
  • docs: c# README lines 104, 111-112, 204-206 still say a hand-run login can't be restored; rust max_retries docs (client_builder.rs:168, tcp_client_config_builder.rs:61) and the python config doc still say retries, one retry is a full sweep now.
  • go: currentServerAddress read without c.mtx in exchange (tcp_core.go:542) and clientAddress in tcp_session_management.go:79 - both reported by -race.
  • go: concurrent requests failing during a reconnect with credentials hang the whole client. skipAutoLoginOnce is client-global: a second failing request's disconnect() flips the state so two Connects run, one eats the flag and the login replay's own Connect auto-logins under its own registerMtx. 8 concurrent pings on a dropped socket with WithAutoLogin hung until the test timeout. pre-existing for WithAutoLogin, remembered logins reach it now. the second Connect also overwrites c.conn and the first conn is never closed.
  • java has no failover dial bound: netty CONNECT_TIMEOUT_MILLIS = connectionTimeout or 3s, always on, tcp only, so a long connectionTimeout stretches every rotation.

Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread core/sdk/src/leader_aware.rs Outdated
Comment thread core/sdk/src/leader_aware.rs Outdated
Comment thread foreign/node/src/client/client.socket.ts
Comment thread foreign/node/src/client/client.connection.ts
Comment thread foreign/node/src/client/client.connection.ts
Comment thread foreign/node/src/client/client.connection.ts
Comment thread foreign/node/src/client/client.connection.ts
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 25, 2026
@numinnex

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants