feat(sdk): fail over to a surviving node when the current one dies - #3944
Open
numinnex wants to merge 13 commits into
Open
feat(sdk): fail over to a surviving node when the current one dies#3944numinnex wants to merge 13 commits into
numinnex wants to merge 13 commits into
Conversation
Contributor
Author
|
Merge after #3934 |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3944 +/- ##
============================================
- Coverage 84.03% 82.98% -1.06%
- Complexity 1358 1395 +37
============================================
Files 1215 1217 +2
Lines 170421 166509 -3912
Branches 138177 133794 -4383
============================================
- Hits 143218 138173 -5045
- Misses 23351 24228 +877
- Partials 3852 4108 +256
🚀 New features to boost your workflow:
|
hubcio
requested changes
Aug 25, 2026
hubcio
left a comment
Contributor
There was a problem hiding this comment.
cross-sdk:
- configured seeds (
failover_addresses) exist only in rust; go/java/c#/node keep a single seed, so a first connect on a dead configured node has nowhere to go. in scope or follow-up? - dial order: rust = current, roster, seeds; the four ports = current, configured seed, roster. every port dials a possibly dead configured node ahead of live survivors on every loss. pick one (rust's). same for reconnection disabled: rust bails on the first failed dial, go still sweeps everything.
StaleClientforget: rust/go/c# forget the remembered credentials, java never does, c# misses the outcome-unknown wrapper, go forgets only after the reconnection-disabled return. same contract in all five or in none.- empty description for a behaviour change across five sdks (remembered credentials, skipped
reestablish_after, the 2s dial bound, replay semantics). write the contract down. the c# readme (InitialDelay, "without credentials a reconnect cannot restore the session") and the python readme (reestablish_after) are now wrong too.
outside the diff:
- rust:
IggyClientBuilder::with_tcp()has nowith_failover_addressespassthrough, so seeds are only reachable viaTcpClient::create(config).max_retriesnow counts full sweeps but the builder doc still says retries. - go:
currentServerAddressis read withoutc.mtxinexchange(line 533) whiledialCandidatewrites it under the lock;time.Sleep(remaining)in the reestablish pause (line 926) ignoresctx. both pre-existing. - node: the
maxRetriesdoc inclient.type.tssays reconnection attempts, it now counts full passes; same for theDefaultReconnectOptioncomment. the readme has no failover mention.
Contributor
Author
|
/ready |
hubcio
requested changes
Aug 25, 2026
hubcio
left a comment
Contributor
There was a problem hiding this comment.
second pass at 4433f9a. most threads are fixed; what's below and the inline comments are still open.
- staleclient rule: replay the remembered login everywhere, like rust does now. an eviction is not caller intent. go, c# and java (with and without builder credentials) should match.
- PR description is still empty. write the contract down: remembered credentials, the 2s dial bound, the
reestablish_afterrotation, what is replayed after a reconnect and what is not. - seeds still exist only in rust, and there only via
TcpClient::create(config): nowith_failover_addressesonIggyClientBuilder::with_tcp(), no connection string option, cli and python can't set them. in scope or follow-up? - dial order still differs: rust = current, roster, seeds; go/java/c#/node = current, configured seed, roster, and the ports now pin their order with tests. pick one.
- reconnection disabled: rust, go and c# sweep every candidate once; java and node don't dial at all (
maxRetries0 /enabled = falseskips the loop). same rule everywhere. - docs: c# README lines 104, 111-112, 204-206 still say a hand-run login can't be restored; rust
max_retriesdocs (client_builder.rs:168,tcp_client_config_builder.rs:61) and the python config doc still say retries, one retry is a full sweep now. - go:
currentServerAddressread withoutc.mtxinexchange(tcp_core.go:542) andclientAddressin tcp_session_management.go:79 - both reported by-race. - go: concurrent requests failing during a reconnect with credentials hang the whole client.
skipAutoLoginOnceis client-global: a second failing request'sdisconnect()flips the state so twoConnects run, one eats the flag and the login replay's ownConnectauto-logins under its ownregisterMtx. 8 concurrent pings on a dropped socket withWithAutoLoginhung until the test timeout. pre-existing forWithAutoLogin, remembered logins reach it now. the secondConnectalso overwritesc.connand the first conn is never closed. - java has no failover dial bound: netty
CONNECT_TIMEOUT_MILLIS=connectionTimeoutor 3s, always on, tcp only, so a longconnectionTimeoutstretches every rotation.
Contributor
Author
|
/ready |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.