Skip to content

feat(simulator): drive the workload under network and replica faults - #3942

Open
krishvishal wants to merge 2 commits into
masterfrom
sim-workload-faults
Open

feat(simulator): drive the workload under network and replica faults#3942
krishvishal wants to merge 2 commits into
masterfrom
sim-workload-faults

Conversation

@krishvishal

@krishvishal krishvishal commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Faults:

  • Network parameters drawn from the seed (swarm), or taken as a named profile. Per-command delivery counters report what a run reached.
  • FaultInjector crashes and restarts replicas behind stability windows, so a crash lasts long enough to need repair.
  • Clients resend on timeout, rotating target. Unblocks the rest: with one in-flight slot per client, a dropped frame stranded it for the run, so primaries had to be spared and view changes were unreachable.
  • SimJournal honours a slot count, drains on checkpoint, advances its snapshot watermark. RangeEvicted and state transfer were unreachable while it answered a constant 0.
  • Restart carries a partition's log across the rebuild. Without it, a rebuilt partition reports commit_offset 0, tripping the monotonicity invariant on a harness artefact.

Oracles:

  • State checker folds every replica's committed metadata into one chain, asserting agreement where they overlap in both directions of “same op iff same prepare.” Reports ops seen on more than one replica, so a green run cannot mean “compared nothing.”
  • Dispatch shell: pre-commit denials, result-framed transients, PAT client wire shape, logout forwarding, eviction re-login.
  • settle_to_stable_view before the leader-relative asserts. A failed drain reports what is outstanding and what each live replica believes.

Two real bugs found:

  • Consumer-offset ops drew consumer_kind as u8::from(bool), so half of every request carried kind 0. WireConsumer rejects it, the partition plane drops the frame with no reply, and the client wedges.
  • Metadata backup gated prepare admission on its sequencer, not its journal head. on_start_view adopts the announced head before bodies arrive, so the backup refused the prepare that would heal its log. The op never reaches quorum and the client is never answered. This affected any rejoining replica.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 21, 2026
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.94081% with 463 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.21%. Comparing base (a65f177) to head (9f0df3e).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
core/simulator/src/bin/workload-fuzz.rs 0.00% 235 Missing ⚠️
core/simulator/src/workload/oracle.rs 46.66% 83 Missing and 5 partials ⚠️
core/simulator/src/packet.rs 19.14% 38 Missing ⚠️
core/simulator/src/workload/mod.rs 89.29% 35 Missing and 3 partials ⚠️
core/simulator/src/workload/state_checker.rs 78.65% 18 Missing and 1 partial ⚠️
core/simulator/src/deps.rs 83.87% 6 Missing and 4 partials ⚠️
core/partitions/src/iggy_partition.rs 59.09% 9 Missing ⚠️
core/partitions/src/journal.rs 0.00% 8 Missing ⚠️
core/shard/src/lib.rs 61.90% 7 Missing and 1 partial ⚠️
core/simulator/src/client.rs 88.00% 3 Missing ⚠️
... and 3 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3942      +/-   ##
============================================
+ Coverage     83.93%   84.21%   +0.28%     
  Complexity     1358     1358              
============================================
  Files          1213     1216       +3     
  Lines        168441   170526    +2085     
  Branches     135999   138209    +2210     
============================================
+ Hits         141373   143615    +2242     
+ Misses        23389    23047     -342     
- Partials       3679     3864     +185     
Components Coverage Δ
Rust Core 85.16% <78.96%> (+0.44%) ⬆️
Java SDK 66.67% <ø> (ø)
C# SDK 74.92% <ø> (-1.61%) ⬇️
Python SDK 90.13% <ø> (ø)
PHP SDK 84.48% <ø> (ø)
Node SDK 95.90% <ø> (ø)
Go SDK 68.29% <ø> (ø)
Files with missing lines Coverage Δ
core/binary_protocol/src/primitives/consumer.rs 100.00% <ø> (ø)
core/partitions/src/lib.rs 0.00% <ø> (ø)
core/simulator/src/executor/mod.rs 90.45% <100.00%> (ø)
core/simulator/src/lib.rs 96.19% <ø> (-0.92%) ⬇️
core/simulator/src/ready_queue.rs 85.29% <100.00%> (ø)
core/simulator/src/replica.rs 98.02% <100.00%> (+1.33%) ⬆️
core/simulator/src/seeds.rs 100.00% <100.00%> (ø)
core/simulator/src/workload/ids.rs 100.00% <100.00%> (ø)
core/simulator/src/workload/invariants.rs 96.22% <100.00%> (+0.07%) ⬆️
core/simulator/src/workload/ops/change_password.rs 94.44% <100.00%> (+30.55%) ⬆️
... and 38 more

... and 82 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants