feat: implements keeper crate - state orchestrator

[bmuddha]

What changed

Added the keeper crate as the orchestration layer over accountsdb, ledger, caches, subscriptions, sysvar seeding, and account snapshot archival.

Why

Runtime code needs one consistency boundary that keeps account state, ledger records, cache state, and subscription updates aligned around execution and slot progress.

Closes #12 #41.

Impact

• Introduces Keeper, KeeperBuilder, storage directory parameters, and keeper-level errors.
• Initializes ledger services and account storage together, restoring accounts from archived snapshots when needed.
• Seeds clock, rent, and slot-hashes sysvars and maintains slot-based caches for signatures and blocks.

Reviewer notes

finalize seals the next superblock and archives the matching account snapshot. Restore keeps a backup before unpacking archived state so failed recovery can roll back.

Follow-up

Transaction processor and runtime integration can build on the keeper API.

[bmuddha]

• feat: add state replicator crate on top of the engine #25 : 2 dependent PRs (#37 , #55 )
• feat: adds top level engine crate for global orchestration #24
• feat: added internal built-in magic root program #23
• feat: implement transaction processor crate #22
• feat: implements keeper crate - state orchestrator #17 👈 (View in Graphite)
• feat: implement superblock based ledger crate #15
• feat: add optimized accountsdb crate #14
• feat: add nucleus engine crate #18
• feat: add svm crate #21
• feat: add program runtime crate #20
• feat: add transaction context crate #19
• feat: forked transaction-view for increased limits #33
• feat: add customized solana-account crate #13
• chore: import upstream Solana runtime crates #31
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8790b830-0cdb-43fa-b1a3-e25d2a09c1bc

📥 Commits

Reviewing files that changed from the base of the PR and between e5f37e1 and 18bd30e.

📒 Files selected for processing (14)

• Cargo.toml
• keeper/Cargo.toml
• keeper/README.md
• keeper/src/accessor.rs
• keeper/src/builder.rs
• keeper/src/cache.rs
• keeper/src/error.rs
• keeper/src/lib.rs
• keeper/src/metrics.rs
• keeper/src/subscriptions.rs
• keeper/src/testkit.rs
• keeper/src/tests/caches.rs
• keeper/src/tests/recovery.rs
• keeper/src/tests/subscriptions.rs

🚧 Files skipped from review as they are similar to previous changes (10)

• keeper/Cargo.toml
• Cargo.toml
• keeper/src/error.rs
• keeper/src/metrics.rs
• keeper/src/tests/caches.rs
• keeper/src/cache.rs
• keeper/src/builder.rs
• keeper/src/lib.rs
• keeper/src/tests/recovery.rs
• keeper/src/accessor.rs

---

📝 Walkthrough

Walkthrough

Added the keeper crate as the coordination boundary for accounts, transactions, blocks, superblocks, recovery, caches, subscriptions, and test utilities. Added the v42 calculator SBF program with postfix evaluation, nested CPI, transfers, and stable errors.

Changes

Keeper engine

Layer / File(s)	Summary
v42 calculator program programs/v42-calculator-program/...	Added the calculator SBF crate, entrypoint dispatch, postfix evaluator, nested self-CPI, account and clock operations, transfer processing, checked arithmetic, and stable custom errors.
Keeper packaging and testkit integration Cargo.toml, keeper/Cargo.toml, keeper/build.rs, keeper/README.md, keeper/src/testkit.rs	Added workspace integration, optional v42 SBF builds, testkit constructors, transaction and account helpers, snapshot helpers, corruption helpers, and Keeper documentation.
Keeper startup and durable state keeper/src/error.rs, keeper/src/builder.rs, keeper/src/lib.rs, keeper/src/metrics.rs	Added Keeper construction, startup account seeding, accounts database validation, snapshot restoration and archival, synchronization, reset handling, typed errors, and Prometheus metrics.
Caches and subscriptions keeper/src/cache.rs, keeper/src/subscriptions.rs	Added account-load coordination, expiring caches, block tracking, keyed subscriptions, bounded channels, periodic cleanup, and shutdown handling.
Execution coordination API keeper/src/accessor.rs, keeper/src/util.rs, keeper/src/lib.rs	Added account, transaction, block, and superblock accessors. Added ledger request handling, execution commit conversion, state transition persistence, replay, finalization, and synchronization operations.
Keeper behavioral validation keeper/src/tests/*	Added tests for cache behavior, concurrent account loading, startup seeding, snapshot recovery, subscription lifecycle, transaction deduplication, and cached transaction status.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Runtime
  participant Keeper
  participant AccountsDB
  participant Ledger
  participant Subscribers
  Runtime->>Keeper: append and execute transaction
  Keeper->>AccountsDB: load and update account state
  Keeper->>Ledger: persist transaction and execution event
  Ledger-->>Keeper: return commit result
  Keeper->>Subscribers: publish status, logs, and account updates
  Subscribers-->>Runtime: deliver notifications

Loading

Possibly related PRs

• magicblock-labs/magicblock-engine#18: Provides the runtime, ledger, shutdown, TLS, metrics, notifier, and testkit primitives used by Keeper.
• magicblock-labs/magicblock-engine#22: Uses Keeper APIs for account loading, execution, and transaction persistence.
• magicblock-labs/magicblock-engine#25: Builds on Keeper ledger, block, snapshot, and synchronization APIs.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: adding the keeper crate as a state orchestrator.
Description check	✅ Passed	The description directly explains the keeper crate, its coordination role, recovery behavior, and intended runtime integration.
Linked Issues check	✅ Passed	The PR adds the keeper API and coordination paths that align accountsdb state with ledger execution and slot data as required by [#12].
Out of Scope Changes check	✅ Passed	The calculator program, testkit, caches, subscriptions, and recovery tests support keeper construction and validation rather than unrelated functionality.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch keeper

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[bmuddha]

@CodeRabbit review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 12

🧹 Nitpick comments (1)

keeper/src/lib.rs (1)

213-217: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

A failed thread spawn leaves the temporary archive file behind.

dst is created at Line 215 before the thread spawn. If thread::Builder::spawn returns an error at Line 217, the .tmp file stays in the superblock directory. The next finalize_superblock truncates it, so this is not a correctness failure, but the stale file remains after a terminal spawn failure.

Remove tmp on the spawn error path.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@keeper/src/lib.rs` around lines 213 - 217, Update the thread creation flow
around thread::Builder::new().name("snapshot-archiver") so a failed spawn
removes the temporary archive at tmp before propagating the spawn error.
Preserve the existing successful spawn behavior and error propagation.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@keeper/src/accessor.rs`:
- Around line 214-215: Make the ledger appender transition at
accessor.rs:214-215 recoverable with the accountsdb mutation by changing the
Event::Execution flow around commit_state_transitions to use a durable
pending-transition or rollback/replay protocol, and publish the event only after
both stores commit. In accessor.rs:309-319, defer block-cache publication and
block subscriptions until update_sysvars and set_slot succeed, using the same
recovery protocol; both sites are in the accessor transition flow and must
preserve consistent ledger and account state after failures.

In `@keeper/src/builder.rs`:
- Around line 270-275: Update the restored-store failure branch in the
validation match to return a corruption-specific error such as
AccountsDBError::Corruption instead of SnapshotError::Missing, while preserving
the existing backup and logging behavior.
- Around line 256-286: In the accountsdb method, explicitly drop the AccountsDB
instance after saving the backup and before calling self.unarchive(ledger),
ensuring the LMDB environment and mapped storage are closed before unpacking the
restored snapshot. Preserve the existing restore and error-handling flow.
- Around line 196-207: Update the SlotHashes construction in the surrounding
builder flow to initialize it from the retained blocks returned by
handle.recv_timeout().await??, rather than seeding it with [Default::default();
SLOTHASH_ENTRIES]. Add each retained block’s slot and hash to the resulting
SlotHashes and preserve the existing account creation and last_block updates.
- Around line 161-170: The program-account construction loop in the builder must
account for the loader-v4 state layout: prepend the required LoaderV4State
header before each ELF (or switch to a loader matching the raw ELF layout), and
compute rent from the complete account data size. Preserve the loader_v4 owner
and executable configuration while ensuring the seeded data begins with valid
loader-v4 state.

In `@keeper/src/cache.rs`:
- Around line 40-48: Enforce the documented 256-slot minimum for the cache
capacity used by Cache::new and the lru_capacity configuration path: reject
values below 256 with a startup error before constructing HashCache, or
consistently update the configuration documentation and default to a lower
minimum. Preserve valid capacities and ensure the behavior matches the chosen
documented contract.

In `@keeper/src/lib.rs`:
- Around line 180-188: Update the final-shutdown logic in sync so every ledger
reader receives its own shutdown signal; do not rely on sending multiple
ReadRequest::Shutdown messages through the shared MPMC channel, since one reader
may consume more than one. Use the existing per-reader or broadcast mechanism if
available, and preserve the subsequent superblocks and accounts database
synchronization.

In `@keeper/src/metrics.rs`:
- Around line 69-72: Update the rustdoc for the metrics `init` function to
describe only one-time metrics registration via
`METRICS.get_or_init(Default::default)`. Remove the claim that gauges are seeded
from current caches, leaving the implementation unchanged.
- Around line 79-82: Update account_cache_eviction() to decrement
m.account_cache_entries when an account cache entry is evicted, while preserving
its existing eviction-counter increment. Keep account_cache_insert() increasing
the same gauge so ACCOUNT_CACHE_ENTRIES reflects current occupancy.

In `@keeper/src/subscriptions.rs`:
- Around line 43-50: Correct the rustdoc comments on the subscription fields:
end the `programs` description with a period instead of a semicolon, and update
the `blocks` description to state that it broadcasts newly committed `Block`
values rather than slots. Leave the field types and other comments unchanged.
- Around line 82-88: Update Subscriptions::send so sending and conditional
removal occur under the same bucket lock, preventing subscribe from inserting
between the send and removal. Prefer remove_if_sync with a predicate that sends
the cloned value and removes the channel when sending fails or oneshot is true;
otherwise use the existing send_sync/read_sync path while preserving atomic
removal semantics.

In `@keeper/src/testkit.rs`:
- Around line 3-5: Update the module-level rustdoc describing the Keeper test
configuration to state the values actually used by keeper_builder: 100 ms
blocktime and superblock 4, preserving the existing description of the other
parameters.

---

Nitpick comments:
In `@keeper/src/lib.rs`:
- Around line 213-217: Update the thread creation flow around
thread::Builder::new().name("snapshot-archiver") so a failed spawn removes the
temporary archive at tmp before propagating the spawn error. Preserve the
existing successful spawn behavior and error propagation.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9274987e-68ef-495d-9594-3228059455ce

📥 Commits

Reviewing files that changed from the base of the PR and between 573431d and e5f37e1.

📒 Files selected for processing (23)

• Cargo.toml
• keeper/Cargo.toml
• keeper/README.md
• keeper/build.rs
• keeper/src/accessor.rs
• keeper/src/builder.rs
• keeper/src/cache.rs
• keeper/src/error.rs
• keeper/src/lib.rs
• keeper/src/metrics.rs
• keeper/src/subscriptions.rs
• keeper/src/testkit.rs
• keeper/src/tests/caches.rs
• keeper/src/tests/mod.rs
• keeper/src/tests/recovery.rs
• keeper/src/tests/subscriptions.rs
• keeper/src/util.rs
• programs/v42-calculator-program/Cargo.toml
• programs/v42-calculator-program/README.md
• programs/v42-calculator-program/src/calculator.rs
• programs/v42-calculator-program/src/error.rs
• programs/v42-calculator-program/src/lib.rs
• programs/v42-calculator-program/src/transfer.rs