Avoid cloud object listings during RocksDB Cloud startup

[thweetkomputer]

Context

During RocksDB Cloud failover, DBCloud::Open() completed in 364 ms but OpenDataStore took 70.272 s. The startup log contained two approximately 34-second gaps: one at SetOptions(disable_auto_compactions=false) and one at SetDBOptions(max_open_files=-1).

Both APIs persist a local OPTIONS file. RocksDB then cleans obsolete OPTIONS files through GetChildren(); after DBCloud::Open(), EloqDS had already restored skip_cloud_files_in_getchildren=false, so each cleanup issued an OBS ListObjectsV2 over the complete DB object prefix. OPTIONS files are local-only, so those cloud listings do not contribute to correctness.

Behavior before and after

Before: startup restores cloud-aware directory listing immediately after DBCloud::Open(). The two subsequent startup option updates can each list every cloud object, extending failover by roughly the size and latency of two full-prefix listings.

After: cloud listings remain skipped until both startup option updates finish. Normal cloud-aware GetChildren() behavior is restored before background work and warm-up resume. Runtime behavior and persisted options are unchanged.

Implementation

• Keep skip_cloud_files_in_getchildren=true through SetOptions() and SetDBOptions().
• Keep background flush and compaction paused while the mutable CloudFileSystem option is temporary.
• Restore the option on every post-open success and failure path.
• Resume background work only after restoring the normal cloud directory view.

Design decisions and alternatives

The change narrows an existing startup-only optimization instead of permanently disabling cloud listings. GetMutableCloudFileSystemOptions() exposes the live option object, and CloudFileSystemImpl::GetChildren() reads this flag on each call, so no reopen is required.

Leaving the flag permanently enabled was rejected because it would change runtime directory visibility. Restoring it immediately after DBCloud::Open() preserves the two expensive startup listings and does not solve the failover delay.

Test plan

• Unit/CTest coverage
• Parent-project integration or manual validation
• Formatting/build checks
• Recovery, compatibility, or performance validation, when relevant
• Documentation updated, when behavior changed

Commands and results:

clang-format-18 -i store_handler/eloq_data_store_service/rocksdb_cloud_data_store.cpp
# completed

git diff --check origin/main...HEAD
# passed

/usr/bin/c++ <bld-rocksdb-cloud data_substrate compile definitions/includes/flags, with third_party/src/rocksdb-cloud/include first> -fsyntax-only store_handler/eloq_data_store_service/rocksdb_cloud_data_store.cpp
# passed

cmake --build bld-rocksdb-cloud --target data_substrate --parallel 16
# not completed: the existing build tree uses a stale installed rocksdb-cloud
# header and fails on pre-existing code because CloudFileSystemOptions lacks
# publish_file_number_guard. The current rocksdb-cloud submodule header contains
# that member; the modified translation unit passes with the matching header.

A production-scale OBS failover timing test was not run in this workspace.

Risk assessment

Low and startup-scoped. The mutable flag is restored before background work resumes, avoiding concurrent background GetChildren() calls observing the temporary value. Failure paths also restore it. No transaction visibility, durability boundary, on-disk format, or cloud object lifecycle behavior changes.

The main residual risk is that an unrecognized synchronous startup operation between DB open and restoration requires cloud-only directory entries. Existing DB open already runs with listings skipped, and the added interval contains epoch validation plus local OPTIONS persistence.

Rollback plan

Revert this PR. That restores the previous timing of the option reset and the two cloud-aware startup cleanups; no data or configuration migration is required.

Reviewer guide

Review RocksDBCloudDataStore::OpenCloudDB() and verify these invariants:

1. skip_cloud_files_in_getchildren stays true across both startup option writes.
2. Every return after successful DB open restores the flag when needed.
3. Background work resumes only after the flag is false.
4. Normal runtime and warm-up run with cloud-aware listing restored.

Follow-up work

Run an OBS deployment failover benchmark and confirm the two approximately 34-second gaps disappear.

Summary by CodeRabbit

• Bug Fixes
  • Improved cloud-backed database startup recovery when initialization encounters an error.
  • Restored normal cloud directory listings after failed startup attempts.
  • Preserved configuration settings during database startup, improving consistency for subsequent operations.
  • Ensured background processing resumes with the correct cloud storage behavior after recovery.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 99b28d8c-1743-475e-ac30-236aef0eec85

📥 Commits

Reviewing files that changed from the base of the PR and between 5a76896 and f2154b2.

📒 Files selected for processing (1)

• store_handler/eloq_data_store_service/rocksdb_cloud_data_store.cpp

---

Walkthrough

StartDB documents the temporary cloud-listing state during OPTIONS rewrites. OpenCloudDB centralizes restoration of normal cloud-directory listing and applies it across initialization failure paths and successful startup.

Changes

Cloud listing restoration

Layer / File(s)	Summary
Startup listing state store_handler/eloq_data_store_service/rocksdb_cloud_data_store.cpp	Startup comments document the disabled cloud-file listing state. OpenCloudDB adds a reusable restoration callback.
Initialization restoration paths store_handler/eloq_data_store_service/rocksdb_cloud_data_store.cpp	Initialization failure paths invoke the restoration callback. Successful initialization restores normal listing before background flush and compaction work resumes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Mergeability Score: ⚪ Minimal · up to f2154

The change is localized to startup behavior and preserves normal runtime directory visibility; no actionable merge-blocking risk remains after normal checks and review.

Possibly related PRs

• eloqdata/tx_service#548: Changes related initialization and file-number handling in the same RocksDB cloud data store implementation.

Poem

A rabbit hops through cloud files bright,
Restoring listings to their usual sight.
If startup stumbles, cleanup runs,
Then flush and compaction greet the sun.
skip_cloud_files_in_getchildren takes flight!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the primary change: preventing cloud object listings during RocksDB Cloud startup.
Description check	✅ Passed	The description covers all required sections, explains the implementation and risks, and documents completed and skipped validation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/avoid-cloud-list-on-open

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}