HDDS-16220. Reduce OM bucket write-lock hold time during FSO directory purge

[yandrey321]

What changes were proposed in this pull request?

DirectoryDeletingService (DDS) drains the FSO deleted-directory backlog by
submitting PurgeDirectories transactions to the OM. Each transaction is
applied on the single OM state-machine thread while holding the bucket write
lock. Two aspects of the old path let a background purge stall interactive
writes and reads:

1. Wide lock hold time. OMDirectoriesPurgeRequestWithFSO.validateAndUpdateCache
   did all of its work — protobuf field extraction, RocksDB key-string
   construction, replicated block-size summation, hsync client-id lookup — while
   holding the bucket write lock.

2. Cross-bucket lock stacking. A single purge transaction could carry paths
   from multiple buckets, so applying it acquired multiple bucket write locks,
   blocking read RPCs (lookupKey, getFileStatus, listStatus) on buckets
   unrelated to the deletion.

This PR addresses both:

Two-phase apply in OMDirectoriesPurgeRequestWithFSO

• Phase 1 (no lock): iterate every PurgePathRequest and precompute each
  entry's delete key, path key, replicated byte size, hsync client id, and
  prepared sub-dir/sub-file tombstones into a PurgeApplyState.
• Phase 2 (bucket write lock): re-validate the bucket object id (snapshot
  chain unchanged), then apply the precomputed cache tombstones, quota deltas,
  and hsync open-key cleanup. Snapshot-namespace copyObject work is moved out
  of the locked region.

Per-bucket grouping in DirectoryDeletingService

• submitPurgePathsWithBatching groups PurgePathRequests by
  (volumeId, bucketId) before submission, so each submitted transaction
  carries paths from exactly one bucket and the apply side takes exactly one
  bucket write lock. Byte-limit batching is preserved within each bucket group.

Wire format is unchanged — no protobuf, RPC, or RocksDB-schema changes.

Generated-by: Claude Code (claude-opus-4-8)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16220

How was this patch tested?

CI: https://github.com/yandrey321/ozone/actions/runs/32983050746/job/98250638151

• TestOMDirectoriesPurgeRequestAndResponse#testLightParsePreservesReplicatedByteSumAndHsyncForSubFiles
  — proves the lock-free phase-1 light parse yields the exact same replicated
  byte sum and hsync open-key handling as the original full OmKeyInfo parse
  (bucket usedBytes decrements to precisely zero).
• TestDirectoryDeletingService#testPurgeDirectoriesGroupedByBucketPerTransaction
  — interleaves paths from two buckets and asserts one bucket per submitted
  PurgeDirectories transaction, order-independently.

Benchmark runs an interactive workload (create/mkdir/rename/filewrite plus reads) against
a live FSO directory purge and reports p50/p99 per operation, first with no
deletion running (control) and then during the drain (under
load).

Configuration: 180k-file delete backlog across 4 buckets, 14 client threads for mixed workload, 32 MB
Ratis appender byte limit. Baseline (pre-change) and fix measured back-to-back.

p99 latency, control (no purge running) — baseline vs fix:

Operation	Baseline	Fix
create	159.9 ms	77.4 ms
mkdir	56.3 ms	41.0 ms
rename	47.8 ms	39.5 ms
filewrite	1213.1 ms	530.3 ms
fileread	49.6 ms	65.6 ms
getfilestatus	3.58 ms	5.38 ms
liststatus	21.96 ms	21.87 ms
infobucket	1.02 ms	1.22 ms
getkeyinfo	2.26 ms	2.09 ms

p99 latency, under load (during the purge drain) — baseline vs fix:

Operation	Baseline	Fix	Fix vs baseline
create	579.8 ms	85.2 ms	6.8× lower
mkdir	62.8 ms	50.4 ms	1.25× lower
rename	55.2 ms	67.0 ms	~noise
filewrite	938.2 ms	523.0 ms	1.79× lower
fileread	33.1 ms	54.6 ms	~noise
getfilestatus	1.55 ms	2.20 ms	sub-ms
liststatus	10.58 ms	15.84 ms	sub-ms/noise
infobucket	0.69 ms	1.05 ms	sub-ms
getkeyinfo	0.95 ms	1.23 ms	sub-ms

Interactive ops completed during the drain window (effectively a fixed ~20 s
observation window):

	Baseline	Fix
Ops completed under load	4917	5466 (+11%)
Drain window	19.66 s	19.93 s
Control batch (fixed, 14 × 400)	5600	5600

[yandrey321]

flame-mixed-all.html
flame-mixed-bythread.html
flame-writecommit.html

[rich7420]

@yandrey321 thanks for the patch!

[Copilot]

Pull request overview

This PR targets OM write-path latency during FSO directory purges by reducing bucket write-lock hold time in the purge apply handler and preventing multi-bucket lock stacking by ensuring purge transactions are submitted per bucket. It does so by splitting the purge apply into a lock-free “prepare” phase and a lock-held “apply” phase, and by grouping DirectoryDeletingService submissions by (volumeId, bucketId).

Changes:

• Refactors OMDirectoriesPurgeRequestWithFSO.validateAndUpdateCache into a two-phase flow (precompute outside the bucket lock; apply tombstones/quota/hsync cleanup under the lock).
• Updates DirectoryDeletingService batching to group purge paths per bucket so each PurgeDirectories transaction takes only one bucket write lock.
• Adds/extends unit tests plus new benchmark-tagged micro/e2e benchmarks (and a small CodecBuffer helper to disable leak detection for perf measurements).

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestDirectoryDeletingService.java	Adds a unit test asserting one bucket per submitted purge transaction.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMDirectoriesPurgeRequestAndResponse.java	Adds a regression test validating light-parse equivalence for replicated bytes + hsync open-key handling.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMDirectoriesPurgeApplyPerf.java	New benchmark-tagged microbenchmark for purge apply-thread time.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java	Groups purge path submissions by bucket while preserving byte-limit batching per bucket.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMDirectoriesPurgeRequestWithFSO.java	Implements two-phase purge apply, adds light-parse helpers, memoizes bucket-info lookups, and batches quota/metric mutations.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMMetrics.java	Adjusts internal delete metric increment to accept a count for bulk updates.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestOmMixedWorkloadUnderDeletionBench.java	New benchmark-tagged end-to-end mixed workload benchmark under deletion load (optional profiling support).
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/CodecBuffer.java	Adds disableLeakDetection() to revert leak-detecting buffer construction for perf-sensitive benchmarks.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jojochuang]

Pre-PR vs post-PR: bucket lock behavior during FSO directory purge

Two distinct lock-hold problems are easy to conflate because both show up as long OM apply times, but they have different causes and different fixes.

1. Large single-bucket batches (still a hotspot, but cheaper)

Before this PR, a single PurgeDirectories transaction for one bucket could contain many sub-dirs/sub-files (e.g. one deleted directory tree). On apply, OMDirectoriesPurgeRequestWithFSO held that bucket's write lock for the entire loop, including:

• full OmKeyInfo.getFromProtobuf() per entry
• sumBlockLengths() on parsed keys
• hsync / open-key lookup
• per-entry getBucketInfo() and quota decrements
• cache tombstones

So large single-bucket batches already caused long write-lock holds on that bucket. This PR does not split those batches further (HDDS-16297), but it does move parse/precompute outside the lock and keeps only tombstones + batched quota work under the lock. Same batch size, shorter hold.

2. Multiple buckets in one transaction (fixed by this PR, AOS path)

Terminology matters here:

• One PurgePathRequest is always one bucket (one volumeId/bucketId).
• One PurgeDirectoriesRequest / OMDirectoriesPurgeRequestWithFSO apply can contain many PurgePathRequests.

Before this PR, DirectoryDeletingService.submitPurgePathsWithBatching packed paths purely by Ratis byte limit, with no bucket boundary. On the AOS path (snapshotId == null), DDS walks the global deleted-dir table and can accumulate paths from different buckets in one worker iteration; the code even notes AOS "could process multiple buckets in one iteration." Those mixed-bucket paths could land in one submitted transaction.

On apply, getBucketLockKeySet() collects every distinct bucket in the request and acquireWriteLocks(BUCKET_LOCK, bucketLockKeys) held all of those write locks for the whole apply loop. A background purge could therefore block read RPCs on unrelated buckets while backlog was drained elsewhere.

Snapshot deep-clean was already single-bucket end-to-end: the deleted-dir supplier is scoped to one (volume, bucket) per snapshot task.

This PR groups DDS submissions by (volumeId, bucketId) before byte-limit batching, so each submitted AOS transaction touches one bucket and apply takes one write lock.

Summary

Scenario	Before	After
Large batch, one bucket	Long write-lock hold (parse + apply under lock)	Still one lock per txn, but shorter hold (apply-only under lock)
One txn spanning multiple buckets (AOS)	Multiple write locks held together	Fixed — one bucket per txn
Snapshot deep-clean	Single bucket (unchanged)	Single bucket (unchanged)
Very large single PurgePathRequest	Long single-bucket hold	Still possible; not split yet (HDDS-16297)

The remaining hotspot called out in review is the first and last rows: dense single-bucket work is cheaper now, but a very large purge path can still hold one bucket write lock for the full apply phase until HDDS-16297 lands.

[rich7420]

Could we drop apply-side chunking here? This patch does not chunk a PurgePathRequest during apply; HDDS-16297 tracks DDS-side splitting.