Skip to content

perf: index EXT4 FileTree children by name to avoid O(n^2) unpack#793

Open
kkilchrist wants to merge 1 commit into
apple:mainfrom
kkilchrist:perf/ext4-filetree-child-index
Open

perf: index EXT4 FileTree children by name to avoid O(n^2) unpack#793
kkilchrist wants to merge 1 commit into
apple:mainfrom
kkilchrist:perf/ext4-filetree-child-index

Conversation

@kkilchrist

@kkilchrist kkilchrist commented Jul 7, 2026

Copy link
Copy Markdown

What

FileTree.lookup resolved each path component by linearly scanning the node's children array. This changes the node's child storage to an OrderedDictionary<String, Ptr<FileTreeNode>> (from swift-collections, which is already a package dependency) keyed by name, so lookup resolves each component in O(1) while iteration keeps the existing insertion order.

Why

EXT4.Formatter.create calls lookup for every entry (and once per ancestor directory). With a linearly-scanned array, populating a directory with K entries is O(K²), which dominates unpack time for images with large directories (node_modules, /usr/lib, apt lists, …).

Design

childrenByName is the single source of truth; children remains available as an ordered view (.values), so existing iterate/map call sites are unchanged. Mutation goes through two helpers on FileTreeNodeaddChild(_:) and removeChild(named:) — used by every mutation site: Formatter.link, Formatter._unlink, Formatter.create, FileTreeNode.init, and the reader's tree build in EXT4+Reader. A single keyed structure also makes duplicate-name children unrepresentable.

(An earlier revision of this PR added a separate childIndex dictionary mirroring the children array; per review feedback it now uses a single OrderedDictionary instead — same lookup performance, less memory, and no two structures to keep in sync.)

Results

Measured on Apple Silicon unpacking louislam/uptime-kuma:1 (40,834 files / 438 MB):

  • Optimized build: ~15s → ~9s (~40% faster).
  • The relative improvement grows with directory size; in unoptimized/debug builds the absolute difference is far larger.

A single-directory microbenchmark (30,000 create calls into one directory, release build) runs in ~0.33s vs ~1.9s before this change, with peak RSS slightly lower than the array+index revision.

Testing

swift test --filter ContainerizationEXT4Tests — all pass (62 tests), including the whiteout, hardlink, symlink, and replace-semantics cases that exercise the mutated sites.

@kkilchrist kkilchrist force-pushed the perf/ext4-filetree-child-index branch 2 times, most recently from 6a5c989 to d196659 Compare July 7, 2026 01:52
@jglogan

jglogan commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@kkilchrist Did you consider replacing Array+Dictionary with OrderedDictionary from swift-collections? Would that improve performance while reducing additional memory consumption?

FileTree.lookup resolved each path component by linearly scanning the
node's `children` array. Because EXT4.Formatter.create calls lookup for
every entry (and once per ancestor directory), populating a directory
with K entries was O(K^2), which dominates unpack time for images with
large directories (node_modules, /usr/lib, apt lists, ...).

Store children in an OrderedDictionary<String, Ptr<FileTreeNode>>
(swift-collections, already a package dependency), keyed by name:
lookup resolves each component in O(1) while iteration keeps the
existing insertion order. `children` remains as an ordered view for
existing call sites; mutation goes through addChild/removeChild.

No behavior change; all ContainerizationEXT4Tests pass. Measured ~40%
faster unpack of louislam/uptime-kuma:1 (40,834 files / 438 MB) on Apple
Silicon in an optimized build (~15s -> ~9s); the relative win grows with
directory size. A 30,000-entry single-directory create microbenchmark
runs in ~0.33s vs ~1.9s before this change.
@kkilchrist kkilchrist force-pushed the perf/ext4-filetree-child-index branch from d196659 to 925d3c9 Compare July 16, 2026 17:43
@kkilchrist

kkilchrist commented Jul 16, 2026

Copy link
Copy Markdown
Author

@jglogan
I hadn't considered it. I kept the original change additive as an outside contributor. But you're right that OrderedDictionary is strictly better.

A small benchmark on my side shows OrderedDictionary has equivalent performance to and lower memory footprint than the Array + Index in my original proposal. (Array+Index / OrderedDictionary both ~0.33 s compared to ~1.9 s for upstream implementation).

Since ext4 materialization is one of the key differences vs overlayfs layer-composition, improving performance here will narrow the latency gap on first-pull cold starts.

PR updated with suggestion. Thanks for the feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants