perf: index EXT4 FileTree children by name to avoid O(n^2) unpack#793
perf: index EXT4 FileTree children by name to avoid O(n^2) unpack#793kkilchrist wants to merge 1 commit into
Conversation
6a5c989 to
d196659
Compare
|
@kkilchrist Did you consider replacing Array+Dictionary with OrderedDictionary from swift-collections? Would that improve performance while reducing additional memory consumption? |
FileTree.lookup resolved each path component by linearly scanning the node's `children` array. Because EXT4.Formatter.create calls lookup for every entry (and once per ancestor directory), populating a directory with K entries was O(K^2), which dominates unpack time for images with large directories (node_modules, /usr/lib, apt lists, ...). Store children in an OrderedDictionary<String, Ptr<FileTreeNode>> (swift-collections, already a package dependency), keyed by name: lookup resolves each component in O(1) while iteration keeps the existing insertion order. `children` remains as an ordered view for existing call sites; mutation goes through addChild/removeChild. No behavior change; all ContainerizationEXT4Tests pass. Measured ~40% faster unpack of louislam/uptime-kuma:1 (40,834 files / 438 MB) on Apple Silicon in an optimized build (~15s -> ~9s); the relative win grows with directory size. A 30,000-entry single-directory create microbenchmark runs in ~0.33s vs ~1.9s before this change.
d196659 to
925d3c9
Compare
|
@jglogan A small benchmark on my side shows OrderedDictionary has equivalent performance to and lower memory footprint than the Array + Index in my original proposal. (Array+Index / OrderedDictionary both ~0.33 s compared to ~1.9 s for upstream implementation). Since ext4 materialization is one of the key differences vs overlayfs layer-composition, improving performance here will narrow the latency gap on first-pull cold starts. PR updated with suggestion. Thanks for the feedback! |
What
FileTree.lookupresolved each path component by linearly scanning the node'schildrenarray. This changes the node's child storage to anOrderedDictionary<String, Ptr<FileTreeNode>>(from swift-collections, which is already a package dependency) keyed by name, solookupresolves each component in O(1) while iteration keeps the existing insertion order.Why
EXT4.Formatter.createcallslookupfor every entry (and once per ancestor directory). With a linearly-scanned array, populating a directory with K entries is O(K²), which dominates unpack time for images with large directories (node_modules,/usr/lib, apt lists, …).Design
childrenByNameis the single source of truth;childrenremains available as an ordered view (.values), so existing iterate/map call sites are unchanged. Mutation goes through two helpers onFileTreeNode—addChild(_:)andremoveChild(named:)— used by every mutation site:Formatter.link,Formatter._unlink,Formatter.create,FileTreeNode.init, and the reader's tree build inEXT4+Reader. A single keyed structure also makes duplicate-name children unrepresentable.(An earlier revision of this PR added a separate
childIndexdictionary mirroring thechildrenarray; per review feedback it now uses a singleOrderedDictionaryinstead — same lookup performance, less memory, and no two structures to keep in sync.)Results
Measured on Apple Silicon unpacking
louislam/uptime-kuma:1(40,834 files / 438 MB):A single-directory microbenchmark (30,000
createcalls into one directory, release build) runs in ~0.33s vs ~1.9s before this change, with peak RSS slightly lower than the array+index revision.Testing
swift test --filter ContainerizationEXT4Tests— all pass (62 tests), including the whiteout, hardlink, symlink, and replace-semantics cases that exercise the mutated sites.