Fix exponential dependency resolution on non-linear DAGs#1
Open
ocharles wants to merge 1 commit into
Open
Conversation
dependenciesWith enumerated every path from the start object (each object's direct successor names followed by the recursive expansion of each name), then deduplicated the result with the quadratic cleanLDups. The number of paths is exponential in the depth of a non-linear dependency graph, so an upgrade over a "diamond ladder" of just 46 migrations took ~55 seconds. Produce the identical ordering in linear time by scanning the virtual path enumeration from the end: walk successor names in reverse order, expand each name's subtree at most once (a repeated expansion occurs earlier in the enumeration, so it cannot contain a last occurrence), and emit each name the first time it is seen, i.e. at its last occurrence. Like the old code, the traversal is name-driven rather than node-driven, so behaviour is preserved even for graphs with duplicate object names. Also replace the assoc-list lookup in dependenciesWith and the list-membership filters in migrationsToApply/migrationsToRevert with Map/Set equivalents. Add a stress test that simulates the upgrade command over a diamond-ladder graph (two migrations per level, each depending on both migrations of the previous level). It now runs in ~3ms where it previously took ~55s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dependencies/reverseDependencies(viadependenciesWith) enumerate every path from the start object through the dependency graph, then deduplicate with the quadraticcleanLDups. The number of paths is exponential in the depth of a non-linear DAG, soupgrade— which runs this resolution once per missing migration — becomes unusably slow as soon as the migration graph isn't a straight line. This is what has forced us to keep the CircuitHub migration history fully serialized.Reproduced with the new stress test in this PR (a "diamond ladder": two migrations per level, each depending on both migrations of the previous level, simulating exactly what
upgradedoes):Fix
The old algorithm is
reverse . keepLastOccurrence . enumerateAllPaths. This PR produces the identical ordering in linear time by scanning the virtual path enumeration from the end: walk successor names in reverse order, expand each name's subtree at most once (a repeated expansion occurs earlier in the enumeration, so it cannot contain a last occurrence), and emit each name the first time it is seen — i.e. at its last occurrence.Ordering-preservation notes:
DependencyTestcases (which pin exact output orders) pass unchanged.Also swapped the assoc-list lookup in
dependenciesWithfor aMap(first entry wins on duplicate names, matchinglookup) and theelemfilters inmigrationsToApply/migrationsToRevertforSetmembership. The remaining growth at very large store sizes isupgraderecomputing an O(n) plan once per missing migration — inherent to the command's shape and negligible at realistic scale.Test
test/NonLinearStressTest.hssimulates theupgradecommand over the diamond ladder against an in-memory recording backend and asserts:Ladder depth is overridable via
STRESS_DEPTHfor experimentation.Note:
FilesystemParseTesthas 3 pre-existing failures when the suite is built against neweryaml/aeson(error-message string differences). They fail identically onmasterand are untouched by this PR.🤖 Generated with Claude Code