Skip to content

Fix exponential dependency resolution on non-linear DAGs#1

Open
ocharles wants to merge 1 commit into
masterfrom
fix-nonlinear-dag-exponential-blowup
Open

Fix exponential dependency resolution on non-linear DAGs#1
ocharles wants to merge 1 commit into
masterfrom
fix-nonlinear-dag-exponential-blowup

Conversation

@ocharles

@ocharles ocharles commented Jul 8, 2026

Copy link
Copy Markdown

Problem

dependencies/reverseDependencies (via dependenciesWith) enumerate every path from the start object through the dependency graph, then deduplicate with the quadratic cleanLDups. The number of paths is exponential in the depth of a non-linear DAG, so upgrade — which runs this resolution once per missing migration — becomes unusably slow as soon as the migration graph isn't a straight line. This is what has forced us to keep the CircuitHub migration history fully serialized.

Reproduced with the new stress test in this PR (a "diamond ladder": two migrations per level, each depending on both migrations of the previous level, simulating exactly what upgrade does):

migrations before after
46 (ladder depth 22) 54.9 s 3 ms
402 (depth 200) — (hours) 0.3 s
2002 (depth 1000) ~10 s

Fix

The old algorithm is reverse . keepLastOccurrence . enumerateAllPaths. This PR produces the identical ordering in linear time by scanning the virtual path enumeration from the end: walk successor names in reverse order, expand each name's subtree at most once (a repeated expansion occurs earlier in the enumeration, so it cannot contain a last occurrence), and emit each name the first time it is seen — i.e. at its last occurrence.

Ordering-preservation notes:

  • All existing DependencyTest cases (which pin exact output orders) pass unchanged.
  • Like the old code, the traversal is name-driven rather than graph-node-driven: the old implementation recursed by looking successor names back up in the name map (first match wins). This matters for the existing test case with duplicate object ids, which fails under a node-driven traversal.

Also swapped the assoc-list lookup in dependenciesWith for a Map (first entry wins on duplicate names, matching lookup) and the elem filters in migrationsToApply/migrationsToRevert for Set membership. The remaining growth at very large store sizes is upgrade recomputing an O(n) plan once per missing migration — inherent to the command's shape and negligible at realistic scale.

Test

test/NonLinearStressTest.hs simulates the upgrade command over the diamond ladder against an in-memory recording backend and asserts:

  • every migration is applied exactly once,
  • every dependency is applied before its dependents,
  • the whole upgrade finishes within 10 s (was 54.9 s before the fix).

Ladder depth is overridable via STRESS_DEPTH for experimentation.

Note: FilesystemParseTest has 3 pre-existing failures when the suite is built against newer yaml/aeson (error-message string differences). They fail identically on master and are untouched by this PR.

🤖 Generated with Claude Code

dependenciesWith enumerated every path from the start object (each
object's direct successor names followed by the recursive expansion of
each name), then deduplicated the result with the quadratic cleanLDups.
The number of paths is exponential in the depth of a non-linear
dependency graph, so an upgrade over a "diamond ladder" of just 46
migrations took ~55 seconds.

Produce the identical ordering in linear time by scanning the virtual
path enumeration from the end: walk successor names in reverse order,
expand each name's subtree at most once (a repeated expansion occurs
earlier in the enumeration, so it cannot contain a last occurrence),
and emit each name the first time it is seen, i.e. at its last
occurrence. Like the old code, the traversal is name-driven rather
than node-driven, so behaviour is preserved even for graphs with
duplicate object names.

Also replace the assoc-list lookup in dependenciesWith and the
list-membership filters in migrationsToApply/migrationsToRevert with
Map/Set equivalents.

Add a stress test that simulates the upgrade command over a
diamond-ladder graph (two migrations per level, each depending on both
migrations of the previous level). It now runs in ~3ms where it
previously took ~55s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant