Skip to content

fix(mm): close TOCTOU race in _restore_incomplete_installs#9142

Open
lstein wants to merge 7 commits into
mainfrom
fix/model-install-restore-race
Open

fix(mm): close TOCTOU race in _restore_incomplete_installs#9142
lstein wants to merge 7 commits into
mainfrom
fix/model-install-restore-race

Conversation

@lstein

@lstein lstein commented May 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes an intermittent unit test failure in test_model_install.py.

  • _restore_incomplete_installs snapshotted active sources under the lock, then iterated tmpdirs against the stale snapshot. A foreground import_model call landing during iteration could be missed, the same source enqueued twice, and the second download would fail with FileNotFoundError when trying to rename the .downloading file the first download had already moved.
  • Fix: move the membership check inside the lock immediately before _install_jobs.append(job) so check-and-append is atomic. Duplicate-tmpdir suppression within a single restore pass stays outside the lock since it doesn't race against import_model.

Fixes #9141.

Test plan

  • pytest tests/app/services/model_install/test_model_install.py::test_simple_download — ran 10× consecutively, all pass (was flaky ~30–50% on main)
  • pytest tests/app/services/model_install/ — 31/31 pass

🤖 Generated with Claude Code

The restore path snapshotted active sources under the lock, then
released the lock and iterated tmpdirs against the stale snapshot.
A foreground import_model call landing during iteration could append
a job the loop never saw, leading to a duplicate enqueue and a
FileNotFoundError when the second download tried to rename the
.downloading file the first had already moved.

Move the membership check inside the lock immediately before the
append so check-and-append is atomic. Fixes #9141.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added python PRs that change python files services PRs that change app services labels May 9, 2026
@lstein lstein added the 6.14.x label May 9, 2026
@lstein lstein moved this to 6.14.x Theme: LIBRARY UPDATES in Invoke - Community Roadmap May 9, 2026

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good when inspecting visually, but I'd also like to see a test added for this in tests/app/services/model_install/test_model_install.py, especially since it's a race condition. It may be a little challenging to get the test right, though:

  1. Create a tmp install dir with an active marker for a remote source.
  2. Monkeypatch _resume_remote_download() so restore pauses at a controlled point.
  3. Insert an active job for the same source into _install_jobs while restore is between marker parsing and the new locked duplicate check.
  4. Let restore continue.
  5. Assert restore did not append a duplicate job and did not enqueue/resume a second download.

That maps directly to the PR’s fix around invokeai/app/services/model_install/model_install_default.py:249.

The only slightly fiddly part is choosing the pause point. Because the fixed code does the duplicate check under _lock immediately before append, the test probably needs to monkeypatch either _guess_source() or _read_install_marker() to signal “restore has observed the marker”, then add the foreground job before restore reaches the locked block. That is manageable with two threading.Events, similar to the existing test_import_waits_for_startup_restore.

The test should not need real network downloads if _resume_remote_download() is monkeypatched to fail the test if called for the skipped duplicate.

@JPPhoto JPPhoto requested a review from Pfannkuchensack as a code owner July 10, 2026 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.x python PRs that change python files services PRs that change app services

Projects

Status: 6.14.x Theme: USER EXPERIENCE

Development

Successfully merging this pull request may close these issues.

Race condition in ModelInstallService._restore_incomplete_installs causes duplicate download enqueue

2 participants