Add dispatch: cron orchestration for the ingest pipeline - #9
Open
CodyCBakerPhD wants to merge 5 commits into
Open
Add dispatch: cron orchestration for the ingest pipeline#9CodyCBakerPhD wants to merge 5 commits into
CodyCBakerPhD wants to merge 5 commits into
Conversation
…ngest
Adds dispatch/dispatch.py, a generic entrypoint intended to be run on a
schedule by the self-hosted runner in data-ingest-runner. Per registered
project (dispatch/projects.yaml) it:
1. dandi-downloads the incoming dandiset from dandi.emberarchive.org
2. diffs discovered sessions against a per-project manifest
(<standardized_dir>/.ingest_state.json) to find unconverted sessions,
also forcing a full reprocess if the conversion script's own contents
(sha256) have changed since the manifest was last written
3. runs the lab's conversion command over new/changed sessions
4. dandi-uploads the standardized output
registry.py validates projects.yaml; state.py owns the manifest read/write.
Includes unit tests (registry validation, manifest round-trip, command
templating/dry-run with subprocess mocked out) wired into CI as a new
Dispatch job in test.yml. Kemere is registered as the first (and so far
only) project, with placeholder dandiset ids to be filled in once assigned.
CodyCBakerPhD
marked this pull request as ready for review
August 6, 2026 20:14
projects.yaml -> projects.json (stdlib json, drops the pyyaml dependency). Session discovery pulled out of the project registry into a new dispatch/sessions.json + sessions.py: a single glob per project didn't generalize (labs may need multiple raw subtrees or exclusions), and that shape can evolve independently of a project's dandiset ids/conversion command. sessions.json is keyed by lab, each entry an include-glob list (unioned) plus an optional exclude-glob list (matched by basename or path relative to the incoming project dir). dispatch.py now loads both registries and looks up each project's session spec by lab name (erroring per-project, not fatally, if one is missing). Tests updated/added for both files; all 21 pass, ruff clean.
CodyCBakerPhD
commented
Aug 6, 2026
Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>
Registers the ai_generated pytest marker repo-wide in the root pyproject.toml and applies it (via module-level pytestmark) to every test in dispatch/tests/, since that whole suite was AI-authored this session. Adds CLAUDE.md documenting the rule for future AI-agent sessions: mark AI-generated tests with this marker (module-level pytestmark by default, per-function @pytest.mark.ai_generated when a module mixes AI-generated and human-written tests), and register new suites against the same root pytest config rather than adding a competing one.
setuptools needed an explicit (empty) [tool.setuptools] py-modules = [] block to build a code-less, dependency-only package; hatchling doesn't need a placeholder section for that -- 'bypass-selection = true' under [tool.hatch.build.targets.wheel] says outright that the wheel ships no files, which is what py-modules = [] was standing in for anyway.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces
dispatch, a cron-driven orchestration system that automates the data ingest pipeline for registered labs. It coordinates downloading raw data from DANDI, detecting new sessions, running lab-specific conversion scripts, and uploading standardized output.Key Changes
dispatch.py: Main orchestration script that:
dandi downloaddandi upload--dry-run,--skip-download,--skip-upload, and--only <lab>flagsregistry.py: Project registry loader that:
projects.yamlto define lab configurationsProjectdataclass with script path resolutionstate.py: Per-project manifest manager that:
.ingest_state.jsonprojects.yaml: Registry configuration with Kemere lab as initial example, defining:
Comprehensive test suite (
dispatch/tests/):Documentation (README.md): Usage guide, layout explanation, credential setup, and project onboarding instructions
Environment declaration (
dispatch/envs/pyproject.toml): Minimal dependencies (pyyaml, pytest for tests)CI integration: Added
Dispatchjob to.github/workflows/test.ymlto run unit tests on every commitNotable Implementation Details
overwrite_flagto handle breaking changes{repo_root},{incoming_dir},{standardized_dir}substitutionhttps://claude.ai/code/session_01NbcD48t2AYkB1YdLA2zQCm