Skip to content

[SYNPY-1895] Support Compute Tasks - #1438

Draft
andrewelamb wants to merge 11 commits into
developfrom
SYNPY-1895
Draft

[SYNPY-1895] Support Compute Tasks#1438
andrewelamb wants to merge 11 commits into
developfrom
SYNPY-1895

Conversation

@andrewelamb

@andrewelamb andrewelamb commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Problem:

Synapse now supports compute tasks: CurationTasks whose properties describe a computation that a Synapse sub-worker performs, rather than metadata a contributor types into a Grid. Two kinds exist today:

  • SampleSheetGenerationExecutionProperties — reshape the annotations behind a file-based task's FileView into a sample sheet, written to a destination RecordSet.
  • RecordSetGenerationExecutionProperties — transform the source files in a folder into a CSV, written to a destination RecordSet.

The client had no way to model or run these:

  • synapseclient.models.curation only knew FileBasedMetadataTaskProperties and RecordBasedMetadataTaskProperties, and _create_task_properties_from_dict() raised ValueError on anything else. The same was true for TaskExecutionDetails, where only GridExecutionDetails was known. So a project containing a single compute task could not be listed or read at all — an unrecognized concreteType anywhere made every other field of that task unreadable, and this would recur every time the service adds a subtype ahead of a client release.
  • There was no way to start the execution job. The async job machinery could only resolve URIs containing {entityId}, and the compute endpoint is /curation/task/{taskId}/execute/async.
  • create_progress_bar() passed Synapse.silent straight to tqdm's disable. When silent is None (its value before login() configures logging), tqdm treats None as "auto" and the bar misbehaves — progress_bar.desc is not even guaranteed to exist, which get_job_async() read unconditionally.

Solution:

Model the new types. CurationTaskProperties is now an ABC with a concrete_type property, and FileBasedMetadataTaskProperties / RecordBasedMetadataTaskProperties are subclasses of it alongside the two new execution-properties types. TaskExecutionDetails gains the same concrete_type property, plus a new ExecutableTaskExecutionDetails base for details that support automated execution (SampleSheetGenerationExecutionDetails, RecordSetGenerationExecutionDetails), carrying async_job_id, started_by, started_on, error_message and error_details.

Forward compatibility instead of a hard failure. Both factories now fall back to UnknownCurationTaskProperties / UnknownTaskExecutionDetails rather than raising. These keep the raw response verbatim, expose the concreteType Synapse sent as a read-only property, and round-trip unchanged on write — the status endpoint replaces executionDetails rather than merging, so a read-modify-write that dropped unmodelled fields would delete them server-side. They are deliberately not ExecutableTaskExecutionDetails, since an unrecognized type may not be executable. delete(delete_source=True) now raises a specific ValueError naming the unrecognized type (and a separate one explaining that a compute task has no source of its own) instead of the old generic message.

Run the task. New CurationTask.execute() / execute_async() submits a ComputeTaskExecutionRequest (a new AsynchronousCommunicator), waits for the job, and returns the resulting execution details — raising SynapseError if the job fails or returns no details. Because a newly created task has no execution details and Synapse will not dispatch it without them, set_execution_details() / set_execution_details_async() was added to attach empty details of the matching type; the existing set_active_session_id_async() was refactored to delegate to it.

Generalize async job URI resolution. _resolve_async_job_uri() replaces the hardcoded {entityId} handling in both send_job_async() and get_job_async(). It discovers placeholders with string.Formatter().parse() — the same parser that fills them — so a URI whose placeholder can't name a request key is rejected locally rather than sent to Synapse with braces intact, and each value is percent-encoded with quote(safe="") so a request value containing a slash or .. cannot redirect the call to a different endpoint. Resolution moved out of get_job_async()'s polling loop, which was recomputing the URI on every iteration.

Progress bar fix. create_progress_bar() coerces disable=bool(silent), and get_job_async() reads desc via getattr(..., None).

mixins/CLAUDE.md documents the placeholder convention for anyone registering a new resource-scoped async job type.

Testing:

  • Unit tests (unit_test_curation_async.py, +1079): the new properties/details dataclasses round-trip through fill_from_dict() and to_synapse_request(); the factories dispatch on concreteType and fall back to the Unknown* types; the Unknown* types preserve unmodelled fields and raise ValueError when serialized without a concreteType; execute_async() success, job-failure and missing-details paths; set_execution_details_async() etag refresh; the new delete(delete_source=True) error paths.
  • Unit tests (unit_test_asynchronous_job.py, +177): _resolve_async_job_uri() across static URIs, placeholder substitution, percent-encoding of path-traversal-shaped values, unusable placeholders, missing request, and missing/None keys.
  • Unit tests (unit_test_transfer_bar.py, +24): disable coercion when silent is None/True/False.
  • Integration tests (test_curation_async.py, +321): create both kinds of compute task against Synapse, attach execution details, execute, and assert on the returned details, plus a validation-error case (test_execute_validation_error_async). conftest.py cleanup now also handles JSONSchema and SchemaOrganization, which these tests create.

@andrewelamb
andrewelamb requested a review from a team as a code owner July 29, 2026 19:53
@andrewelamb
andrewelamb marked this pull request as draft July 29, 2026 19:54
@thomasyu888 thomasyu888 changed the title Synpy 1895 [SYNPY-1895] Support Compute Tasks Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant