[SYNPY-1895] Support Compute Tasks - #1438
Draft
andrewelamb wants to merge 11 commits into
Draft
Conversation
andrewelamb
marked this pull request as draft
July 29, 2026 19:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
Synapse now supports compute tasks:
CurationTasks whose properties describe a computation that a Synapse sub-worker performs, rather than metadata a contributor types into a Grid. Two kinds exist today:SampleSheetGenerationExecutionProperties— reshape the annotations behind a file-based task's FileView into a sample sheet, written to a destination RecordSet.RecordSetGenerationExecutionProperties— transform the source files in a folder into a CSV, written to a destination RecordSet.The client had no way to model or run these:
synapseclient.models.curationonly knewFileBasedMetadataTaskPropertiesandRecordBasedMetadataTaskProperties, and_create_task_properties_from_dict()raisedValueErroron anything else. The same was true forTaskExecutionDetails, where onlyGridExecutionDetailswas known. So a project containing a single compute task could not be listed or read at all — an unrecognizedconcreteTypeanywhere made every other field of that task unreadable, and this would recur every time the service adds a subtype ahead of a client release.{entityId}, and the compute endpoint is/curation/task/{taskId}/execute/async.create_progress_bar()passedSynapse.silentstraight to tqdm'sdisable. WhensilentisNone(its value beforelogin()configures logging), tqdm treatsNoneas "auto" and the bar misbehaves —progress_bar.descis not even guaranteed to exist, whichget_job_async()read unconditionally.Solution:
Model the new types.
CurationTaskPropertiesis now an ABC with aconcrete_typeproperty, andFileBasedMetadataTaskProperties/RecordBasedMetadataTaskPropertiesare subclasses of it alongside the two new execution-properties types.TaskExecutionDetailsgains the sameconcrete_typeproperty, plus a newExecutableTaskExecutionDetailsbase for details that support automated execution (SampleSheetGenerationExecutionDetails,RecordSetGenerationExecutionDetails), carryingasync_job_id,started_by,started_on,error_messageanderror_details.Forward compatibility instead of a hard failure. Both factories now fall back to
UnknownCurationTaskProperties/UnknownTaskExecutionDetailsrather than raising. These keep the raw response verbatim, expose theconcreteTypeSynapse sent as a read-only property, and round-trip unchanged on write — the status endpoint replacesexecutionDetailsrather than merging, so a read-modify-write that dropped unmodelled fields would delete them server-side. They are deliberately notExecutableTaskExecutionDetails, since an unrecognized type may not be executable.delete(delete_source=True)now raises a specificValueErrornaming the unrecognized type (and a separate one explaining that a compute task has no source of its own) instead of the old generic message.Run the task. New
CurationTask.execute()/execute_async()submits aComputeTaskExecutionRequest(a newAsynchronousCommunicator), waits for the job, and returns the resulting execution details — raisingSynapseErrorif the job fails or returns no details. Because a newly created task has no execution details and Synapse will not dispatch it without them,set_execution_details()/set_execution_details_async()was added to attach empty details of the matching type; the existingset_active_session_id_async()was refactored to delegate to it.Generalize async job URI resolution.
_resolve_async_job_uri()replaces the hardcoded{entityId}handling in bothsend_job_async()andget_job_async(). It discovers placeholders withstring.Formatter().parse()— the same parser that fills them — so a URI whose placeholder can't name a request key is rejected locally rather than sent to Synapse with braces intact, and each value is percent-encoded withquote(safe="")so a request value containing a slash or..cannot redirect the call to a different endpoint. Resolution moved out ofget_job_async()'s polling loop, which was recomputing the URI on every iteration.Progress bar fix.
create_progress_bar()coercesdisable=bool(silent), andget_job_async()readsdescviagetattr(..., None).mixins/CLAUDE.mddocuments the placeholder convention for anyone registering a new resource-scoped async job type.Testing:
unit_test_curation_async.py, +1079): the new properties/details dataclasses round-trip throughfill_from_dict()andto_synapse_request(); the factories dispatch onconcreteTypeand fall back to theUnknown*types; theUnknown*types preserve unmodelled fields and raiseValueErrorwhen serialized without aconcreteType;execute_async()success, job-failure and missing-details paths;set_execution_details_async()etag refresh; the newdelete(delete_source=True)error paths.unit_test_asynchronous_job.py, +177):_resolve_async_job_uri()across static URIs, placeholder substitution, percent-encoding of path-traversal-shaped values, unusable placeholders, missing request, and missing/Nonekeys.unit_test_transfer_bar.py, +24):disablecoercion whensilentisNone/True/False.test_curation_async.py, +321): create both kinds of compute task against Synapse, attach execution details, execute, and assert on the returned details, plus a validation-error case (test_execute_validation_error_async).conftest.pycleanup now also handlesJSONSchemaandSchemaOrganization, which these tests create.