Skip to content

[SYNPY-1840] Add ability to set column order when creating file and record based tasks. - #1443

Merged
andrewelamb merged 9 commits into
developfrom
SYNPY-1840
Aug 13, 2026
Merged

[SYNPY-1840] Add ability to set column order when creating file and record based tasks.#1443
andrewelamb merged 9 commits into
developfrom
SYNPY-1840

Conversation

@andrewelamb

@andrewelamb andrewelamb commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Problem:

The curator metadata task creators gave callers no control over the left-to-right order of the columns that contributors see in the grid.

  • create_record_based_metadata_task built its CSV template from the JSON Schema properties in whatever order extract_schema_properties_from_web returned them, and only moved the upsert_keys to the front.
  • create_file_based_metadata_task hard-coded a single ordering in _create_json_schema_entity_view: name, id, createdBy were reordered to positions 0, 1, 2 and everything else kept the order Synapse gave it.

JSON Schema property order is not reliably preserved by downstream applications, so there was no dependable way to make contributor-relevant metadata appear first. Pinning createdBy in third position made this worse for file-based tasks — a system column was always placed ahead of every curation column.

Solution:

Add an optional keyword-only column_order: list[str] | None parameter to both create_record_based_metadata_task and create_file_based_metadata_task.

Ordering semantics are shared by both functions and live in two new helpers in synapseclient/extensions/curator/utils.py:

  • validate_column_order_list() — the cheap shape check (is a list, entries are non-empty strings, no duplicates). Called at the top of both public functions so malformed input fails before any entity is created in Synapse.
  • resolve_column_order_list() — produces the final order: pinned columns, then the requested columns that were not already pinned, then the remaining available columns in their existing relative order. A column never appears twice, so naming a pinned column is a no-op rather than a duplication. This is also where the unknown-column check happens.

The result is that callers only name the columns that need intentional placement; everything else stays visible and is appended afterwards. Pinned columns are the upsert_keys for record-based tasks and name + id for file-based tasks.

Notable details:

  • createdBy is no longer pinned for file-based tasks. Only name and id are. createdBy, modifiedOn, and the other Synapse-managed columns are now orderable like any other column, which is what lets curation metadata precede system metadata.

Docs: a new "Controlling the order of the columns" section in docs/guides/extensions/curator/metadata_curation.md, plus a worked example in each function's docstring.

Testing:

Unit tests added to tests/unit/synapseclient/extensions/unit_test_curator.py:

  • validate_column_order_list()None, non-list input, non-string and empty-string entries, duplicates, valid input passthrough.
  • resolve_column_order_list() — pinned-only, partial request, request naming a pinned column (no duplication, stays pinned), unknown column raises ValueError, remaining columns keep relative order.
  • Record-based: column_order produces the expected CSV template header (patientId,specimenID,diagnosis,age,assay from a partial order that redundantly names an upsert key), and a malformed column_order raises before the schema is fetched.
  • File-based: column_order is forwarded to _create_json_schema_entity_view, the resulting view.columns is in the expected order, a malformed column_order raises before the entity view is created, and an unknown column name deletes the created view before raising — including the case where the cleanup delete itself fails, which logs and still raises the ValueError.

Existing call-site assertions were updated for the new column_order=None argument.

Two existing unittest.TestCase classes in this file (TestRecordBasedHelperFunctions, TestFileBasedHelperFunctions) were converted to plain pytest classes so the new cases could use pytest.mark.parametrize and fixtures, in line with the test conventions in tests/CLAUDE.md.

@andrewelamb
andrewelamb requested a review from a team as a code owner August 13, 2026 14:45
@andrewelamb
andrewelamb marked this pull request as draft August 13, 2026 14:45
@andrewelamb andrewelamb changed the title add ability to set column order [SYNPY-1840] Add ability to set column order when creating file and record based tasks. Aug 13, 2026
@andrewelamb
andrewelamb marked this pull request as ready for review August 13, 2026 15:04
Synapse.get_client(synapse_client=syn).logger.exception(
f"Could not delete the created EntityView {view.id}. It "
"must be deleted manually."
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "deleted manually" mean here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm changing this to

f"Could not delete the created EntityView {view.id}. Delete it "
"yourself, either from the Synapse web UI, or with the Python "
f"client: EntityView(id='{view.id}').delete()"

Comment thread synapseclient/extensions/curator/file_based_metadata_task.py Outdated
@andrewelamb
andrewelamb merged commit 3833ee8 into develop Aug 13, 2026
17 of 23 checks passed
@andrewelamb
andrewelamb deleted the SYNPY-1840 branch August 13, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants