Skip to content

[ESSREDUCE] Minimal implementation-independent workflow specifications - #690

Open
SimonHeybrock wants to merge 1 commit into
mainfrom
653-minimal-workflow-spec
Open

[ESSREDUCE] Minimal implementation-independent workflow specifications#690
SimonHeybrock wants to merge 1 commit into
mainfrom
653-minimal-workflow-spec

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

Fresh counter-proposal for #653, as announced in the review on #689 — I propose closing that PR in favor of this one.

The core difference to #689: the spec is pure interface. No workflow factory, no sciline keys, no use_workflow_default — nothing that presumes a particular implementation or execution location. The same spec must make sense whether the workflow runs as a local sciline pipeline, behind a web service, or as a cluster job; compute binding is a parallel mechanism, deliberately out of scope here.

Main decisions (full rationale in ADR 0001, included in this PR):

  • One pydantic params model per workflow instead of per-sciline-key Parameter entries. Enables cross-parameter validation and JSON Schema for free. The graph-derived "select outputs → see relevant params" feature is dropped, not ported: it treats output selection as workflow slicing, which only the sciline implementation can express.
  • title and description mandatory; identity is name + version — no instrument, no WorkflowId class. Scoping and global uniqueness are the enumerating context's responsibility.
  • Two forms with a one-way projection: WorkflowSpec (in-process, holds the params model class, full validation) and SerializedWorkflowSpec (plain data, params as JSON Schema, JSON-round-trippable). No inverse — validators don't survive JSON Schema, so authoritative validation stays with the process owning the model class; remote consumers validate optimistically. This matches the announcement-as-contract design from Rethink workflow registration: move schema ownership to domain packages esslivedata#889.
  • Outputs declared structurally via ArraySpec (dims, unit, coord units) — plain data, no scipp, replacing sc.DataArray default-factory templates. Scalars with units are the 0-d case.
  • Scipp-free shared parameter vocabulary (unit enums, range/edges models with cross-field validation); scipp conversions quarantined in spec.conversions. Value defaults are set by workflow authors at the use site, not by the vocabulary.
  • No registry: enumeration mechanisms are out of scope, any works against the same spec type.
  • Explicit goal: ess.livedata WorkflowSpec eventually inherits from this, gated on its planned ArraySpec migration; base field names are a strict subset of livedata's for that reason.

Adds pydantic>=2.5 as an essreduce dependency (already present in the locked CI environments). The existing ess.reduce.parameter/workflow/widgets machinery is superseded but untouched; removal is a later hard break — no one depends on it currently.

🤖 Generated with Claude Code

WorkflowSpec describes a workflow's user-facing interface (identity,
title/description, one pydantic params model, structural output
descriptions) without factories, sciline keys, or registries, so generic
UIs can be generated from it regardless of where compute happens.
serialize() projects one-way onto SerializedWorkflowSpec (params as JSON
Schema) for cross-process consumers; authoritative validation stays with
the process owning the model class.

Includes a scipp-free shared parameter vocabulary (unit enums,
range/edges models with cross-field validation) with scipp conversions
quarantined in spec.conversions, and ADR 0001 recording the design and
its rationale (see #653, scipp/esslivedata#889).

Adds pydantic as an essreduce dependency. The existing
ess.reduce.parameter/workflow machinery is superseded but untouched;
removal is a later hard break.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

essreduce Issues for essreduce.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant