[ESSREDUCE] Minimal implementation-independent workflow specifications

packages/essreduce/docs/developer/adr/0001-minimal-workflow-spec.md

@@ -0,0 +1,158 @@
+# ADR 0001: Minimal implementation-independent workflow specifications
+
+- Status: proposed
+- Deciders: Simon
+- Date: 2026-08-05
+
+## Context
+
+Three mechanisms currently describe workflow interfaces to users, with
+overlapping purpose and no shared shape:
+
+- `ess.reduce.parameter` / `ess.reduce.workflow`: per-sciline-key `Parameter`
+  dataclasses in a global registry, with parameters discovered by walking the
+  pipeline graph from selected outputs. Drives the ipywidgets GUI.
+- `ess.livedata.config.workflow_spec.WorkflowSpec`: one pydantic params model
+  per workflow, plus output descriptions, driving the live-data dashboard.
+- `ess.nmx.configurations`: standalone pydantic models for batch reduction.
+
+Consolidation was analyzed at length in
+[scipp/ess#653](https://github.com/scipp/ess/issues/653) and
+[scipp/esslivedata#889](https://github.com/scipp/esslivedata/issues/889). Two
+earlier attempts stalled, both for the same reason: scope. A universal
+`ess.schemas` catalog ("all workflows for all instruments, imported by
+everyone") turned shared conventions into a cross-team release-coordination
+problem; a rewrite of the essreduce widget layer
+([scipp/ess#689](https://github.com/scipp/ess/pull/689)) kept sciline keys,
+workflow factories, and widget concerns inside the spec, so the spec could not
+outlive or precede any particular implementation.
+
+The goal is the minimal layer that lets a *generic* user interface — ipywidgets,
+a web dashboard, or a command-line tool — be generated from a workflow
+description alone. Compute is deliberately abstracted away: the same spec must
+make sense whether the workflow runs as a local sciline pipeline, behind a web
+service, or as a cluster job. Compute is not part of this work, but it shapes
+the design: nothing implementation-bound may appear in the spec.
+
+## Decision
+
+A new module `ess.reduce.spec` defines the spec layer. Its only dependency
+beyond the standard library is pydantic (a new essreduce dependency); the one
+scipp-facing piece is quarantined in a submodule.
+
+### The spec is pure interface: no factory, no keys, no registry
+
+`WorkflowSpec` holds identity (`name`, `version`), display metadata (`title`,
+`description`, both mandatory), a params model, and output descriptions.
+Nothing else. In particular it holds *no* workflow factory and *no* sciline
+keys: a spec describes *what a user can configure and what they get back*, not
+how it is computed. Binding a spec to an executor — conceptually a mapping from
+spec identity to `Callable[[BaseModel], Mapping[str, Any]]`, or a remote
+service holding the same spec — is a parallel mechanism, intentionally
+undefined here. This is what keeps the spec valid across local, service, and
+cluster execution.
+
+How specs are enumerated (module-level tuples, entry points, esslivedata's
+per-instrument registration) is likewise out of scope. Any mechanism works
+against the same spec type; prescribing one here would recreate the catalog
+problem that sank the `ess.schemas` plan.
+
+### One params model per workflow
+
+Parameters are a single pydantic model class per workflow
+(`params: type[BaseModel]`), not per-key entries in a registry. This enables
+cross-parameter validation, gives JSON Schema for free, and removes the
+implementation coupling of key-addressed parameters. The graph-derived
+"select outputs, then see only relevant parameters" feature of
+`ess.reduce.workflow.get_parameters` does not survive: it treats output
+selection as workflow slicing, which only the sciline implementation can
+express. If output-dependent parameter sets are needed, they are distinct
+workflows (distinct specs).
+
+The field defaults to `NoParams` (a closed model with no fields), so consumers
+never branch on params being absent, and sending parameters to a workflow that
+takes none is a validation error rather than silently ignored.
+
+### Two forms, one-way projection
+
+`WorkflowSpec` is the in-process form: it holds the params model *class*, so
+same-process consumers (ipywidgets, a CLI wrapping a local pipeline) get full
+pydantic validation including custom validators. `spec.serialize()` projects
+onto `SerializedWorkflowSpec`, a plain-data pydantic model with params as JSON
+Schema (`model_json_schema()`), which round-trips through JSON and is what a
+service announces to remote consumers.
+
+There is deliberately no inverse. Validators do not survive JSON Schema, so a
+deserialized spec would be a lie about its own validation. Instead, validation
+authority sits with the process owning the model class: in-process UIs validate
+directly; remote UIs validate optimistically against the schema and the owning
+service accepts or rejects authoritatively. This matches the
+announcement-as-contract design adopted for esslivedata in
+[scipp/esslivedata#889](https://github.com/scipp/esslivedata/issues/889): the
+serialized spec is the entire cross-process surface, and where a model class is
+*defined* is invisible to consumers.
+
+### Identity is `name` + `version`; scoping is the enumerator's problem
+
+No `instrument` field and no `WorkflowId` class at this level. Instrument is
+meaningless for technique-level batch workflows, and a spec cannot guarantee
+global uniqueness of anything — only the context that enumerates or deploys
+specs can. esslivedata keeps keying workflows by `(instrument, name, version)`,
+supplying the instrument from its registration context. Data-provenance
+identity (which spec, params, and input datasets produced a dataset) similarly
+composes spec identity with deployment context; the spec's contribution is
+being serializable and versioned.
+
+### Outputs are declared, structurally, without scipp
+
+`outputs` maps output names to `OutputSpec` (mandatory title, description,
+optional `ArraySpec`). `ArraySpec` describes dims, unit, and coordinate units —
+plain data, so it serializes, replacing the `sc.DataArray` default-factory
+templates esslivedata currently uses for plotter selection. Output *selection*
+(choosing which sciline targets to compute) is not modeled: like parameter
+slicing, it is an implementation notion. Declaration order is meaningful
+(consumers show outputs in order, primary output first). Livedata-specific
+output machinery (`OutputView`, `Temporality`, windowing) stays in esslivedata.
+
+### Shared parameter vocabulary, scipp-free
+
+`ess.reduce.spec.parameters` provides constrained unit enums and range/edges
+models with cross-field validation (`stop > start`, log-scale positivity) —
+the models previously duplicated between esslivedata and package-specific
+code. They contain no scipp: conversion of validated values into scipp objects
+(`edges_to_variable`, `range_to_variables`) lives in
+`ess.reduce.spec.conversions`, imported by workflow implementations only. This
+keeps the vocabulary JSON-Schema-clean and the spec layer importable without
+touching scipp. Value defaults (start/stop/bin counts) are set by workflow
+authors at the use site, not by the vocabulary — sensible values are a
+workflow/instrument decision, and a generic default is a wrong default.
+
+### Convergence with esslivedata
+
+Explicit goal: `ess.livedata.config.workflow_spec.WorkflowSpec` eventually
+inherits from this spec, adding its live-data fields (`instrument`, `group`,
+`source_names`, `aux_sources`, `device_outputs`, reset flags). The base spec's
+field names and semantics (`name`, `version`, `title`, `description`,
+`params`) are a strict subset of esslivedata's today for exactly this reason.
+The blocking difference is `outputs`: esslivedata's `sc.DataArray` templates
+must first migrate to `ArraySpec` (already planned independently in
+scipp/esslivedata#889). The import edge is free — the esslivedata backend
+already depends on essreduce, and its dashboard is decoupled via the
+serialized-spec announcement, not via imports.
+
+## Consequences
+
+- Generic UIs (including a command-line interface) can be generated from
+  `WorkflowSpec` alone, and from `SerializedWorkflowSpec` across process
+  boundaries, with no knowledge of the workflow implementation.
+- essreduce gains a pydantic dependency.
+- `ess.reduce.parameter`, `ess.reduce.workflow`, and the widgets built on them
+  are superseded and will be removed in a later hard break; they are untouched
+  for now. The graph-derived parameter discovery they provide is dropped, not
+  ported.
+- `ess.nmx.configurations` and esslivedata migrate to the shared vocabulary
+  and spec incrementally, per package, with no coordination requirement — a
+  package that never migrates costs the others nothing.
+- The executor binding and spec enumeration remain to be designed when a
+  concrete consumer needs them; the spec layer does not constrain either
+  beyond being addressable by `(name, version)`.

packages/essreduce/docs/developer/adr/index.md

@@ -0,0 +1,17 @@
+# Architecture Decision Records
+
+Lightweight records of load-bearing design decisions and their rationale. Each
+ADR captures one decision. Accepted text is not rewritten: corrections and
+extensions land as a dated amendment section at the bottom, flagged in the
+status line, so the original stays readable as the reasoning of its time.
+Reversing or replacing a decision gets a new ADR that links back. Format
+follows [scipp's ADR convention](https://github.com/scipp/scipp/tree/main/docs/development/adr).
+
+```{toctree}
+---
+maxdepth: 1
+glob: true
+---
+
+0*
+```

packages/essreduce/docs/developer/index.md

@@ -14,4 +14,5 @@ getting-started
 coding-conventions
 dependency-management
 gui
+adr/index
 ```

packages/essreduce/pyproject.toml

@@ -32,6 +32,7 @@ dynamic = ["version"]
 dependencies = [
     "dask>=2022.1.0",
     "graphviz>=0.20",
+    "pydantic>=2.5",
     "sciline>=25.11.0",
     "scipp>=26.3.1",
     "scippneutron>=26.6.0",

packages/essreduce/src/ess/reduce/spec/__init__.py

@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2026 Scipp contributors (https://github.com/scipp)
+"""
+Implementation-independent workflow specifications for UI generation.
+
+See :mod:`ess.reduce.spec._workflow_spec` for the design;
+ADR 0001 (docs/developer/adr) for the rationale.
+"""
+
+from ._workflow_spec import (
+    ArraySpec,
+    NoParams,
+    OutputSpec,
+    SerializedWorkflowSpec,
+    WorkflowSpec,
+)
+
+__all__ = [
+    'ArraySpec',
+    'NoParams',
+    'OutputSpec',
+    'SerializedWorkflowSpec',
+    'WorkflowSpec',
+]

packages/essreduce/src/ess/reduce/spec/_workflow_spec.py

@@ -0,0 +1,165 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2026 Scipp contributors (https://github.com/scipp)
+"""
+Workflow specifications: implementation-independent workflow metadata.
+
+A :class:`WorkflowSpec` describes a workflow's user-facing interface — identity,
+display metadata, parameters, and outputs — without reference to how or where
+the workflow is computed. User interfaces (widgets, dashboards, command-line
+tools) are generated from the spec alone; the binding from a spec to an
+executor is a separate, parallel mechanism deliberately not defined here.
+
+Two forms exist, related by a one-way projection:
+
+* :class:`WorkflowSpec` is the in-process form. It holds the params *model
+  class*, so consumers in the same process get full pydantic validation,
+  including cross-field validators.
+* :class:`SerializedWorkflowSpec` is the plain-data form produced by
+  :meth:`WorkflowSpec.serialize`, with params as JSON Schema. It is what a
+  service announces to remote consumers, which can render forms and validate
+  optimistically against the schema. There is intentionally no inverse:
+  validators do not round-trip through JSON Schema, and authoritative
+  validation always happens in the process owning the model class.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict, Field
+
+
+class NoParams(BaseModel):
+    """
+    Params model for workflows that take no configuration.
+
+    Workflows always have a params model, so consumers never branch on its
+    absence; "takes no parameters" is expressed as a model with no fields.
+    Extra fields are rejected so that sending params to such a workflow is an
+    error rather than silently ignored.
+    """
+
+    model_config = ConfigDict(extra='forbid')
+
+
+class ArraySpec(BaseModel, frozen=True):
+    """
+    Structural description of an array-valued workflow output.
+
+    Describes shape-independent structure — dimensions, unit, and coordinate
+    units — sufficient for a consumer to prepare for the data (e.g., select a
+    plotter) before any has been computed. A scalar value with a unit is the
+    0-d case: ``ArraySpec(dims=(), unit='counts')``.
+    """
+
+    dims: tuple[str, ...] = Field(description="Dimension names, outermost first.")
+    unit: str | None = Field(
+        default=None, description="Unit of the array values, if any."
+    )
+    coords: dict[str, str | None] = Field(
+        default_factory=dict,
+        description="Coordinate names mapped to their units (None for unitless).",
+    )
+
+
+class OutputSpec(BaseModel, frozen=True):
+    """Description of a single named workflow output."""
+
+    title: str = Field(min_length=1, description="Display title of the output.")
+    description: str = Field(default='', description="Description of the output.")
+    array: ArraySpec | None = Field(
+        default=None,
+        description=(
+            "Structural description of the output data, if array-valued and known."
+        ),
+    )
+
+
+def _default_outputs() -> dict[str, OutputSpec]:
+    return {'result': OutputSpec(title='Result', description='Workflow output.')}
+
+
+class _SpecFields(BaseModel, frozen=True):
+    """Metadata fields shared by both forms of the workflow spec."""
+
+    name: str = Field(
+        min_length=1,
+        description=(
+            "Machine-readable workflow identifier. Unique within the context "
+            "that enumerates the spec; global uniqueness is the enumerator's "
+            "responsibility, not the spec's."
+        ),
+    )
+    version: int = Field(
+        ge=1,
+        description=(
+            "Version of the workflow interface. Increment on any change a "
+            "consumer could observe: params model, outputs, or semantics."
+        ),
+    )
+    title: str = Field(min_length=1, description="Display title of the workflow.")
+    description: str = Field(
+        min_length=1, description="Description of what the workflow computes."
+    )
+
+
+class WorkflowSpec(_SpecFields, frozen=True):
+    """
+    Implementation-independent specification of a workflow's user interface.
+
+    Holds identity and display metadata, the pydantic model class defining the
+    workflow's parameters, and descriptions of its outputs. Contains no
+    factory, no executor, and no reference to any workflow implementation;
+    pairing a spec with something that computes it is a separate mechanism.
+    """
+
+    params: type[BaseModel] = Field(
+        default=NoParams,
+        description=(
+            "Pydantic model class defining the workflow parameters. Defaults "
+            "to :class:`NoParams` for workflows that take no configuration."
+        ),
+    )
+    outputs: dict[str, OutputSpec] = Field(
+        default_factory=_default_outputs,
+        description=(
+            "Named outputs the workflow produces. Order is meaningful: "
+            "consumers present outputs in this order and may auto-select the "
+            "first, so put the primary output first."
+        ),
+    )
+
+    def serialize(self) -> SerializedWorkflowSpec:
+        """
+        Project to the plain-data form with params as JSON Schema.
+
+        The projection is one-way: pydantic validators do not survive it, so
+        a consumer of the serialized form can validate only optimistically.
+        Authoritative validation happens where the model class lives.
+        """
+        return SerializedWorkflowSpec(
+            name=self.name,
+            version=self.version,
+            title=self.title,
+            description=self.description,
+            params_schema=self.params.model_json_schema(),
+            outputs=self.outputs,
+        )
+
+
+class SerializedWorkflowSpec(_SpecFields, frozen=True):
+    """
+    Plain-data form of a workflow spec, safe to send across process boundaries.
+
+    Produced by :meth:`WorkflowSpec.serialize`; round-trips through JSON. Params
+    are represented as JSON Schema, sufficient for form generation and
+    optimistic validation but not for authoritative validation — that remains
+    with the process owning the params model class.
+    """
+
+    params_schema: dict[str, Any] = Field(
+        description="JSON Schema of the workflow's params model."
+    )
+    outputs: dict[str, OutputSpec] = Field(
+        description="Named outputs the workflow produces, in display order."
+    )