Show the result builder as a node in the Hamilton UI - #1678
Open
charitarthchugh wants to merge 5 commits into
Open
Show the result builder as a node in the Hamilton UI#1678charitarthchugh wants to merge 5 commits into
charitarthchugh wants to merge 5 commits into
Conversation
The SDK tracker is about to register a synthetic node standing in for the driver's built result. NodeTemplate.classifications is a constrained choice field, so the server has to know the value before it can be sent one. Postgres gets a new migration; the sqlite initial migration is edited in place, matching how 0002's unique_together is already carried there.
The union is the frontend's mirror of the server's choice field, so it has to widen with it or the new classification fails to type-check on arrival. No styling, icon or filter is added for it -- the node renders through the existing paths like any other classification.
The combined result a driver returns is assembled by a result builder after the last node finishes, outside the dataflow. There is no node to hang a data summary on, so what a run actually produced never reached the UI. Add a synthetic node template standing in for it. The template carries no dependencies -- which nodes fed the result varies per run, so they are recorded per run instead -- and typing.Any for its output, since the type varies with the result builder. Registering it is opt-in. A caller that registers the node without also emitting a task run for it would render it as never-executed on every run, as the legacy Driver in this module would. The node is folded into the DAG hash when it is registered, because register_dag_template_if_not_exists matches on that hash alone: without it, a template registered before this node existed would be reused and runs would log against a node it does not have. Underscore-prefixed functions never become nodes, but names that do not come from a function do -- an external input, a decorator-generated name. NodeTemplate is unique on (name, dag_template), so a collision would fail registration outright. Detect it and skip the node with a warning instead: skipping costs the run its node, failing would cost the user their run.
Register the synthetic node and log a run for it on every successful tracked run, so the combined result gets the same data observability as any other node. Both the sync and async trackers do this; the payload is built once in a shared, I/O-free helper and each sends it its own way. Dependencies are recorded per run, from the result itself where it can say what went into it. Driver.materialize asks pre_graph_execute for final_vars + materializer_vars but hands post_graph_execute only the final_vars slice, so recording the requested list unfiltered would credit the node with materializers it never saw. Narrowing is only sound when the result is keyed by node name, so a dataframe or a custom builder's own dict keeps the full list rather than be credited with nothing. Nothing is emitted when the name collided and no template was registered, or when the run failed -- a failed run leaves the node not-executed, like any node the run never reached. A builder that returns None still counts as having run. Emission failures are logged and swallowed. This runs just before log_dag_run_end, so an exception escaping here would leave an otherwise successful run rendering as still-running forever.
Covers what the node is, that nothing is needed to enable it, and the four things worth knowing: the result is profiled twice, only Driver.execute sees a real built result, failed runs leave it not-executed, and a name collision makes the tracker step aside. Also notes that the node is part of what identifies a DAG version, so the first tracked run after upgrading registers a new version of each dataflow.
charitarthchugh
added a commit
to charitarthchugh/su26-ai301-contribution
that referenced
this pull request
Aug 4, 2026
The Hamilton contribution shipped: PR apache/hamilton#1678 is open against upstream main with five per-layer commits, 15 new tests and a green suite. The README still described Phase III as in progress and pointed at commit hashes that no longer exist after the branch was rebuilt. Rewrites it against the branch as submitted: real commit hashes and file lists, test names and verified counts (148/7 vs a 133/7 baseline on main), the end-to-end cases and before/after evidence, and a Pull Request section with the PR summary, acceptance criteria and a dated maintainer-feedback log. Also corrects two things earlier phases got wrong: the frontend touchpoint is friendlyApi.ts rather than DAGViz.tsx, and the backend enum is NodeTemplate.NodeType. Documents the two visible history gaps -- the analysis window waiting on maintainer approval, and the pre-PR rebase that dates all five commits to one day -- rather than leaving a reader to find them in the reflog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #1150
The combined result returned by
execute()is assembled by a result builder (DictResult,PandasDataFrameResult, ...) indo_build_result, after the last node finishes. That runsoutside the dataflow, so there is no node to attach a data summary to. Every individual output
is profiled in the UI; the object the caller actually receives is not.
This adds a synthetic node named
_result_builderto stand in for that object, so it gets thesame data observability as any other node. Nothing is needed to enable it — attach a
HamiltonTrackeras usual.Before / after
Before: 4 nodes, the task table ends at
average_squared. After: 5 nodes, with_result_builder(typing.Any, success) alongside them. Tags, outputs and duration areotherwise identical.
average_squared, input_numbers, squared, sum_squaredaverage_squared, input_numbers, _result_builder, squared, sum_squaredOpening the node shows the object the caller received:
A
PandasDataFrameResultgives a full DataFrame profile in the same view, since the builtresult goes through the existing
process_resultpath unchanged.Dataflow used for the comparison
Run twice against the same local UI and project — once with the SDK from
main, once withthis branch. Both returned
{'average_squared': 11.0, 'sum_squared': 55.0}.Changes
Five commits, one per layer.
Tracking server.
NodeTemplate.classificationsis anArrayFieldconstrained byTextChoices, not a free-form field, so the SDK cannot send a value the server does not know:result_builderto those choices.migrations_sqlite/0001_initial.pyis edited in place, matching how0002'sunique_togetheris already carried there.Frontend. The
Classificationunion mirrors that server field, so it has to widen with it.result_builderto the union. No styling, icon or filter — the node renders through theexisting paths.
SDK, node template. Which nodes feed the result varies per run, so a fixed dependency list
on the template would be wrong for any subset run.
typing.Anyfor the output type, since it varies with the result builder. The real typeshows up in the per-run summary.
never-executed on every run, which is what the legacy
Driverin that module would do.register_dag_template_if_not_existsmatcheson that hash alone without inspecting the nodes posted with it, so otherwise a template
registered before this node existed gets reused and runs log against a node it does not have.
NodeTemplateisunique on
(name, dag_template), so registering both would fail the run.SDK, task run. The combined result only exists at
post_graph_executetime.reach their client.
log_dag_run_end, so anexception escaping here would leave a successful run rendering as still-running.
Docs. Add a "The result builder node" subsection to
docs/hamilton-ui/ui.rst.How I tested this
pytest ui/sdk/tests/test_driver.py ui/sdk/tests/test_adapters.py -q— 22 passed, 3 skipped.pre-commit run --files <changed files>postgres:
DictResultPandasDataFrameResultmaterialize()Notes
Dependency narrowing.
Driver.materializeaskspre_graph_executeforfinal_vars + materializer_varsbut handspost_graph_executeonly thefinal_varsslice, sorecording the requested list unfiltered would credit the node with materializers it never saw.
Narrowing is only sound when the result is keyed by node name, so a dataframe or a custom
builder's own dict keeps the full list rather than be credited with nothing.
Not every path sees a built result. Only
Driver.execute()does.raw_execute()andmaterialize()never call a result builder, and async drivers build the result after the trackerhas logged. The node is still emitted in those cases, summarizing the raw dict of outputs. No
lifecycle hook reports which result builder ran, or whether one ran at all.
The node is part of the DAG hash, so the first tracked run after upgrading registers a new
version of each dataflow. Existing versions and their runs are untouched.
It renders unconnected in the DAG view. The template carries no dependencies by design and
the DAG view builds edges from template dependencies, so the node sits off to one side. Noted in
the docs.
The result is profiled twice. Each output node is already summarized and the combined result
contains those same objects, so expect roughly twice the profiling time and payload.
Not refactored:
_result_attribute/_result_attributesduplicate the inlineattribute-shaping already in both trackers'
post_node_execute. Folding those together meansediting working code outside this change, so I left it. Happy to do it as a follow-up.
cc @skrawcz
Checklist