feat(http task): let next_page write record context

[Divyanshu Tiwari (divyanshu-tiwari)]

Problem

next_page could read record context but never write it. Every value it saw was fixed at the moment the upstream task emitted the record, so pagination could not carry state that a single response does not contain.

This blocks any API where pages are nested inside something that also moves — e.g. a page token scoped to the exact query that issued it, plus a cap on how many items one query can walk. Exhausting a token means re-querying with a new filter and then paging that, and the token loop has to reproduce the current filter. With a frozen query it reproduces the original one, and the API rejects the replayed token.

Change

A context map in the object returned by next_page is now applied to the record, before the next iteration renders its templates. 13 lines in processItem.

next_page: |
  [inputs] as $input |
  ($input[0].data | fromjson) as $body |
  "{{ context "current_query" }}" as $query |
  if ($body.next_token // "") != "" then
    { endpoint: ("…?" + $query + "&page_token=" + ($body.next_token | @uri)) }
  else
    (($query | sub("&after_id=[^&]*"; "")) + "&after_id=" + $last_id) as $next_query |
    { endpoint: ("…?" + $next_query), context: { current_query: $next_query } }
  end

Values are JSON-encoded to match how task.Base sets context (task.go:138-143), so {{ context "key" }} renders identically regardless of which one wrote the key. Writes go through the existing record.SetContextValue.

Compatibility

Opt-in and unreachable for every pipeline as it stands — I scanned all 793 YAMLs in data-airflow, and zero next_page blocks currently return a context key. Nothing else about the task changes.

Verification

test/pipelines/next_page_context_test.yaml walks three pages where each one republishes the offset for the next:

{"ids":[1,2,3],"query_used":"_limit=3&_start=0"}
{"ids":[4,5,6],"query_used":"_limit=3&_start=3"}
{"ids":[7,8,9],"query_used":"_limit=3&_start=6"}

Each page reads back what the previous one wrote, and the value differs per emitted record downstream. page_id cannot substitute — the offset advances by the item count actually received.

During development this was also covered by Go tests against a mock that rejected a page token replayed under a different query, with a negative control proving the frozen-query form fails there. Those were dropped in favour of the test pipeline, so this behaviour has no automated regression guard.

Docs

New Pagination section in the task README, which also writes down things that were previously undocumented or easy to get wrong:

• Both JQ inputs, including the page counter — [inputs][1].page_id, 1-indexed on the page about to be requested, so 2 on the first evaluation. Used by ~30 pipelines, documented nowhere.
• inputs is a one-shot iterator. A second [inputs] yields an empty list, and jq's null <= 50 is true, so a page bound written that way becomes an infinite loop. This hung a test while I was writing it.
• The context map takes values, not expressions — a string is stored verbatim, so context: { cursor: ".data | fromjson | .next" } silently stores that text.
• Writing a key creates it; reading one requires it to exist already, including in a branch that never executes, since template substitution is textual and happens before JQ parses the expression.

Follow-up, not in this PR

A next_page render failure returns from processItem, which under the default fail_on_error: false is only printed (pipeline.go:243-249). The run reports success having emitted page 1 alone. This PR makes it slightly easier to trip, because reading a carried key means depending on an upstream seed. Worth making such a failure fatal regardless of fail_on_error, so truncated pagination cannot pass as success.

🤖 Generated with Claude Code