feat: Add environment ID support for hooks. - #484
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
kinyoklion
left a comment
There was a problem hiding this comment.
Note
These comments were generated by Claude (multi-agent review: general / security / adversarial agents), run and posted by @rlamb.
No crash or flag-data-integrity issues found — thread safety, None-safety, backward compat, and the eventsource header contract all checked out under adversarial testing (full report available on request). Two findings worth addressing, posted inline:
- Environment ID is recorded from error responses and exposed to hooks before initialization — diverges from the HOOK spec and the .NET implementation this PR mirrors (proven with a runnable repro).
- The two new polling tests use unbounded
ready.wait()— a regression in the fallback path hangs CI instead of failing (proven by mutation).
| log.info("StreamingUpdateProcessor initialized ok.") | ||
| self._ready.set() | ||
| elif isinstance(action, Fault): | ||
| record_environment_id(self._data_source_update_sink, action.headers) |
There was a problem hiding this comment.
[Claude] Medium: env ID is latched from error responses and exposed to hooks pre-initialization
This Fault-path record means the env ID is captured from failed responses: per ld_eventsource, Fault.headers is populated exactly when the response was non-2xx or wrong content type. The same happens in FDv2 from failed-initializer headers (fdv2.py:226) and from every synchronizer update including INTERRUPTED/OFF (fdv2.py:425), and the value is never cleared once set.
Proven with a repro: a 401 (invalid SDK key) response carrying X-LD-EnvID: env-from-401 leaves the SDK permanently uninitialized (variation() returns defaults), yet every hook receives environment_id == 'env-from-401' for the process lifetime.
This diverges from both references:
- The hooks spec says the environment ID is "provided by LaunchDarkly and only available once initialization has completed."
- The .NET implementation this PR mirrors (feat: Add environment ID support for hooks. dotnet-core#81) records it only as store init metadata written together with successfully applied data; its error handlers read only
x-ld-fd-fallback. Under the same 401, .NET hooks getnull.
Practical impact: any middlebox that synthesizes an error page on the SDK↔LD path (TLS-intercepting proxy, LB in front of a Relay Proxy) can permanently poison the value that downstream OTel hooks emit as feature_flag.set.id — even after the real connection recovers.
Suggestion: record only from success paths — Start here, the _Success basis in _run_initializers (drop the _Fail-headers record at fdv2.py:226), and gate fdv2.py:425 on VALID updates — then update/remove test_records_environment_id_from_error_response_headers and test_environment_id_from_initializer_error_headers, which currently codify the error-latching. If latching from errors is intentional, it deserves an explicit callout since it's a Python-only behavioral divergence.
Rider: FDv2._record_environment_id (fdv2.py:511) only skips None, so a present-but-empty X-LD-EnvID overwrites a previously correct value; the FDv1 helper filters '' (datasource_common.py). Worth applying the same isinstance(str) and != '' guard there for consistency.
There was a problem hiding this comment.
Agreed — fixed in b9fcb4c. Env ID is now only recorded from success paths: Start in FDv1 streaming (dropped the Fault record), the _Success basis in _run_initializers (dropped the _Fail-headers record), and synchronizer updates gated on DataSourceState.VALID. FDv2._record_environment_id now applies the same isinstance(str) and != '' guard as the FDv1 helper. The two tests that codified error-latching now assert the opposite (test_does_not_record_environment_id_from_error_response_headers, test_environment_id_is_not_recorded_from_initializer_error_headers), plus a new test_environment_id_is_not_recorded_from_non_valid_updates.
| sink = DataSourceUpdateSinkImpl(store, Listeners(), Listeners()) | ||
| config._data_source_update_sink = sink | ||
| setup_processor(config) | ||
| ready.wait() |
There was a problem hiding this comment.
[Claude] Medium: unbounded ready.wait() turns a regression into a CI hang
This wait (and the one at line 180) has no timeout. If the external-requester fallback in PollingUpdateProcessor._get_all_data_with_headers regresses, the resulting exception is swallowed by _poll's generic handler, ready is never set, and the test hangs forever — the repo configures no pytest-timeout, so CI hangs until the job limit rather than reporting a failure.
Verified by mutation: replacing the fallback with an unconditional self._requester.get_all_data_with_headers() call makes test_environment_id_is_none_when_requester_provides_no_headers hang rather than fail.
Suggestion: assert ready.wait(2) in both new tests, matching the bounded waits already used elsewhere in this file (e.g. ready.wait(0.5) at lines 115/137).
There was a problem hiding this comment.
Done in b9fcb4c — both new polling tests now use assert ready.wait(2).
|
@devin rebase this PR on main and resolve the conflicts |
Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
b51e14c to
585459b
Compare
|
Rebased on |
**Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/master/CONTRIBUTING.md#submitting-pull-requests) - [x] I have validated my changes against all supported platform versions **Related issues** Follows the hooks spec requirement that `EvaluationSeriesContext` carries the environment ID reported by LaunchDarkly. Companion SDK work: launchdarkly/python-server-sdk#484. **Describe the solution you've provided** `servicedef.EvaluationSeriesContext`/`TrackSeriesContext` already had an `environmentId` field, but nothing exercised it and there was no capability for SDKs to declare support. This adds: - the `hook-environment-id` capability, documented in `docs/service_spec.md` - `DataSourceOptionEnvironmentID(...)`, which wraps the mock streaming/polling handler so responses carry an `X-LD-EnvID` header - an evaluation-series test that configures that header and asserts `evaluationSeriesContext.environmentId` matches on `beforeEvaluation` **Describe alternatives you've considered** Setting the header inside `mockld.StreamingService`/`PollingService` themselves rather than via a handler wrapper; the wrapper avoids touching both services and works for either. **Additional context** Track hooks (`trackSeriesContext.environmentId`) are described in the docs but not yet asserted by a test; that can follow once an SDK supports both capabilities. Link to Devin session: https://app.devin.ai/sessions/bfe54128e2804a96bb100e6120e9a3ef Requested by: @kinyoklion <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Adds the **`hook-environment-id`** capability so SDKs can declare that they propagate LaunchDarkly’s environment ID from streaming/polling into hook series contexts, with matching documentation in `docs/service_spec.md`. > > The mock data source can now emit **`X-LD-EnvID`** via `DataSourceOptionEnvironmentID`, using a small handler wrapper so both streaming and polling paths behave like production. Hook integration tests gain **`provides the environment ID`**, which checks `evaluationSeriesContext.environmentId` on **`afterEvaluation`** for default and polling data sources when the capability is present. Hook test setup is extended so data source options can be passed through `createClientForHooksWithDataSourceOptions`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ccd1596. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
**Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/master/CONTRIBUTING.md#submitting-pull-requests) - [x] I have validated my changes against all supported platform versions **Related issues** Ports #410 (v2) to v3. Companion SDK work: launchdarkly/python-server-sdk#484, launchdarkly/cpp-sdks#594. **Describe the solution you've provided** Same capability and test as #410, adapted to the v3 data system: - the `hook-environment-id` capability, documented in `docs/service_spec.md` - `DataSystemOptionEnvironmentID(...)`, a top-level `SDKDataSystemOption` that makes every mock service of the data system (initializers and synchronizers, in all connection modes) send an `X-LD-EnvID` response header - an evaluation-series test (`hooks/evaluation/provides the environment ID`) asserting `evaluationSeriesContext.environmentId` on `afterEvaluation`, run against the SDK's default data source plus an explicit polling subtest for client-side/`server-side-polling` SDKs Unlike v2, where the option wrapped the single `SDKDataSource` handler, the header is applied in `createEndpoints`, so both FDv2 initializers and synchronizers report it — matching production, where the header is on every flag-delivery response. **Describe alternatives you've considered** Making the option per-connection-mode. Environment ID is a property of the environment, not of a connection mode, so a single top-level option keeps call sites simple and covers every mode. **Additional context** Verified against the FDv2 contract services of python-server-sdk (#484) and cpp-sdks (#594): `hooks/evaluation/provides the environment ID` passes for both, in default (streaming) and polling modes. Track hooks (`trackSeriesContext.environmentId`) are documented but not yet asserted, same as in v2. Link to Devin session: https://app.devin.ai/sessions/bfe54128e2804a96bb100e6120e9a3ef Requested by: @kinyoklion <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Adds the **`hook-environment-id`** capability so SDKs that read LaunchDarkly’s `X-LD-EnvID` from streaming/polling responses must surface it on hook series contexts (`evaluationSeriesContext.environmentId`, and `trackSeriesContext` when track hooks are supported). The harness documents this in `service_spec.md` and gates a new evaluation hook test on the capability. > > **Mock flag delivery** now supports `DataSystemOptionEnvironmentID(...)`, a top-level data-system option that wraps every initializer and synchronizer endpoint (all connection modes) to emit `X-LD-EnvID`, matching production flag-delivery responses in the v3 data system. > > The test **“provides the environment ID”** evaluates a flag and asserts `environmentId` on the `afterEvaluation` hook payload, under the default data source and an explicit polling subtest for client-side / server-side-polling SDKs. Track-hook `environmentId` is documented but not asserted yet (same as v2). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8331c29. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
**Requirements** - [ ] I have added test coverage for new or changed functionality - [x] I have followed the repository's pull request submission guidelines - [x] I have validated my changes against all supported platform versions **Related issues** Intermittent `Download failed` failures in SDK contract-test jobs, e.g. launchdarkly/python-server-sdk#484 where 4 of 5 matrix jobs died at the same second while the rest passed on identical code. **Describe the solution you've provided** `downloader/run.sh` resolves the release version through the GitHub API (authenticated when `GITHUB_TOKEN` is set), but then downloads the release asset with a single un-retried `curl --fail -s`, and `-s` hides the reason for a failure: ```diff -curl --fail -s -L -o "${TEMP_DIR}/archive.${EXTENSION}" "${DOWNLOAD_URL}" || (echo "Download failed" >&2; exit 1) +curl --fail -sS -L --retry 5 --retry-delay 2 \ + -o "${TEMP_DIR}/archive.${EXTENSION}" "${DOWNLOAD_URL}" \ + || { echo "Download failed" >&2; exit 1; } ``` `--retry` covers transient 5xx/408/429 responses (consistent with several matrix jobs failing at once while others succeed), and `-sS` means the next failure reports the actual curl error instead of a bare `Download failed`. **Describe alternatives you've considered** - `--retry-all-errors`: rejected, it requires curl 7.71+ and this script also runs on older macOS runners. - Sending the token on the asset download: the asset URL redirects to an unauthenticated object store, so an `Authorization` header there is at best useless and can break the redirect. **Additional context** The `contract-tests` GitHub action reads this script from the `v2` branch by default, so this also covers repos running the v3 harness through that action. `main` has the same line and should get the same change. Link to Devin session: https://app.devin.ai/sessions/bfe54128e2804a96bb100e6120e9a3ef Requested by: @kinyoklion <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Contract-test jobs intermittently failed with a bare **Download failed** when fetching the `sdk-test-harness` release asset from GitHub. > > The download `curl` in `downloader/run.sh` now uses **`--retry 5 --retry-delay 2`** for transient HTTP failures (aligned with matrix jobs failing together while others pass) and **`-sS`** instead of `-s` so failures include the actual curl error message. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 2c23020. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
Requirements
Related issues
Implements the
environmentIdfield ofEvaluationSeriesContextfrom the hooks spec, which the OTel tracing hook uses forfeature_flag.set.id. Mirrors launchdarkly/dotnet-core#81.Describe the solution you've provided
EvaluationSeriesContextgains an optionalenvironment_id, populated from theX-LD-EnvIDresponse header sent by LaunchDarkly. Both data systems are supported, and each exposes it through a newDataSystem.environment_idproperty that the client reads when building the series context. Per the spec, it is only recorded from successful responses, so hooks never see an environment ID scraped off an error page.Basis.environment_id/Update.environment_idbut discarded it;FDv2now latches the last value seen on a success path (a_Successbasis, or an update whose state isVALID).Startaction of the SSE client, polling reads it from the response headers, and both record it onDataSourceUpdateSinkImpl.Sketch of the FDv1 path:
Describe alternatives you've considered
Following the .NET implementation more literally, where the environment ID is stored as init metadata on the data store, would require optional extension interfaces on the public
FeatureStore/DataSourceUpdateSinktypes. Instead, header handling stays inside the data sources and is surfaced by the data system, so externally implemented stores, sinks, feature requesters, and update processors continue to work unchanged (they simply report no environment ID).Additional context
launchdarkly-eventsource: it already exposes response headers onStartandFault(unlike the .NET event source, which needed feat: Report event source headers on open. dotnet-eventsource#104).environmentIdsupport in the OTel tracing hook lives inpython-server-sdk-oteland is a follow-up.Link to Devin session: https://app.devin.ai/sessions/bfe54128e2804a96bb100e6120e9a3ef
Requested by: @kinyoklion
Note
Overview
Adds optional
environment_idonEvaluationSeriesContextso evaluation hooks (e.g. OTelfeature_flag.set.id) can see which LaunchDarkly environment the SDK is connected to, matching the hooks spec.The ID comes from the
X-LD-EnvIDheader on successful data-source responses only—not from error responses. FDv1 records it via streamingStartheaders and polling response headers intoDataSourceUpdateSinkImpl; FDv2 latches values already present on successfulBasis/VALIDUpdateobjects.DataSystem.environment_idexposes the value andLDClientpasses it when building hook series context.Custom
FeatureRequesterimplementations withoutget_all_data_with_headersstill work (no env ID). Contract tests add thehook-environment-idcapability.Reviewed by Cursor Bugbot for commit 7ce7b6a. Bugbot is set up for automated code reviews on this repo. Configure here.