Skip to content

Accept experiment context in the analytics event ingest payload #37017

Description

@erickgonzalez

Description

Part 1, Track 2 of #36763 — teach the analytics ingest pipeline to read experiment context off the event payload and write it to the columns added by the schema issue.

Implemented in dotCMS/dot-ca-event-manager. Tracked here for sprint and team ownership.

No new endpoint, no contract change

Worth stating up front because the epic's framing suggests otherwise: nothing is created here. POST /v1/event/ingest already exists and is already the target of core's EventAnalyticsProxyResource. The URL, the auth, the batch shape, and the response contract all stay identical. What widens is only the payload model — one new optional block inside Context, and three more columns in the INSERT.

Payload shape

The experiment block sits in Context, not in event.data, so it is shared across every event in a batch. A visitor is in one experiment at a time, so attaching it once per batch is simpler than repeating it per event.

{
  "context": {
    "session_id": "abc-123",
    "site_id":    "site-xyz",
    "user_id":    "user-456",
    "device":     { "screen_resolution": "1920x1080", "language": "en" },
    "experiment": {
      "id":         "exp-789",
      "running_id": "run-001",
      "variant":    "B"
    }
  },
  "events": [ ... ]
}

When no experiment is running the block is omitted entirely and the event is byte-identical to today's.

Files

Both verified present on main:

File Change
src/main/java/com/dotcms/analytics/ingestion/web/UserEventPayload.java Experiment nested record + experiment field on Context
src/main/java/com/dotcms/analytics/ingestion/repository/SaveEventRepository.java 3 columns and placeholders in INSERT_SQL; writes in insertChunk

Follow the existing Device record for the nested-record pattern, and the existing str() null-to-empty-string helper for the writes so DEFAULT '' semantics hold even when a sub-field is missing.

Validation

The block is optional, but not partially optional. null when no experiment is running is fine; a block that is present with any of id, running_id, or variant missing or blank is a client bug and must be rejected rather than silently written as '' — a half-attributed event is worse than an unattributed one, because it lands in the experiment's result set with a missing dimension. Enforce with @NotBlank on all three fields so Bean Validation rejects before any DB write.

Acceptance Criteria

Payload model

  • Experiment nested record added to UserEventPayload with @NotBlank @JsonProperty on id, running_id, variant
  • @JsonProperty("experiment") Experiment experiment added to the Context record
  • experiment is optional at block level — a payload with no block is accepted unchanged

Persistence

  • experiment_id, running_id, variant added to INSERT_SQL with matching ? placeholders
  • insertChunk writes all three after conversion_name, using str() so a missing sub-field becomes '' rather than null
  • No change required to the JSON schema validators under src/main/resources/analytics/validators/ — they cover event.data sub-maps only, and the experiment block lives in Context. Confirm this rather than assume it.

Tests — new ExperimentIngestionIT

A dedicated class extending AbstractIntegrationTest, following UserAgentIngestionIT. Auth and generic bean validation are already covered by EventIngestionControllerIT; this class covers only experiment-block behaviour.

  • Full experiment block → all three columns land correctly in analytics.events; awaitCount + assertReplicatedCount confirm replication to ch-02
  • No experiment block → all three columns are '', no null, no error
  • Partial block (id present, variant blank or absent) → 400, and nothing is written to the database
  • Multi-event batch → every event in the batch carries the same experiment values from Context

Priority

High

Additional Context

Blocked by the schema issue — the three columns must exist before the INSERT can reference them.

This does not make experiment data flow end to end. Nothing sets context.experiment until the SDK work in Part 3, and that developer is unavailable this sprint. Until then the only producers are the synthetic fixture (tracked separately) and ExperimentIngestionIT itself. That is expected for this sprint, but it means "ingest works" is demonstrated by tests, not by real traffic.

Today's experiment events do not travel this path at all. They go browser → /api/v1/eventEventLogWebInterceptorExperimentEventsPayloadEventLogSubmitter → the Jitsu collector at analyticsApp.getAnalyticsProperties().analyticsWriteUrl(). Retiring that path is Part 3, not this issue.

Parent epic: #36763

Metadata

Metadata

Assignees

Type

Projects

Status
Current Sprint Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions