diff --git a/docs/web-console-docs/holdout/_category_.json b/docs/web-console-docs/holdout/_category_.json new file mode 100644 index 00000000..ce4c37e3 --- /dev/null +++ b/docs/web-console-docs/holdout/_category_.json @@ -0,0 +1,6 @@ +{ + "position": 9, + "collapsible": true, + "collapsed": true, + "label": "Holdouts" +} diff --git a/docs/web-console-docs/holdout/overview.mdx b/docs/web-console-docs/holdout/overview.mdx new file mode 100644 index 00000000..d15e7794 --- /dev/null +++ b/docs/web-console-docs/holdout/overview.mdx @@ -0,0 +1,118 @@ +--- +sidebar_position: 1 +title: "Overview" +description: "Holdouts measure the combined impact of all your experiments by keeping a small group of visitors out of every experiment." +--- + +import Image from "../../../src/components/Image"; + +# Overview + +## What is a holdout + +Individual experiments tell you the impact of a single change. A **holdout** tells you the +impact of everything you shipped, together. + +A holdout keeps a small, randomly selected group of visitors out of every experiment on your +platform. Their behaviour becomes a baseline. By comparing that baseline with visitors who are +exposed to your experiments as normal, you can measure the cumulative effect of your entire +experimentation programme over a period of months rather than the effect of one change at a time. + +This answers questions that individual experiment results cannot: + +- Do the wins we recorded across dozens of experiments add up at the aggregate level? +- Are the effects of our changes still present months after launch, or did they decay? +- Is there an interaction between our experiments that no single test would reveal? + +The Holdouts page listing one holdout group per tracking unit + +## One holdout per tracking unit + +Each tracking unit has its own holdout group. If your platform tracks visitors by User ID, Device ID +and Anonymous ID, you get three separate holdouts, each configured and started independently. + +This mirrors how experiments work. An experiment that randomises on Device ID can only be held out +from a group defined by Device ID, so a holdout can only cover experiments that share its tracking +unit. Holdouts on other units run alongside it, without overlapping. + +A holdout does not have to be running on every unit. Configure and start the ones that are useful to +you and leave the rest stopped. + +:::info +To understand how tracking units are defined on your platform, see +[Tracking Unit](/docs/web-console-docs/experiments/creating-an-experiment#tracking-unit). +::: + +## How the comparison works + +A holdout report compares two groups of equal size: + +| Group | Role | What it sees | +| --- | --- | --- | +| **Holdout** | Control | No experiments. The baseline experience. | +| **No Holdout** | Variant 1 | Experiments as normal, exactly like the rest of your population. | + +The **No Holdout** group is not simply "everyone else". It is a sample drawn from the full +experiment-exposed population and sized to match the holdout group. Comparing a 5% holdout against +the remaining 95% directly would put a small group next to a very large one, and the imbalance costs +you statistical power. Sampling an equal-size comparison group keeps the two sides balanced and the +comparison fair. + +The relative impact reported on each metric therefore reads as: how much the experiment-exposed +group differs from the holdout baseline. + +## Traffic allocation + +A holdout can be allocated between **1% and 10%** of the traffic for its tracking unit, and defaults +to **5%**. The allocated traffic is split equally between the two groups above. + +The size of the allocation is a trade-off. A larger holdout detects smaller cumulative effects, but +it also means more visitors are kept away from the improvements you have shipped. Between 3% and 5% +is usually the right balance for most platforms. + +## Which experiments are included + +Every experiment running on the holdout's tracking unit is included automatically. There is no +opt-in step when you create an experiment, and no way for an experiment to join a holdout by +accident, because inclusion follows the tracking unit. + +Users with the appropriate permissions can exclude individual experiments from a holdout. This is +worth doing sparingly. Each exclusion means the holdout group has seen that experiment after all, so +the baseline drifts a little further from "the product as it was". Reserve exclusions for cases +where withholding a change is not acceptable, such as a legal or compliance requirement, or a fix +that must reach all visitors. + +## Periods and resets + +A holdout runs for a fixed **duration of 3 or 6 months**, chosen at setup. When that period ends, +the holdout can restart automatically: a new period begins immediately with the same configuration, +and visitors are randomised again from scratch. + +Fresh randomisation matters. A visitor who was held out during one period is unlikely to be held out +in the next, which stops the same people from being excluded from your product improvements +indefinitely, and prevents the holdout group from slowly diverging into an unrepresentative +population. + +Each period is an **independent dataset** with its own results. Periods are not merged or averaged. +When a period ends its results are archived and stay accessible, so you can look back at how the +cumulative impact of your experimentation changed from one quarter or half-year to the next. + +## Reading results before the period ends + +You do not have to wait 3 or 6 months to learn something. Holdouts are analysed once per day using **Fully +Sequential testing**, so results are valid whenever you look at them and can be checked at any point +during the period. + +## Metrics + +The metrics used to evaluate a holdout (max 5 of them) are selected from your existing metric catalogue, filtered to +those compatible with the holdout's tracking unit. A metric defined on User ID cannot be computed +for a Device ID holdout, so it is not offered. + +Because a holdout measures broad, long-run impact rather than the effect of one change, favour your +top-line business metrics over narrow, feature-specific ones. + +## Next steps + +- [Setting up a holdout](./setting-up-a-holdout) walks through the configuration. +- [Reading holdout results](./reading-holdout-results) explains the report and the period history. diff --git a/docs/web-console-docs/holdout/reading-holdout-results.mdx b/docs/web-console-docs/holdout/reading-holdout-results.mdx new file mode 100644 index 00000000..77ecfc78 --- /dev/null +++ b/docs/web-console-docs/holdout/reading-holdout-results.mdx @@ -0,0 +1,98 @@ +--- +sidebar_position: 3 +title: "Reading Holdout Results" +description: "Interpret the holdout report, and browse the archived results of previous periods." +--- + +import Image from "../../../src/components/Image"; + +# Reading Holdout Results + +A holdout report shows the cumulative impact of all your experiments on a tracking unit, measured +against a group of visitors who were kept out of every one of them. + +Open a holdout from the **Holdouts** page to see its detail view. The **Overview** tab shows the +results of a single period, and the **Iterations** tab lists every period the holdout has run. + +## The Overview tab + +The Overview tab of a holdout showing the metrics table for the current period + +The header names the period you are looking at, its status and its start date. Use the arrows on the +right to move between periods without leaving the tab. + +Below it, a summary card repeats the holdout's configuration: tracking unit, traffic allocation, +duration and the scheduled start and end dates of this period. + +### The two variants + +Every metric is reported for the same two groups. + +| Variant | Name | What it represents | +| --- | --- | --- | +| Control | **Holdout** | Visitors kept out of all experiments. The baseline. | +| Variant 1 | **No Holdout** | A sample of the experiment-exposed population, sized to match the holdout group. | + +The two rows carry equal participant counts by design. **No Holdout** is sampled from the full +population rather than being every remaining visitor, so that the comparison is between two groups +of the same size. A **Sampling** toggle above the metrics indicates that this equal-size sampling is +in effect. + +### Reading a metric + +Each metric card lists both variants with their participants, value and mean, then the statistics +for the comparison: + +- **Confidence** — how confident the test is that the observed difference is real. +- **Relative impact** — how much the experiment-exposed group differs from the holdout baseline. +- **Impact per day** — the effect expressed as a daily rate. + +Because Control is the holdout, a **positive relative impact means your experiments are performing +better than the unchanged product**. That is the outcome you are hoping for: the changes you shipped +add up to a measurable gain over doing nothing. + +A negative or flat result is equally informative. It suggests the individual wins you recorded did +not accumulate, whether through decay over time, interaction between experiments, or effects that +were smaller than they first appeared. + +:::info +For a fuller explanation of confidence, impact and the statistics behind them, see +[Interpreting metrics in experiment results](/docs/web-console-docs/experiments/Interpreting-metrics-in-experiment-results). +::: + +### Checking results before the period ends + +Holdouts are analysed once per day using Fully Sequential testing, so the numbers are valid whenever +you look at them. You can check a running holdout at any point during its 3 or 6 month period +without waiting for it to finish, and without the risk of an inflated false positive rate that +repeatedly checking a fixed horizon test would carry. + +Because results refresh daily rather than continuously, the figures you see reflect the most recent +analysis rather than the last few minutes of traffic. + +## The Iterations tab + +The Iterations tab listing completed and running holdout periods + +Each period the holdout has run appears here, newest first, with its status, date range and total +participants. The current period shows as **Running**; earlier ones show as **Completed**. + +Expand any period to see its results. + +### Periods are independent + +Each period is a separate dataset with its own randomisation and its own results. Periods are never +merged or averaged, because the holdout group is drawn fresh each time and the experiments running +during one period are not the same as those running during the next. + +Read the sequence as a trend rather than a single accumulating number. A run of periods each showing +a positive impact is strong evidence that your experimentation programme is delivering; a series +that flattens over time is a signal worth investigating. + +Completed periods stay accessible indefinitely, so the archive builds into a long-run record of what +your experimentation has been worth. + +## Related + +- [Holdout overview](./overview) +- [Setting up a holdout](./setting-up-a-holdout) diff --git a/docs/web-console-docs/holdout/setting-up-a-holdout.mdx b/docs/web-console-docs/holdout/setting-up-a-holdout.mdx new file mode 100644 index 00000000..0544006d --- /dev/null +++ b/docs/web-console-docs/holdout/setting-up-a-holdout.mdx @@ -0,0 +1,109 @@ +--- +sidebar_position: 2 +title: "Setting up a Holdout" +description: "Configure a holdout group's traffic allocation, metrics and schedule, then start it." +--- + +import Image from "../../../src/components/Image"; + +# Setting up a Holdout + +A step by step guide to configuring a holdout group and starting it. + +Holdout groups are not created from scratch. One already exists for each tracking unit on your +platform, waiting to be configured. Open the **Holdouts** page from the left navigation to see them +all, along with the status, traffic allocation, metrics and schedule of each. + +The Holdouts page listing one holdout group per tracking unit + +Select the holdout you want to configure to open its settings. + +The Configure Holdout page + +## General + +Give the holdout a **Name** and an optional **Description**. These appear on the Holdouts page and at +the top of the report, so describe what the holdout covers, for example +`Holdout group for all experiments tracking visitors by device`. + +## Configuration + +### Tracking unit + +The **tracking unit** is fixed for each holdout and cannot be changed. Only experiments that track +visitors with this unit are part of this holdout. + +If you want to hold out visitors identified by a different unit, configure that unit's own holdout +instead. + +### Traffic allocation + +Choose what share of this unit's traffic the holdout uses, between **1% and 10%**. The default is +**5%**, and 3% to 5% is recommended for most platforms. + +The allocated traffic is split equally between the two holdout variants: the group kept out of all +experiments, and the equal-size sample of the general population it is compared against. + +:::tip +Increasing the allocation lets you detect smaller cumulative effects, but keeps more visitors away +from the improvements you have shipped. If you are unsure, start at the 5% default. +::: + +## Metrics + +Select the metrics used to evaluate the holdout, from your existing metric catalogue. A holdout can +use up to **5 metrics**. + +Only metrics compatible with the holdout's tracking unit are shown, and the active filter is +displayed above the selector. A metric defined on a different unit cannot be computed for this +holdout, so it does not appear in the list. + +Choose metrics that reflect the overall health of your product rather than the success of any one +feature. A holdout measures the combined effect of everything you shipped, so top-line business +metrics are the ones worth watching. + +## Schedule + +### Duration + +Choose a **duration** of **3 months** or **6 months**. Set a **start date**, and the end date is +calculated for you. + +A longer period accumulates more experiments and more data, which makes small cumulative effects +easier to detect. A shorter period gives you a completed, archived result sooner and refreshes the +randomisation more often. Three months suits platforms running many experiments; six months suits +those with fewer experiments or slower-moving metrics. + +### Restart automatically when holdout ends + +When this is enabled, a new holdout period with the same configuration begins immediately once the +current one ends, and visitors are randomised again from scratch. A visitor who was in the holdout +group during one period might not be in it during the next. + +Automatic restart is disabled when a holdout is stopped manually. + +## Starting and stopping + +Save the configuration, then start the holdout when you are ready. Running holdouts show a +**Running** status on the Holdouts page, along with their current period's dates. + +:::caution +Changes saved while a holdout is running only apply to the **next** period. The current period keeps +the configuration it started with, so its dataset stays internally consistent. To change a running +holdout immediately, stop it first, then save. +::: + +## Excluding an experiment + +Every experiment on the holdout's tracking unit is included automatically, with no opt-in step +during experiment creation. + +Users with the appropriate permissions can exclude an individual experiment from the holdout. Do +this only when withholding a change is genuinely not acceptable, such as a legal or compliance +requirement, or a fix that has to reach every visitor. Each exclusion means the holdout group has +been exposed to that experiment, which moves the baseline a little further from the unchanged +product and weakens the comparison. + +## Next steps + +Once the holdout is running, see [Reading holdout results](./reading-holdout-results). diff --git a/static/img/holdout/configure-holdout.png b/static/img/holdout/configure-holdout.png new file mode 100644 index 00000000..dd5d945a Binary files /dev/null and b/static/img/holdout/configure-holdout.png differ diff --git a/static/img/holdout/holdout-iterations.png b/static/img/holdout/holdout-iterations.png new file mode 100644 index 00000000..bd4486e5 Binary files /dev/null and b/static/img/holdout/holdout-iterations.png differ diff --git a/static/img/holdout/holdout-results.png b/static/img/holdout/holdout-results.png new file mode 100644 index 00000000..52eb8fef Binary files /dev/null and b/static/img/holdout/holdout-results.png differ diff --git a/static/img/holdout/holdouts-list.png b/static/img/holdout/holdouts-list.png new file mode 100644 index 00000000..24865ce8 Binary files /dev/null and b/static/img/holdout/holdouts-list.png differ