docs(holdout): add Holdouts product documentation

docs/web-console-docs/holdout/_category_.json

@@ -0,0 +1,6 @@
+{
+  "position": 9,
+  "collapsible": true,
+  "collapsed": true,
+  "label": "Holdouts"
+}

docs/web-console-docs/holdout/overview.mdx

@@ -0,0 +1,118 @@
+---
+sidebar_position: 1
+title: "Overview"
+description: "Holdouts measure the combined impact of all your experiments by keeping a small group of visitors out of every experiment."
+---
+
+import Image from "../../../src/components/Image";
+
+# Overview
+
+## What is a holdout
+
+Individual experiments tell you the impact of a single change. A **holdout** tells you the
+impact of everything you shipped, together.
+
+A holdout keeps a small, randomly selected group of visitors out of every experiment on your
+platform. Their behaviour becomes a baseline. By comparing that baseline with visitors who are
+exposed to your experiments as normal, you can measure the cumulative effect of your entire
+experimentation programme over a period of months rather than the effect of one change at a time.
+
+This answers questions that individual experiment results cannot:
+
+- Do the wins we recorded across dozens of experiments add up at the aggregate level?
+- Are the effects of our changes still present months after launch, or did they decay?
+- Is there an interaction between our experiments that no single test would reveal?
+
+<Image maxWidth="48rem" centered img="holdout/holdouts-list.png" alt="The Holdouts page listing one holdout group per tracking unit" />
+
+## One holdout per tracking unit
+
+Each tracking unit has its own holdout group. If your platform tracks visitors by User ID, Device ID
+and Anonymous ID, you get three separate holdouts, each configured and started independently.
+
+This mirrors how experiments work. An experiment that randomises on Device ID can only be held out
+from a group defined by Device ID, so a holdout can only cover experiments that share its tracking
+unit. Holdouts on other units run alongside it, without overlapping.
+
+A holdout does not have to be running on every unit. Configure and start the ones that are useful to
+you and leave the rest stopped.
+
+:::info
+To understand how tracking units are defined on your platform, see
+[Tracking Unit](/docs/web-console-docs/experiments/creating-an-experiment#tracking-unit).
+:::
+
+## How the comparison works
+
+A holdout report compares two groups of equal size:
+
+| Group | Role | What it sees |
+| --- | --- | --- |
+| **Holdout** | Control | No experiments. The baseline experience. |
+| **No Holdout** | Variant 1 | Experiments as normal, exactly like the rest of your population. |
+
+The **No Holdout** group is not simply "everyone else". It is a sample drawn from the full
+experiment-exposed population and sized to match the holdout group. Comparing a 5% holdout against
+the remaining 95% directly would put a small group next to a very large one, and the imbalance costs
+you statistical power. Sampling an equal-size comparison group keeps the two sides balanced and the
+comparison fair.
+
+The relative impact reported on each metric therefore reads as: how much the experiment-exposed
+group differs from the holdout baseline.
+
+## Traffic allocation
+
+A holdout can be allocated between **1% and 10%** of the traffic for its tracking unit, and defaults
+to **5%**. The allocated traffic is split equally between the two groups above.
+
+The size of the allocation is a trade-off. A larger holdout detects smaller cumulative effects, but
+it also means more visitors are kept away from the improvements you have shipped. Between 3% and 5%
+is usually the right balance for most platforms.
+
+## Which experiments are included
+
+Every experiment running on the holdout's tracking unit is included automatically. There is no
+opt-in step when you create an experiment, and no way for an experiment to join a holdout by
+accident, because inclusion follows the tracking unit.
+
+Users with the appropriate permissions can exclude individual experiments from a holdout. This is
+worth doing sparingly. Each exclusion means the holdout group has seen that experiment after all, so
+the baseline drifts a little further from "the product as it was". Reserve exclusions for cases
+where withholding a change is not acceptable, such as a legal or compliance requirement, or a fix
+that must reach all visitors.
+
+## Periods and resets
+
+A holdout runs for a fixed **duration of 3 or 6 months**, chosen at setup. When that period ends,
+the holdout can restart automatically: a new period begins immediately with the same configuration,
+and visitors are randomised again from scratch.
+
+Fresh randomisation matters. A visitor who was held out during one period is unlikely to be held out
+in the next, which stops the same people from being excluded from your product improvements
+indefinitely, and prevents the holdout group from slowly diverging into an unrepresentative
+population.
+
+Each period is an **independent dataset** with its own results. Periods are not merged or averaged.
+When a period ends its results are archived and stay accessible, so you can look back at how the
+cumulative impact of your experimentation changed from one quarter or half-year to the next.
+
+## Reading results before the period ends
+
+You do not have to wait 3 or 6 months to learn something. Holdouts are analysed once per day using **Fully
+Sequential testing**, so results are valid whenever you look at them and can be checked at any point
+during the period.
+
+## Metrics
+
+The metrics used to evaluate a holdout (max 5 of them) are selected from your existing metric catalogue, filtered to
+those compatible with the holdout's tracking unit. A metric defined on User ID cannot be computed
+for a Device ID holdout, so it is not offered.
+
+Because a holdout measures broad, long-run impact rather than the effect of one change, favour your
+top-line business metrics over narrow, feature-specific ones.
+
+## Next steps
+
+- [Setting up a holdout](./setting-up-a-holdout) walks through the configuration.
+- [Reading holdout results](./reading-holdout-results) explains the report and the period history.

docs/web-console-docs/holdout/reading-holdout-results.mdx

@@ -0,0 +1,98 @@
+---
+sidebar_position: 3
+title: "Reading Holdout Results"
+description: "Interpret the holdout report, and browse the archived results of previous periods."
+---
+
+import Image from "../../../src/components/Image";
+
+# Reading Holdout Results
+
+A holdout report shows the cumulative impact of all your experiments on a tracking unit, measured
+against a group of visitors who were kept out of every one of them.
+
+Open a holdout from the **Holdouts** page to see its detail view. The **Overview** tab shows the
+results of a single period, and the **Iterations** tab lists every period the holdout has run.
+
+## The Overview tab
+
+<Image maxWidth="48rem" centered img="holdout/holdout-results.png" alt="The Overview tab of a holdout showing the metrics table for the current period" />
+
+The header names the period you are looking at, its status and its start date. Use the arrows on the
+right to move between periods without leaving the tab.
+
+Below it, a summary card repeats the holdout's configuration: tracking unit, traffic allocation,
+duration and the scheduled start and end dates of this period.
+
+### The two variants
+
+Every metric is reported for the same two groups.
+
+| Variant | Name | What it represents |
+| --- | --- | --- |
+| Control | **Holdout** | Visitors kept out of all experiments. The baseline. |
+| Variant 1 | **No Holdout** | A sample of the experiment-exposed population, sized to match the holdout group. |
+
+The two rows carry equal participant counts by design. **No Holdout** is sampled from the full
+population rather than being every remaining visitor, so that the comparison is between two groups
+of the same size. A **Sampling** toggle above the metrics indicates that this equal-size sampling is
+in effect.
+
+### Reading a metric
+
+Each metric card lists both variants with their participants, value and mean, then the statistics
+for the comparison:
+
+- **Confidence** — how confident the test is that the observed difference is real.
+- **Relative impact** — how much the experiment-exposed group differs from the holdout baseline.
+- **Impact per day** — the effect expressed as a daily rate.
+
+Because Control is the holdout, a **positive relative impact means your experiments are performing
+better than the unchanged product**. That is the outcome you are hoping for: the changes you shipped
+add up to a measurable gain over doing nothing.
+
+A negative or flat result is equally informative. It suggests the individual wins you recorded did
+not accumulate, whether through decay over time, interaction between experiments, or effects that
+were smaller than they first appeared.
+
+:::info
+For a fuller explanation of confidence, impact and the statistics behind them, see
+[Interpreting metrics in experiment results](/docs/web-console-docs/experiments/Interpreting-metrics-in-experiment-results).
+:::
+
+### Checking results before the period ends
+
+Holdouts are analysed once per day using Fully Sequential testing, so the numbers are valid whenever
+you look at them. You can check a running holdout at any point during its 3 or 6 month period
+without waiting for it to finish, and without the risk of an inflated false positive rate that
+repeatedly checking a fixed horizon test would carry.
+
+Because results refresh daily rather than continuously, the figures you see reflect the most recent
+analysis rather than the last few minutes of traffic.
+
+## The Iterations tab
+
+<Image maxWidth="48rem" centered img="holdout/holdout-iterations.png" alt="The Iterations tab listing completed and running holdout periods" />
+
+Each period the holdout has run appears here, newest first, with its status, date range and total
+participants. The current period shows as **Running**; earlier ones show as **Completed**.
+
+Expand any period to see its results.
+
+### Periods are independent
+
+Each period is a separate dataset with its own randomisation and its own results. Periods are never
+merged or averaged, because the holdout group is drawn fresh each time and the experiments running
+during one period are not the same as those running during the next.
+
+Read the sequence as a trend rather than a single accumulating number. A run of periods each showing
+a positive impact is strong evidence that your experimentation programme is delivering; a series
+that flattens over time is a signal worth investigating.
+
+Completed periods stay accessible indefinitely, so the archive builds into a long-run record of what
+your experimentation has been worth.
+
+## Related
+
+- [Holdout overview](./overview)
+- [Setting up a holdout](./setting-up-a-holdout)

docs/web-console-docs/holdout/setting-up-a-holdout.mdx

@@ -0,0 +1,109 @@
+---
+sidebar_position: 2
+title: "Setting up a Holdout"
+description: "Configure a holdout group's traffic allocation, metrics and schedule, then start it."
+---
+
+import Image from "../../../src/components/Image";
+
+# Setting up a Holdout
+
+A step by step guide to configuring a holdout group and starting it.
+
+Holdout groups are not created from scratch. One already exists for each tracking unit on your
+platform, waiting to be configured. Open the **Holdouts** page from the left navigation to see them
+all, along with the status, traffic allocation, metrics and schedule of each.
+
+<Image maxWidth="48rem" centered img="holdout/holdouts-list.png" alt="The Holdouts page listing one holdout group per tracking unit" />
+
+Select the holdout you want to configure to open its settings.
+
+<Image maxWidth="40rem" centered img="holdout/configure-holdout.png" alt="The Configure Holdout page" />
+
+## General
+
+Give the holdout a **Name** and an optional **Description**. These appear on the Holdouts page and at
+the top of the report, so describe what the holdout covers, for example
+`Holdout group for all experiments tracking visitors by device`.
+
+## Configuration
+
+### Tracking unit
+
+The **tracking unit** is fixed for each holdout and cannot be changed. Only experiments that track
+visitors with this unit are part of this holdout.
+
+If you want to hold out visitors identified by a different unit, configure that unit's own holdout
+instead.
+
+### Traffic allocation
+
+Choose what share of this unit's traffic the holdout uses, between **1% and 10%**. The default is
+**5%**, and 3% to 5% is recommended for most platforms.
+
+The allocated traffic is split equally between the two holdout variants: the group kept out of all
+experiments, and the equal-size sample of the general population it is compared against.
+
+:::tip
+Increasing the allocation lets you detect smaller cumulative effects, but keeps more visitors away
+from the improvements you have shipped. If you are unsure, start at the 5% default.
+:::
+
+## Metrics
+
+Select the metrics used to evaluate the holdout, from your existing metric catalogue. A holdout can
+use up to **5 metrics**.
+
+Only metrics compatible with the holdout's tracking unit are shown, and the active filter is
+displayed above the selector. A metric defined on a different unit cannot be computed for this
+holdout, so it does not appear in the list.
+
+Choose metrics that reflect the overall health of your product rather than the success of any one
+feature. A holdout measures the combined effect of everything you shipped, so top-line business
+metrics are the ones worth watching.
+
+## Schedule
+
+### Duration
+
+Choose a **duration** of **3 months** or **6 months**. Set a **start date**, and the end date is
+calculated for you.
+
+A longer period accumulates more experiments and more data, which makes small cumulative effects
+easier to detect. A shorter period gives you a completed, archived result sooner and refreshes the
+randomisation more often. Three months suits platforms running many experiments; six months suits
+those with fewer experiments or slower-moving metrics.
+
+### Restart automatically when holdout ends
+
+When this is enabled, a new holdout period with the same configuration begins immediately once the
+current one ends, and visitors are randomised again from scratch. A visitor who was in the holdout
+group during one period might not be in it during the next.
+
+Automatic restart is disabled when a holdout is stopped manually.
+
+## Starting and stopping
+
+Save the configuration, then start the holdout when you are ready. Running holdouts show a
+**Running** status on the Holdouts page, along with their current period's dates.
+
+:::caution
+Changes saved while a holdout is running only apply to the **next** period. The current period keeps
+the configuration it started with, so its dataset stays internally consistent. To change a running
+holdout immediately, stop it first, then save.
+:::
+
+## Excluding an experiment
+
+Every experiment on the holdout's tracking unit is included automatically, with no opt-in step
+during experiment creation.
+
+Users with the appropriate permissions can exclude an individual experiment from the holdout. Do
+this only when withholding a change is genuinely not acceptable, such as a legal or compliance
+requirement, or a fix that has to reach every visitor. Each exclusion means the holdout group has
+been exposed to that experiment, which moves the baseline a little further from the unchanged
+product and weakens the comparison.
+
+## Next steps
+
+Once the holdout is running, see [Reading holdout results](./reading-holdout-results).