Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/APIs-and-SDKs/SDK-Documentation/getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Before you start, make sure you have:
<InitializeBasic />

<details>
<summary>Using a custom event logger</summary>
<summary id="using-a-custom-event-logger">Using a custom event logger</summary>

The SDK fires events internally for things like exposures and goal tracking. By default
these are handled automatically, but you can plug in a custom event logger if you need to:
Expand Down
2 changes: 1 addition & 1 deletion docs/APIs-and-SDKs/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ This is the right surface for:
- Bulk imports/exports
- Programmatic governance — auditing experiments, metrics or permissions

See the [API examples](/docs/APIs-and-SDKs/Web-Console-API/Examples/Slack-Integration)
See the [API examples](/docs/APIs-and-SDKs/Web-Console-API/examples/slack-integration)
for end-to-end integrations such as a custom Slack notifier.

### MCP Server
Expand Down
2 changes: 1 addition & 1 deletion docs/Third-party-integrations/Intercom-Integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ example, to measure whether a product change reduces the number of support conve
| `X-Environment` | The environment name matching your ABsmartly Web Console (e.g. `production`) |
| `X-Agent` | `intercom-webhook` |

7. Configure the **request body** with key-value pairs. Use the **{...}** menu to insert
7. Configure the **request body** with key-value pairs. Use the **`{...}`** menu to insert
dynamic Intercom attributes. Build the following payload:

```json
Expand Down
19 changes: 19 additions & 0 deletions docs/Third-party-integrations/SSO/Azure SAML Setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,31 @@ to get it set up!

1. Go to the [Microsoft Entra admin center](https://entra.microsoft.com/) and sign in with your Azure account.
2. [Click this link](https://entra.microsoft.com/#view/Microsoft_AAD_IAM/AppGalleryBladeV2) or head to **Identity** > **Applications** > **Enterprise Applications** > **All applications** > **New application**.

<Image
img="azure/new_application.png"
alt="A screenshot of the New application button"
/>

3. In the **Browse Microsoft Entra Gallery** page, click **+ Create your own application**.

<Image
img="azure/create_application.png"
alt="A screenshot of the Create application button"
/>

4. Name the application **ABsmartly** and leave the radio button on **Integrate any other application you don't find in the gallery (Non-gallery)**.

<Image
img="azure/application_info.png"
alt="A screenshot of the completed 'New application' form"
/>

5. Click **Create** and you will be redirected to the Overview page of your new application.

:::info
Feel free to add [the ABsmartly logo](/img/logo.png) to the application in **Properties** > **Logo**.

<Image
img="logo.png"
alt="The ABsmartly logo"
Expand All @@ -36,35 +43,43 @@ Feel free to add [the ABsmartly logo](/img/logo.png) to the application in **Pro
:::

6. In the inner left pane, click on **Single sign-on** and **SAML**.

<Image
img="azure/sign_on_saml.png"
alt="A screenshot of the Single sign-on and SAML buttons"
/>

7. Next to **Basic SAML Configuration**, click **Edit** and input the following info:
- **Identifier (Entity ID)**: `absmartly-<your-assigned-subdomain>`
- **Reply URL (Assertion Consumer Service URL)**: `https://<your-assigned-subdomain>.absmartly.com/auth/saml/azure/callback`
- **Sign on URL**: `https://<your-assigned-subdomain>.absmartly.com`

<Image
img="azure/basic_saml_edit.png"
alt="A screenshot of the Basic SAML Configuration panel and the Edit button"
/>

<Image
img="azure/basic_saml_values.png"
alt="A screenshot of the Basic SAML Configuration panel with the Identifier, Reply URL, and Sign on URL filled in"
/>

8. Click **Save** and close the **Basic SAML Configuration** panel.
9. In the **User Attributes & Claims** section, click **Edit** and edit the **Unique User Identifier (Name ID)** (click on its name) with the following:
- **Name identifier format**: `Email address`
- **Source**: `Attribute`
- **Source attribute**: `user.mail`

<Image
img="azure/user_attributes_edit.png"
alt="A screenshot of the User Attributes & Claims panel and the Edit button"
/>

<Image
img="azure/unique_id_edit.png"
alt="A screenshot of the list of attributes with the Unique User Identifier (Name ID) highlighted"
/>

<Image
img="azure/unique_id_values.png"
alt="A screenshot of the Unique User Identifier (Name ID) panel with the Name identifier format, Source, and Source attribute filled in"
Expand All @@ -84,20 +99,24 @@ Feel free to add [the ABsmartly logo](/img/logo.png) to the application in **Pro
/>

12. The **SAML Based Sign-on** page should now look like this:

<Image
img="azure/single_sign_on_result.png"
alt="A screenshot of the completed SAML Based Sign-on page"
/>

13. Lastly, in the **SAML Certificates** section, click to download your
**Federtion Metadata XML** file, send it to us through email or via Slack
and we will have your SSO setup ASAP!

<Image
img="azure/saml_certificates.png"
alt="A screenshot of the SAML Certificates section with the Download button highlighted"
/>

:::info
Don't forget to assign users to the application in the **Users and groups** section!

<Image
img="azure/users_and_groups.png"
alt="A screenshot of the Users and groups section"
Expand Down
5 changes: 5 additions & 0 deletions docs/Third-party-integrations/SSO/Google SAML Setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,16 @@ to get it set up!

1. Go to the [Google Admin Console](https://admin.google.com) and log in as an administrator.
2. In the sidebar, navigate to `Apps > Web and Mobile Apps`.

<Image
img="google/sidebar.png"
alt="Google Admin Console sidebar"
maxWidth={300}
centered={false}
/>

3. At the top, click `Add app > Add custom SAML app`.

<Image
img="google/add_app.png"
alt="Google Admin Console Add custom SAML app button"
Expand Down Expand Up @@ -45,12 +48,14 @@ You can use the ABsmartly logo by uploading [this image](/img/logo.png).
/>

5. Click `Download Metadata` and save the file somewhere safe. At the end of this guide you will send your metadata to us so that we can get you set up quickly. Click `Continue`.

<Image
img="google/create_app_2.png"
alt="Google Admin Console Create App Step 2"
maxWidth={800}
centered={false}
/>

6. For these fields input:
- **ACS URL**: `https://<your-assigned-subdomain>.absmartly.com/auth/saml/google/callback`
- **Entity ID**: `absmartly-<your-assigned-subdomain>`
Expand Down
10 changes: 5 additions & 5 deletions docs/glossary.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ To avoid such bias, ABsmartly recommends pre-registering the [decision criteria]
### Confidence level
The probability that the [confidence interval](#confidence-interval) procedure will capture the true value, across many hypothetical repetitions of the experiment.

Common choices such as 90, 95 percent or 99 percent define how strict you are about uncertainty and directly relate to the [significance level](#significance-level).
Common choices such as 90, 95 percent or 99 percent define how strict you are about uncertainty and directly relate to the [significance level](#significance-level-alpha).
A higher confidence level reduces the risk of [false positive](#false-positive) but requires more data (wider intervals).

### Continuous metric
Expand Down Expand Up @@ -550,7 +550,7 @@ This outlier can shift the average upward, making the treatment look better than
Peeking refers to looking at experiment results before the test is completed, especially to check for statistical significance and
making decisions based on those early results without proper statistical adjustments.

Peeking inflates the [false positive rate](##false-discovery-rate-fdr), making it more likely that you'll incorrectly conclude a treatment is effective when it’s not.
Peeking inflates the [false positive rate](#false-discovery-rate-fdr), making it more likely that you'll incorrectly conclude a treatment is effective when it’s not.
This happens because repeatedly checking increases the chance that random noise appears significant at least once.

**Example**:
Expand Down Expand Up @@ -615,7 +615,7 @@ P-hacking severely inflates false positives and creates misleading “evidence

### Sample size
The number of visitors included in an experiment.
Sample size, together with [variance](#variance) and [effect size](#effect-size), determines [power](##experimentation-power) and the time needed to reach a conclusion.
Sample size, together with [variance](#variance) and [effect size](#effect-size), determines [power](#experimentation-power) and the time needed to reach a conclusion.

### Secondary metrics
Additional metrics tracked in an experiment to understand side effects or support interpretation of the primary metric.
Expand All @@ -624,7 +624,7 @@ Secondary metrics reveal trade-offs and help explain why a primary metric change
**Example:** Monitoring average order value and gross conversions while the primary metric is net conversion rate.

### Significance level (alpha)
The maximum acceptable probability of a [Type I error](#type-1-error) that you are willing to tolerate in a single hypothesis test.
The maximum acceptable probability of a [Type I error](#type-i-error) that you are willing to tolerate in a single hypothesis test.
It determines the threshold at which you consider a result statistically significant.

**Example:** If you set α = 0.05, and your p-value is below 0.05, you declare the test result statistically significant — i.e., there's enough evidence to reject the null hypothesis.
Expand All @@ -646,7 +646,7 @@ ABsmartly automatically checks for SRM and reports any issue to the experimenter

### Standard deviation
A measure of how spread out or variable your data is. It tells you, on average, how far each data point is from the mean (average).
Standard deviation is central to many formulas for [confidence intervals](#confidnce-interval), [z-scores](#z-score) and sample size calculations.
Standard deviation is central to many formulas for [confidence intervals](#confidence-interval), [z-scores](#z-score) and sample size calculations.

- If your data points are close to the mean, the standard deviation is small.
- If your data points are widely spread out, the standard deviation is large.
Expand Down
2 changes: 1 addition & 1 deletion docs/platform-release-notes/2025/02.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ designed to help you track your experimentation program more effectively.
## Beta Feature: Velocity Report

We're introducing the
[Velocity Report (Beta)](/docs/web-console-docs/experiments/Experiment-reports) to
[Velocity Report (Beta)](/docs/web-console-docs/experiments/experiment-reports) to
give teams better visibility into experimentation speed and execution
trends. This report helps you understand how efficiently experiments move
through different stages, providing insights to optimize your
Expand Down
2 changes: 1 addition & 1 deletion docs/platform-release-notes/2025/04.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ In this release, we focused mainly on improvements to the **Velocity** and **Dec
Earlier this month we launched the new [Decisions Reports (Beta)](/docs/web-console-docs/experiments/experiment-reports/#decisions-overview). In this release we are introducing a few improvements to provide better insights into each decision.

- **Secondary & Guardrail metrics:** Like with the primary metric, you can now see the experiment results on the secondary and guardrail metrics. This helps understand the rationale behind each decision.
- **Health checks:** We are now surfacing any [health check violations](/docs/web-console-docs/experiments/Experiment-health-checks) in the reports so you can get better insights into the quality of each decision.
- **Health checks:** We are now surfacing any [health check violations](/docs/web-console-docs/experiments/experiment-health-checks) in the reports so you can get better insights into the quality of each decision.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ weekly or monthly.

#### Filter Events

As mentioned in the [Event Filters section](#event-filters), you can filter
As mentioned in the [Filter Events section](#filter-events), you can filter
your events to fine tune what kind of events are exported. The `Start At`
field is required and is the date and time of the earliest event that you want
to export. For recurring exports, after the first export, the `Start At` event
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ created and edited in the [Dashboard Settings](/docs/web-console-docs/configurat

Select which unit types you want to see events from. Common unit types are `user_id`
or `anonymous_id`, but they could be anything depending on your setup. Unit
Types can be created in the [Dashboard Settings](/docs/web-console-docs/configuration/settings#units).
Types can be created in the [Dashboard Settings](/docs/web-console-docs/configuration/units).

### Event Type

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,12 +93,12 @@ If the experiment is to run on a part of the product where visitors are not logg
ABsmartly does not create nor use any third-party cookie. It is the responsibility of the experimenters to provide a unique identifier for the visitors of the experiment and if needed to store it in a first-party cookie.
:::

You can create new tracking units (there is no limit to the number of identifier you create) in your [dashboard settings](/docs/web-console-docs/Configuration/settings#units).
You can create new tracking units (there is no limit to the number of identifier you create) in your [dashboard settings](/docs/web-console-docs/configuration/units).
### Applications

**Applications** defined where this experiment will run. An experiment can be running on several platforms at the same time.

These could be `android`, `ios` and `web`, for example, and are also created in your [dashboard settings](/docs/web-console-docs/Configuration/settings#applications).
These could be `android`, `ios` and `web`, for example, and are also created in your [dashboard settings](/docs/web-console-docs/configuration/applications).
### Targeting Audience

In this section you can define which particular audience will be exposed to the experiment. For example, you could decide to only run the experiment for visitors on an `android` device or visitors whose language is set to `english`.
Expand Down Expand Up @@ -281,7 +281,7 @@ The metadata step is where you can fill in details about your experiment that ma

### Metadata

The metadata section allows you to select the owners of this experiment, assign it to a particular team and add tags to help with searching and filtering later on. These fields may be required, optional or hidden depending on your company's [platform settings](/docs/web-console-docs/Configuration/settings#platform-settings).
The metadata section allows you to select the owners of this experiment, assign it to a particular team and add tags to help with searching and filtering later on. These fields may be required, optional or hidden depending on your company's [platform settings](/docs/web-console-docs/configuration/settings#platform-settings).

#### Tags

Expand All @@ -296,7 +296,7 @@ These allow for you to filter your experiments in the experiments list page, but

### Description

When creating a new experiment, the description section acts as your contract for the test. Allowing you to define for yourself and your team why you are running this experiment, what you hope for the result to be and what will be done after any of the experiment's possible outcomes. The following are our default fields, but they may be different depending on your company's [platform settings](/docs/web-console-docs/Configuration/settings#platform-settings).
When creating a new experiment, the description section acts as your contract for the test. Allowing you to define for yourself and your team why you are running this experiment, what you hope for the result to be and what will be done after any of the experiment's possible outcomes. The following are our default fields, but they may be different depending on your company's [platform settings](/docs/web-console-docs/configuration/settings#platform-settings).

#### Hypothesis

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Knowing that something does not have the expected impact on customer behaviour c
In the case of **early stopped experiments** (also known as aborted experiments), the decisions to abort can happen because of bugs in the implementation,
early worrying negative signals, strategic reasons or for any other reason.
Depending on the reason for early stopping, aborted experiments are typically restarted once the underlying issue is fixed.
Read our [When to abort an experiment?](Aborting-experiments) guide to understand more about aborting experiments.
Read our [When to abort an experiment?](aborting-experiments) guide to understand more about aborting experiments.

While aborting experiments is common and part of the process, **early full on**, should be rare as they indicate that changes were pushed without supporting evidence.
This can sometimes happen for strategic or legal reasons but because the experiment did not complete, it does not provide the reliable evidence needed to support or discard the underlying hypothesis.
Expand Down Expand Up @@ -160,7 +160,7 @@ This typically happens when the evidence does not support the hypothesis.
This widget provides a view on how many `Abort` decisions were made in the reporting period.
Aborting means stopping an experiment before it is completed. Aborting can happen when a bug is found or
when early negative signals indicate a possible degradation in certain key metrics but it could also be a strategic choice to stop early.
Read our [When to abort an experiment?](Aborting-experiments) guide to understand more about aborting experiments.
Read our [When to abort an experiment?](aborting-experiments) guide to understand more about aborting experiments.
Like with `Keep current` decisions, `Abort` decisons means that the current experience remains but unlike `Keep current` decisions it does not say anything about the hypothesis being tested or not.

## Decisions history
Expand Down
8 changes: 4 additions & 4 deletions docs/web-console-docs/experiments/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ When a user reaches an experiment, ABsmartly assigns them to a variant using a d
- consistent behaviour during the entire experiment
- predictable control of traffic allocation

[Exposure events](../Events/exposure-events) are sent automatically by the SDK, and these events define when users become part of the analysis dataset.
[Exposure events](../events/exposure-events) are sent automatically by the SDK, and these events define when users become part of the analysis dataset.

### Variants

Expand All @@ -65,7 +65,7 @@ context.track("add_to_cart", { product_id: "ABC123" });
context.track("view_item", { item_id: "XYZ987" });
```

Metrics take your [goal events](../Events/goal-events) and turn them into meaningful measurements of user behaviour.
Metrics take your [goal events](../events/goal-events) and turn them into meaningful measurements of user behaviour.
They let you answer questions like:

- how many times something happened
Expand All @@ -77,14 +77,14 @@ ABsmartly handles all the computation and presents the results in a clear, compa

### Guardrails and monitoring

Before experiment results can be trusted, ABsmartly performs several [health checks](Experiment-health-checks) automatically:
Before experiment results can be trusted, ABsmartly performs several [health checks](experiment-health-checks) automatically:
- Sample Ratio Mismatch at the experiment level
- assignment and exposure conflicts
- guardrail metric thresholds
- unexpected behaviour in exposure or goal events
- data quality anomalies

Guardrails help detect harmful side effects and make it possible to [abort experiments](Aborting-experiments) if needed.
Guardrails help detect harmful side effects and make it possible to [abort experiments](aborting-experiments) if needed.

You can also configure guardrail metrics with [thresholds](../goals-and-metrics/metrics/create#metric-threshold-alert) that alert you when impact crosses meaningful limits,
for example when a performance metric becomes slower or when a business KPI shows a potential drop.
Expand Down
Loading