Experiments

An experiment wraps the part of a page that changes. You write every version of it, called a variant, right in the page. When you publish, Mockly builds each variant ahead of time. When someone visits, the server picks a variant for them and sends it as finished HTML, so the page never flickers from one version to another.

Each experiment is judged on one goal: the action you want visitors to take, such as clicking a button or sending a form. See Goals for the ways to record one, and Reading results for what the numbers mean.

Variants in blocks#

Put each version in its own <Variant>:

import { Experiment, Page, Variant } from "mockly"; export default function Home() {  return (    <Page title="Acme">      <Experiment name="hero" goal="signup">        <Variant name="control">          <h1>Ship faster</h1>          <p>Write the page once and test it as you go.</p>        </Variant>        <Variant name="bold">          <h1>Launch in an afternoon</h1>          <p>Every version is built ahead of time, so nothing flickers.</p>        </Variant>      </Experiment>      <a href="/signup" track="signup">        Start free      </a>    </Page>  );}

• name identifies the experiment in the studio and in its results.
• goal is the goal it's judged on. Here the link records signup when it's clicked.
• Exactly one variant is named control. It's the version the others are compared with, and the one everyone sees while the experiment is paused.

A variant can hold any markup: text, elements, components, and interactive parts. An experiment has between two and eight variants.

Weights#

Without weights, traffic splits evenly between the variants. To split it another way, give every variant a weight:

<Experiment name="pricing-headline" goal="signup">  <Variant name="control" weight={70}>    <h2>Simple pricing</h2>  </Variant>  <Variant name="value" weight={30}>    <h2>Costs less than one hour of engineering time</h2>  </Variant></Experiment>

Weights are relative, so 7 and 3 split traffic the same way as 70 and 30. Each is a positive whole number, and either every variant has one or none does (MK4007). Mockly turns them into whole percentages that add up to 100, which is how the studio shows them.

The weights in your code are where an experiment starts. After the first publish, you change the split in the studio instead, as described under Traffic split below.

Inline variants#

When only a word, a class, or a link changes, copying the whole block into each variant is hard to read, and a fix made in one copy can miss the others. Instead, list the variants in variants and give the experiment one child: a function of the variant's name.

import { Experiment, Page } from "mockly"; export default function Home() {  return (    <Page title="Acme">      <h1>Ship faster</h1>      <Experiment name="cta-color" goal="signup" variants={{ control: 50, green: 50 }}>        {(v) => (          <a href="/signup" track="signup" class={v === "green" ? "bg-green-600 text-white" : "bg-blue-600 text-white"}>            Start free          </a>        )}      </Experiment>    </Page>  );}

The function runs once per variant when you publish, each time with v set to that variant's name. The result is the same experiment, built the same way, as one written with <Variant> blocks. You can use v anywhere an expression goes: an attribute, text, a condition, or a prop of a component.

Weights or an even split#

variants takes one of two forms:

• An object of names and weights, such as { control: 50, green: 50 }. The weights follow the same rules as weight.
• A list of names, such as ["control", "benefit", "social"], for an even split.

A list pairs well with a lookup when each variant changes only the words:

import { Experiment, Page } from "mockly"; const headlines = {  control: "Start your free trial",  benefit: "Launch your first experiment today",  social: "Join the teams that test every launch",}; export default function Signup() {  return (    <Page title="Sign up · Acme">      <Experiment name="signup-headline" goal="signup" variants={["control", "benefit", "social"]}>        {(v) => <h1>{headlines[v]}</h1>}      </Experiment>    </Page>  );}

The variant is typed#

Mockly knows the variant names, so v can only be one of them. A misspelled name is an error before you publish:

<Experiment name="cta-color" goal="signup" variants={{ control: 50, green: 50 }}>  {(v) => <a href="/signup" class={v === "gren" ? "bg-green-600" : "bg-blue-600"}>Start free</a>}</Experiment>

The comparison reports TS2367: "This comparison appears to be unintentional because the types '"control" | "green"' and '"gren"' have no overlap." The studio underlines it as you type.

Rules#

The compiler checks every experiment. Each rule has a diagnostic with a stable code:

Names#

Experiment, variant, and goal names use lowercase letters, digits, and dashes, start with a letter or a digit, and are at most 63 characters long: hero, pricing-headline, cta-2. A name such as Hero or cta_color is MK4002.

The name ties an experiment to its history in the studio. If you rename one in your code and publish, the old experiment is archived and a new one starts with fresh results.

One experiment in several places#

Experiments with the same name are one experiment, on one page or across several. Each has its own content for each variant, and a visitor sees the same variant in all of them. Use it to test one idea everywhere it shows up, such as a pricing message on the home page and on the pricing page:

import { Experiment, Page, Variant } from "mockly"; export default function Home() {  return (    <Page title="Acme">      <h1>Ship faster</h1>      <Experiment name="pricing-message" goal="signup">        <Variant name="control">          <p>Plans start at $8 a month.</p>        </Variant>        <Variant name="value">          <p>Plans cost less than one hour of engineering time.</p>        </Variant>      </Experiment>    </Page>  );}

import { Experiment, Page, Variant } from "mockly"; export default function Pricing() {  return (    <Page title="Pricing · Acme">      <Experiment name="pricing-message" goal="signup">        <Variant name="control">          <h1>Simple pricing</h1>        </Variant>        <Variant name="value">          <h1>Less than one hour of engineering time</h1>        </Variant>      </Experiment>    </Page>  );}

Every declaration must have the same goal, the same secondary goals, and the same variants with the same weights, in any order (MK4017). Weights are compared as shares, so 7 and 3 agree with 70 and 30. One declaration can use <Variant> blocks while another uses variants.

Experiments in components#

An experiment can live in a component, and the component takes props as usual:

import { Experiment } from "mockly"; export function Cta({ href }: { href: string }) {  return (    <Experiment name="cta-label" goal="signup" variants={["control", "trial"]}>      {(v) => (        <a href={href} track="signup" class="rounded-md bg-blue-600 px-4 py-2 text-white">          {v === "trial" ? "Start my trial" : "Create account"}        </a>      )}    </Experiment>  );}

import { Page } from "mockly";import { Cta } from "../components/cta"; export default function Home() {  return (    <Page title="Acme">      <header>        <Cta href="/signup" />      </header>      <h1>Ship faster</h1>      <Cta href="/signup" />    </Page>  );}

When a component is rendered twice, as here in the header and below the headline, both copies show the same variant, and the visitor is counted once.

Who sees what#

Sticky assignment#

A visitor is assigned a variant the first time they're served a page with the experiment on it. From then on they keep it, on every page and on later visits, because Mockly remembers it in a cookie that lasts a year. Assignments are independent: being in bold for one experiment says nothing about the variant of another.

A visitor who clears their cookies, or comes back in another browser, is a new visitor and is assigned again.

Changing the traffic split doesn't move anyone. Visitors who already have a variant keep it, and the new split applies to visitors who arrive after.

Bots, such as search engine crawlers and link previews, get a variant like anyone else, but they aren't counted in results. An experiment with targeting assigns only the visitors it's for.

Previewing a variant#

The studio's preview shows control by default. Above it, there's a control for each experiment on the page that picks the variant to show, and Outline experiments draws a border around each experiment, labeled with its name and variant. In the Experiments tab beside the preview, clicking a variant's name shows it in the preview too.

Nothing you do in the preview counts toward results. To see the variants of a version you published earlier, use the Pages view in History. See Publishing and history.

Running an experiment#

Publishing starts each new experiment at the weights in your code. From then on, the studio controls it. Each experiment has a card in the Experiments tab beside the preview, with its goal, its revision, its traffic split, and raw counts of visitors and conversions. Full results, above the cards, opens the statistics on the Analytics page, explained in Reading results.

Status#

Switch between Running and Paused, or use the Ship menu, which asks which variant to ship. The change takes effect on the next page view, without publishing.

Traffic split#

Traffic, at the bottom of the card, opens a slider and a percentage for each variant. They're linked, so the total is always 100%, and Even split resets them. Simulate runs 10,000 random visitors through the real assignment and shows the split they got next to the target.

Before an experiment has results, a new split saves when you let go of the slider. Once it has results, a new split restarts them, because the visitors so far were split the old way, and mixing two splits can mislead. So the change waits until you click Save and restart results. Everyone keeps their variant, and the results count only visitors who arrive from then on.

Targeting#

Targeting, at the bottom of the card, sets who the experiment is for. Leave a part open to include everyone:

Visitors it isn't for see control and aren't counted, as if the experiment were paused for them. A visitor is checked once, when they would first be assigned. After that they keep their variant, even on a later visit that wouldn't match, such as a new visitor coming back. The source is where the visit came from, so it still counts on the visit's later pages, which arrive from your own site. The card shows who the experiment is for.

Changing who an experiment is for restarts its results, as a new split does: the visitors so far were a different group. Everyone keeps their variant. A publish that starts a new revision keeps the targeting, and shipping a variant shows it to everyone.

Scheduling changes#

Schedule, at the bottom of the card, starts, pauses, or ships a variant at a date and time you pick, in your own time zone. Use it to launch a test on Monday morning, or to stop one before a sale. The card lists the changes waiting, each with Cancel, and says what's next even when the list is closed.

The studio makes each change within a minute of its time. If the studio isn't running then, it makes the change as soon as it starts again. A change that no longer applies is skipped, and says why:

• The experiment is already in that state, such as a start for a running experiment.
• A publish changed its variant names or its goal after it was scheduled, which makes it a new experiment.
• It was removed from your code.

Shipping a variant decides the test, so it cancels the changes still waiting: a pause scheduled for later would otherwise take the winner away from everyone. The Ship menu warns you first.

When publishing restarts an experiment#

The studio keeps an experiment's status and split across publishes, until a publish changes the experiment's variant names or its goal. Then the experiment starts a new revision:

• Visitors are assigned again, from scratch.
• The weights in your code apply again, and the status goes back to running.
• Results start fresh.

The publish plan marks each experiment as New, Unchanged, Restarts, or Archived (removed from your code), so you see what will happen before you confirm. Changing what a variant shows, or its secondary goals, doesn't restart anything.

Variants that look like control#

A variant whose HTML comes out exactly like control's on every page can't win, because visitors who get it see no difference. It happens with a forgotten edit, or an edit made to the wrong variant. Publishing checks for it and warns with MK4018, and so does mockly build. The experiment's card and its results then carry a note:

"bold" looks exactly like control on every page, so it can't win. Change what it shows, or remove it.

The preview can't tell, since it shows one page at a time. Publishing renders every variant of every page, so it can.