Run an A/B test

Hey - by the end of this page you'll have a two-variant test running in your app: each install sees one variant, you can force either one from Settings → Developer tools, and every person who sees a variant is counted once. The experiments layer is core code, so the same screen works with every analytics option. What changes is where the split comes from: PostHog's feature flag, or a split on the device.

SetupCopy Anchor Link

Register the experiment in src/experiments.ts. The first variant is the control:

import { defineExperiments } from "@/lib/experiments";

export const experiments = defineExperiments({
  "paywall-copy": { variants: ["control", "short"] },
});

The key is the name you'll see in PostHog and in the event, so use lowercase words joined by hyphens. Two optional fields: weights gives each variant a share (one number per variant, an equal split when you leave it out), and local: true keeps the split on the device even when PostHog is selected.

With PostHog selected (bun run setup --analytics posthog), create the same experiment there. Open Feature flags, create a flag with the same key, choose multiple variants and add variant keys that match your registry exactly: control and short. Then create an experiment on that flag and start it. A flag with another key, or a plain on/off flag, gives undefined and everyone sees the control.

With analytics/none or analytics/amplitude there is no flag to create. Amplitude's experiments are a separate SDK that this template doesn't wire up, so the split runs on the device.

UsageCopy Anchor Link

Read the variant in a component with useExperiment, then branch on the variant you're testing:

import { Text } from "@/ui";
import { useExperiment } from "@/lib/experiments";

export function PaywallCopy() {
  const copy = useExperiment("paywall-copy");
  return <Text>{copy === "short" ? "Go Pro in one tap" : "Unlock everything with Pro"}</Text>;
}

The component re-renders when the variant changes: when PostHog's flags load, when the device's install id loads, and when you force a variant.

undefined means unknown: the flags haven't loaded, the person hasn't given analytics consent, the flag is missing, or the install id isn't on the device yet. Everyone in that state should see the control, which is why the snippet checks for "short" and lets everything else fall through. Don't branch on === "control", because undefined would then look like a control that isn't one.

The first time a variant is shown in a session, useExperiment tracks experiment_exposure with experiment, variant and source. source is local for the device's own split and override for a forced variant. Remote variants aren't tracked here: PostHog records its own $feature_flag_called when it serves one, so counting them twice would skew the results. Outside React, getVariant("paywall-copy") reads the variant without recording an exposure.

To force a variant while you test, open Settings → Developer tools. Each experiment has a row with the key and the variant you see. Press it to step through automatic, then each variant in turn, then automatic again. Looking at the rows doesn't record an exposure. A forced variant wins over PostHog and the local split, and it's kept on the device until you step back to automatic.

GotchasCopy Anchor Link

• Names must match. A variant renamed in src/experiments.ts but not in PostHog stops matching, so devices get undefined and see the control. The registry is typed, so a key that isn't in it is a type error in your code.
• Changing weights moves people. The split is a fixed hash of each install, so changing the weights of a running test moves some installs to another variant. Decide the split before you start, or start a new key.
• The local split is per install. It follows the device, not the account, so the same person on two devices can see two variants. Reinstalling the app starts the split over.