AB Testing — Cheat Sheet

Tags: ab-testing, experimentation, statistics, feature-flags, analytics, conversion-rate

Level: intermediate

TL;DR: Controlled randomized experiment comparing variant B against baseline A to validate hypotheses using statistically significant business and behavioral metrics.

Key concepts

Compare control group A with variant B using randomized user assignment.

Formulate a clear null hypothesis (H0H_0H0) and alternative hypothesis (H1H_1H1).

Determine sample size beforehand based on power (1−β1-\beta1−β) and significance (α\alphaα).

Statistical significance (α\alphaα, usually 0.05) bounds false positive risk (Type I error).

Statistical power (1−β1-\beta1−β, usually 0.80) ensures detecting real effects (Type II error).

Minimum Detectable Effect (MDE) defines smallest relative change worth detecting reliably.

Assignment unit (user ID, session ID, device ID) dictates randomization consistency.

Core metrics split into primary evaluation metric, secondary metrics, and guardrail metrics.

Guardrail metrics safeguard system latency, error rates, and key revenue channels.

Run tests across complete business cycles to account for day-of-week seasonality.

Typical questions and ready answers

Q: How do you choose between client-side and server-side AB testing?R: I pick client-side for rapid UI/copy tests via tag managers, but I rely on server-side for business logic, performance-critical paths, and zero-flicker stability.

Q: Why shouldn't you stop an AB test as soon as p-value drops below 0.05?R: Early stopping inflates false positives through the peeking problem; I keep the test running until the planned sample size and duration are reached.

Q: What is an A/A test and when should you run one?R: An A/A test compares identical experiences to verify tooling health, check sample ratio mismatch, and confirm the false-positive rate matches alpha.

Q: How do you handle Sample Ratio Mismatch (SRM)?R: I run a chi-square goodness-of-fit test on sample allocations; if SRM is present, I invalidate the test and investigate tracking or redirect biases.

Pitfalls and common mistakes

Peeking at intermediate results and terminating runs before reaching calculated sample size.

Ignoring Sample Ratio Mismatch caused by redirect latency or tracking dropped events.

Over-testing multiple variants without family-wise error rate corrections like Bonferroni.

Changing traffic split percentages midway through an active test without bucket resetting.

Confusing novel novelty or primacy effects with sustainable long-term conversion lift.

Running experiments underpowered against tiny variations that require unrealistic sample volume.

Snippets and commandsTypeScript

// Deterministic bucket assignment using MurmurHash3 / SHA-256
import { createHash } from "crypto";

function getVariant(userId: string, experimentKey: string, splitRatio = 0.5): "control" | "treatment" {
  const hash = createHash("sha256")
    .update(`${userId}:${experimentKey}`)
    .digest("hex");
  const bucket = parseInt(hash.substring(0, 8), 16) % 10000;
  return bucket < splitRatio * 10000 ? "control" : "treatment";
}

SQL

```-- Chi-Square Sample Ratio Mismatch (SRM) check (50/50 expected split)

WITH counts AS (

SELECT

SUM(CASE WHEN variant = 'control' THEN 1 ELSE 0 END) AS obs_control,

SUM(CASE WHEN variant = 'treatment' THEN 1 ELSE 0 END) AS obs_treatment,

COUNT(*) AS total_obs

FROM experiment_assignments

WHERE experiment_id = 'exp_checkout_flow_v2'

)

SELECT

obs_control,

obs_treatment,

POWER(obs_control - (total_obs / 2.0), 2) / (total_obs / 2.0) +

POWER(obs_treatment - (total_obs / 2.0), 2) / (total_obs / 2.0) AS chi_square_stat

FROM counts;

```

To review

[ ] Calculate sample size formulas for binomial metrics vs continuous revenue metrics.

[ ] Multi-armed bandit trade-offs compared to fixed-horizon frequentist experiments.

[ ] Sequential testing methods to enable continuous monitoring without alpha inflation.

[x] Standard z-test and t-test difference-in-proportions evaluation.

[x] Deterministic hashing patterns for consistent bucket allocation across microservices.

References

Trustworthy Online Controlled Experiments (Ron Kohavi, Diane Tang, Ya Xu)

Evan Miller Sample Size and AB Testing Statistical Calculators

Jarvis Interviews
Jarvis