# Experiments Experiments are a controlled loop, in exactly this order, and each step gates the next: 1. **Pre-register.** Write the decision contract — question, arms, metric, guardrails, fixed horizon, decision rule — as a numbered markdown doc under `docs/experiments/`, before any API experiment, any variant flow content, or any traffic exists. See "Pre-registration format" below. 2. **Publish.** Author the treatment arm's flow content per the pre-registration (the variant gate on a screen — see below), and create the experiment against the API with `weir_experiment_start` (see "Current build status" for what's real versus still pending). 3. **Traffic.** Real assigned users flow through both arms; `variant_assigned` events land per user; guardrails (SRM, completion parity, reliability) are monitored continuously, not decided on. 4. **Decide.** After the API reports the fixed horizon reached (`ready: true`), read results exactly once and apply the pre-registered rule mechanically. No peeking before that, no re-reading after a decision is made. The decision rule belongs in the pre-registration **before any result exists** — writing the rule after seeing data is exactly the failure mode fixed-horizon pre-registration exists to prevent. ## Current build status — read this before assuming any of this is live Being honest about what's shipped versus designed matters more here than anywhere else in the toolkit, so: - **Sticky assignment, SRM guardrails, peeking guardrails, and the results API** are built and tested against a real Weir API. `variant_assigned` is a first-class event in the instrumentation contract (see [Reference → Event vocabulary](/docs/reference/#event-vocabulary)). - **The variant→config serving seam.** A screen can declare an optional `variant` gate (see [Reference → Experiments and the variant gate](/docs/reference/#experiments-and-the-variant-gate)); the headless engine and the in-app runtime both bucket a user with the same hash function, walk only the assigned arm's screen graph, and emit `variant_assigned`. One published config honestly carries every arm instead of two flow specs silently overwriting each other at `/manifest`. Assignment is recorded and server-verified from the ingested `variant_assigned` event, so it needs no SDK round trip to `POST /assign`. - **The `weir_experiment` MCP verbs (`weir_experiment_start`/`_status`/`_decide`).** The agent-facing lifecycle commands are registered tools — check [Reference → MCP tools](/docs/reference/#mcp-tools) for the live `ALL_TOOLS` list, the source of truth. `start` creates the experiment against the Weir API from a pre-registration doc, after validating the doc's mechanical fields and that the flow spec's `experiments[]` entry matches the doc's arms exactly; it refuses if the experiment already exists (pre-registration is immutable). `status` renders arms/SRM/drift/day-of-horizon and labels results INTERIM before the fixed horizon. `decide` refuses before the horizon (reporting the exact unlock date) and, once reached, applies the doc's decision rule mechanically — it reports the verdict and the exact `weir_release` follow-up but never publishes itself. `weir_experiment_results` still exists in code but stays deliberately **unregistered**; it predates the lifecycle verbs and is superseded by `weir_experiment_status`. - **Current operating state: two experiments are RUNNING now.** `exp_cob_commitment_first` (CutOrBulk, flow `cob_intake`) and `exp_psd_scan_demo_first` (PSD, flow `onboarding`) are live in production with 14-day horizons ending around 2026-08-17. Per-arm funnel reads stay 425-gated before horizon (the peeking guard); decide a winner only via `weir experiment-decide`, only at horizon, only per each experiment's pre-registration doc. This states that the experiments are running and enrolled — it is not a conversion-lift claim, and no winner has been read yet. - **EXP-001 is closed, unpowered (history).** It ran on the pre-pivot webview delivery path, and a production client bug on that path (a stale embedded bundle sending every event with no user id) starved it of traffic — assignments froze at 5 against a 50-per-arm minimum. SRM itself was clean (the split wasn't broken, there just wasn't enough traffic), and the experiment proved the delivery/assignment/ingest/fixed-horizon-gate pipeline works end to end on real infrastructure. No product decision was taken from it. The v4 native pivot replaced the entire delivery path that caused the starvation; the two running experiments above are the properly powered follow-ons. ## Worked example: EXP-001 (CutOrBulk intake) `docs/experiments/exp-001-cob-intake.md` is the first pre-registration and the worked example for the format below. It removed the `credibility` interstitial from `cob_intake` and measured per-user paid-subscription conversion. **It is closed, not running** — see "Current build status" above for why, and the closure note for the full record. Read it as a worked example of the pre-registration format, not as a live or decision-ready experiment. ## Pre-registration format Every pre-registration doc under `docs/experiments/` follows the same sections, in the same order, because the format is what makes "the rule was written before the data" checkable by inspection. Quoting the shape from `exp-001-cob-intake.md` directly (do not invent a different shape per experiment): 1. **Title + status line** — begin with `Status: pre-registered — not created on an API, not published, and no results exist ().` When the experiment is created, update only the operational status (or append an operational addendum); never edit the hypothesis, arms, metrics, horizon, thresholds, or decision rule below it (see "Immutable record"). 2. **Question and hypothesis** — the exact question being asked, and a one-paragraph hypothesis for the direction of effect. EXP-001: "Does removing the full-screen `credibility` interstitial ... increase the rate at which an assigned user starts a paid subscription." 3. **Exact arms** — a table: arm name, id, allocation weight, and the *exact* change, plus the literal JSON experiment declaration (`id`, `variants[].id`/`weight`, `holdout`) that will be passed to the API. EXP-001 pins allocation at 50/50, no holdout, and states plainly: "All other spec fields, app routing, Superwall placement, products, analytics, and native post-intake behavior must be identical." 4. **Metric contract** — a **primary metric** defined precisely enough to compute independently from raw events (EXP-001: per-user `purchase_result: purchased` for the pinned `flowId`, deduped by `userId`, matching `GET /experiments/:id/results` exactly), plus **guardrails**: assignment integrity (SRM not flagged), completion parity (a bounded percentage-point floor per arm, computed independently from `/events`, not the results endpoint), and reliability (fallback/native-call-error ceilings that invalidate the whole result if breached). 5. **Fixed horizon and decision rule** — exact `minSamplePerArm`/`minDurationMs` values, an explicit "do not read before `ready: true`, then freeze and apply once" rule, and an ordered, numbered decision rule (invalid/kill → ship treatment → ship control → no call) with concrete thresholds — not "when the effect looks significant." EXP-001 requires both an absolute 10-percentage-point difference **and** a 1.5× relative lift/drop before it will ship either arm; anything else is "no call," recorded as inconclusive, and the experiment is not extended or re-opened under a different rule. 6. **Rehearsal fixtures** — a small table of synthetic control/treatment counts with the decision each fixture must produce (known winner, no difference, harm), each including a multi-session user so the rehearsal proves per-`userId` dedup rather than per-session inflation. This is what a local rehearsal script asserts against before any real traffic exists. 7. **Immutable record** — a closing rule: do not alter hypothesis, arms, metrics, horizon, thresholds, or decision rule once an API experiment is created from this doc. If reality reveals a contract gap, the document is marked blocked and a **new numbered pre-registration** is written — the original is never edited in place to fit what happened. ## Freeze rules — why "no peeking" is enforced, not just requested The service is built with fixed-horizon guardrails, not sequential/always-valid inference — the standing rules restate this explicitly: experiments statistics stay fixed-horizon, no sequential inference. Concretely, from EXP-001's own rules (the pattern every pre-registration doc follows): - **No interim decisions.** Reading results before the API reports `ready: true` is not allowed to produce a product call — you can look, but you cannot ship or kill on what you see. - **Freeze after the first eligible read.** Once `ready: true` is reached, the pre-registered rule is applied **exactly once**. The experiment is not re-read, re-extended, or re-decided under a different rule afterward. - **Guardrail breach invalidates the read entirely.** SRM flagged, a guardrail failed, or either arm under its minimum sample makes the result **uninterpretable** — the rule says "make no product call... do not interpret conversion rates," not "be extra cautious." Delivery returns to `control` and the defect is logged, not shipped around. - **Immutable pre-registration.** See "Immutable record" above — the rule can't be rewritten once data exists, which is the actual mechanism that prevents a "significant-looking" post-hoc threshold from replacing the pre-registered one. ## Guardrails, restated for the operating view Make no call when sample-ratio mismatch is flagged, completion degrades beyond the pre-registered floor, or fallback/native-call errors breach the pre-registered limit. A dashboard chart informs the review; the dashboard's **Decide** action and `weir_experiment_decide` both apply the pre-registered rule mechanically — neither lets a human override that rule or read a verdict before the fixed horizon.