#Experiments

Experiments are a controlled loop, in exactly this order, and each step gates the next:

1. Pre-register. Write the decision contract — question, arms, metric, guardrails, fixed horizon, decision rule — as a numbered markdown doc under docs/experiments/, before any API experiment, any variant flow content, or any traffic exists. See "Pre-registration format" below.
2. Publish. Author the treatment arm's flow content per the pre-registration (the variant gate on a screen — see below), and create the experiment against the API with weir_experiment_start (see "Current build status" for what's real versus still pending).
3. Traffic. Real assigned users flow through both arms; variant_assigned events land per user; guardrails (SRM, completion parity, reliability) are monitored continuously, not decided on.
4. Decide. After the API reports the fixed horizon reached (ready: true), read results exactly once and apply the pre-registered rule mechanically. No peeking before that, no re-reading after a decision is made.

The decision rule belongs in the pre-registration before any result exists — writing the rule after seeing data is exactly the failure mode fixed-horizon pre-registration exists to prevent.

#Current build status — read this before assuming any of this is live

Being honest about what's shipped versus designed matters more here than anywhere else in the toolkit, so:

• Sticky assignment, SRM guardrails, peeking guardrails, and the results API are built and tested against a real Weir API. variant_assigned is a first-class event in the instrumentation contract (see Reference → Event vocabulary).
• The variant→config serving seam. A screen can declare an optional variant gate (see Reference → Experiments and the variant gate); the headless engine and the in-app runtime both bucket a user with the same hash function, walk only the assigned arm's screen graph, and emit variant_assigned. One published config honestly carries every arm instead of two flow specs silently overwriting each other at /manifest. Assignment is recorded and server-verified from the ingested variant_assigned event, so it needs no SDK round trip to POST /assign.
• The weir_experiment MCP verbs (weir_experiment_start/_status/_decide). The agent-facing lifecycle commands are registered tools — check Reference → MCP tools for the live ALL_TOOLS list, the source of truth. start creates the experiment against the Weir API from a pre-registration doc, after validating the doc's mechanical fields and that the flow spec's experiments[] entry matches the doc's arms exactly; it refuses if the experiment already exists (pre-registration is immutable). status renders arms/SRM/drift/day-of-horizon and labels results INTERIM before the fixed horizon. decide refuses before the horizon (reporting the exact unlock date) and, once reached, applies the doc's decision rule mechanically — it reports the verdict and the exact weir_release follow-up but never publishes itself. weir_experiment_results still exists in code but stays deliberately unregistered; it predates the lifecycle verbs and is superseded by weir_experiment_status.
• Current operating state: two experiments are RUNNING now. exp_cob_commitment_first (CutOrBulk, flow cob_intake) and exp_psd_scan_demo_first (PSD, flow onboarding) are live in production with 14-day horizons ending around 2026-08-17. Per-arm funnel reads stay 425-gated before horizon (the peeking guard); decide a winner only via weir experiment-decide, only at horizon, only per each experiment's pre-registration doc. This states that the experiments are running and enrolled — it is not a conversion-lift claim, and no winner has been read yet.
• EXP-001 is closed, unpowered (history). It ran on the pre-pivot webview delivery path, and a production client bug on that path (a stale embedded bundle sending every event with no user id) starved it of traffic — assignments froze at 5 against a 50-per-arm minimum. SRM itself was clean (the split wasn't broken, there just wasn't enough traffic), and the experiment proved the delivery/assignment/ingest/fixed-horizon-gate pipeline works end to end on real infrastructure. No product decision was taken from it. The v4 native pivot replaced the entire delivery path that caused the starvation; the two running experiments above are the properly powered follow-ons.

#Worked example: EXP-001 (CutOrBulk intake)

docs/experiments/exp-001-cob-intake.md is the first pre-registration and the worked example for the format below. It removed the credibility interstitial from cob_intake and measured per-user paid-subscription conversion. It is closed, not running — see "Current build status" above for why, and the closure note for the full record. Read it as a worked example of the pre-registration format, not as a live or decision-ready experiment.

#Pre-registration format

Every pre-registration doc under docs/experiments/ follows the same sections, in the same order, because the format is what makes "the rule was written before the data" checkable by inspection. Quoting the shape from exp-001-cob-intake.md directly (do not invent a different shape per experiment):

1. Title + status line — begin with Status: pre-registered — not created on an API, not published, and no results exist (<date>). When the experiment is created, update only the operational status (or append an operational addendum); never edit the hypothesis, arms, metrics, horizon, thresholds, or decision rule below it (see "Immutable record").
2. Question and hypothesis — the exact question being asked, and a one-paragraph hypothesis for the direction of effect. EXP-001: "Does removing the full-screen credibility interstitial ... increase the rate at which an assigned user starts a paid subscription."
3. Exact arms — a table: arm name, id, allocation weight, and the exact change, plus the literal JSON experiment declaration (id, variants[].id/weight, holdout) that will be passed to the API. EXP-001 pins allocation at 50/50, no holdout, and states plainly: "All other spec fields, app routing, Superwall placement, products, analytics, and native post-intake behavior must be identical."
4. Metric contract — a primary metric defined precisely enough to compute independently from raw events (EXP-001: per-user purchase_result: purchased for the pinned flowId, deduped by userId, matching GET /experiments/:id/results exactly), plus guardrails: assignment integrity (SRM not flagged), completion parity (a bounded percentage-point floor per arm, computed independently from /events, not the results endpoint), and reliability (fallback/native-call-error ceilings that invalidate the whole result if breached).
5. Fixed horizon and decision rule — exact minSamplePerArm/minDurationMs values, an explicit "do not read before ready: true, then freeze and apply once" rule, and an ordered, numbered decision rule (invalid/kill → ship treatment → ship control → no call) with concrete thresholds — not "when the effect looks significant." EXP-001 requires both an absolute 10-percentage-point difference and a 1.5× relative lift/drop before it will ship either arm; anything else is "no call," recorded as inconclusive, and the experiment is not extended or re-opened under a different rule.
6. Rehearsal fixtures — a small table of synthetic control/treatment counts with the decision each fixture must produce (known winner, no difference, harm), each including a multi-session user so the rehearsal proves per-userId dedup rather than per-session inflation. This is what a local rehearsal script asserts against before any real traffic exists.
7. Immutable record — a closing rule: do not alter hypothesis, arms, metrics, horizon, thresholds, or decision rule once an API experiment is created from this doc. If reality reveals a contract gap, the document is marked blocked and a new numbered pre-registration is written — the original is never edited in place to fit what happened.

#Freeze rules — why "no peeking" is enforced, not just requested

The service is built with fixed-horizon guardrails, not sequential/always-valid inference — the standing rules restate this explicitly: experiments statistics stay fixed-horizon, no sequential inference. Concretely, from EXP-001's own rules (the pattern every pre-registration doc follows):

• No interim decisions. Reading results before the API reports ready: true is not allowed to produce a product call — you can look, but you cannot ship or kill on what you see.
• Freeze after the first eligible read. Once ready: true is reached, the pre-registered rule is applied exactly once. The experiment is not re-read, re-extended, or re-decided under a different rule afterward.
• Guardrail breach invalidates the read entirely. SRM flagged, a guardrail failed, or either arm under its minimum sample makes the result uninterpretable — the rule says "make no product call... do not interpret conversion rates," not "be extra cautious." Delivery returns to control and the defect is logged, not shipped around.
• Immutable pre-registration. See "Immutable record" above — the rule can't be rewritten once data exists, which is the actual mechanism that prevents a "significant-looking" post-hoc threshold from replacing the pre-registered one.

#Guardrails, restated for the operating view

Make no call when sample-ratio mismatch is flagged, completion degrades beyond the pre-registered floor, or fallback/native-call errors breach the pre-registered limit. A dashboard chart informs the review; the dashboard's Decide action and weir_experiment_decide both apply the pre-registered rule mechanically — neither lets a human override that rule or read a verdict before the fixed horizon.