The form is complete but the causal question is unclear.
Cause: No Product decision or eligible population is defined.
Fix: Write the decision, inclusion criteria, assignment unit, and exposure rule before metrics.
Pre-launch experiment design
A/B Test Plan Generator
Turn a Product experiment into a launch-ready decision contract: define the causal question, eligible population, randomization and exposure, primary metric, guardrails, power assumptions, data-quality checks and pre-committed ship/revert rules.
Readiness, not form completion
The Experiment Readiness Gate shows exactly what is ready, missing, unsupported or still needs a decision.
No fake statistical precision
The built-in calculator only returns a number for the statistical design it actually supports. Unsupported designs get no fabricated sample size.
Decision contract
Pre-commit practical thresholds, guardrails, SRM/data-quality actions and what win, loss or inconclusive evidence means before results arrive.
No login. Browser save. Markdown, JSON and Metrics CSV export. Share links contain the experiment configuration.
Synthetic example:
Share privacy: This link contains the experiment configuration and metric assumptions. Anyone with the link can inspect it.
Experiment Readiness Gate
No score. Inspect the exact missing, unsupported, or review items before launch.
Causal question
missing- Product decision: The Product decision this experiment informs is missing.
- Observation & evidence: Observation or evidence source is missing.
- Intervention & mechanism: Define the intervention and why it should change behavior.
Population
missing- Inclusion criteria: Who can enter the experiment is missing.
- Exclusions: Record exclusions or explicitly state none.
Assignment
missing- Randomization unit: Randomization unit is missing.
- Why this unit: Explain why this is the correct independent unit.
- Final allocation: 50/50 allocation is internally valid.
- Contamination/interference review: Contamination/interference risk has not been reviewed.
Exposure
missing- Exposure event: Define the event/trigger that means a randomized unit had a chance to experience the treatment.
- Analysis population: Choose assigned, exposed, or custom analysis population.
Metrics
missing- Primary metric: Exactly one primary metric is required.
- Denominator / analysis unit: Not required by the built-in binary contract.
- What could get worse: Answer what could improve the primary metric while making the customer, Product, or business worse.
- Guardrail contract: No guardrail is defined; confirm that this is intentional after reviewing harm.
Power & horizon
unsupported- Supported method: Continuous/custom metrics require a validated external power analysis.
- Sample calculation: Sample calculation is not ready.
- Why the MDE matters: Explain why this is the smallest effect that would change the Product decision.
- Eligible units / duration: Sample can be planned without traffic, but duration cannot.
- Planned horizon: External method controls horizon planning.
- Stopping rule: Analysis mode is fixed horizon.
Decision
missing- Practical threshold: Define the effect magnitude that would actually change the Product decision.
- Ship / expand: Define ship/expand conditions beyond p-value alone.
- Keep control / revert: Define when to keep control or revert.
- Inconclusive: Define what insufficient evidence means and what happens next.
- Data-quality failure: Define what happens if SRM or instrumentation fails.
Data quality
missing- Assignment QA: Status is still unknown.
- Exposure QA: Status is still unknown.
- Primary metric QA: Status is still unknown.
- Guardrail QA: Status is still unknown.
- Allocation QA: Status is still unknown.
- SRM plan: Status is still unknown.
- Experiment collision review: Status is still unknown.
- SRM owner & failure action: Assign who runs the SRM statistical check and what happens if it fails.
1. Decision & context
2. Hypothesis contract
3. Population, assignment & exposure
4. Control & treatment
5. Primary metric contract
6. Secondary metrics & guardrails
Secondary metrics are explanatory/supporting. They do not silently replace a missed primary. Guardrails protect against explicit harm.
Secondary
Guardrails
7. Power, sample & horizon
Analysis mode: Fixed horizon. The built-in calculator supports binary conversion, two independent groups, two-sided frequentist analysis and a 50/50 final allocation. Need valid early stopping or another metric/design? Record a validated external power method instead.
Built-in binary power math is disabled for this metric type. Record a sample/horizon calculated by a validated external method; CraftUp will not fabricate a number.
8. Pre-committed decision contract
Do not reduce this to “p < .05 = ship.” Combine effect magnitude/uncertainty, practical threshold, guardrails and data quality.
9. Data-quality gate
One acknowledgement checkbox is not evidence of readiness. Mark each item explicitly.
SRM policy: compare observed assignment counts with the configured allocation using a statistical SRM test before interpreting treatment effects. A raw percentage deviation alone is not an SRM test.
10. Pre-registered segments
Only add segments that could change the Product decision. More segments create multiplicity and subgroup-power problems; there is no arbitrary statistical “maximum 3” rule.
11. Risks & mitigations
What sample-size calculation this tool supports
The built-in calculator is intentionally narrow: binary conversion metrics, two independent groups, a two-sided frequentist fixed-horizon design, and a 50/50 final allocation. Alpha and power are explicit inputs and both affect the result. The tool uses a standard two-proportion normal approximation and shows control, treatment and total required sample separately.
Continuous metrics, custom outcomes, unequal final allocation, clustered/paired designs and sequential stopping require a validated external power method. Record that external plan in the tool rather than forcing the binary calculator to return a plausible-looking number.
Absolute vs relative MDE
MDE is the smallest effect worth designing the experiment to detect under the stated assumptions-not the lift you hope to get. Use Product economics, customer impact, implementation cost and opportunity cost to decide what effect would actually change the decision.
Absolute: an 8% baseline plus 2 percentage points implies a 10% treatment rate.
Relative: an 8% baseline plus 15% relative lift implies a 9.2% treatment rate.
The UI uses human percentages: enter 8 for 8%, not 0.08.
Eligible traffic vs total traffic
Duration depends on new experiment units that can actually enter the defined population. Pageviews, sessions or site-wide traffic are wrong when the experiment randomizes only a subset of users, accounts, devices or requests. The plan therefore separates eligibility, randomization unit and eligible units per day.
Fixed horizon vs sequential testing
The built-in analysis mode is fixed horizon: specify the sample/horizon before launch and use that planned readout for the decision. Repeatedly checking a standard fixed-horizon test and stopping the first time it becomes significant can inflate false positives. If valid early stopping matters, use a sequential-testing method in your experimentation platform and record that external method; CraftUp does not compute sequential boundaries here.
Why Sample Ratio Mismatch blocks interpretation
SRM asks whether observed assignment counts are inconsistent with the allocation specified before launch. A raw split such as 48/52 is not automatically a failure: the evidence depends on sample size. Run a statistical SRM check against the configured allocation and investigate assignment, exposure loss or instrumentation before interpreting treatment effects when it fails.
Statistical significance is not the Product decision
A pre-launch contract should state the practical effect threshold, how uncertainty will be read, which guardrails must remain acceptable, and which data-quality checks must pass. A statistically significant effect can still be too small or too harmful to ship. A non-significant result is not automatically proof of no effect; it can mean the planned evidence was insufficient at the chosen sensitivity.
When an A/B test may be the wrong learning method
Reconsider the design when eligible traffic is far below the required sample, randomization cannot isolate treatment, network/interference effects dominate, outcomes mature after the decision window, the change cannot safely be randomized, or the main question is qualitative rather than causal. The readiness gate should expose these constraints instead of encouraging a low-information test.
Experiment-planning rules that survive review
- Define the Product decision first. An experiment that only asks which version performs better is underspecified.
- Choose the smallest effect that would change the decision; MDE is not the lift you hope to see.
- Power the population that can actually enter the experiment, not total site or app traffic.
- Match the analysis unit to the randomization unit, especially for shared B2B accounts and collaborative workflows.
- Define exposure before launch so assignment, exposure, and outcome populations cannot drift after results arrive.
- Use one primary metric as the pre-committed decision target; secondary metrics explain rather than rescue a missed primary.
- Treat guardrails as harm constraints with explicit direction, units, window, and action-not decorative metrics.
- For this built-in calculator, use the pre-specified fixed horizon. Valid early stopping requires a separate sequential method.
- Run an SRM statistical check against the configured allocation before trusting treatment-effect analysis.
- Pre-register only segments that could change the Product decision; subgroup claims usually need more power and multiplicity planning.
Common planning failures
A sample number appears for a non-binary metric.
Cause: Conversion math was applied to counts, revenue, or time.
Fix: Choose the metric type explicitly; use a validated external power analysis for continuous/custom metrics.
12% is entered as 0.12.
Cause: Decimal-rate and human-percentage conventions were mixed.
Fix: Enter 12 for 12%. The tool normalizes percentages internally.
Guardrail says 'must not drop >5%'.
Cause: Relative percent vs percentage points is ambiguous.
Fix: Choose threshold direction/type/value and observation window explicitly.
A significant positive result is treated as an automatic ship.
Cause: Practical threshold, uncertainty, guardrails, or data quality were omitted.
Fix: Pre-commit a decision rule that includes all four.
An inconclusive fixed-horizon test is extended after looking at results.
Cause: The stopping rule changed post hoc.
Fix: Treat the result as insufficient evidence; redesign a new experiment or use a sequential method that was specified up front.
48/52 is treated as an SRM failure regardless of sample size.
Cause: Raw allocation deviation was used instead of a statistical SRM check.
Fix: Run an SRM test against the configured allocation and investigate before reading effects if it fails.
Many post-hoc segments produce a winner.
Cause: Subgroup slicing and multiple comparisons were not planned.
Fix: Pre-register decision-relevant segments and plan subgroup power/multiplicity when they drive shipping decisions.
FAQ
What statistical method does the built-in calculator support?
Binary conversion metrics only: two independent groups, a two-sided frequentist fixed-horizon design, configurable alpha and power, and a 50/50 final allocation. The calculation uses a standard two-proportion normal approximation with separate null and alternative variance terms. Unequal allocation, continuous metrics, sequential boundaries, cluster designs, and other custom methods must be powered with a validated external tool.
Why does the tool sometimes refuse to show a sample size?
Because a plausible-looking number can be worse than no number. A numeric estimate is withheld when the baseline or MDE is missing/invalid, the implied treatment rate is impossible, the final allocation is not the supported 50/50 design, or the primary metric is not binary. The readiness gate tells you exactly what input or method is missing.
What does MDE mean here?
Minimum detectable effect is the smallest effect the experiment is designed to reliably detect under the stated alpha, power, baseline and method. It should come from the Product decision: customer impact, economics, implementation cost or opportunity cost. It is not a prediction of the lift you expect, and the tool does not invent a default effect size.
What is the difference between absolute and relative MDE?
An absolute MDE is measured in percentage points: an 8% baseline plus 2 percentage points implies 10%. A relative MDE scales the baseline: an 8% baseline plus 15% relative implies 9.2%. The tool makes you choose one mode and shows the implied treatment rate so the units cannot silently conflict.
Why separate eligible traffic from total traffic?
Power and duration depend on new experiment units that can actually be randomized into the defined population, not every pageview or app session. If only a subset qualifies, total traffic will understate duration. The tool therefore asks for eligible experiment units per day and separately records the randomization unit and eligibility contract.
Can I stop when the result becomes significant?
Not with the built-in fixed-horizon method. Repeatedly reading a standard fixed-horizon test and stopping the first time a threshold is crossed inflates false positives. If valid early decisions are important, use a sequential-testing method in an experimentation platform and record that external method rather than pretending this tool computes sequential boundaries.
What is SRM and how should I use it?
Sample Ratio Mismatch checks whether observed assignment counts are inconsistent with the allocation configured before launch. It should be evaluated statistically because the same raw ratio can mean very different things at different sample sizes. Treat a failed SRM check as a data-quality blocker: investigate assignment, exposure loss or instrumentation before interpreting treatment effects.
Are the examples benchmarks?
No. The three built-in examples are explicitly synthetic learning designs. Their baselines, MDEs, guardrail thresholds and traffic values are chosen only to demonstrate the workflow and calculator behavior. They are not CraftUp results, industry averages, recommended thresholds, or evidence that a similar experiment in your Product should use the same values.
What does the share link contain?
The share link contains the experiment configuration itself, compressed into the URL. Compression is not encryption. Anyone who receives the link can inspect the hypothesis, variants, metric assumptions and other plan content. Do not use the share feature for information your team is not comfortable placing in a link; local browser save stays on the device.
Learn the underlying experiment judgment
Use the plan generator for the pre-launch contract; use CraftUp learning content when the underlying hypothesis, metric or experimentation concept still needs work.
Ready means ready under explicit assumptions
If the readiness gate still shows missing, unsupported or review items, fix the design before launch. If the causal question itself is weak, go back to hypothesis or discovery work rather than filling more fields.
Freshness
Last updated: 2026-09-16
- 2026-09-16: Rebuilt the tool as an Experiment Readiness Gate with explicit population, assignment, exposure, metric, data-quality and decision contracts.
- 2026-09-16: Replaced unsafe sample math with a validated binary two-proportion fixed-horizon method; unsupported cases now return no numeric sample.
- 2026-09-16: Added v1 migration, structured guardrails, SRM policy, privacy-safe sharing copy, and richer stakeholder exports.