Supports
The outcome reaches the practical threshold with acceptable guardrails and no data-quality flag.
Action: Move to rollout/decision review; a supported hypothesis still needs prioritization before roadmap capacity.
Product experimentation workspace
Product Experiment Hypothesis Generator
Turn a real product decision into a testable, falsifiable experiment contract, before anyone asks for sample size.
“If we do X, metric Y will improve” is not yet a hypothesis. This workspace forces the missing parts into the open: eligible population, observable intervention, causal mechanism, defined outcome, direction, your practical threshold, guardrails, falsifiability and what each result path triggers. It never invents an uplift, sample size, duration or significance claim.
Decision first
Every contract starts from the decision the evidence would change, not from a feature idea.
Falsifiable by construction
A descriptive inspector shows exactly what would let evidence weaken the claim.
No statistics fiction
Practical thresholds stay user-owned; power, sample size and significance belong in the A/B plan.
Falsifiability review
Not yet testable
Not yet testable: no product decision, uncertainty is missing, eligible population is vague, intervention is not observable, causal mechanism is missing, primary outcome is missing, expected direction is missing, practical decision threshold is missing.
No login. The snapshot autosaves in this browser. Share links encode the snapshot in the URL, so anyone with that URL can read it. Avoid confidential product data. This workspace never invents effect sizes, sample sizes, durations or significance.
Hypothesis contract
The hypothesis contract is not ready: resolve the blocking checks below before treating this as a testable statement.
Falsifiability inspector
- • Observable outcome: missing, definition missing.
- • Direction: missing.
- • Eligible population: missing.
- • Practical threshold (yours, not invented): Threshold unknown, needs decision/economics work.
- • Window: Observation window not specified.
- • Guardrails: No guardrail named.
Quality lab, inspectable checks, no score
blockerNo Product decision
The hypothesis is not linked to a decision it could change.
Fix: Name the decision this evidence would inform, e.g. whether reducing setup friction is worth implementation capacity.
blockerUncertainty is missing
Without an explicit uncertainty, the experiment cannot be scoped.
Fix: State what is unknown, e.g. whether mandatory setup delays first value for new self-serve accounts.
blockerEligible population is vague
“All users” hides who can actually experience the intervention.
Fix: Name lifecycle state, surface and eligibility, e.g. new self-serve accounts entering onboarding on web.
blockerIntervention is not observable
The treatment cannot be built or recognized from this wording.
Fix: Describe one observable change, e.g. remove two non-essential setup steps before first value.
blockerCausal mechanism is missing
There is no explanation of why this intervention should move the outcome.
Fix: State the mechanism separately from the metric, e.g. users reach first value with less setup burden in the first session.
blockerPrimary outcome is missing
No observable Product outcome is named.
Fix: Name one primary metric plus its event/definition, e.g. 7-day activation: new account reaches first value within 7 days.
blockerExpected direction is missing
Not every experiment aims for “increase”: some protect a bound or reduce harm.
Fix: Choose increase, decrease, or maintain-within-bound explicitly.
blockerPractical decision threshold is missing
Without a threshold, any movement can be narrated as support.
Fix: Enter the smallest outcome that would change the decision, e.g. at least +2 pp 7-day activation. The number is yours; this tool never invents it.
infoNo guardrail named
Optional for trivial changes; risky to omit when refunds, errors, latency, complaints or downstream retention could move.
Fix: Name harms to watch with an explicit bound, or record why no guardrail is needed.
cautionResult actions are incomplete
A contract that only defines the success path turns every other result into improvisation.
Fix: Pre-commit what happens on support, weaken and inconclusive before running.
Result interpretation matrix, decide before running
Weakens
The outcome misses the threshold or moves against the expected direction without a data-quality explanation.
Action: Revisit the mechanism or intervention strength; do not relabel the feature as “bad”.
Inconclusive
Evidence cannot distinguish the threshold from noise within the planned sensitivity.
Action: Record what evidence is still missing; redesign only if the decision is worth another test.
Invalid
Assignment, exposure, instrumentation or SRM checks fail.
Action: Do not interpret treatment effects. Fix data quality first; rerun only under a valid design if required.
Guardrail fail
A protected harm exceeds its bound even if the primary metric moves favorably.
Action: Pause rollout and investigate the harm before calling the test a Product win. Do not ship on the primary metric alone.
Method: Choose the method after the contract
A controlled A/B test fits only when population, exposure and outcome are observable at sufficient volume; otherwise use a prototype, pilot or qualitative test.
Open Choose the method after the contractNext: Repair the blocking checks above
Most blocking failures are framing problems: vague population, unobservable treatment, missing mechanism or undefined outcome.
Continue to next stepBoundaries, what this contract does not do
- • This contract does not calculate sample size, duration, power or significance.
- • It does not predict an effect size and never invents an uplift.
- • A supporting result does not automatically justify roadmap capacity; use prioritization.
- • It does not certify causality, validation, or a shipping decision.
- • Language stays at test / support / weaken / inconclusive, never prove, validate, or winner/loser.
What makes a product hypothesis testable?
A testable product hypothesis connects a decision to an observation. Each clause below earns its place: remove one and the contract can no longer fail honestly.
Decision + uncertainty
What will be decided, and what unknown blocks it?
Eligible population
Who can actually experience the treatment?
Observable intervention
What one change does the population experience?
Causal mechanism
Why should this change move the outcome?
Primary outcome
Which one metric, with event, denominator and window?
Direction
Increase, decrease, or stay within a bound?
Practical threshold
Smallest outcome that changes the decision, yours, never invented.
Guardrails + actions
Which harms stop a rollout, and what follows each result?
Hypothesis is not assumption, goal, requirement or target
Assumption → hypothesis
“Setup feels too long” becomes: removing two setup steps for eligible new accounts will increase 7-day activation by at least your threshold, because users reach first value sooner.
Goal / target → hypothesis
“Improve retention” and “activation above 35%” are targets. A hypothesis adds the population, intervention and mechanism that could move them.
Requirement → hypothesis
“We need a dashboard” is a solution request. Ask which behavior, for whom, why, and what would change the decision before testing anything.
Practical vs statistical
A change can be statistically significant but too small to matter, or practically important but inconclusive at current sensitivity. This page owns the practical threshold; the A/B Test Plan Generator owns alpha, power, MDE and sample size.
When not to write an experiment hypothesis
Not every uncertainty deserves A/B language. The workspace says so explicitly and routes you to discovery when the question is whether users have the problem, what they do today, which market or segment to serve, or whether the work is feasible at all.
Unknown problem
“Do users have this problem?” needs interviews, surveys or problem framing, not a hypothesis sentence.
Strategy choice
Which market, segment, business model or positioning to pursue may use experiment evidence, but the strategy owner holds the decision model.
Bundled redesign
A package test is sometimes intentional, but attribution across components stays weaker. Label it.
Five illustrative contracts, including when to refuse
All examples below are fictional teaching scenarios. Thresholds and metric values are user-entered illustrations, never benchmarks or CraftUp results. Load any of them from the workspace above.
onboarding setup reduction
New self-serve accounts, two setup steps removed, 7-day activation with a support-contact guardrail.
Correct behavior: Testable contract with an explicit +2 pp practical threshold, falsifiability checklist and a support/weaken/inconclusive action for each path.
pricing message change
Trial visitors from comparison pages, annual-savings exposure, trial-start outcome with a refund guardrail.
Correct behavior: Contract keeps the practical threshold user-owned and routes sample-size work to the A/B plan.
lifecycle reminder
Dormant consumer accounts, contextual reminder, week-4 return with an unsubscribe guardrail.
Correct behavior: Bundled-change and proxy-metric checks stay visible; the threshold remains user-entered.
B2B admin setup
New 1–50 seat workspaces, deferred role setup, signup completion with a permission-error guardrail.
Correct behavior: Population and unit-of-analysis wording anticipates the A/B plan handoff without duplicating statistics.
not experiment-ready
“Do users have this problem?” with no intervention, population or outcome, correctly routed to discovery.
Correct behavior: The workspace refuses A/B language and routes to interviews instead of generating a fake hypothesis.
Inconclusive is a real outcome
Before running, the contract defines supports, weakens, inconclusive, invalid and guardrail-fail paths with explicit actions. Language stays at test, support, weaken and inconclusive, never prove, validate, winner or loser. “Experiment failed” becomes the precise alternative: hypothesis not supported under this test, evidence weakened the mechanism, result inconclusive, test invalid, or guardrail unacceptable.
Where this fits in the experimentation workflow
| Job | Owner | Boundary |
|---|---|---|
| Frame the problem | Problem Statement Generator | Clarify actor, obstacle and consequence before hypothesizing. |
| Gather behavioral evidence | Interview Script / Survey Builder | Learn what happened; do not pre-write the answer. |
| Write the decision contract | This tool | Define what is claimed and what would change the decision. |
| Design the controlled test | A/B Test Plan Generator | Owns assignment, exposure, sample size, duration, SRM, QA and analysis. |
| Define metric architecture | KPI Tree / North Star | Owns instrumentation and metric decomposition; this tool names one outcome. |
| Choose among supported bets | Prioritization | A supported hypothesis still competes for capacity on cost, fit and opportunity cost. |
Experiment-hypothesis tips
- Start from the decision, not the feature idea. A hypothesis that cannot change a decision is documentation, not an experiment.
- Write the mechanism separately from the metric. “Because activation will improve” is circular.
- Keep one primary outcome. Secondary signals explain; they must not rescue a missed primary.
- Enter your own practical threshold. The tool never invents an uplift, sample size or duration.
- Pre-commit support, weaken and inconclusive actions before running. Missing paths get improvised after results arrive.
- Name the eligible population narrowly. Power depends on units that can actually enter the test, not total traffic.
- Treat guardrails as harm constraints with bounds and actions, not decorative metrics.
- A supporting result still needs prioritization: cost, strategy fit and opportunity cost remain.
- Use test / support / weaken / inconclusive language. Nothing here proves, validates or declares a winner.
- Route non-experimentable questions to interviews, surveys or problem framing instead of forcing A/B language.
Common hypothesis mistakes
The hypothesis is “if we build X, users will like it.”
Why: No population, mechanism, outcome or threshold.
Fix: Fill the full contract: eligible population, observable treatment, mechanism, defined metric, direction and your threshold.
The mechanism repeats the metric.
Why: Causal explanation collapsed into the outcome.
Fix: Describe the user-level change that precedes metric movement.
The treatment lists three redesigns at once.
Why: Bundled changes.
Fix: Test the package intentionally and label attribution as weaker, or split hypotheses.
Any metric movement counts as support.
Why: No practical threshold.
Fix: Enter the smallest outcome that would change the decision, or mark it unknown and do the economics work.
A significant result is treated as an automatic ship.
Why: Practical threshold, guardrails and data quality omitted.
Fix: Pre-commit the full decision rule and hand statistics to the A/B plan.
“Do users have this problem?” becomes an A/B test.
Why: Discovery question forced into experiment language.
Fix: Route to interviews, surveys or problem framing first.
Clicks are the primary outcome with no downstream link.
Why: Proxy metric without a user-value argument.
Fix: State which outcome makes the proxy useful and guard the downstream metric.
A strategy choice is disguised as a feature test.
Why: Market, segment or positioning decision.
Fix: Let the experiment inform the strategy owner; do not let one test decide the strategy.
FAQ
What is a product experiment hypothesis?
A decision-linked, falsifiable claim: for a defined eligible population, an observable intervention will move one defined outcome in an expected direction by at least your practical threshold, because of a stated mechanism, with guardrails and pre-committed actions for support, weaken, inconclusive, invalid and guardrail-fail paths.
How do I write an A/B test hypothesis?
Name the decision and uncertainty, the eligible population, one observable treatment, the causal mechanism, one primary metric with its event/definition/window, the expected direction, your practical threshold, guardrails, and what each result path would trigger. Then hand power, sample size, duration and analysis to the A/B Test Plan Generator.
What should a product hypothesis contain?
Decision, uncertainty, population, intervention, mechanism, outcome, direction, practical threshold, guardrails, falsifiability review and pre-defined result actions. This workspace requires each field before calling the contract testable.
Do I need an expected uplift?
No. You need a practical decision threshold: the smallest outcome that would change your decision. The tool never invents that number. If you do not know it, mark it unknown and do the decision/economics work first.
What if I don't know the threshold?
Leave it explicitly unknown. The contract stays honest but weaker, and the next step becomes implementation-cost and opportunity-cost analysis rather than launching.
What is the difference between a hypothesis and an assumption?
An assumption is an untested belief, e.g. setup feels too long. A hypothesis makes it experimentable: removing two setup steps for eligible new accounts will increase 7-day activation by at least your threshold because users reach first value sooner.
Hypothesis vs experiment plan?
The hypothesis contract defines what is claimed and what would change the decision. The A/B Test Plan owns statistical design: assignment, exposure, sample size, duration, significance, SRM, QA and analysis.
What metric should I use?
One primary behavioral or outcome metric with an explicit event, denominator and window. Bare labels like engagement need definitions; proxies like clicks need a stated link to user value. Metric architecture help lives in the KPI Tree Builder.
What is a guardrail metric?
A harm constraint with an explicit bound and action, e.g. refund rate must not worsen beyond your bound or rollout pauses. Guardrails prevent shipping a primary-metric win that harms something important.
How do I interpret an inconclusive result?
As insufficient evidence at the planned sensitivity, not as proof of no effect. Keep control, record what evidence is still missing, and redesign only if the decision justifies another test.
Does a significant result mean I should ship?
No. Statistical significance and practical importance differ. Ship only when the effect clears your practical threshold, guardrails are acceptable and data-quality checks pass, then still confirm prioritization.
When should I use interviews instead?
When the uncertainty is whether users have the problem, what they do today, or why behavior happens. This workspace says so explicitly and routes to the Interview Script Generator or Survey Builder.
Does CraftUp calculate sample size or duration?
Not here. This workspace records an observation window conceptually. Sample size, duration, power and significance belong in the A/B Test Plan Generator, which supports binary conversion designs and refuses unsupported cases rather than inventing numbers.
What is stored when I use the tool or share a link?
The workspace autosaves the current snapshot in this browser's local storage. A share link compresses the snapshot into the URL, so anyone with the link can read it and the URL may appear in history or logs. Avoid confidential product or customer information.
Last materially updated: 2026-09-23
- Rebuilt from a three-field sentence template into a decision-linked falsifiable contract workspace.
- Added decision, uncertainty, mechanism, direction, user-owned practical threshold, guardrails and pre-committed result actions.
- Added a falsifiability inspector, bundled-change and proxy-metric checks, and a discovery/method router.
- Removed invented guardrail and uplift text from the legacy generic runner path.
Learn the method, not just the sentence
The A/B plan and low-traffic guide own deeper statistical methodology; the courses own broader discovery and validation capability. This page stays focused on the live hypothesis-contract workflow.
The next step depends on what the contract says
Contract testable? Hand it to the A/B plan for statistical design. Uncertainty still qualitative? Gather discovery evidence. Outcome metric undefined? Decompose it in the KPI tree. Supported result competing for capacity? Prioritize explicitly.