Product Management Foundations · Module 5 of 15
Test the assumption, not the whole idea.
Discovery gave you a plausible opportunity, one or more solution directions and unresolved assumptions. This chapter teaches how to choose the uncertainty that could change the decision, match it to a credible test, interpret mixed evidence and decide whether an MVP is even necessary.
The direct answer
Validation is not proving that an idea is right. It is deliberately collecting evidence about an assumption that matters to a product decision. Start with the decision and the assumption, choose a test that can observe the relevant behavior or mechanism, define what the result can and cannot support, then update the decision based on the evidence.
An MVP is one possible learning vehicle, not the default answer to every uncertainty. Interviews, prototypes, technical evaluations, fake doors, manual pilots and controlled experiments answer different questions. Learning can happen before, during and after product implementation.
The chapter question
What exactly am I uncertain about, what evidence would change my decision, and what is the least costly credible test that can produce that evidence?
You came from
A bounded opportunity, candidate directions and explicit assumptions are now visible.
You are here
Which assumption matters next, and what test can reduce that uncertainty?
Next
What do you choose when evidence remains incomplete and trade-offs still conflict?
Preview context
The published first lesson contains older “test before build” framing and several broad claims that this chapter does not repeat. The preview below keeps only a real source question whose answer remains accurate: problem validation asks whether a problem exists and matters. The deeper problem-evidence gate belongs to Master Problem Validation.
Real CraftUp lesson
Try the quiz questions taken directly from the published CraftUp lesson for this module.
Quick practiceWant another challenge? Practice another round.
Real CraftUp curriculum · no sign-up required
Core mental model
Start from the product decision, not from a favorite validation tactic. The method earns its place only by producing evidence that is relevant to the assumption.
Name the product choice that could change: continue exploring a mechanism, fund automation, run a pilot, widen exposure, narrow a segment or stop.
“Validate the idea” is too vague. A test is useful only if its result can update a real decision.
State what must be true for that choice to make sense. It may concern desirability, usability, feasibility, viability, trust, safety or another material constraint.
There is not always one objectively riskiest assumption. Choose the uncertainty whose resolution is most useful now.
Choose a method that can observe something relevant to the assumption with proportionate cost, exposure and user risk.
A cheap test that cannot discriminate between competing explanations is not automatically a good test.
Record what happened, the context in which it happened, contradictory observations and what the method could not establish.
Evidence can increase or reduce confidence. It does not certify a belief as universally true.
Continue, narrow, revise, run a different test, pause, stop, expand, inspect data quality or reopen the opportunity or strategy.
The evidence informs judgment. It does not make the product decision on behalf of the team.
Make the assumption explicit
These lenses are useful prompts, not a universal taxonomy. A product direction can depend on several at once, and safety or feasibility can dominate even when another assumption feels more uncertain.
Do relevant users care enough about the outcome to change behavior in the context that matters?
Can people understand and use the proposed interaction sufficiently for the intended job?
Can the system perform the required behavior reliably enough under the relevant technical constraints?
Can the direction work within the economic, operational or organizational constraints that matter?
Can the learning method and eventual solution avoid unacceptable harm, trust failure or unsafe exposure?
10× learning asset
Work through one fully hypothetical field-service scheduling case: proactive conflict warnings after same-day schedule changes. Open each reveal only after deciding what the current evidence can justify. The lab produces a decision trace, not a score.
Hypothetical field-service platform: should the team invest further in proactive conflict warnings for same-day schedule changes?
Reveal the reasoning
This is a decision with an investment consequence. “Validate conflict warnings” would hide what could actually change.
Conflicts occur often enough to matter; dispatchers miss them early; earlier visibility changes action; warnings can be generated reliably; alert burden is tolerable; teams trust the warning enough to use it.
Reveal the reasoning
Several assumptions are coupled. The goal is not to multiply impact and confidence into a synthetic winner.
Suppose historical data does not yet show whether overlaps are frequent enough to justify engineering attention.
Reveal the reasoning
Resolve frequency first because a negative answer could change the investment decision before interaction design matters.
Support has seen overlapping-job questions and dispatchers report manual reconciliation, but frequency and denominator are unclear.
Reveal the reasoning
The evidence is directionally relevant but cannot establish how common the issue is. Keep provenance and missing denominator visible.
A historical event analysis could show that true same-day overlaps are rare in the target workflow or concentrated in a segment the current decision does not serve.
Reveal the reasoning
Pre-defining counter-evidence makes it harder to move the goalposts after seeing a result.
For frequency, inspect historical schedule events, support incidents and workflow context. A clickable prototype is a poor instrument for this uncertainty.
Reveal the reasoning
Method fit matters more than a universal hierarchy of “strong” versus “weak” evidence.
The test should make actual conflict incidence, context and detectability inspectable-not merely ask whether dispatchers like the idea of warnings.
Reveal the reasoning
The observable should map to the assumption. A convenient metric is not useful if the mapping is weak.
Historical frequency cannot tell you whether dispatchers understand the warning, whether earlier visibility changes behavior or whether automated detection is reliable.
Reveal the reasoning
Every test leaves uncertainty behind. Naming the limitation prevents one positive signal from becoming “the idea is validated.”
Use the least costly method that can produce decision-relevant evidence without unnecessary user exposure. Small does not mean weak or arbitrary.
Reveal the reasoning
The evidence bar depends on stakes, reversibility, uncertainty, evidence quality and the cost of being wrong.
Classify the evidence as supporting, contradicting, inconclusive or invalid/unreliable for this assumption and decision.
Reveal the reasoning
An inconclusive result is legitimate. It may mean the test was insensitive, the assumption was poorly framed or the context was too narrow.
Continue, narrow, revise, run a more discriminating test, pause, stop or inspect evidence quality. If frequency is supported, the next uncertainty may be comprehension, behavior, feasibility or trust.
Reveal the reasoning
Validation succeeds when it changes the evidence state enough to improve a product decision-even when the answer is “do not proceed yet.”
Worked decision readout
Should we invest in automating proactive conflict warnings?
Earlier, sufficiently reliable warnings change dispatcher behavior during same-day conflicts.
A bounded manual pilot produces warnings from real scheduling data and presents them in the live dispatcher workflow.
In this hypothetical trace, dispatchers use some warnings to resolve conflicts earlier, while other warnings are ignored because context is missing.
The pilot does not establish automated detection reliability, scalable operations, long-term alert fatigue or broad adoption.
The behavior is compatible with the value assumption in this bounded context, but the evidence remains mixed and does not settle feasibility.
Run a narrow technical feasibility evaluation before investing in automated production delivery; revisit the direction if reliable detection is not achievable.
This example is intentionally hypothetical and descriptive. It is a teaching record, not a benchmark or a downloadable validation tool.
Evidence fit
There is no universal evidence ladder. A payment may be highly relevant to willingness to pay and almost irrelevant to usability. An interview may be excellent for reconstructing prior workflow while saying little about population prevalence. Fit the evidence to the question.
Useful when
Context, motivations, prior behavior, workflow, alternatives and how people explain a decision or constraint.
Does not automatically establish
They do not by themselves establish population prevalence, future adoption or causal product effect.
Useful when
Comprehension, navigation, interaction and task performance under the conditions of the test.
Does not automatically establish
It does not by itself establish sustained value, market demand or willingness to pay.
Useful when
Response to a proposition, channel or bounded low-cost action under a particular audience, message and context.
Does not automatically establish
A signup does not automatically establish usage, retention, willingness to pay or product-market fit.
Useful when
Behavior at a real product surface and relative interest in a proposed capability when the setup is ethically appropriate.
Does not automatically establish
It does not establish value after use, successful implementation or willingness to pay; trust and disclosure matter.
Useful when
Whether delivering the outcome creates value, how the workflow behaves, repeated use and-when designed for it-commitment or payment.
Does not automatically establish
It does not automatically establish scalable economics, automated feasibility or behavior after automation.
Useful when
Feasibility, performance, integration, architecture or reliability uncertainty.
Does not automatically establish
It can answer a technical question without telling you whether users want or can use the resulting product.
Useful when
The causal effect of a defined intervention when assignment, exposure, instrumentation, measurement and decision rules are valid.
Does not automatically establish
It is not the default for every uncertainty and may be inappropriate when scale, horizon, contamination or measurement make inference weak.
If a randomized controlled product experiment is actually the chosen method, use the A/B Test Plan Generator for the detailed experiment contract.
Practice · match the test
Comprehension
Dispatchers do not understand the proposed conflict warning.
Prototype / usability test
Observe whether people can identify the conflict, understand why it exists and choose an appropriate action. A technical spike, landing page or market-sizing exercise does not answer comprehension well.
Feasibility
Current scheduling data may not contain enough information to detect overlaps reliably.
Technical / offline feasibility evaluation
Inspect the available data and detection behavior against relevant requirements. A survey cannot establish whether the system can perform the job.
Live behavior
Dispatchers may act earlier when a warning appears during the live workflow.
Bounded concierge or Wizard-of-Oz pilot
A carefully governed manual backend can expose the real workflow and action change before full automation. A concept interview alone cannot establish live behavior.
Causal placement effect
Does changing warning placement causally increase successful resolution in a live high-volume workflow?
Controlled experiment-only if the design conditions are met
When valid assignment, exposure, instrumentation, outcome measurement and sufficient observation are feasible, a randomized experiment may fit. Use the A/B Test Plan Generator for the production experiment contract.
Interpret before you celebrate
Define before the test what observations would increase confidence, what would weaken it, what the test cannot distinguish and what action follows each interpretable outcome. Not every method needs a numerical threshold; the decision criteria should fit the method.
The evidence increases confidence enough to justify a decision update in this context. It is not permanent certification.
The current belief loses support. Revisit the direction, the assumption or the scope rather than explaining the evidence away.
The test did not discriminate enough. Improve the design, choose a better instrument or stop investing in the question if resolution is not worth the cost.
Execution, instrumentation, data quality, context or contamination prevents responsible interpretation. Fix the evidence problem before making the result carry more weight.
Evidence is what happened. Interpretation is what it suggests about the assumption. The decision combines that interpretation with strategy, risk, constraints, alternatives and opportunity cost. Pre-commitment can reduce goalpost movement, but a threshold cannot make a weak measure meaningful.
Practice · what are you really testing?
It may support a claim about response to the proposition and CTA under that traffic, audience and context.
Limit: It does not establish retention, value after use or sustainable product demand.
It may support a claim about comprehension or usability under the test conditions.
Limit: It does not establish market demand or repeated real-world use.
That can be meaningful evidence about value, adoption and willingness to pay for that segment and setup.
Limit: It does not by itself establish automated feasibility or scalable economics.
That supports a feasibility claim about the tested technical mechanism and conditions.
Limit: It does not establish desirability or usability.
MVP restraint
An MVP is a deliberately limited product or service implementation used to create real learning about value, behavior or a critical product assumption while avoiding unnecessary scope. Not every validation test is an MVP: an interview, survey, technical spike, fake door or isolated prototype can be an experiment without being a product.
“Minimum” means the smallest scope that still creates the learning conditions needed for the decision. Sometimes that is one complete end-to-end workflow rather than many partial features. “Viable” is context-dependent: the experience needs enough quality, trust and functionality that the learning is interpretable, but it does not have to be polished.
Research / experiment method, not an MVP.
Usually a prototype or test artifact, not necessarily an MVP.
It could function as an MVP when enough of the real value loop exists to create the learning the decision requires.
It could be an MVP, pilot or experiment depending on the learning objective and operating context.
Minimum lovable product is one practitioner label, not a universal successor to MVP. The durable question is: what minimum experience quality is necessary for this learning question?
Trust and exposure
Use a fake door only when user harm is low, disclosure after the interaction is prompt and clear, no irreversible commitment is created, and billing, data and trust implications are controlled. Do not charge people for nonexistent functionality.
A Wizard-of-Oz setup can manually perform backend work behind a realistic experience, but privacy, sensitive-data handling, consent or disclosure, safety and legal constraints still apply. Do not deliberately make outputs wrong to imitate a hypothetical automated system.
AI in validation
AI can help draft test materials, generate alternative copy, prototype interactions, challenge assumptions, suggest possible confounds and summarize real evidence. Synthetic users are not validation evidence for real need, willingness to pay, actual workflow, prevalence or adoption.
AI-generated landing copy, prototype content and interview prompts can themselves bias a test. Human review should focus on whether AI improves the learning workflow without contaminating the evidence.
Decision-ready test contract
What must we learn for this decision?
What method could produce relevant evidence?
What would we actually observe?
What will remain unresolved?
What resources or user risk does the test impose?
What changes after each interpretable result?
High-stakes contexts
In healthcare, finance, regulated or safety-critical contexts, fake doors, manual backends and limited implementations may be inappropriate or require specialized review. Validation design must respect the domain’s safety, legal, privacy and ethical constraints. This chapter is a product-learning framework, not regulatory advice.
Full curriculum
The real lesson order remains unchanged. The phase labels add an orientation layer while the public chapter corrects rigid rules, unsupported thresholds and outdated implementation claims rather than promoting them as universal guidance.
What belief is the product decision relying on?
What evidence can this method actually provide?
What is the least costly credible way to observe the thing that matters?
How much real product or service must exist for the learning to be interpretable?
What evidence would update the decision, and what remains unresolved?
How does one test change the next product decision?
Use the specialist owner when the job changes
Master Problem Validation owns the narrower problem-evidence gate: pursue, narrow, reframe, stop or gather one missing signal. C09 starts after a candidate direction exists.
Use the Experiment Hypothesis Generator to turn a testable intervention assumption into a structured hypothesis. This chapter teaches why the assumption matters and what evidence could change the decision.
Use the A/B Test Plan Generator for assignment, exposure, primary and guardrail metrics, sample assumptions, horizon, data-quality checks and decision rules.
Use the interview sample-size and saturation owner when the method is qualitative research and you need a defensible stopping logic rather than a universal interview count.
Use the triangulation owner when complementary evidence is needed to compare competing explanations. C09 teaches the need; the specialist guide owns the methodology.
Use the Opportunity Solution Tree when the relationship between outcome, opportunity, solution direction and assumptions still needs structure before you test a branch.
Use the Interview Script Generator for the actual research artifact. C09 only decides whether an interview fits the uncertainty.
Use the Survey Builder & Scorer when self-report is relevant and question, sample, timing and interpretation need an explicit survey contract.
Use Product Discovery for ongoing discovery methods, cadence and opportunity mapping. This chapter stays focused on assumption-to-test judgment.
Ongoing feedback cadence belongs to Customer Feedback and Continuous Discovery. Deeper analytics belongs to Metrics, Analytics, and Data Literacy.
Validation questions beginners get wrong
Collect relevant evidence about an assumption for a decision. It does not prove that an idea is correct.
No. A prototype may test comprehension, workflow or interaction without being a usable product or service implementation.
No. Use the smallest credible method that can observe the uncertainty. Sometimes that is research, a prototype, a data analysis or a technical evaluation.
When the question concerns context, prior behavior, motivation, workflow or alternatives that people can meaningfully reconstruct.
When a causal intervention comparison is the real question and the experiment can be designed and measured validly.
No. It is one bounded behavior under a specific proposition, audience, acquisition source and context.
Do not force a pass or fail. Decide whether a more discriminating test is worth the additional cost and exposure.
Strategic Decision-Making for PMs, because validation changes the evidence state without removing every trade-off or uncertainty.
Next learning step
Once validation has changed what you know, the next PM job is choosing among imperfect options, trade-offs and opportunity costs. Do not ask the data to make that strategic choice automatically.
Continue to Strategic Decision-Making for PMs →Build your product skills with short, practical lessons you can fit around your day.
Start with our free courses and upgrade anytime.
