How should Product and Engineering design a product experiment?
Design a product experiment around one consequential assumption and a decision it could change. Choose the smallest safe method that can test it, define observable evidence and a stopping point, then compare the result with alternative explanations. Product owns the value decision; Engineering owns technical feasibility and safe execution. Prototype research, limited pilots and controlled comparisons answer different questions, so select the method to fit the uncertainty.
Key takeaways
- Name one assumption and the decision it informs.
- Choose a method matched to the uncertainty and exposure.
- Agree evidence, guardrails and a stopping point in advance.
- Interpret results before deciding to build, revise or stop.
A team may know the customer problem but still be uncertain which solution will work. Building a full Feature to answer that question can consume capacity and create avoidable risk. A small experiment can produce decision-useful evidence if its assumption, method and limits are agreed before the result is known.
Start with the decision at risk
An experiment is a bounded test designed to reduce uncertainty for a specific decision. State the customer problem, the proposed change and the assumption that could make the choice wrong. A prototype may test comprehension; a technical spike may test feasibility; a limited live trial may test behaviour under real conditions. The GOV.UK alpha guidance recommends testing the riskiest assumptions with only enough prototype to learn. That is an example of the principle, not a universal phase model.
Use established research and measurement methods
Start with existing user research or operational evidence when it can answer the question. For live changes, define the intended outcome, affected population, baseline if available and possible harms. GOV.UK guidance on measuring services combines performance evidence with user research. A controlled comparison can help isolate a change when feasible, but a small sample or biased selection may leave the result inconclusive. Do not present an observed movement as causal proof without a credible design.
Design the smallest useful test together
Product sets the decision threshold and what customer value would justify further investment. Engineering checks instrument quality, exposure, reversibility and dependencies. Research, accessibility, security or operations specialists should join when the question or risk requires them. Choose a test that can produce enough evidence at acceptable cost and risk. Write down the hypothesis, method, success and harm signals, duration or stopping condition, and who can halt the test.
Read the result and decide
Compare the result with the original decision rule, then inspect the user segments and operational effects. Ask whether seasonality, selection or another concurrent change could explain what happened. An inconclusive result may call for a better test or a narrower question. Record whether the team will proceed, revise, investigate further or stop. AI may help organise notes or analysis, but people must check source data and make the investment call.
Example
Hypothetically, a membership team believes a simpler renewal reminder will reduce missed renewals. Product first tests two message concepts with members to check comprehension. Engineering verifies that the reminder can be sent reliably and that opt-out and failure paths work. A limited pilot then tracks completed renewals alongside complaints. The team has a pre-agreed stop condition for erroneous messages and decides whether wider delivery is justified after reviewing the evidence.
FAQs
-
Does every experiment need an A/B test?
No. Interviews, prototypes, technical spikes and pilots can test different assumptions; use a controlled comparison when the question and conditions justify it.
-
What if the sample is small?
Treat the finding as limited, report uncertainty and avoid a broad claim about all users.
-
Who decides whether to continue?
Product owns the value and priority decision, informed by Engineering, research and relevant risk owners.
