A product team I worked with had a problem they could not quite name.
Their conversion rate had improved by eleven per cent over the quarter. The dashboards looked clean. Leadership was pleased. The number had moved in the right direction.
The first question was attribution. That quarter had included three meaningful changes: a redesigned signup flow, a new pricing page, and a paid acquisition channel that brought in a different kind of user. Sequencing the changes, holding back a control, or tracking the new channel as its own cohort would have made their effects easier to separate. Standard hygiene, and it mattered.
That was the surface issue. The deeper question was what they wanted to be able to claim. Did they want to say conversion had increased? That was true. Did they want to say the signup redesign had caused the increase? The evidence could not carry that. Did they want to say the people who converted got more value afterwards? That needed a different kind of evidence again, and nobody had collected it.
The team had measurement. What they lacked was evidence for the claim they wanted to make.
It is one of the situations I meet most often in product organisations, including teams with mature analytics and experimentation. They have data and dashboards in quantity, and little sense of what the data allows them to say, because measurement gets treated as evidence for whichever claim the team was hoping to make.
Every metric carries a claim
The claim is usually unspoken, often unexamined, and frequently stronger than the data can support.
Conversion implies that an experience improved enough to change behaviour. Conversion also moves with pricing, traffic mix, seasonality, a campaign, a competitor’s outage, or a change in how the event is logged, so the number can be accurate while the interpretation is wrong. Satisfaction implies that people valued the service. It also reflects the wording of the question, the moment the survey appeared, and the absence of everyone unhappy enough to leave before they could be asked. In a social programme, movement on an outcome measure implies that something changed for participants because of the intervention. It may equally reflect life events, regression to the mean, or the enrolment of people who were about to improve anyway.
Each of these is a real signal about a specific thing. Trouble starts when it is read as evidence for something adjacent.
The dashboard makes this worse, and the dashboard is a designed object. A green arrow reads as good news. A neat line implies clarity. A percentage change presented confidently in a steering meeting feels like a conclusion. Most people who look at a dashboard for thirty seconds act on the impression, and the visual authority of the display does the persuading that the method underneath cannot. A metric without a clear claim is a number waiting to be overinterpreted.
What claim discipline asks
Claim discipline is a short run of questions asked before the measurement work begins, and the order matters.
It starts with the statement the team wants to be able to make at the end, to whom, and in support of which decision. The choice of metric comes after. Then comes the standard of evidence that statement requires, set by the decision it will inform. A roadmap conversation can rest on directional signals, a scaling decision needs more, and a claim that enters a regulatory submission, a grant report or an investment case needs more again.
The most useful question comes next: what weaker statement could the team honestly make today? A team may want to say that a change improved activation, while the evidence supports only that activation increased during the period in which the change was released. Those are different sentences, and the distance between them is the work. Last comes what would have to change to support the stronger one. Sometimes that is instrumentation, a comparison group, implementation data that tells one variant from another, a result defined in advance, or simply enough time to see whether the effect persists.
Often the answer is structural. More data of the same kind cannot repair a design that was never able to support the claim. A test run on an audience that does not represent the population the team cares about will not generalise however clean the statistics. A charity that surveys participants only at the end of a programme cannot make a robust contribution claim on that alone, however warm the responses.
Methods are protections
Seen this way, the methods a team can reach for are protections against different failures of inference, and choosing well means knowing which failure the moment cannot afford.
A prototype protects against building something people do not understand or want. A usability test protects against assuming a journey works because the concept makes sense. A concierge test protects against committing to infrastructure before the value is established. A pilot protects against mistaking desirability for operational viability: whether the team can actually run the thing, and what breaks first. A controlled experiment protects against relying on opinion or hierarchy when behaviour can be observed. A quasi-experimental comparison protects against reading change over time as the intervention’s effect when randomisation is unavailable. A formal randomised evaluation protects against harder forms of selection bias when the stakes justify the cost.
Every result also has a boundary. An A/B test shows that a change moved a defined behaviour, for the people included, on the metric chosen, over the period run. That boundary is what makes the result trustworthy, and it is also why the result cannot answer the questions that follow it: why it worked, whether it lasts, whether it transfers, whether anyone was better off. Those are reasonable next questions. Treating them as findings is where most overclaiming happens, and it compounds. A roadmap is a sequence of decisions, each resting on the evidence before it, and a team that is mildly loose about what each test proves can spend a year building on claims that would not survive scrutiny one at a time.
The claim is a design material
This matters most for designers, because design sits where claims are formed implicitly all the time.
Designers define the variants, entry points, populations, journeys, success states and moments of measurement. A confusing treatment can make a sound idea look ineffective. A treatment with several simultaneous changes makes attribution impossible however careful the analysis. A test that runs only on highly engaged users describes highly engaged users, and that choice is usually made in the design before it appears in the analysis. The reporting layer is design output too, and it mediates every decision taken from the data.
So the claim can be shaped like any other material. A vague one says a tool will help teams make better decisions. A shaped one says that teams using it can identify priority issues faster, and with more confidence, than they could with the previous process. The second forces the useful questions: faster by how much, confidence measured how, compared with what, for which teams, over what period, and what would count as failure. It also protects the organisation from overstating what it learns. Perhaps the tool improves confidence and leaves speed unchanged. Perhaps it helps experienced teams and confuses new ones. Perhaps the evidence supports a usability claim and not yet an outcome claim. That is good measurement doing its job.
Every experiment claims its risk is acceptable
One more claim sits inside every test, and it is usually the least examined.
In social programmes it is visible. Participants may be vulnerable, and choices about comparison groups, consent and data collection shape people’s real circumstances. Product experiments carry the same claim more quietly. A pricing test, a ranking change, an onboarding flow or an AI-assisted workflow can change what people see, understand, choose, disclose, pay for or trust. A test can lift conversion by reducing comprehension, raise engagement by raising anxiety, or improve one group’s experience while quietly excluding another. Diffuse consequences are still consequences.
The question a mature team asks of a winning result is what behaviour it produced, for whom, at what cost, and whether the team would be comfortable scaling that behaviour.
What precision buys
When claims are vague, evidence becomes political. Teams argue about whether the numbers are good enough, whether a result counts, whether the board or the funder will accept the story. When claims are precise, the organisation can see what is known, what is unknown, and what kind of evidence would reduce the uncertainty that matters. In the teams I have seen handle this well, someone asks early what exactly is being claimed and what would falsify it. Often it is a designer, a researcher or a product lead, and it matters more than how sophisticated the analytics are.
It is slower at the start of the work, and much faster, and more honest, across its life.
Defining the claim before the evidence is old ground: the separation of claim, data and warrant in Toulmin’s model of argument; the typology of validity in Shadish, Cook and Campbell, which sets out the boundaries every result carries, and Campbell’s law on what happens to indicators once decisions lean on them; contribution analysis, in Mayne, for what a programme can honestly claim when attribution is out of reach; measurement as the reduction of uncertainty for a decision, in Hubbard; the working standard for trustworthy online experiments, in Kohavi, Tang and Xu; and pre-registration, which brings the same discipline to research.
The point here concerns design. The people shaping variants, populations and dashboards are part of the evidential apparatus, and the claim is something they can shape as deliberately as an interface.