AI Helps Us Build Products Faster. But Are We Understanding Users Faster?

A working prototype can answer one question: “Can this idea be built?” It does not yet answer a more important one: “Does anyone actually need it?”

The gap between those questions deserves attention as product teams bring AI into their workflows. We can create interfaces, write code, and explore alternatives faster. But the speed at which we produce a solution does not establish that we understand the problem behind it.

My view is that AI's greatest advantage is not simply helping teams build more. It is helping us test earlier, recognize mistaken assumptions sooner, and spend fewer resources on things people do not need.

1. Faster building is a real advantage—but be clear about what is being measured

In an experiment published in 2023, developers were asked to build an HTTP server in JavaScript. The group using GitHub Copilot completed the task 55.8% faster than the group without it. This is evidence of a speed benefit on a specific programming task, not evidence that AI accelerates the entire product-development process by the same amount. The study did not measure market demand, retention, or revenue. Study: The Impact of AI on Developer Productivity.

That distinction matters because product teams face two different kinds of uncertainty. One concerns execution: how do we make a feature work correctly? The other concerns value: do people need it, understand it, and incorporate it into their actual work?

Resolving the first does not resolve the second. A well-built feature can address a problem that is not important enough. A workflow can meet its requirements perfectly while those requirements rest on an incomplete understanding of the user.

The number of features shipped therefore cannot substitute for evidence that a product is becoming more useful.

2. A reasonable idea can still fail to improve the experience

This problem did not begin with AI.

In a 2013 account of experimentation at Bing, Microsoft reported that fewer than a third of ideas evaluated in controlled experiments improved the metrics they were intended to improve; the proportion was even lower in an already optimized environment such as Bing. That was an observation in Microsoft's context at the time, not a universal startup failure rate. It nevertheless illustrates why even experienced teams cannot rely on intuition alone to predict the value of a change. Bing: Large Scale Experimentation.

A concrete example appears in a 2017 study of experimentation at Microsoft. The Bing team tested light-grey and dark-grey backgrounds against the usual white background, expecting the direction to be worth exploring. Both variants degraded key metrics. The result helped the team decide against a broad rollout and against prioritizing further work in that direction. The Benefits of Controlled Experimentation at Scale.

These examples do not demonstrate that AI makes product decisions worse. They expose an existing risk: we can be highly confident in a solution that has not been validated.

My inference is that this risk needs careful management as building becomes cheaper. If teams spend all the time saved on implementing more ideas without strengthening validation, they may accumulate more changes whose value remains uncertain.

AI does not turn an assumption into a fact. It can turn that assumption into software faster.

3. Understanding users is not the same as generating a convincing persona

A customer profile can be detailed and persuasive: a job title, goals, anxieties, and obstacles at work. But until those details are checked against real people, they remain hypotheses.

Nielsen Norman Group distinguishes synthetic users—AI-generated simulations—from research with actual users. Its 2024 analysis suggests that simulations can support desk research and hypothesis generation, but should not replace research with real people. This is professional guidance, not an experiment establishing a universal error rate for all AI models. NN/g: Synthetic Users.

Consider a hypothetical team building an AI document-processing tool for businesses. A persona might say that customers want to save time synthesizing information. That goal is still too broad to guide design. Are employees allowed to upload documents to an external service? Do they need a quick summary, or precise citations to send to an approver? Who is accountable if the output contains an error?

If the real obstacle is permission to use the data or the ability to verify the output, adding more summary formats may solve very little.

Understanding users means understanding constraints specific enough to change a product decision. Sometimes the most valuable discovery is not what people want added, but why they cannot yet use what has already been built.

4. Behavioral data tells us where to investigate, not automatically why something happens

Suppose an AI product records substantial drop-off at the document-upload step. This is an illustrative scenario, not data from a particular project.

A team might conclude that uploading is too complicated and immediately use AI to redesign the screen. Yet the same behavior could have several explanations: users do not have a document ready, do not understand the supported formats, worry about privacy, or do not see enough value in the promised result.

Each explanation implies a different response. Fewer clicks will not address a concern about data. More privacy guidance will not help if the actual problem is a broken upload.

The approach I propose is to use behavioral data to locate an anomaly, then combine observation, conversations, and technical checks to narrow down its cause. Only then should the team choose a change to test.

A UX audit can identify friction and prioritize questions. Its conclusions should still distinguish between problems supported by evidence and problems that have only been inferred.

There is a meaningful difference between “we have found the answer” and “we have found the most useful hypothesis to test next.”

5. AI's biggest opportunity is shortening the path from a question to evidence

The argument is not for slowing product development down. It is for using the new speed to learn faster.

Instead of a vague assignment to improve onboarding, a team can start with a specific hypothesis: new users stop because they cannot picture the output they will receive after uploading a document.

AI can help produce a prototype that includes an example output. The team can ask target users to perform a concrete task with it, observe whether they understand the next step, and identify what still makes them hesitate. If the hypothesis is not supported, the team can revise or discard the approach before investing in a full implementation.

When traffic and measurement conditions are sufficient, a controlled experiment can help evaluate a change's impact. For products with few users, direct usability testing and limited trials can still reveal problems, but findings from a small group should not become confident claims about the entire market.

A rapidly built prototype has research value when it helps answer a clear question. Without knowing what we are testing, we can generate ten alternatives and still have no sound reason to choose one.

6. Teams should track what they have learned, not only what they have completed

Alongside “What did we ship this week?”, I believe teams should be able to answer: what did we learn about users, what evidence changed an assumption, and which decision changed because of that evidence?

An experiment that stops a team from building an unnecessary feature can still represent progress. An observation that locates the problem in an approval process rather than an interface can prevent a redesign aimed at the wrong issue.

The number of interviews or hypotheses tested should not become another vanity metric. What matters is whether the work improves decisions and ultimately creates value for users.

Expertise matters here—not as a substitute for evidence, but as the ability to frame useful questions, choose appropriate methods, interpret results, and take responsibility for decisions. That expertise may sit with a founder, an internal specialist, or an external partner. Experience does not exempt anyone from testing their assumptions.

AI should help us guess less, not just produce more

None of the research cited here establishes a general rule that AI makes teams understand users less well. That is not the conclusion of this article.

The evidence illustrates that execution speed, the quality of assumptions, and real-world value are different things. Improving the first does not automatically guarantee the other two.

The opportunity lies in what we do with the time we save. We can use it to build more of what we already believe is right. Or we can use it to discover sooner what we have not yet understood.

AI helps us build products faster. The more durable advantage will belong to teams that turn that speed into better evidence and better decisions.

The question is not only “How much have we built?” It is “How much less are we guessing about our users?”

Further reading

For related design considerations, explore our guides to reducing cognitive load in UX and designing clearer app onboarding.

‍