When AI creates the idea and judges it, what have you learned?

An innovation team can generate a set of concepts, ask an AI system to assess them and arrive at a ranked shortlist remarkably quickly. That can be a useful way to organize an early discussion. The difficult moment comes when a ranking starts to look like evidence that customers will want the winners.

When AI creates an idea and then judges it, the team has learned how well the idea matches the criteria it was scored against, which is a different thing from learning that customers want it.

An assessment can be coherent and still answer a different question from the one the business needs to resolve. What matters is what the score has been shown to mean.

A good answer to the brief can be a weak opportunity

Consider an illustrative food brand exploring products for busy weekday lunches. A team asks AI to generate ideas and score them on convenience, novelty, brand fit and appeal.

A personalized subscription comes out ahead. Its description explains each benefit fluently and anticipates the obvious objections.

The evaluation may have identified the concept that best expresses the criteria. It has not necessarily established whether people want another subscription, how they would compare it with buying lunch locally, or whether the proposed price leaves enough value for the business.

Those questions could change the recommendation even if every score in the table was applied consistently.

The criteria also deserve scrutiny. Convenience could mean less planning, fewer decisions or being free to change plans at the last minute. A proposition might improve one while making another worse. Until the team understands which matters in the intended occasion, a high convenience score can hide the choice it needs to make.

This problem is familiar in human evaluation too. A persuasive concept and a well-run scoring meeting do not automatically establish demand. AI makes the distinction more pressing because it can produce both the proposition and an apparently complete case for it in the same exchange.

Useful criticism has a different job from validation

AI can help a team notice missing information, compare an idea with a stated brief, consider other interpretations or find questions worth investigating.

Used this way, its value lies in improving the next decision or piece of work.

It can also help evaluate ideas in defined settings. The relevant issue is whether performance has been established for the task at hand. Agreement with expert ratings on a particular set of material is evidence about that comparison. It is not automatically evidence about repeat purchase, commercial viability or a different category.

Research on scientific ideation shows why both the opportunity and the limitation matter. In a paper presented at ICLR 2025, more than 100 researchers in natural language processing wrote ideas and blind-reviewed them alongside AI-generated ones. The AI-generated ideas were judged more novel than the experts’ ideas, though slightly weaker on feasibility.

Among the open problems, the authors reported “failures of LLM self-evaluation.” That research concerns scientific proposals and expert judgments. It does not establish how well AI predicts demand for a consumer product.

A system that helps expose a weak explanation may be valuable even when it cannot settle which product customers would buy.

A second answer is not necessarily a second source

Asking the model to take a customer’s perspective can produce useful objections. It does not turn those objections into observations of customers.

Asking another model may broaden the critique, though the answers could still draw on similar material or share the same assumptions.

The team needs to know what is independent of the idea being judged: whether the evaluation has met information capable of overturning the original story, or has mainly rearranged the same description into a more authoritative format.

That does not mean every early idea needs a concept test with customers. Existing research, market evidence and experience may already resolve important questions. Some concepts can sensibly be set aside because they are infeasible or outside the brief. The distinction matters most when a recommendation depends on a claim about people that has not actually been examined.

Match the conclusion to the commitment

A shortlist for further exploration can tolerate uncertainties that would be unacceptable in a recommendation to launch.

An AI assessment may be enough to help decide which rough concepts deserve human attention. It may contribute to a more consequential decision alongside relevant evidence and an understood evaluation process.

What it should not do is silently move the work from “worth exploring” to “validated.” The business needs to understand what would have to be true for the recommendation to stand, and which of those conditions have been examined.

The useful output may be a better question, a stronger alternative or a clearer account of why an idea is promising. Those are substantial contributions. A score becomes more valuable when the team can explain what it permits them to do next, and what it leaves open.