The difficulties of assessing qualitative data
For decades, businesses have been taught that good decisions come from numbers. Revenue. Conversion rates. Market size. Survey percentages. Statistical significance.
Those metrics matter.
But they also create a dangerous illusion: that everything important can be measured directly.
In reality, many of the most valuable business decisions are made long before meaningful quantitative data exists. When you’re evaluating a new startup idea, entering a new market, or identifying an emerging customer problem, the data simply isn’t there yet.
This is where qualitative evidence becomes essential—and where many founders struggle.
The Procurement Lesson
One of the biggest misconceptions about qualitative evidence is that it is somehow “less scientific” than quantitative data.
Professionals in procurement face this problem constantly.
Imagine evaluating two suppliers.
One supplier offers the cheapest price. The other demonstrates stronger governance, better communication, superior technical capability, and lower operational risk.
How do you assign a numerical value to “excellent communication”?
How many points should “responsive leadership” be worth?
Can you objectively score organisational culture?
You can’t measure these things with a ruler.
Yet experienced procurement professionals know these factors often determine whether a contract succeeds or fails.
The challenge isn’t that qualitative evidence lacks value.
The challenge is that humans naturally want to reduce everything to a single number—even when doing so removes much of the context.
The Problem with Startup Validation
Startup founders often fall into the same trap.
They ask questions like:
These are perfectly reasonable questions.
Unfortunately, they often don’t have reliable numerical answers at the idea stage.
Instead, what exists are conversations.
People complaining online.
Users describing awkward workarounds.
Businesses spending hours manually solving problems.
Competitors quietly abandoning certain features.
Customers repeatedly asking the same questions.
Individually, none of these tells you very much.
Collectively, they begin to paint a picture.
Looking for Patterns, Not Votes
One person complaining on Reddit proves almost nothing.
Fifty people independently describing the same frustration across different communities is far more interesting.
Not because fifty is a magic number.
Because independent repetition increases confidence that you’re observing a genuine market behaviour rather than an isolated opinion.
This is where qualitative analysis becomes evidence-based rather than anecdotal.
You’re no longer looking at individual comments.
You’re looking for recurring themes across independent sources.
Evidence of Evidence
One concept that’s surprisingly useful is what might be called evidence of evidence.
Sometimes the strongest signal isn’t the original problem itself.
It’s the fact that multiple independent people are all pointing to the same underlying issue without knowing each other.
A customer writes a lengthy complaint.
Another creates a spreadsheet workaround.
A third builds a small internal tool.
Someone else asks whether software already exists.
A competitor quietly releases a feature addressing exactly that pain point.
Each piece alone is weak.
Together, they become evidence of evidence—multiple independent indicators that suggest the market itself is repeatedly generating the same conclusion.
You’re not simply collecting opinions.
You’re observing consistent behavioural patterns emerging from unrelated sources.
Why AI Makes This More Interesting
Historically, analysing qualitative evidence at scale was incredibly difficult.
Humans can only read so many forum posts, reviews, discussions and support threads before fatigue sets in.
AI changes that.
Not because AI magically knows the answer.
But because it can process enormous amounts of unstructured information, identify recurring themes, compare conflicting viewpoints, and highlight patterns that would take humans weeks to assemble manually.
The important distinction is that AI should be identifying evidence—not replacing judgement.
Good decisions still require human interpretation.
The Danger of False Precision
One of the biggest mistakes in analytics is assigning highly precise numbers to fundamentally uncertain information.
“We estimate a 73.6% chance of success.”
Really?
Compared to what?
Based on which evidence?
Precision is not the same as accuracy.
A model producing three decimal places often appears more credible than one expressing uncertainty.
But certainty without supporting evidence is simply confidence wearing a spreadsheet.
Sometimes the most honest conclusion is:
“The available evidence consistently suggests demand exists, but there is insufficient evidence to estimate its size confidently.”
That isn’t a weakness.
It’s an accurate representation of reality.
Intelligence Beats Simple Validation
This is ultimately why market intelligence matters more than simple idea validation.
Validation asks:
“Is this a good idea?”
Market intelligence asks:
Those questions rarely produce neat percentages.
But they produce something much more valuable.
Better decisions.
Because in the earliest stages of building a business, understanding the quality of the evidence is often far more important than pretending certainty exists where it doesn’t.