Diagnosing the missing recommendation: retrieval, familiarity or reputation?
Your brand appears in an AI answer about your category, yet disappears when a buyer asks which product to choose. That difference matters: recognition is not the same as commercial consideration. To improve AI search recommendations, first establish whether relevant evidence is missing from retrieval, the brand is poorly understood, or its suitability lacks credible support.
This worked example uses a hypothetical software vendor, not a customer case or measured outcome. It demonstrates what to record, which tests to run, and how to choose an intervention. The objective is a defensible diagnosis - not a claim to know an AI system’s internal reasoning.
Separate three causes of a missing recommendation
Treat retrieval, familiarity, and reputation as competing hypotheses. Observable answers can help distinguish them, but no single response proves a cause. A fourth possibility - genuine mismatch with the buyer’s requirements - must remain open.
| Hypothesis | Observable behavior | What would strengthen the diagnosis? |
|---|---|---|
| Retrieval gap | Relevant product evidence is not visibly used; answers miss documented capabilities. | Supplying the relevant evidence changes the assessment. |
| Familiarity gap | The system misidentifies the brand, category, audience, or capabilities. | Neutral identity questions repeatedly produce incomplete or inconsistent descriptions. |
| Reputation-evidence gap | The brand is described accurately, but suitability is questioned or unsupported. | The answer identifies missing independent validation, outdated information, or substantiated concerns. |
These conditions can coexist. A product might be understood correctly while its current security documentation remains undiscovered. Record a primary hypothesis, alternatives, and confidence rather than assigning a permanent brand-level label.
Establish the buyer-prompt symptom
Consider this illustrative request: “Which project-management tools should a 50-person agency shortlist for client approvals, guest access, and EU data hosting?” Assume the vendor genuinely meets those requirements. The following record shows how to document a hypothetical omission without inventing customer results.
| Field | Illustrative record |
|---|---|
| Exact prompt | Preserve the wording above unchanged. |
| Intent | Commercial shortlist with three explicit requirements. |
| Environment | Record engine, model if visible, date, locale, and search mode. |
| Answer structure | Shortlist, comparison table, and buying caveats. |
| Named alternatives | Copy every actual product name from the captured answer. |
| Sources | Save cited URLs and the claims they support. |
| Brand status | Absent from prose; documentation cited; not recommended. |
| Initial hypothesis | Evidence may be discovered without establishing comparative fit. |
Record presence, citation, and recommendation separately. A mention names the brand; a citation references its material; a recommendation explicitly endorses considering it for the stated need. These are separate fields, not mutually exclusive stages.
Repeat the unchanged prompt in fresh sessions, recording every run rather than selecting favorable examples. Then test related buyer requests separately so wording changes do not contaminate the baseline. Retain the complete answers for human review.
Test retrieval before rewriting everything
1. Match evidence to the implied sub-question
Translate the request into evidence requirements: how approvals work, what guests can do, and which hosting arrangements apply. Inspect whether existing pages answer those questions directly. A broad “built for agencies” statement does not establish any of them.
2. Check specificity and discoverability
Look for current documentation, eligibility conditions, limitations, and supporting product details. Check page access, indexing directives, rendered content, and links from relevant product or support pages. An AI-readiness check can help identify technical obstacles, but passing one does not prove retrieval occurred.
3. Compare unassisted and evidence-assisted answers
Keep the buyer requirements fixed, then provide a relevant excerpt or accessible page and ask for reassessment. If the assessment changes, an evidence-access or presentation problem becomes more plausible. This is a diagnostic probe, not proof: supplying evidence also changes the conversational context.
A citation confirms visible use of a source, while an absent citation does not establish that no retrieval happened. Without retrieval logs, describe the finding as a suspected access or evidence-use gap. The BEIR benchmark’s heterogeneous retrieval datasets also illustrate why testing across different tasks matters rather than relying on one query.
Interpret citations without treating them as endorsements
The supplied research brief reports: “In a study of 158,099 AI-search answers, ChatGPT retrieval explained 32% of within-site citation variation, compared with 2.8% explained by page properties.” The reported citation-variation finding has not been independently verified here. Treat it as provisional rather than a validated benchmark.
Even if verified, that statistic concerns citation variation - not recommendation probability or causal effects. It would not establish that retrieval and page properties explain the entire recommendation mechanism. Nor would it justify predicting commercial outcomes from citation counts.
In the example, documentation might support a definition of guest access while another product receives the shortlist position. The cited page has supplied information, not necessarily evidence of superior buyer fit. Compare the claim supported by each citation with the reason given for each recommendation.
Identify familiarity and reputation evidence
Once relevant evidence is visibly used, examine what the answer understands and what it can substantiate. Familiarity concerns identification and positioning; reputation evidence concerns credible support for trust and suitability. Avoid interpreting silence as negative sentiment.
Use this checklist:
- Consistent entity facts: Align names, product descriptions, ownership, and capabilities across maintained profiles.
- Clear category positioning: Explain the buyer, use case, and boundaries of product fit.
- Independently verifiable proof: Support applicable claims with accessible, current evidence.
- Accurate third-party information: Request corrections to demonstrably outdated or incorrect listings.
- Customer outcome evidence: Publish only authorized, documented outcomes, including context and limitations.
Separate missing proof from adverse proof. The former calls for substantiation; the latter may require correcting an actual product or service issue before changing communications. An audit of brand descriptions and their supporting sources can organize that distinction.
Choose the next intervention
Follow this decision tree before assigning work:
- Does the product meet the buyer’s requirements? If not, accept the exclusion or address the product gap.
- Is relevant evidence accessible and visibly used? If not, prioritize technical access, direct answers, and supporting links.
- Is the brand described accurately? If not, strengthen consistent entity facts and category evidence.
- Is suitability supported by credible proof? If not, address missing validation, inaccurate information, or substantiated concerns.
Change one evidence area at a time, then rerun the original prompt set under comparable conditions. Track recommendations separately from mentions and citations. Improvement supports the intervention’s usefulness, but does not isolate causality.
FAQ
Can one answer establish a reputation problem?
No. Inspect its stated reasons and supporting sources before drawing that conclusion. Require repeated, corroborated evidence rather than treating generated criticism as fact.
Should the model diagnose its own omission?
Use its explanation to generate hypotheses, not settle them. Research cautioning against using LLMs as relevance judges reinforces the need for independent assessment. Check claims against source material.
Start with ten buyer prompts
Classify ten high-intent buyer prompts using this framework, retaining “uncertain” where evidence is insufficient. Run one focused intervention for each supported diagnosis instead of launching an undifferentiated AI-search campaign. The objective is to resolve a demonstrated barrier - not simply accumulate citations.
