From v1 to v5: Building an AI Visibility Scoring System
Our AI visibility scoring system took five versions to become an instrument we would defend in front of a client. Version 1 produced a single visibility score from single runs. Its database contained a citation field that was never computed — it stored null while reports implied three metrics. Version 4 made citation real and split it in two, adding recommendation rate as a third angle. Version 5 fixed the sampling: three runs per prompt per platform, appearance rates instead of verdicts, and a methodology note in every report. This is the build log, including the parts we would rather skip.
v1: one number, honestly computed, wrongly framed
The first version answered the question every client asks first — “how visible are we in AI?” — with a single score. It ran buyer questions across AI platforms, recorded whether the brand appeared, and averaged. As far as it went, the arithmetic was honest. The framing was not. A single score asserts that visibility is one thing, and the assertion survives every meeting because nobody can see inside the average to argue with it. A brand invisible on five platforms and dominant on one received the same class of number as a brand mediocre everywhere. The per-platform breakdowns existed in the data; the headline metric trained everyone to ignore them. Lesson one, which recurs in every version of this story: the most dangerous number in a report is the one at the top.
The field that stored nothing
The database schema was more ambitious than the code. From early on, the data model carried a citation field next to the mention data — the plan had always been to measure whether the brand's own pages were used as sources. The plan stayed a plan. Version after version, the citation field was written as null, because computing it required parsing each answer's source list against the brand's domains, and something else was always more urgent. Reports did not show a citation rate, so nothing looked missing — which is exactly the failure mode worth recording. An unmeasured metric that nobody displays is invisible; an unmeasured metric implied by the framework around it is a quiet falsehood. When we finally wired it up, the first real citation numbers explained several mysteries the mention data never could — starting with brands that were mentioned constantly and sourced never.
v4: citation becomes real — and splits in two
Version 4 computed citation rate properly: for every answer, the sources were extracted and matched against the brand's own pages. The moment the number existed, it misbehaved in an instructive way. Brands with healthy citation rates were still losing the recommendation sentence to competitors — cited as background, passed over in the verdict. Published research by Lily Ray quantified the pattern: vendors cited yet excluded from recommendations in 69% of audited cases in June 2026, and 83% by September 2026. So v4 split the measurement again: alongside the citation check, the system began scoring recommendation — whether the answer's text near the brand's mention actually puts it forward, in language like “best for” or an explicit pick. That made the measurement standard three angles — mention, citation, recommendation — crossed with two objects, the brand and each product or SKU. The three angles fail independently and are fixed independently, so they are reported independently, per platform, never blended.
v5: the sampling fix, and the fine print becomes a feature
Version 5 fixed the deepest flaw, one this blog documents as a failure in its own right: through v4, every prompt ran once. Since AI answers are regenerated each run — Kevin Indig's research found three identical runs produce consistent citations only 2.2% of the time — single-run scores were substantially noise. Full scans now run every prompt three times per platform and report appearance rates, 0 of 3 through 3 of 3, separating never from sometimes from always. The platforms measured are the six products buyers actually use — ChatGPT, Perplexity, Copilot, Gemini, Grok, and Claude — tested as products, not as raw model endpoints, because the product's browsing and source-selection layers are what decide a brand's fate. And every report carries its methodology note: the question set, the sample counts, the platforms, the dates. We treat that note as part of the product, not as legal cover. A measurement that cannot state how it was taken has not finished being taken.
Sources & method. Version history is from our own build and audit records for the tracking workspace used in this practice: citation field stored null through the early versions; real citation computation plus the recommendation split in v4 (recommendation rate recorded alongside citation rate per scan); three-sample full scans, appearance rates, and methodology notes in v5. External figures are published research: Lily Ray (Algorythmic) for cited-but-not-recommended rates (69%, June 2026; 83%, September 2026), Kevin Indig (Growth Memo) for the 2.2% three-run consistency figure.
© 2026 Jun Liu · JUN & GEO. This article is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0) — https://creativecommons.org/licenses/by-sa/4.0/. You may share and adapt it with attribution and under the same license.