10 questions to ask before you buy AI visibility measurement

Question 1

Can each metric be broken down into its numerator and denominator?

Accepted Answer

A composite score nobody can explain cannot explain its own rises and falls either. With the numerator and denominator written out, you know what question the number answers, and you can check the arithmetic yourself. How ask10ais does it: No composite score. Mention rate, average position and model coverage are reported separately, with each metric's numerator and denominator on the methodology page; the table view of the sample report gives the numerator, denominator and interval for every rate.

Question 2

Do the numbers come with error margins? How large is the sample?

Accepted Answer

Model answers are partly random: ask the same question twice and the list can change. Without a sample size and an interval, a percentage cannot tell you whether it is a stable finding or the luck of one draw. How ask10ais does it: In a monitoring project each model is asked 35 times per scenario and language. Mention rates carry 95% Wilson intervals; failed calls are left out of the denominator and the report states the actual valid count. The public ranking index asks each model once per question and gives no interval, as the methodology page says.

Question 3

Is a change between two periods tested for significance, or just shown as an arrow?

Accepted Answer

A few percentage points between two periods can be sampling noise. Draw an arrow without a test and noise gets read as change — and budget and content decisions follow it. How ask10ais does it: Changes in mention rate are judged with a two-proportion test at the 95% level, with one of three verdicts: significant rise, no significant change, significant fall. If the prompts, the panel or the method changed, there is no verdict, only a note on why.

Question 4

What happens when a model is updated? Can past numbers be rewritten?

Accepted Answer

Model makers update their models, and measurement providers swap models and change formulas. Unmarked, the numbers before and after join into one line and it looks as if the brand changed when the ruler did. Rewrite old numbers, and past reports can no longer be checked. How ask10ais does it: Every model swap, formula change and normaliser change is recorded in the panel-changes table on the methodology page, which is append-only. Trend charts in monitoring reports mark each change with a vertical line, and the two sides are not compared directly. Published records are not altered; errors go into the corrections log.

Question 5

Does it measure what models remember, or what they answer with web search? Are the two reported separately?

Accepted Answer

Without web access a model answers from its training data, which barely moves between releases. With web access it reads pages first, and its answers move with whatever was just published. They measure different things; pooled into one number, nobody can say where a change came from. How ask10ais does it: Public rankings and certificates use only the model-only panel. The web-enabled panel is used only for client projects, in its own table — never subtracted from the other, never merged into one score.

Question 6

Can you see the raw answers, one by one?

Accepted Answer

Summary numbers are a reading of the raw answers, and the reading can go wrong: brand names merged wrongly, parsing that missed something, answers cut off. Only the originals let you judge whether a number holds up. How ask10ais does it: Every raw response behind a public ranking or certificate can be read on the verification page, without logging in. Every number in a monitoring report comes with that period's raw model answers.

Question 7

Who sets the prompts, and are they frozen before the run?

Accepted Answer

Change the wording slightly and the list can change. If prompts can still be adjusted after the results are in, the numbers can be picked; frozen and versioned before the run, two periods are measured with the same ruler. How ask10ais does it: In client projects the client sets the questions and scenarios, which are frozen before the run and archived for checking. Public ranking questions are written by us, published verbatim on every ranking and verification page, and never reworded for a particular model.

Question 8

Which models are used, and why these? Can the brand remove models or change weights?

Accepted Answer

Which models are asked, and how much each one counts, decides the result. If the paying party can drop models that are unkind to it or raise the weight of kind ones, the report shows only what the payer wants to see. How ask10ais does it: The panel is fixed at 7 head seats and 5 long-tail seats. Selection rules and weight sources are published on the methodology page and read from the configuration in use. Brands choose the questions and scenarios but cannot remove models or change weights; such requests go into the refusal log.

Question 9

Does the provider also sell you services to improve the numbers, or take money from those who do?

Accepted Answer

When the party paid to push a number up is also the party reporting it, interests collide. Fees tied to results, or referral fees, do the same: the measurement then moves the measurer's income. How ask10ais does it: Measurement only — no content placement or any service aimed at getting models to recommend a brand more. Prices are fixed and nothing is charged on results; we neither pay nor accept referral fees; reports are never white-labelled.

Question 10

In multilingual markets, is each language's prompt set written independently, or translated?

Accepted Answer

A translated prompt is often not what local buyers actually say, so what gets measured is the answer to translationese. Merge the languages into one number, and a change in one of them hides behind the others. How ask10ais does it: In monitoring projects the prompts for each language are written independently rather than translated, and each language is computed separately, never merged. The sample report shows the Chinese and English prompts side by side.