Prompt set monitoring: the same questions each month, a stable trend and logged model updates

Q1Core and rotating questions

Split the set into a fixed core, roughly three quarters, which never changes, and a rotating part for new products, seasons or emerging questions. Trends are read from the core; the rotating part explores.

Q2Rules for changing a question

• Never edit a core question in place; retire it and add a new one with a new ID.
• Mark the month of every change so charts show a break, not a false jump.
• Keep wording safe for work and free of brand names, as in the original set.

Q3Log what the assistants change

Assistants update models and search features often, sometimes without much notice. Note the date of any announced change and any visible shift in answer format. When visibility moves sharply, the log is the first place to look before crediting or blaming your own work.

Q4A simple record format

Q5Read the trend, not the month

Judge direction over three or more months, within the margins, and only on core questions. One good month is a data point; three in a row is a pattern. The method behind each run is in our report on measuring AI visibility, and ongoing monitoring is part of every plan on pricing.

Q6A monitoring sheet

Q7When to raise an alert

A sensible rule: raise an alert when a share moves beyond its margin in two consecutive rounds, or when a new factual error about your brand appears in several answers. One-off swings within the margin are noise. An alert should trigger a quick investigation: did the assistant change, did a source change, or did something in your own business change?

Q8Keeping costs under control

Monitoring does not need full audits every month. A smaller core set, run monthly on the assistants that matter most, with full rounds each quarter, keeps costs reasonable while catching sudden shifts. Automate the running and storage where providers' terms allow, but keep the coding reviewed by people, at least for a sample.

Q9An annual review

Once a year, review the whole set: retire questions buyers no longer ask, add new ones from support and search data, check the rival list and update the weights if your audience's assistant use has changed. Record every change, and re-run the old and new sets side by side for one round, so the trend can continue across the change.

Q10Monitoring adult topics

Assistants' handling of adult topics changes with their policies. OpenAI's adult mode, announced in 2025 and delayed in March 2026, is one example of a change that could alter answers quickly if it launches. Track refusal rates as a separate series; a sudden drop or rise often signals a policy change before any announcement. The basics of the method are in our guide to measuring AI visibility.

Q11Monthly or quarterly

Q12Rotating questions, done properly

About a quarter of the set can rotate, letting you explore new topics without disturbing the core. Mark rotating questions clearly, report them separately, and promote one into the core only after several rounds show it matters. Never quietly swap a core question; the trend line breaks without anyone noticing.

Q13Who owns monitoring

Give one named person clear responsibility for running the set, logging changes and raising alerts. Without an owner, monitoring drifts: questions are edited, runs are skipped, and the trend becomes unreliable. The owner does not need to code every answer, but should check a sample each round and sign off the report each round.

Q14Monitoring checklist

• Core set unchanged since the last round, or changes logged.
• Runs completed on every assistant, from the agreed locations.
• Assistant changes noted with dates.
• Shares reported with margins; alerts checked against the two-round rule.
• Refusal rates tracked as their own series.

Share calculations are in the share of voice guide, and source tracking in citation source analysis.

Q15Interpreting a sudden change

When a share jumps or drops beyond its margin, check three things in order: did the assistant announce a change, did a major source in your category change, and did anything change in your own business, such as a product launch or a site update? Most sudden shifts trace to one of these. Note the cause in the log, so the next person reading the trend understands it.

Q16Monitoring several markets

Brands in several countries should keep a separate core set for each market and language, run from the right locations. Results often differ sharply between markets. A single combined monitor hides the markets where a brand is slipping. Keep the method the same across markets so comparisons are fair.

Q17Monitoring tools versus analysts

Automated tools can run questions and count mentions at scale; analysts check coding, spot errors and explain what changed. The best monitoring uses both: tools for collection, people for judgment. Whatever the mix, keep the raw answers available, so every figure can be checked.

Q18In short

Fixed core questions, a few clearly marked rotating ones, a log of assistant changes, shares with margins, alerts only for changes that last two rounds, and an annual review of the whole set. Monitoring done this way turns noisy answers into a trend you can actually act on.

Q19A first-year monitoring plan

1. Month one: pilot, then baseline on the full set.
2. Months two to three: monthly core runs; first alerts tested.
3. Month four: first full quarterly round compared with the baseline.
4. Months five to eleven: monthly core runs, quarterly full rounds.
5. Month twelve: annual review of questions, rivals and weights.

The measurement behind each step is described in measuring AI visibility, and rivals in AI competitor analysis.

Q20Keeping the archive

Keep every round's answers, coding sheet and report in one archive, labelled clearly by date and set version. When a question arises months later, the archive answers it quickly, and it lets a new team member pick up the series without guesswork or lost history.

FAQQuestions