I Tracked 70K+ ChatGPT and AI Overview Responses to Measure Their Volatility

It's no secret that asking the same question to the same AI platform, seconds apart, can produce both different brand recommendations and cited links.

But just how volatile are the results, and what, if anything, remains consistent?

I first researched this topic in February, when I analyzed the same prompts hundreds of times. Exact matches were rare, but the most prominently named brands appeared frequently. 

I wanted to see if that held up today, checking over a longer timeline. 

The Dataset

In this new research, I've looked at over 70,000 responses, checking 1,300+ prompts daily in both ChatGPT and AI Overviews, to see how they shift over four weeks.

I find a lot of value in larger studies, including those by my colleagues at Ahrefs, but since individual brands aren't likely to track millions of prompts on their own, I think this angle has value.

Prompts were primarily commercial, based on keywords with traditional search volume, then converted into a more conversational style. They include questions like: "What are the best binoculars for stargazing?" Best Products, What's the best digital marketing agency in Sydney?" Agency, and "I'm looking for the best 3D design software" Software.

I'm not claiming these prompts are perfect. They're not.

While I've covered in detail how to come up with prompts to track, I don't have exact data here, and real-world requests often come as follow-ups in conversations, rather than always starting from a new chat.

The prompts you track should be specific to your business and what you're focused on. 

While I don't want to downplay the effort put into this research, view this as someone actively working in this space who was curious to pull some numbers for themself.

I'm always trying to learn and expand my knowledge, and sharing data like this is one way I do that. 

The top brands for a prompt show up on most days

For a typical prompt, the brand each platform names most often shows up on most days, rather than rotating in and out at random.

On ChatGPT, the leading brand appeared on 82% of days.

On Google AI Overviews, that figure is higher, at 89%.

To check this wasn't just a result of picking the winner after the fact, I also identified each prompt's leader using only the first 14 days of data. That brand still appeared on 79% (ChatGPT) and 85% (AI Overviews) of the following 14 days.

To show that visually, below, each row is a brand named on at least two days (sorted by most-present first). Each column is a day, with the oldest on the left:

ChatGPT

Sage Intacct 27

Featured brands

1. 2 18
2. 3 16
3. 4 15
4. 5 14
5. 6 14
6. 7 12
7. 8 11
8. 9 10
9. 10 10
10. 11 10
11. 12 10
12. 13 7
13. 14 7
14. 15 7
15. 16 7
16. 17 7
17. 18 6
18. 19 5
19. 20 5
20. 21 4
21. 22 4
22. 23 4

Brands named on two or more days, aligned with the other engine.

AI Overviews

Sage Intacct 21

Featured brands

1. 2 19
2. 3 15
3. 4 12
4. 5 11
5. 6 9
6. 7 7
7. 8 7
8. 9 7
9. 10 6
10. 11 5
11. 12 5
12. 13 4
13. 14 4
14. 15 3
15. 16 3
16. 17 3
17. 18 2
18. 19 2
19. 20 2
20. 21 2
21. 22 2
22. 23 2

Brands named on two or more days.

The top spot is close. The gap between the first and second brand is typically around two days out of 28, and the leading brand changed between the first and second fortnight on roughly half of prompts.

For example, for a prompt asking for advice on the best financial services software on ChatGPT, Sage Intacct appeared on 27 of the 28 days. No other brand appeared on more than 18 days.

18% of ChatGPT prompts had at least one brand named on every single day, compared to 29% of AI Overviews. 

The "core" of a response barely moves, but the tail turns over daily

For each prompt I sorted every brand named over the 28 days into two groups: core brands, named on at least 80% of days, and the tail (everything else). Then I compared each day's list with the previous day, counting both brands that dropped out and those that newly appeared.

On ChatGPT, the core brands changed by 13% from one day to the next, against 78% for the tail. On AI Overviews, they changed by 11% and 65%.

Cited domains followed the same pattern: 14% vs 74% on ChatGPT, and 12% vs 61% on AI Overviews.