This week's biggest news in search visibility did not come from Google. A piece of research published a few days ago, built on months of analyzing ChatGPT's internal data, showed that OpenAI is running its own search index: a crawler that explores the web on its own, a cache that stores copies of pages, and a family of specialized indexes, all aimed at depending less and less on Google and Bing. OpenAI has not confirmed it publicly, but the technical evidence is hard to read any other way. And if your AI visibility plan still boils down to "just rank well on Bing", you will want to keep reading.
What exactly was discovered?
A team of European analysts spent May through July watching the data ChatGPT sends to its own servers while it answers questions. That is where an internal retrieval system nicknamed Labrador showed up, along with traces of a fairly serious indexing operation: the reports circulating this week describe an index in preview phase holding around 3.2 billion pages.
The context helps explain why it exists. During the antitrust trial against Google, it came out that OpenAI asked for access to Google's search index and Google said no. Without that deal, OpenAI had two options: keep renting third-party results forever, or build an index of its own. It picked the second one, and judging by the data, it started building quite a while ago.
How does it work under the hood?
The first thing that stands out is that Labrador does not operate as one giant index. The research identified specialized indexes for the general web, news (with separate layers by recency), PDFs, YouTube videos, academic papers, local business listings, finance, legal and medical topics, shopping, and images. If that architecture sounds familiar, it is because it is the same one Google has spent two decades polishing.
The second thing is the cache. For every page, the system stores the full content, the date its crawler visited it, and the publication date. When ChatGPT answers with search turned on, it is often reading that stored copy, because fetching every page live is too slow at its scale. It works like a librarian who photocopied your book the last time he walked through your neighborhood: if you updated the book yesterday and he came by three weeks ago, what everyone reads is the three-week-old version.
And the third thing is that the system is hybrid. ChatGPT combines its own index with results from Google, from Bing, and from a list of external data providers (including Yelp and TripAdvisor for local businesses), and decides which source to use depending on the query. It even runs experiments with names as subtle as prefer-index-over-serp-v3, which translates to "prefer my index over search engine results". All of this feeds the fan-out searches we already know about, which, as we saw in the study on AI's favorite brands, tend to favor the brands the model already knows.
Why does it matter, if ChatGPT still uses Google and Bing?
Because of the direction of travel. According to the research, the internal goal is to answer close to 80% of queries without relying on external search engines. Crawl data points the same way: an analysis of more than 7 billion log records by Botify showed that OpenAI's crawling tripled since the launch of GPT-5. The search bot grew the most, 3.5 times. It is still a small operation next to Google, around 4% of its crawl volume. Then again, a year ago it was just over 1%.
For your strategy, the point is this. Until now, the recipe for showing up in ChatGPT was largely inherited: rank on Bing, be quotable, keep your brand profile tidy. I have written before about optimizing for LLMs like ChatGPT, and all of that still applies. What is new is an additional indexing surface, with its own crawler, its own rules, and its own cache. Ranking well on Bing no longer guarantees you are well represented in ChatGPT's index, and the reverse is also true.
OpenAI's three bots (and why mixing them up is costly)
OpenAI operates three bots with different jobs. GPTBot collects content to train models. OAI-SearchBot is the one building the search index. ChatGPT-User is the live visit that happens when a user asks ChatGPT to open a specific page.
The distinction matters for a very practical reason. Between 2023 and 2024, thousands of sites blocked GPTBot in their robots.txt to avoid giving away training content, and quite a few blocked anything that smelled like OpenAI along the way. That inherited block can now mean being left out of the index ChatGPT uses to answer searches, which according to industry measurements concentrated 56.7% of traffic from AI assistants in August. Blocking training and blocking search are two separate decisions, and they deserve to be reviewed separately.
What should you check on your site this week?
None of this requires budget, just an hour or two from someone on the team.
1. Open your robots.txt and look for inherited blocks. Make sure OAI-SearchBot has access, even if you decide to keep GPTBot blocked. If at some point someone copied a list of "AI bots to block" from a 2023 article, this is a good moment to review what ended up in there.
2. Look for OAI-SearchBot in your logs. In your server logs or your CDN panel, filter by user agent. Three useful questions: does it show up, how often does it come back, and which pages does it visit. A site OAI-SearchBot has not visited in a month is a site that index has outdated (or missing).
3. Run the cache test on your important pages. Ask ChatGPT, with search turned on, about your brand, your products, or your prices. If it answers with information you already changed, you are looking at the stored copy and now you know how old it is. A simple routine works well here: pick your ten most important queries, run them monthly, and log what ChatGPT gets wrong.
4. Get your dates in order. Visible publication and update dates on every piece of content, and a sitemap lastmod that tells the truth. ChatGPT's cache stores each page's publication date, and its news indexes are organized by freshness, so dates carry quite a bit more weight than we are used to.
5. Strengthen your external sources. The system fills in its answers with third-party data, like Yelp or TripAdvisor. Depending on your market, that translates into having your Google Business Profile, your industry directories, and your press mentions in good shape, the same brand-profile logic we covered when talking about structured data.
What comes next
My read is that this is the most important infrastructure news of the year for AI visibility, precisely because it is pure plumbing, the kind that never gets announced in a keynote. Once the index leaves its preview phase, the rules for who shows up in ChatGPT will depend less on Bing and more on this in-house pipeline, and the sites that are already well crawled, with a fresh cache and tidy dates, will start ahead.
If you are already measuring your AI impressions with the Search Console AI report, add the five checks above. It is a couple of hours of work, and it leaves you ready for a scenario where your organic visibility plays out across more than one index at a time.