<\/p>\n
Since the launch of GPT-5 in August 2025, OpenAI\u2019s web crawling activity has sharply increased, with data indicating a shift towards live retrieval for fast-moving search queries, although overall crawl volume remains modest compared to Google.<\/p>\n<\/div>\n
OpenAI\u2019s web crawling activity has risen sharply since GPT-5 arrived in August 2025, according to an analysis by Botify and guest author Chris Long, with the company\u2019s search-related bot now appearing more active than its training crawler in the firm\u2019s enterprise log data. The shift suggests that OpenAI is leaning more heavily on live retrieval for some prompts, even as its overall crawl footprint still trails the biggest search engines by a wide margin.<\/p>\n
Long, who co-founded the SEO consultancy Nectiv, examined about 7 billion OpenAI-bot log events drawn from Botify\u2019s enterprise customer base between November 2024 and March 2026. In that dataset, OAI-SearchBot, which is used when ChatGPT searches the web, increased to roughly 3.5 times its previous level after August 2025, while GPTBot, OpenAI\u2019s training crawler, rose by about 2.9 times. Botify says that before GPT-5 the two crawlers were running at near parity, but after the launch search activity moved ahead.<\/p>\n
The strongest growth was concentrated in sectors where users are more likely to want current information. Healthcare sites recorded about 740% more OAI-SearchBot activity after the launch, while media and publishing saw a 702% increase. Marketplaces, software and retail also posted substantial gains, while travel showed a much smaller rise. Long and Botify say the pattern points to a split in how prompts are handled, with news and other fast-moving queries more likely to trigger live search, and health or product queries often drawing more on trained knowledge.<\/p>\n
The dataset also showed a decline in ChatGPT-User log events, which fell 28% between December 2025 and March 2026. Because that user agent is triggered when ChatGPT fetches a page on behalf of a user, the drop may indicate fewer real-time page requests, though Botify\u2019s team also suggested OpenAI could be relying more on stored or indexed material instead. Long did not choose between those explanations.<\/p>\n
Even with the post-GPT-5 surge, OpenAI\u2019s crawl volume remains far below Google\u2019s. In Botify\u2019s latest 30-day window, Googlebot accounted for 18.2 billion events, compared with 887 million from OpenAI\u2019s crawlers combined, or about 4% of Google\u2019s volume. A year earlier, OpenAI\u2019s share was closer to 1.38%, which indicates the gap is narrowing, albeit from a very low base. Bingbot still logged far more traffic than OpenAI as well.<\/p>\n
The findings fit with other recent reports suggesting that AI search crawling and AI training crawling are diverging. Earlier analyses from Alli AI, Hostinger and Akamai pointed to different patterns in how OpenAI\u2019s bots are behaving across the web, while CNBC reported in August 2025 that GPT-5\u2019s release boosted enterprise use cases and helped accelerate adoption in developer tools. Taken together, the reports suggest website owners may need to treat search-oriented AI crawlers as a distinct class of visitor rather than assuming that blocking training bots alone is enough.<\/p>\n
Inspired by headline at:<\/strong> [1]<\/a><\/sup><\/p>\n