<\/p>\n
This summer\u2019s plunge in news traffic was not just about Google\u2019s AI Overviews rewriting how people find information. It was a perfect storm: Google cutting clicks at the front end, and publishers, through Cloudflare\u2019s default bot blocking, cutting off the back-end pathways that help stories \u201cgo legs.\u201d It is the digital equivalent of my dad refusing mobiles for years, fear keeping you isolated just when connection matters most.
\nPublishers want to blame Google alone. The reality is harsher. Many are hurting themselves by blocking the very crawlers that carry their work into the wider web. Case studies show the cost. One tech site lost half its traffic overnight from a robots.txt error. Recipe sites vanished from search after misconfigured responses. Even e-commerce giants have crashed rankings from accidental bot blocks. Meanwhile, industry data shows publishers losing 1 to 25 percent of Google referrals in just eight weeks as AI Overviews rolled out. Together, that is a double hit: fewer front-end clicks, fewer back-end routes for discovery.
\nThe devastation is real, budgets evaporating, stories buried before they spread. And the truth is painful. Much of this damage is self-inflicted.
\nThe Numbers Do Not Lie
\nGoogle\u2019s new AI layer is swallowing clicks. Research shows a 34 percent drop in click-through when an Overview appears. Pew found just 8 percent of users who saw an AI summary clicked any link, compared with 15 percent for traditional search. Only 1 percent clicked the source links inside the summaries. Sessions are ending faster too, 26 percent terminate after an AI Overview compared with 16 percent without.
\nPress Gazette reported that MailOnline saw its click-through rates fall by as much as 56 percent when AI Overviews appeared. That single figure should terrify every publisher.
\nCloudflare tells the other half of the story. In June 2025, Claude\u2019s crawl-to-referral ratio stood at 70,900 to 1. OpenAI\u2019s was 1,700 to 1. Even Google now crawls 14 times for every referral it sends. Publishers understandably resent this imbalance. But blanket blocking does not just punish AI, it cuts off the propagation that helps stories move.
\nAnd so the question becomes: if we cannot win on clicks, can we at least get paid for the crawls themselves?
\nWill Pay-Per-Crawl Save Us?
\nCloudflare\u2019s Pay-Per-Crawl promises to turn scraping into revenue, charging bots a flat fee, sometimes as low as $0.001 per crawl. For B2B publishers, that is not transformative income. Worse, most bots can mimic human traffic and slip through undetected, while some of the crawlers being blocked may actually send useful visitors.
\nNow compare that to what publishers pay to attract traffic. Google Ads clicks cost 3 to 4 dollars on average, and often more for business and finance keywords. In other words, the industry pays Google thousands of times more to deliver a reader than it would ever earn from refusing one through Pay-Per-Crawl.
\nThat imbalance highlights the core problem. Pay-Per-Crawl might claw back pennies, but it will not restore the economics of journalism. The real value comes from readers, people who click, share, subscribe, and let stories gain influence. Blocking that discovery for fractions of a cent is a false victory.
\nJournalism\u2019s Real Mission
\nStories are not meant to sit still. A great investigation matters because it spreads, picked up, validated, expanded, inspiring other stories and wider impact. That is the press at its best. Yet many publishers are retreating into digital fortresses, cutting off the very mechanisms that give journalism influence.
\nOf course AI firms should pay fairly, and Cloudflare\u2019s Pay-Per-Crawl may be one step towards that. But shutting the door entirely is self-defeating. My dad once refused to buy a mobile, convinced it was a fad. For a decade he missed out. Now he is addicted to WhatsApp and Facebook. Publishers cannot afford the same mistake.
\nTwo Choices
\nThe instinct in hard times is either to block and shrivel, or to grow and thrive. Blocking bots may give momentary satisfaction, the machine cannot read my words, but at the cost of readership, influence, and future revenue. Growth demands a more nuanced approach:
\n \u2022 Block bad actors, but keep pathways open for legitimate discovery.
\n \u2022 Build direct audience channels while still engaging with search and social.
\n \u2022 Push for fair compensation, but do not confuse protection with proliferation.
\nThe bottom line: good journalism needs to proliferate to matter. A story locked away, whether behind a paywall or a robots.txt barricade, cannot change anything. The perfect storm of AI Overviews and bot blocking is real. But it is survivable if the industry remembers its purpose, not just to publish, but to spread.
\nAt Tomorrow\u2019s Publishers we welcome bots. We have published, you have read it, and we do not begrudge anyone taking our news and making the most of it wherever they want. This is a new age of hyper-intelligence, and we want to be at the forefront of it, not throwing the machines into the canal.<\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"
This summer\u2019s plunge in news traffic was not just about Google\u2019s AI Overviews rewriting how people find information. It was a perfect storm: Google cutting clicks at the front end, and publishers, through Cloudflare\u2019s default bot blocking, cutting off the back-end pathways that help stories \u201cgo legs.\u201d It is the digital equivalent of my dad<\/p>\n","protected":false},"author":1,"featured_media":7563,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[118],"tags":[],"class_list":{"0":"post-7562","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-publishing-news"},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/posts\/7562","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/comments?post=7562"}],"version-history":[{"count":1,"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/posts\/7562\/revisions"}],"predecessor-version":[{"id":7564,"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/posts\/7562\/revisions\/7564"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/media\/7563"}],"wp:attachment":[{"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/media?parent=7562"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/categories?post=7562"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sandbox.hbmadvisory.com\/amplify\/wp-json\/wp\/v2\/tags?post=7562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}