AI scraping now makes up 70.9% of malicious bot traffic analysed by DataDome, up 185.2% over 12 months. Publishers are no longer treating this as a simple blocklist problem—they are lobbying for disclosure rules on both sides of the Atlantic.
Crawlers are getting harder to catch
At a media strategies event in London, People Inc. Chief Innovation Officer Jon Roberts said the publisher lets through just 38 bots a day while blocking “tens of millions” of AI crawler requests. The crawlers keep changing tactics: some name themselves, some hide, some imitate Google, and some route through home internet connections to avoid detection.
The Financial Times is also working to separate harmful bots from useful ones. Fiona Spooner, managing director of customer revenue, said one account alone tried to access about a million articles in eight hours from roughly 700,000 IP addresses. Reach CEO Piers North said the company recorded more than 100 million successful scrapes of its content in the past six months.
Impersonation makes voluntary signals weak
Verification data shows how routinely crawlers misrepresent themselves. Centinel found that one in seven requests claiming to be Googlebot failed verification on a typical site. Fake claims were even higher for AI crawlers: 84% of GPTBot requests and 99% of Perplexity-User requests were not genuine. A separate Piano analysis of Cloudflare data put unverified bot traffic at 70%.
That is why publishers say robots.txt is not enough. People Inc. switched to a default-block approach in January after months of chasing crawlers one by one. “Robots.txt does nothing,” Roberts said. The blocking has to happen at the CDN level rather than through a list of names.
A transatlantic push for disclosure
The News/Media Alliance organized more than 300 publishing executives to lobby Congress in September for the Stealth Bot Prohibition Act, which would require stealth crawlers to identify themselves and their purpose. In October, U.K. and U.S. publishers including The Times, Financial Times and Condé Nast urged the U.K. government to back similar legislation.
Danielle Coffey, CEO of the News/Media Alliance, said the British bill is modeled on the U.S. one and that industry groups are working closely with U.K. counterparts. The goal is greater transparency and control over how AI companies and third-party scrapers access and use publisher content—and more leverage in AI licensing deals.
Why this matters for media and marketing teams
The fight is about value as much as traffic. Roberts pointed to Exa, which he said raised $85 million at a $900 million valuation last September and followed with a $250 million Series C at a $2.2 billion valuation this May. That investor appetite shows strong demand for content, but the original publishers often get little return.
For publishers and brands that rely on original content, the sequence is:
- Block by default at the CDN layer rather than chasing individual crawler names.
- Treat crawler identity and purpose as a business requirement.
- Support disclosure rules if you want leverage over AI content licensing.
Legal experts expect the pressure to grow. Aaron Rubin of Gunderson Dettmer said content rightsholders want a stronger framework than voluntary measures such as robots.txt directives, with GDPR-style parallel movement possible across the Atlantic.
Source: Digiday
