Stopping the Bad Bot Blitz: The Fight for News Transparency
Automated bots account for more than half of all internet traffic, outnumbering human users online.
The Rising Threat of Stealth Crawlers
Internet traffic has officially crossed a tipping point. Researchers note that automated bots now generate the majority of global web traffic, and a significant portion of those programs consist of bad or stealth bots. According to definitions highlighted by media organizations, a bad bot is a crawler that intentionally conceals or falsifies its identity or purpose, bypasses access controls, ignores rights reservations, or actively impersonates human traffic.
For modern newsrooms, this invisible flood creates devastating operational friction. Uninvited crawlers hit publisher websites millions of times in a single day. This relentless automated activity slows page loads for human readers, triggers expensive bandwidth overage charges, and occasionally causes total outages. In August 2025, a UK technology site was forced completely offline after being scraped 1.6 million times within a 24-hour window. The infrastructure strain has since escalated globally; the Wikimedia Foundation blocks or throttles roughly a quarter of all automated requests hitting its servers today, amounting to billions of blocked visits daily from crawlers that ignore established access policies.
Independent local news providers, operating on razor-thin margins, find themselves unable to absorb these mounting technical costs.
The Billion-Dollar Reseller Market and Content Theft
The economic damage extends far beyond server bandwidth. Malevolent actors deploy stealth bots to harvest original journalism—including deep behind paid subscriber paywalls—and funnel it into what media analysts describe as a billion-dollar reseller market. In this shadow economy, artificial intelligence developers purchase scraped data directly from bot operators rather than licensing content legitimately from journalists and publishers. Those stolen assets then power AI products that compete directly with the original reporting sources.
A.G. Sulzberger captured the severity of the crisis, stating that this theft happens not because publishers leave their content exposed, but when it is safely locked inside the house. News Corp Chief Executive Robert Thomson has similarly characterized stealth bots as the silent scavengers of the internet, while Condé Nast CEO Roger Lynch warned that disguised bots scrape original reporting with zero accountability.
Publishers have attempted to deploy sophisticated blockers independently, but stopping malicious actors who actively disguise themselves as ordinary humans remains nearly impossible without legislative backing.
Legislative Convergence on Both Sides of the Atlantic
Recognizing that individual publishers cannot win this arms race alone, governments are beginning to step in. In the United States, a bipartisan proposal named the Stealth Bot Prohibition Act establishes a straightforward principle: automated crawlers must explicitly state who they are and why they are visiting a website. The proposal requires AI bad bots to disclose their identity and purpose, granting content creators the visibility needed to defend their intellectual property.
Simultaneously, the United Kingdom is advancing on a parallel track via the Automated Online Software (Access and Transparency) Bill. Introduced as a Private Member’s Bill by MP Damian Hinds and backed by the News Media Association, the legislation requires any entity running a bot that systematically copies UK website content to disclose operator identities, corporate ownership, and the intended use of the harvested data.
This independent convergence across two major Western legislatures signals that bot transparency has evolved from a niche technical grievance into an absolute prerequisite for a functioning digital economy.
The Foundation for a Viable Licensing Market
Transparency acts as the mandatory first step toward any sustainable licensing framework. Publishers cannot negotiate commercial agreements with entities they cannot identify, nor can they protect copyrighted assets when they cannot see who is extracting them. Across Europe, the EU Copyright in the Digital Single Market Directive grants rightsholders the explicit legal right to reserve their content from text and data mining in a machine-readable format. As of 2 August 2026, AI developers face severe penalties of up to €15 million or 3% of global annual turnover for ignoring these machine-readable reservations.
However, legal rights remain hollow if automated scrapers are permitted to lie about their identities at the digital door. An opt-out mechanism only functions if the reading crawler honestly identifies itself, making enforcement directly dependent on separating truthful scrapers from deceptive ones.
The continuous strip-mining of original journalism threatens the financial models that sustain public-interest reporting. Establishing mandatory disclosure rules and enforceable penalties for deception offers a clear path toward a balanced digital marketplace that fairly rewards publishers, technology developers, and readers alike.