The internet's gatekeeper just threw up a wall to block AI chatbots

Cybersecurity company Cloudflare announced on Tuesday it is adding a new feature that allows users to entirely block AI chatbots from crawling their websites or charge a fee to grant bots access.   

“The web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value,” Matthew Prince, co-founder and CEO of Cloudflare, said in a statement on Tuesday. 

The new feature will give users three options: they can allow AI bots full, free access to their websites; they can block all AI chatbots from their website; or they can charge a “flat, per-request price across their entire site,” according to the company’s statement. 

Users are able to bypass charges for some AI chatbots if they want. 

“This is particularly helpful if you want to allow a certain crawler through for free, or if you want to negotiate and execute a content partnership outside the pay per crawl feature,” the statement said. 

The statement refers to this new feature as a “technical shift in how content is controlled online” to help give users “control over who accesses their work.”

AI training crawl activity has increased by 65% over the past 6 months, the company said in a separate statement on Tuesday. This move could make it more difficult for developers to quickly train AI chatbots, but will let content creators make money from AI chatbot activity.

“That content is the fuel that powers AI engines, and so it's only fair that content creators are compensated directly for it,” Prince said in his statement. 

Cloudflare introduced a different feature in July of last year that let users block AI chatbots with a single click. The company said that more than one million users enabled the feature.

The company estimates that about 16% of all internet traffic around the world operates through Cloudflare, according to its 2023 Q3 Browser Market Share Report.