Media corporations have gotten fairly good at blocking AI bots. They’ve been much less good at deciding which of them needs to be allowed in. Reports say most main publications nearly universally block crawlers from main AI corporations like OpenAI and Anthropic, and lots of small to midsize publishers mirror that, configuring their robots exclusion protocol settings, or robots.txt, to maintain them out.
It’s an comprehensible stance, and it’d even be applicable. Nevertheless it additionally crudely compresses a posh concern right into a binary: Block or don’t block. And the reasoning is usually simply as simple, pointing to the truth that most AI platforms don’t give again important human site visitors. On the similar time, the variety of bot crawls may be excessive, excessive sufficient that price turns into an element. From an ROI standpoint, it’s a simple name.
There are exceptions, in fact. Google is the apparent sophisticated one, since AI is more and more constructed into Search itself, and blocking Googlebot means opting out of much more than AI answers. Google says AI Overviews and AI Mode are part of Search, and that web site homeowners management inclusion by means of the identical Search crawling and preview controls they already use. And positively, in case your publication has a licensing take care of a selected platform, you’d let its bots by means of.
{“blockType”:”mv-promo-block”,”information”:{“imageDesktopUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to u003Cemu003EThe Media Copilotu003C/emu003Eu003C/strongu003E”,”dek”:”Need extra about how AI is altering media? By no means miss an replace from Pete Pachal by signing up for u003Cemu003EThe Media Copilotu003C/emu003E. To study extra, go to u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”textual content”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}
But when direct viewers or income is the one criterion for permitting bots in, that’s too slim a window to actually reap the benefits of the chance AI presents. In the end, blocking is only one element of an AI-forward technique that understands the various kinds of bots, how they join your content material to audiences, the strategic significance of treating bots as part of the audience, and how one can current content material in AI experiences so that you keep in management.
Block the best bots
Let’s begin with the reality that completely different bots do various things. There are a number of varieties, however broadly three matter most to publishers: coaching bots, search bots, and retrieval bots. Not less than these are the kinds which can be a part of the “reliable” info ecosystem, in that they’re normally declared and documented, and customarily respect robots.txt preferences. Let’s put unauthorized scrapers apart for the second.
It’s the retrieval bots, which generally have “-user” as a part of their title, that publishers ought to pay shut consideration to and think about giving the inexperienced gentle. These are the bots that go and get info on the fly when a consumer varieties a query into an AI chatbot. Their exercise is modest and predictable, normally hitting solely a small variety of pages, and so they can usually be rate-limited or served cached content material. Cloudflare, actually, has made it simpler to tell apart between makes use of like search indexing, real-time AI enter, and AI coaching, and its AI Crawl Control tools now embody choices to permit, cost, or block particular AI crawlers.
Permitting retrieval bots, and doubtlessly a rate-limited variety of search bots, can have a significant impact on presence in AI summaries.
“So what?” is the query many publishers retort—you’ll be able to’t take AI presence to the financial institution. Which is true, but it surely misses the strategic level: In case you’re not within the reply, another person is. That cedes authority on a topic to a competitor, as a result of the extra audiences see them cited, the extra they’ll develop thoughts share, and what little referral site visitors there’s will go to them. Successful in AI means understanding that authority is the prize, not site visitors.
Present the worth, don’t give it away
Successful that authority, nevertheless, doesn’t imply profitable in any respect prices. Your most beneficial property—whether or not they be content material, neighborhood, or experiences—want to stay yours, not given freely to AI techniques. Strategic content material specifically requires a cautious stability between what you enable AI to see and what readers want to come back on to you to find. In case you intently handle metadata, snippets, entry controls, and clear directions for LLMs, you’ll be able to assist bots perceive that worthwhile materials exists with out giving it away solely.
For instance, a web site specialised within the agricultural business might need an in depth, data-rich report on the consequences of pesticides on soybean demand. A snippet would possibly describe precisely what sort of information is inside and why the information is related to sure forms of analysis whereas not revealing the information or conclusions. The report itself would possibly clarify what is obtainable, what’s restricted, and the place approved customers can entry the complete work.
Ideally, the consequence can be that the existence of the report is thought to AI engines, however anybody who wished to truly see it will want to go to the writer and interact deeper, with a subscription or at the least an e mail gate. The objective isn’t to cover solely from AI reply engines. It’s to make them conscious of the writer’s worth with out letting them reproduce that worth.
Your archive is your alpha
There’s an analogy right here with what Palantir cofounder Alex Karp stated in a widely shared CNBC interview, the place he suggested enterprise AI clients to cease utilizing fashions from the key AI labs, because it was successfully giving them their “alpha”—the corporate information that offers them an edge over opponents. That is, in a way, what media corporations have been compelled to do from the start of the AI revolution, since publishers’ information is basically public and was an enormous a part of what the fashions initially educated on.
Now the creators of these fashions are successfully competing with publishers by providing reply engines to the general public, which maintain customers on the portal as a substitute of sending them to websites. That is, in fact, the foundational concept behind the numerous lawsuits and licensing offers between the AI corporations and the media.
To offer customers the very best and most recent info, although, you’ll want to get it in the intervening time of the question, which is why the market has shifted from coaching information to retrieval. Rob Kelly has a useful analysis of this shift, exhibiting that coaching rights are not a given in publicly introduced AI licensing offers. In his database, solely about 4 in 10 public 2026 offers embody coaching rights, an indication that the market is shifting from “purchase content material to construct higher fashions” towards “license content material to ship higher solutions.”
The unhappy fact is that right now the overwhelming majority of publishers don’t have the leverage to safe their very own licensing offers with OpenAI, et al. However that doesn’t imply there aren’t methods to extract the worth of your content material in an AI market. It begins with making your content material machine-readable, a separate idea from bot blocking.
The variety of locations publishers can market their content material to AI experiences is rising, together with Factiva and Microsoft’s Writer Content material Market, which Microsoft describes as a approach to assist licensed entry to premium content material whereas preserving writer management, independence, and sustainable income. Lastly, constructing your individual agentic layer that reliable clients or companions might entry through MCP (mannequin context protocol) is probably the most forward-looking choice.
That takes some technical sophistication, usually requiring a associate, but it surely provides the writer management over the expertise. Quite than counting on some third-party bot to scrape and interpret your content material, you’ll have achieved that work already, so approved exterior bots will get what your inner bot has already processed, like a relay. That’s the entire concept behind AI media initiatives like Reuters’ new MCP server, which permits clients to go looking, retrieve, and use the Reuters content material they subscribe to inside AI workflows.
As I’ve argued before about publisher-built agents and AI-ready archives, when you’ve achieved the laborious work of formatting, ingesting, and processing your archive for AI, the writer turns into one thing extra like a device vendor, not only a content material provider. Reuters clearly acknowledges that publishers who don’t construct some form of managed retrieval layer over their archives are letting third-party crawlers set the foundations. Ultimately, that can result in negotiating from weak point.
I haven’t stated a lot concerning the unhealthy actors thus far—the unauthorized scrapers that hoover up large elements of what publishers produce after which promote that information in numerous black and grey markets. Actually, that is the place blocking needs to be inspired, and publishers want each dependable instruments for doing so in addition to broader ecosystem assist, reminiscent of what Cloudflare has achieved to higher establish bots and doubtlessly monetize their exercise. As I wrote earlier this year, unauthorized AI crawling is rampant, and publishers want greater than wishful considering and a robots.txt file to take care of it.
From protection to leverage
However blocking is simply the defensive half of the technique. The reply is to not throw the doorways open or nail them shut. It’s to construct a gate with guidelines. Let the retrieval bots see sufficient to determine your authority. Preserve the coaching crawlers and unauthorized scrapers away from something they don’t have any proper to soak up. Use snippets, metadata, paywalls, and fee limits to separate discovery from entry. After which do the tougher work: Make your archive clear, structured, machine-readable, and obtainable by means of channels you management.
Most publishers usually are not going to get the large AI licensing deal. Which is okay. That was by no means going to be a marketing strategy anyway. The higher play is to make your work legible to AI techniques with out making it free, so when the market does come searching for trusted solutions, you’re not begging to be included. You’re already the gate.
{“blockType”:”mv-promo-block”,”information”:{“imageDesktopUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to u003Cemu003EThe Media Copilotu003C/emu003Eu003C/strongu003E”,”dek”:”Need extra about how AI is altering media? By no means miss an replace from Pete Pachal by signing up for u003Cemu003EThe Media Copilotu003C/emu003E. To study extra, go to u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”textual content”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}
