Our WorkAll ServicesAI VisibilitySEO AutomationProduct ManagementClaude AI ServicesZatrovo BookingStart a Project
ai-search · 10 min read · 2 November 2026

AI Training Opt-Out in robots.txt: Should UK Firms Block It

Google and Bing back a shared robots.txt AI training opt-out from 2027. What UK small businesses gain or lose by blocking it, plus a checklist for this week.

Jacob Horgan, Founder, Irvale Studio
Jacob Horgan
Founder, Irvale Studio
UK small business owner checking website settings on a laptop in a high street office.

On 30 October 2026, Google said it will support a shared robots.txt rule that lets websites opt out of AI training across several providers in one line, aiming to switch it on in 2027. Microsoft Bing announced the same day that it will start honouring new training preferences from 1 February 2027. For a UK small business, the real question is not how to write the rule. It is whether blocking AI training helps you or quietly costs you visibility in AI answers.

This post explains what changed, what the rule does and does not touch, and how to make the call for your own site, with a checklist you can finish this week.

What did Google and Bing announce on 30 October 2026?

Google and Microsoft both backed a shared AI training preference built at the IETF, written in robots.txt as user-agent: __usage_ai-training. Google aims to support it in 2027. Bing starts honouring new training and grounding preferences on 1 February 2027.

In a Google Search Central blog post by John Mueller dated 30 October 2026, Google said it plans to support a shared AI training control based on work from the IETF's AIPREF working group. Instead of writing separate rules for each AI developer, a site owner sets one broad preference that participating providers read. Google wrote: "We aim to support this new control along with other AI providers in 2027."

Google's common crawlers documentation names the signal as user-agent: __usage_ai-training and says Google intends to support it "in early 2027."

On the same day, Microsoft Bing published its own announcement, written by Krishna Madhavan, Principal Product Manager. Bing gives a working example: User-Agent: __usage_ai-training followed by Disallow: / to block training across a whole site, or Disallow: /example-path/ for one section.

30 Oct 2026Google and Bing announce support for a shared AI training preference
Source: Google Search Central blog; Microsoft Bing blog
1 Feb 2027Bing begins honouring new training and grounding preferences as pages are crawled
Source: Microsoft Bing blog
1 Mar 2027Bing stops reading NoArchive and NoCache as AI preferences
Source: Microsoft Bing blog
3%UK consultation respondents who supported an opt-out copyright exception for AI
Source: gov.uk Report on Copyright and AI, March 2026

What is the difference between AI training and AI grounding?

Training is when your pages help build a future AI model. Grounding is when an AI assistant fetches your page live to answer a question and, ideally, links to you. Training brings no traffic. Grounding is where citations and clicks come from.

This distinction drives every decision in this post, so it is worth being precise.

Training happens in the background. Your text becomes part of the data a model learns from. Nobody is sent to your site as a result, and you are not paid.

Grounding happens at the moment a customer asks a question. The assistant looks up current pages, uses them to answer, and may show a link. If someone asks an AI assistant for "an emergency electrician near Sheffield open on Sundays", grounding is how your site gets named. That is the part that behaves like search marketing.

Bing now treats these as two separate settings. Its blog describes training as a preference you can set in robots.txt or per page, and grounding as a page-level preference with three values: Allow (the default), DisallowExtended, which limits use to Microsoft's own experiences, and Disallow. Bing states plainly that choosing either restrictive grounding option "will reduce brand visibility in AI experiences" and adds that "Neither choice guarantees inclusion, traffic, or payment."

Does blocking AI training hurt your Google rankings?

No. Google's documentation says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Bing says its new preferences do not change eligibility for traditional Bing results. Your organic rankings are safe either way.

Google's crawler documentation is direct on this: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

The documentation does not mention AI Overviews in that section, so do not assume either way that this rule changes how your pages appear in AI Overviews. What it does say is the next point, and it matters more.

Why does the Google-Extended rule matter more than it looks?

On Google, training and Gemini Apps grounding share one switch. Google-Extended covers both, and Google will read a shared training opt-out as a Google-Extended block unless you say otherwise. Blocking training on Google can therefore cost you Gemini citations too.

Google's documentation says Google-Extended governs whether crawled content can be used to train future Gemini models. It also covers grounding in Gemini Apps, where Search index content is supplied to Gemini at the moment someone asks a question.

Now add the line from Google's 30 October post: if a site uses the shared token to opt out of AI training and does not explicitly address Google-Extended, Google plans to treat it as if Google-Extended were disallowed. The same post says sites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks via grounding in Gemini Apps.

Put those together and the trap is clear. An owner who adds the shared training block, thinking it only affects model training, may also be stepping out of Gemini Apps answers. On Bing you can split the two. On Google, in practice, it is one decision.

Which UK small businesses should block AI training, and which should not?

If your website exists to win enquiries, leave AI training allowed. If your content is the thing you sell, an opt-out is worth considering, with grounding judged separately. Most UK service businesses fall into the first group.

Use this if/then checklist. Pick the first line that describes you.

  • If you sell services locally (trades, salons, clinics, solicitors, accountants, estate agents): leave training allowed. Your pages hold facts you want repeated, and there is no payment or ranking reward for blocking.
  • If you sell content (paid recipes, online courses, licensed photography, research reports, premium guides): consider blocking training, but decide on grounding first. On Bing you can keep grounding. On Google, an explicit Google-Extended allow keeps Gemini Apps grounding but also keeps training.
  • If you run ecommerce: leave training allowed on product and category pages. If you publish original buying guides that cost real money to produce, consider a path-level block on that folder only, using Bing's Disallow: /example-path/ pattern.
  • If you already use a "block all AI bots" template: audit it now (see below). It may be blocking grounding crawlers you want.
  • If your site uses NoArchive or NoCache tags: note Bing's 1 March 2027 cut-off, after which those tags stop counting as AI preferences.
  • If you are unsure: change nothing yet. No new rule takes effect before 2027, and an uninformed block is harder to spot later than an informed delay.

For background on how AI answers differ from classic rankings, see this guide to AEO, SEO, GEO and LLMO for UK small businesses.

What does the decision look like in pounds?

For a service business, blocking training saves nothing measurable, while losing even one AI-sourced job a month can cost thousands of pounds a year. For a content business, the sums can point the other way.

These figures are illustrative, to show the method. Swap in your own numbers.

Example 1: a bathroom fitter in Bristol. Suppose AI assistants send 3 enquiries a month, the fitter wins 1 in 3, and the average job is £6,000 at a 25% margin. That is one job a month, or £1,500 of profit. Over a year, £18,000 of profit depends on staying visible in AI answers. If a blanket training block on Google also removed Gemini Apps grounding and halved that flow, the cost would be around £9,000 a year. The benefit of blocking training is £0, because nobody pays for it.

Example 2: an online baking school in the Cotswolds. Suppose it sells £45 recipe courses and makes 80 sales a month, £3,600 in revenue. If people increasingly get its method from an AI answer instead of buying, even a 10% drop is £360 a month, or £4,320 a year. Here, blocking training on Bing while keeping grounding, and making a deliberate choice about Google-Extended, is a reasonable trade.

The method is the same for both: estimate the revenue that comes from AI visibility, estimate the revenue your content generates directly, and protect whichever is larger.

UK law has not changed. The government's March 2026 report dropped a broad opt-out exception as its preferred route, and the existing text and data mining exception covers non-commercial research only. robots.txt preferences are a technical signal, not a UK legal regime.

The UK government's Report on Copyright and Artificial Intelligence, published on 18 March 2026, states that "a broad copyright exception with opt-out is no longer the government's preferred way forward." The report says only 3% of consultation respondents supported that option.

The existing UK exception for text and data mining, section 29A of the Copyright, Designs and Patents Act 1988, applies to "non-commercial research only," according to the report. The government also said it does not plan to intervene in the licensing market "at this stage" and will gather further evidence.

The practical meaning for an owner: the new robots.txt token is a voluntary industry signal. It does not create a UK right to payment, and it is not what decides whether a UK use is lawful. Treat it as a marketing and content-control setting, not legal protection.

What are the risks of copying a "block all AI" robots.txt template?

Generic templates often list every AI crawler they can find, including ones that fetch pages for live answers. Copying one can remove you from AI citations without improving anything. Many small business sites carry these rules without the owner knowing.

These templates spread through plugin settings, developer starter kits and forum posts. The shared token makes the problem easier to create, because one line now reaches several providers at once.

Three risks stand out. First, you may block grounding or citation crawlers along with training ones, which cuts you out of answers customers are already using. Second, on Google, a shared training block without an explicit Google-Extended line will be treated as a Google-Extended block, per Google's post. Third, nobody notices for months, because organic rankings stay the same and the loss only shows up as fewer AI-referred enquiries, which are already hard to track.

If you want a wider view of how your business shows up across AI assistants, the AI visibility service page sets out what can be measured.

What should you do this week?

Check your robots.txt and page tags, decide which group your business is in, write the decision down with a date, and set a reminder for February 2027. It takes under an hour.
  1. Open yoursite.co.uk/robots.txt in a browser. List every user-agent line. Look for Google-Extended, GPTBot, OAI-SearchBot, ClaudeBot, CCBot, __usage_ai-training or any other AI-sounding name, and note whether each is allowed or disallowed. Leave Googlebot and Bingbot alone: they power normal search results.
  2. Ask whoever built the site whether any plugin or template added AI blocking rules. Get the answer in writing.
  3. Check for NoArchive or NoCache tags in your page source (search for "noarchive"). If present, note Bing's 1 March 2027 cut-off.
  4. Pick your line from the if/then checklist above. Service businesses: allow training. Content businesses: decide grounding first, then training.
  5. Run your own version of the pounds example. Ten minutes with last year's enquiry data is enough.
  6. Write the decision in one sentence, with today's date, in your business notes. For example: "Allow AI training and grounding on all pages, reviewed 2 November 2026."
  7. Set a calendar reminder for 1 February 2027, when Bing begins honouring the new preferences, to confirm nothing changed without your knowledge.
Next stepMake AI visibility part of a measured revenue systemHow AI citations, enquiry tracking and website content fit into one system for UK small businesses.

The short version: the new control is useful for businesses whose content is the product, and mostly a risk for businesses whose website exists to win work. Decide on purpose, write it down, and check again when the rules go live in 2027.

Common Questions

AI Training Opt — FAQ

What is the new shared AI training control?

It is a single robots.txt rule, written as user-agent: __usage_ai-training, that tells participating AI providers whether your site's content can be used to train their models. It comes from work at the IETF AIPREF working group. Google announced on 30 October 2026 that it aims to support it in 2027, and Microsoft Bing announced the same day that it will begin honouring new training preferences from 1 February 2027. The point is convenience: one line instead of a separate rule for each AI company. Company-specific rules, such as Google-Extended, keep working alongside it, so you can set a broad default and then make exceptions for individual providers.

Will blocking AI training stop my business appearing in Google Search?

No. Google's crawler documentation says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Bing's announcement says the same for traditional Bing results. The catch is elsewhere. Google's documentation says Google-Extended also covers grounding in Gemini Apps, which is how Gemini pulls live web content into answers and links to sources. Google's blog says that if you use the shared token to opt out of training and do not mention Google-Extended, Google will treat that as a Google-Extended block. That could reduce your chances of being cited in Gemini Apps, even though your normal rankings are untouched.

Should a local trades or service business block AI training?

For most UK service businesses, such as plumbers, salons, clinics, accountants and estate agents, the answer is no. Your website exists to win enquiries, and AI assistants are becoming another place customers ask for recommendations. Blocking training brings no payment and no ranking benefit, and on Google it can come bundled with reduced Gemini Apps grounding. The content on a typical service site, such as opening hours, service areas and prices, is information you want repeated accurately. Keep training allowed, make sure the facts on the site are correct, and review the decision if your business model changes.

Who should consider opting out of AI training?

Businesses whose content is the product. That includes paid recipe sites, online course providers, publishers of original research or guides, photographers who license images and anyone selling written expertise. If an AI model can reproduce the value of your pages without sending visitors, training is a real cost. Even then, think about grounding separately. On Bing you can block training and still allow grounding, because they are separate settings. On Google, Google-Extended covers both training and Gemini Apps grounding, so blocking one tends to mean losing the other. Decide which matters more before changing anything.

Do I need to do anything before 2027?

Nothing is mandatory yet. Google says it aims to support the shared control in 2027, and Bing starts honouring the new preferences from 1 February 2027. The one firm deadline that may affect you is Bing's: if your site uses NoArchive or NoCache tags, Bing says those will no longer be read as a training or grounding preference after 1 March 2027. Use the next few weeks to check your robots.txt, find out whether a developer or plugin added AI blocking rules years ago, and write down a deliberate decision. That way you change settings once, on purpose, instead of reacting later.

Next stepGet this run for youWe run Claude AI, websites, booking and SEO for UK small businesses. From £495 a month.
Start a Project