What actually happened
On October 30, OpenAI, Google and Microsoft supported a common robots.txt preference for the use of website content in generative AI training. The proposal builds on work from the IETF AI Preferences group. It aims to reduce the need to maintain a separate rule for every participating AI provider.
This is a planned control, not a switch every crawler already honors. OpenAI expects implementation in the coming months; Google plans support in 2027. Existing provider-specific robots.txt controls are staying in place.
Microsoft's announcement goes further. It describes separate preferences for AI Training, AI Grounding and Limited Grounding. Its new preferences begin taking effect as pages are crawled from February 1, 2027. Sites currently using Microsoft's older NoArchive or NoCache restrictions face a March 1, 2027 transition deadline to preserve their chosen treatment.
Why a small team cares
Picture a specialist WordPress publisher selling research reports and publishing selected free guides. The owner may be comfortable having a guide cited in an AI answer because it brings qualified prospects, but unwilling to allow all original work to train future general-purpose models. Those are different commercial decisions.
Grounding means content is used to inform a particular answer, often with a link or citation. Blocking it can mean fewer opportunities for that content to appear in AI answers. Traditional Bing search eligibility is separate under Microsoft's announced controls, but reduced AI visibility remains a real trade-off.
The separation is not identical across vendors. Google says its existing Google-Extended control affects both Gemini model training and certain Gemini grounding uses. A general claim that training opt-outs never affect answer visibility would therefore be wrong.
Hype vs useful
Shared robots.txt preferences are a meaningful step toward simpler policy management, but they are voluntary and implementation dates differ. No publisher should assume that adding a proposed token today instantly creates enforceable, universal protection.
Microsoft also has transitional details that matter only for affected sites. Its temporary grounding preference in Bing Webmaster Tools is intended for sites relying on NoArchive or NoCache, and the migration has a deadline. A website without those older controls does not need to invent migration work.
For now, the useful choice is a documented content policy: distinguish permission to train from permission to ground answers, understand which existing vendor rule controls which use, and avoid blocking valuable citations without evidence of a commercial benefit.
