Silence Is Not a No. Not Even in robots.txt

Quick question: when did you last open your website's robots.txt?

For most of us the honest answer is "when Google complained about something". That tiny file is about to decide whether AI companies can train on your work. And if you write nothing in it, you have said nothing at all.

Four companies, one switch

On October 30, OpenAI said it will honor a shared signal in robots.txt that lets websites state whether their content may be used for AI training. According to The Verge, Google, Microsoft and Anthropic say they will support it as well. The proposal comes from an IETF draft written by Kevin Kelley at Anthropic.

The timelines are not the same. Anthropic says it has already started. OpenAI says "in the coming months". Google and Microsoft are aiming for next year. And the draft is exactly that, a draft. It is not an IETF standard, and nothing forces a crawler to respect it.

Still, four of the biggest model makers agreeing on one switch is rare. Until now every company had its own crawler name, and opting out properly meant knowing all of them. Like saying "no thanks" in four languages and hoping you didn't forget a fifth.

For those of us in Germany there's a local twist. Our copyright law has let rights holders reserve text and data mining for years, but § 44b UrhG says that for works online, the reservation only counts if it is machine-readable. A sentence in your Impressum saying "no AI, please" is a clear human wish. Whether it counts as machine-readable is something lawyers still argue about. A shared, boring, standard line in robots.txt is much harder to misunderstand.

Silence is not a no

This is the part that stuck with me. The default.

If a site publishes no preference, the crawler sees no preference. Not a yes, not a no. Just nothing. And in practice, nothing gets treated as permission.

I have watched this exact pattern in engineering teams for years.

The engineer who quietly hates being on the payments team and never says it. The reviewer who thinks the design is wrong but approves with "LGTM, small nits". The senior who wants to move into management and waits for someone to notice. The manager who expects people to raise their hand for the promotion track, while half of them are waiting to be asked.

Everyone in those stories had a preference. None of them published it. So the system did what systems always do: it went with the default.

A preference nobody can read is not a preference. It's a private feeling.

Where this is going

Robots.txt has been one of the most boring files on the internet for more than thirty years. I think it gets interesting now.

Expect CMS tools and SEO plugins to grow an "AI training" toggle. Expect legal teams to suddenly ask engineers what the company's file actually says. And expect a long argument about the difference between training a model and showing your content inside an AI answer, because one signal for training does not settle that.

I also expect a lot of teams to discover they never had a policy. Just a file someone copied from a tutorial in 2015.

What we can do this week

1. Open your robots.txt today. Company site, docs portal, personal blog. Read it like a stranger would. Does it say what you actually want?
2. Make the decision on purpose. Training yes or no is a product and legal decision, not a developer guess at 6 pm. If you're the engineer, bring the question to the right people with the options written down.
3. Commit it with a reason. A message like "opt out of AI training, agreed with legal" saves the next person an archaeology project.
4. Do the same for yourself. Pick one preference you've kept private: the project you want, the feedback you disagree with, the role you're aiming for. Say it clearly to someone who can act on it. Writing it down helps.
5. If you lead people, ask instead of waiting. A team where nobody states a preference is not a team without opinions. It's a team that learned nobody reads them.

The internet is finally getting a way to say "no" that machines understand. The harder part, on the web and at work, is the same as always: deciding what you want, and then actually saying it.

Sources: The Verge: OpenAI, Google, Microsoft and Anthropic back one AI training opt-out, OpenAI: Helping websites express AI training preferences, FourWeekMBA: OpenAI, Google, Microsoft, Anthropic back one AI opt-out, AI TL;DR: AI training opt-out token in robots.txt, Gesetze im Internet: § 44b UrhG Text und Data Mining