Ask AI crawlers not to use your content
Adds lines to your robots.txt asking the crawlers that collect training data for AI models (OpenAI, Anthropic, Google, Apple, Meta, Common Crawl, ByteDance) to stay away. Normal search engines are not affected.
<?php
add_filter(
'robots_txt',
static function ( $output, $public ) {
if ( ! $public ) {
return $output;
}
$bots = array( 'GPTBot', 'ClaudeBot', 'anthropic-ai', 'Google-Extended', 'Applebot-Extended', 'meta-externalagent', 'CCBot', 'Bytespider' );
$output .= "\n# AI training crawlers\n";
foreach ( $bots as $bot ) {
$output .= "User-agent: {$bot}\nDisallow: /\n\n";
}
return $output;
},
20,
2
);
Good to know
robots.txt is a request, not a lock: well-behaved crawlers follow it. Only works while there’s no real robots.txt file on the server. Search features such as Google’s AI Overviews use the normal Googlebot and aren’t affected.
In these packs
- SEO basics: Meta descriptions, social sharing tags, article structured data and cleaner indexing, for sites without an SEO plugin.