Class: Abqari::Robots

Inherits:
Object
  • Object
show all
Defined in:
lib/abqari/robots.rb

Constant Summary collapse

AI_BOTS =

AI training crawlers that explicitly identify themselves and respect robots.txt. Blocked when block_ai_scraping: true is set in config (the default). Bots that ignore robots.txt won't be stopped by anything in this file — but the major AI training crawlers (OpenAI, Anthropic, Google's AI carve-out, Meta, Apple, Perplexity, Common Crawl, ByteDance, Cohere) all honour it.

Notes:

- `Google-Extended` is Google's separate user-agent for AI
training. Blocking it does NOT affect regular Googlebot
search indexing.
- `Applebot-Extended` is the same idea for Apple.
- `CCBot` is Common Crawl, used by many third-party trainers.
- User-facing fetchers (ChatGPT-User, Perplexity's on-demand
fetcher) ARE included — block-all-AI is the safer default;
remove specific bots from this list if you want their search
citations.
%w[
  GPTBot
  ChatGPT-User
  OAI-SearchBot
  ClaudeBot
  Claude-Web
  anthropic-ai
  Google-Extended
  Applebot-Extended
  Meta-ExternalAgent
  FacebookBot
  PerplexityBot
  Perplexity-User
  YouBot
  cohere-ai
  Bytespider
  MistralAI-User
  CCBot
  Diffbot
  omgili
  omgilibot
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(site) ⇒ Robots

Returns a new instance of Robots.



45
46
47
 
# File 'lib/abqari/robots.rb', line 45

def initialize(site)
  @site = site
end

Instance Method Details

#write ⇒ Object 



49
50
51
 
# File 'lib/abqari/robots.rb', line 49

def write
  File.write(File.join(@site.output_dir, 'robots.txt'), build)
end