Class: LlmDocsBuilder::Comparator

Inherits:
Object
  • Object
show all
Defined in:
lib/llm_docs_builder/comparator.rb

Overview

Compares content sizes between human and AI versions

Helps quantify context window savings by comparing:

  • Remote URL with different User-Agents (human vs AI bot)

  • Remote URL with local markdown file

Examples:

Compare remote versions

comparator = LlmDocsBuilder::Comparator.new('https://example.com/docs/page.html')
result = comparator.compare
puts "Reduction: #{result[:reduction_percent]}%"

Compare remote with local file

comparator = LlmDocsBuilder::Comparator.new('https://example.com/docs/page.html',
  local_file: 'docs/page.md'
)
result = comparator.compare

Constant Summary collapse

HUMAN_USER_AGENT =

Default User-Agent for simulating human browser

'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0'
AI_USER_AGENT =

Default User-Agent for simulating AI bot

'Claude-Web/1.0 (Anthropic AI Assistant)'
MAX_REDIRECTS =

Maximum number of redirects to follow before raising an error

UrlFetcher::MAX_REDIRECTS

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(url, options = {}) ⇒ Comparator

Initialize a new comparator

Parameters:

  • url (String) —

    URL to fetch and compare

  • options (Hash) (defaults to: {}) —

    comparison options

Options Hash (options):

  • :local_file (String) —

    path to local markdown file for comparison

  • :human_user_agent (String) —

    custom User-Agent for human version

  • :ai_user_agent (String) —

    custom User-Agent for AI version

  • :verbose (Boolean) —

    enable verbose output



46
47
48
49
50
51
52
 
# File 'lib/llm_docs_builder/comparator.rb', line 46

def initialize(url, options = {})
  @url = url
  @options = {
    human_user_agent: HUMAN_USER_AGENT,
    ai_user_agent: AI_USER_AGENT
  }.merge(options)
end

Instance Attribute Details

#options ⇒ Hash (readonly)

Returns comparison options.

Returns:

  • (Hash) —

    comparison options



36
37
38
 
# File 'lib/llm_docs_builder/comparator.rb', line 36

def options
  @options
end

#url ⇒ String (readonly)

Returns URL to compare.

Returns:

  • (String) —

    URL to compare



33
34
35
 
# File 'lib/llm_docs_builder/comparator.rb', line 33

def url
  @url
end

Instance Method Details

#compare ⇒ Hash

Compare content sizes and calculate reduction

Returns:

  • (Hash) —

    comparison results with keys:

    • :human_size [Integer] size of human version in bytes

    • :ai_size [Integer] size of AI version in bytes

    • :reduction_bytes [Integer] bytes saved

    • :reduction_percent [Integer] percentage reduction

    • :factor [Float] compression factor

    • :human_tokens [Integer] estimated tokens in human version

    • :ai_tokens [Integer] estimated tokens in AI version

    • :token_reduction [Integer] estimated tokens saved

    • :token_reduction_percent [Integer] percentage of tokens saved

    • :human_source [String] source description (URL or file)

    • :ai_source [String] source description (URL or file)



68
69
70
71
72
73
74
 
# File 'lib/llm_docs_builder/comparator.rb', line 68

def compare
  if options[:local_file]
    compare_with_local_file
  else
    compare_remote_versions
  end
end