Module: LlmDocsBuilder

Defined in:
lib/llm_docs_builder.rb,
lib/llm_docs_builder/cli.rb,
lib/llm_docs_builder/config.rb,
lib/llm_docs_builder/errors.rb,
lib/llm_docs_builder/parser.rb,
lib/llm_docs_builder/helpers.rb,
lib/llm_docs_builder/version.rb,
lib/llm_docs_builder/generator.rb,
lib/llm_docs_builder/validator.rb,
lib/llm_docs_builder/comparator.rb,
lib/llm_docs_builder/url_fetcher.rb,
lib/llm_docs_builder/html_detector.rb,
lib/llm_docs_builder/text_compressor.rb,
lib/llm_docs_builder/token_estimator.rb,
lib/llm_docs_builder/bulk_transformer.rb,
lib/llm_docs_builder/output_formatter.rb,
lib/llm_docs_builder/markdown_transformer.rb,
lib/llm_docs_builder/html_to_markdown_converter.rb,
lib/llm_docs_builder/transformers/base_transformer.rb,
lib/llm_docs_builder/transformers/link_transformer.rb,
lib/llm_docs_builder/transformers/heading_transformer.rb,
lib/llm_docs_builder/transformers/whitespace_transformer.rb,
lib/llm_docs_builder/transformers/enhancement_transformer.rb,
lib/llm_docs_builder/html_to_markdown/table_markup_renderer.rb,
lib/llm_docs_builder/transformers/content_cleanup_transformer.rb,
lib/llm_docs_builder/helpers/squeeze_blank_lines_outside_fences.rb,
lib/llm_docs_builder/html_to_markdown/figure_code_block_renderer.rb,
lib/llm_docs_builder/helpers/prune_trailing_unsafe_link_separator.rb

Overview

Build and optimize documentation for LLMs

This gem provides tools for generating llms.txt files and transforming markdown documentation to be AI-friendly. It can reduce token consumption by 67-95% while preserving essential documentation content.

Defined Under Namespace

Modules: Errors, Helpers, HtmlToMarkdown, Transformers Classes: BulkTransformer, CLI, Comparator, Config, Generator, HtmlDetector, HtmlToMarkdownConverter, MarkdownTransformer, OutputFormatter, ParsedContent, Parser, TextCompressor, TokenEstimator, UrlFetcher, Validator

Constant Summary collapse

VERSION =

Current version of the LlmDocsBuilder gem

'0.13.0'

Class Method Summary collapse

Class Method Details

.bulk_transform(docs_path, options = {}) ⇒ Array<String>

Bulk transforms multiple markdown files to be AI-friendly

Examples:

Bulk transform with direct options

LlmDocsBuilder.bulk_transform('./docs',
  base_url: 'https://myproject.io',
  suffix: '.ai',
  excludes: ['**/private/**', 'draft-*.md']
)

Bulk transform using config file

LlmDocsBuilder.bulk_transform('./docs', config_file: 'llm-docs-builder.yml')

Parameters:

  • docs_path (String) —

    path to documentation directory

  • options (Hash) (defaults to: {}) —

    transformation options

Options Hash (options):

  • :config_file (String) —

    path to YAML config file (auto-finds llm-docs-builder.yml)

  • :base_url (String) —

    base URL for expanding relative links (overrides config)

  • :convert_urls (Boolean) —

    convert HTML URLs to markdown format (overrides config)

  • :suffix (String) —

    suffix for transformed files (default: ‘.llm’, overrides config)

  • :excludes (Array<String>) —

    glob patterns for files to exclude (overrides config)

  • :verbose (Boolean) —

    enable verbose output (overrides config)

Returns:

  • (Array<String>) —

    paths of transformed files



118
119
120
121
122
123
 
# File 'lib/llm_docs_builder.rb', line 118

def bulk_transform(docs_path, options = {})
  config = Config.new(options[:config_file])
  merged_options = config.merge_with_options(options)

  BulkTransformer.new(docs_path, merged_options).transform_all
end

.generate_from_docs(docs_path = nil, options = {}) ⇒ String

Generates llms.txt from existing markdown documentation

Examples:

Generate from docs directory

LlmDocsBuilder.generate_from_docs('./docs')

Generate using config file

LlmDocsBuilder.generate_from_docs(config_file: 'llm-docs-builder.yml')

Generate with config file and overrides

LlmDocsBuilder.generate_from_docs('./docs',
  config_file: 'my-config.yml',
  title: 'Override Title'
)

Parameters:

  • docs_path (String, nil) (defaults to: nil) —

    path to documentation directory or file (optional if config_file provided)

  • options (Hash) (defaults to: {}) —

    generation options

Options Hash (options):

  • :config_file (String) —

    path to YAML config file (auto-finds llm-docs-builder.yml)

  • :base_url (String) —

    base URL for converting relative links (overrides config)

  • :title (String) —

    project title (auto-detected if not provided, overrides config)

  • :description (String) —

    project description (auto-detected if not provided, overrides config)

  • :output (String) —

    output file path (default: ‘llms.txt’, overrides config)

  • :convert_urls (Boolean) —

    convert HTML URLs to markdown format (overrides config)

  • :verbose (Boolean) —

    enable verbose output (overrides config)

  • :content (String) —

    raw markdown content (used for remote sources)

Returns:

  • (String) —

    generated llms.txt content



52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/llm_docs_builder.rb', line 52

def generate_from_docs(docs_path = nil, options = {})
  # Support config-first usage: generate_from_docs(config_file: 'path.yml')
  if docs_path.is_a?(Hash) && docs_path.key?(:config_file)
    options = docs_path
    docs_path = nil
  end

  config = Config.new(options[:config_file])
  merged_options = config.merge_with_options(options)

  # Use docs_path param or config file docs setting
  final_docs_path = docs_path || merged_options[:docs]

  Generator.new(final_docs_path, merged_options).generate
end

.parse(file_path) ⇒ Parser

Parses an existing llms.txt file

Parameters:

  • file_path (String) —

    path to the llms.txt file to parse

Returns:

  • (Parser) —

    parsed llms.txt object



129
130
131
 
# File 'lib/llm_docs_builder.rb', line 129

def parse(file_path)
  Parser.new(file_path).parse
end

.transform_markdown(file_path, options = {}) ⇒ String

Transforms a markdown file to be AI-friendly

Examples:

Transform with direct options

LlmDocsBuilder.transform_markdown('README.md',
  base_url: 'https://myproject.io',
  convert_urls: true
)

Transform using config file

LlmDocsBuilder.transform_markdown('README.md', config_file: 'llm-docs-builder.yml')

Parameters:

  • file_path (String) —

    path to markdown file

  • options (Hash) (defaults to: {}) —

    transformation options

Options Hash (options):

  • :config_file (String) —

    path to YAML config file (auto-finds llm-docs-builder.yml)

  • :base_url (String) —

    base URL for expanding relative links (overrides config)

  • :convert_urls (Boolean) —

    convert HTML URLs to markdown format (overrides config)

  • :verbose (Boolean) —

    enable verbose output (overrides config)

Returns:

  • (String) —

    transformed markdown content



87
88
89
90
91
92
 
# File 'lib/llm_docs_builder.rb', line 87

def transform_markdown(file_path, options = {})
  config = Config.new(options[:config_file])
  merged_options = config.merge_with_options(options)

  MarkdownTransformer.new(file_path, merged_options).transform
end

.validate(content) ⇒ Boolean

Validates llms.txt content

Parameters:

  • content (String) —

    the llms.txt content to validate

Returns:

  • (Boolean) —

    true if content is valid, false otherwise



137
138
139
 
# File 'lib/llm_docs_builder.rb', line 137

def validate(content)
  Validator.new(content).valid?
end