Class: Ask::RAG::Loader::Directory

Inherits:
Base
  • Object
show all
Defined in:
lib/ask/rag/loader/directory.rb

Overview

Walks a directory tree and loads all supported file types.

Automatically detects file types by extension and dispatches to the appropriate loader. Ignores common non-source directories and files.

Examples:

loader = Ask::RAG::Loader::Directory.new
docs = loader.load("path/to/docs/")
# Loads all .md, .txt, .html, .csv, .pdf files recursively

With glob pattern

loader = Ask::RAG::Loader::Directory.new(pattern: "**/*.md")
docs = loader.load("docs/")
# Only loads Markdown files

Constant Summary collapse

EXTENSION_MAP = 
{
  ".txt" => :text,
  ".md" => :markdown,
  ".markdown" => :markdown,
  ".html" => :html,
  ".htm" => :html,
  ".csv" => :csv,
  ".json" => :json,
  ".pdf" => :pdf
}.freeze
SKIP_DIRS =

Directories to skip when walking (matches basename).

%w[
  .git .svn .hg .bzr _darcs node_modules .bundle vendor
  .yardoc coverage tmp log public/assets
].freeze

Instance Method Summary collapse

Methods inherited from Base

#lazy_load

Constructor Details

#initialize(pattern: nil, skip_dirs: SKIP_DIRS) ⇒ Directory

Returns a new instance of Directory.

Parameters:

  • pattern (String, nil) (defaults to: nil) —

    optional glob pattern (e.g., "**/*.md") When nil, walks all files with known extensions.

  • skip_dirs (Array<String>) (defaults to: SKIP_DIRS) —

    directory names to skip



42
43
44
45
 
# File 'lib/ask/rag/loader/directory.rb', line 42

def initialize(pattern: nil, skip_dirs: SKIP_DIRS)
  @pattern = pattern
  @skip_dirs = skip_dirs.map(&:freeze).freeze
end

Instance Method Details

#load(path) ⇒ Array<Ask::Document>

Load all supported files from a directory tree.

Parameters:

  • path (String) —

    root directory path

Returns:

  • (Array<Ask::Document>) —

    all loaded documents



51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/ask/rag/loader/directory.rb', line 51

def load(path)
  docs = []
  files(path).each do |file|
    loader = loader_for(file)
    next unless loader

    docs.concat(loader.load(file))
  rescue StandardError => e
    warn "[ask-rag] Skipping #{file}: #{e.class}: #{e.message}"
  end
  docs
end