Class: Jekyll::Documents::TextExtractionManifest

Inherits:
Object
  • Object
show all
Defined in:
lib/jekyll/documents/text_extraction_manifest.rb

Overview

Manages the text extraction manifest — a persistent JSON file that tracks which documents have had their text extracted, keyed by SHA-256 digest.

The manifest lives in <site.source>/<cache_dir>/text-extraction-manifest.json and extracted text is stored in separate files under <cache_dir>/text/ (sharded by the first 2 hex chars of the digest to avoid huge directories).

This design borrows from jekyll-imgflow's ManifestManager:

  • Atomic writes (temp file + rename)
  • SHA-256 content digests for cache invalidation
  • Cleanup of entries for deleted source files
  • Survives jekyll clean (stored in site source, not .jekyll-cache/)

Constant Summary collapse

MANIFEST_FILENAME = 
"text-extraction-manifest.json"
TEXT_SUBDIR = 
"text"

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(site, cache_dir) ⇒ TextExtractionManifest

Returns a new instance of TextExtractionManifest.

Parameters:

  • site (Jekyll::Site) —

    the Jekyll site instance

  • cache_dir (String) —

    relative path from site source for cache directory



29
30
31
32
33
34
35
36
37
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 29

def initialize(site, cache_dir)
  @site = site
  @cache_dir = cache_dir
  @cache_root = File.join(site.source, cache_dir)
  @manifest_path = File.join(@cache_root, MANIFEST_FILENAME)
  @text_dir = File.join(@cache_root, TEXT_SUBDIR)
  @manifest = load_manifest
  @dirty = false
end

Instance Attribute Details

#cache_dir ⇒ Object (readonly)

Returns the value of attribute cache_dir.



25
26
27
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 25

def cache_dir
  @cache_dir
end

#manifest_path ⇒ Object (readonly)

Returns the value of attribute manifest_path.



25
26
27
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 25

def manifest_path
  @manifest_path
end

Instance Method Details

#cached?(rel_path, digest) ⇒ Boolean

Check if a document is in the manifest with a matching digest.

Parameters:

  • rel_path (String) —

    relative path of the source document

  • digest (String) —

    SHA-256 hex digest of the source file

Returns:

  • (Boolean) 


119
120
121
122
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 119

def cached?(rel_path, digest)
  entry = @manifest[rel_path]
  !!(entry && entry["digest"] == digest)
end

#cleanup_deleted(current_rel_paths) ⇒ Integer

Remove manifest entries and their text files for source documents that no longer exist.

Parameters:

  • current_rel_paths (Array<String>) —

    relative paths of current source files

Returns:

  • (Integer) —

    number of entries removed



77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 77

def cleanup_deleted(current_rel_paths)
  current_set = current_rel_paths.to_set
  removed = 0

  @manifest.each_key do |rel_path|
    next if current_set.include?(rel_path)

    entry = @manifest[rel_path]
    delete_text_file(entry["text_file"]) if entry
    @manifest.delete(rel_path)
    removed += 1
    @dirty = true
  end

  removed
end

#get(rel_path, digest) ⇒ String?

Get cached text for a document if the digest matches.

Parameters:

  • rel_path (String) —

    relative path of the source document

  • digest (String) —

    SHA-256 hex digest of the source file

Returns:

  • (String, nil) —

    extracted text if cache hit, nil otherwise



43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 43

def get(rel_path, digest)
  entry = @manifest[rel_path]
  return nil unless entry
  return nil unless entry["digest"] == digest

  text_file = text_file_path(entry["text_file"])
  return nil unless File.file?(text_file)

  File.read(text_file, encoding: "UTF-8")
rescue StandardError => e
  ::Jekyll.logger.warn "jekyll-documents",
                       "Manifest read failed for #{rel_path}: #{e.message}"
  nil
end

#save ⇒ void

This method returns an undefined value.

Save the manifest to disk if it has changed. Uses atomic write (temp file + rename) to prevent corruption.



97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 97

def save
  return unless @dirty

  content = JSON.pretty_generate(@manifest)
  return if File.exist?(@manifest_path) && File.binread(@manifest_path) == content

  FileUtils.mkdir_p(@cache_root)
  temporary_path = "#{@manifest_path}.tmp-#{Process.pid}-#{Thread.current.object_id}"
  File.open(temporary_path, "wb") do |file|
    file.write(content)
    file.flush
    file.fsync
  end
  File.rename(temporary_path, @manifest_path)
ensure
  FileUtils.rm_f(temporary_path) if defined?(temporary_path) && temporary_path
end

#set(rel_path, digest, text) ⇒ void

This method returns an undefined value.

Store extracted text for a document.

Parameters:

  • rel_path (String) —

    relative path of the source document

  • digest (String) —

    SHA-256 hex digest of the source file

  • text (String) —

    the extracted text



63
64
65
66
67
68
69
70
71
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 63

def set(rel_path, digest, text)
  text_file = write_text_file(digest, text)
  @manifest[rel_path] = {
    "digest" => digest,
    "text_file" => text_file,
    "extracted_at" => Time.now.to_i
  }
  @dirty = true
end

#size ⇒ Integer

Number of entries in the manifest.

Returns:

  • (Integer) 


126
127
128
 
# File 'lib/jekyll/documents/text_extraction_manifest.rb', line 126

def size
  @manifest.size
end