Module: Wp2txt::RactorWorker

Defined in:
lib/wp2txt/ractor_worker.rb

Overview

Ractor-based parallel processing for Wikipedia article conversion

Ractor allows true parallelism by bypassing Ruby's GVL (Global VM Lock), enabling significant speedups for CPU-intensive text processing.

REQUIREMENTS: Ruby 4.0+ (Ractor API stabilized in Ruby 4.0) For Ruby 3.x, the Parallel gem is used instead (process-based parallelism).

Performance: Typically 1.5-2x speedup with 4 workers on multi-core systems.

Usage:

pages = [["Title1", "wiki text..."], ["Title2", "wiki text..."]]
results = RactorWorker.process_articles(pages, config: config)

Constant Summary collapse

MINIMUM_RUBY_VERSION =

Minimum Ruby version required for stable Ractor support

"4.0"
OPERATIONS =

Registry of available operations

%i[process_article double fib].freeze

Class Method Summary collapse

Class Method Details

.available? ⇒ Boolean

Check if Ractor is available and usable Requires Ruby 4.0+ for stable Ractor support

Returns:

  • (Boolean) —

    true if Ractor can be used



32
33
34
35
36
 
# File 'lib/wp2txt/ractor_worker.rb', line 32

def available?
  return @available if defined?(@available)

  @available = check_ractor_available
end

.check_ractor_available ⇒ Boolean

Internal method to check Ractor availability

Returns:

  • (Boolean) —

    true if Ractor can be used



40
41
42
43
44
45
46
47
48
49
50
 
# File 'lib/wp2txt/ractor_worker.rb', line 40

def check_ractor_available
  return false unless ruby_version_sufficient?
  return false unless defined?(Ractor)

  # Test basic Ractor functionality with Ruby 4.0 API
  r = Ractor.new { 1 + 1 }
  r.join
  r.value == 2
rescue StandardError
  false
end

.deep_freeze(obj) ⇒ Object

Deep freeze an object for Ractor sharing

Parameters:

  • obj (Object) —

    Object to freeze

Returns:

  • (Object) —

    The frozen object



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
 
# File 'lib/wp2txt/ractor_worker.rb', line 198

def deep_freeze(obj)
  case obj
  when Hash
    obj.transform_keys { |k| deep_freeze(k) }
       .transform_values { |v| deep_freeze(v) }
       .freeze
  when Array
    obj.map { |v| deep_freeze(v) }.freeze
  when String
    obj.frozen? ? obj : obj.dup.freeze
  when Symbol, Integer, Float, TrueClass, FalseClass, NilClass
    obj
  else
    obj.freeze rescue obj
  end
end

.optimal_workers ⇒ Integer

Calculate optimal number of workers based on CPU cores

Returns:

  • (Integer) —

    Recommended concurrency level



186
187
188
189
190
191
192
193
 
# File 'lib/wp2txt/ractor_worker.rb', line 186

def optimal_workers
  cores = Etc.nprocessors
  case cores
  when 1..4 then cores
  when 5..8 then cores - 1
  else (cores * 0.8).to_i
  end
end

.parallel_process(items, operation:, config: {}, num_workers: nil) ⇒ Array

Process items in parallel using map-join-value pattern (Ruby 4.0+)

Parameters:

  • items (Array) —

    Items to process

  • operation (Symbol) —

    Operation to perform (:process_article, :double, :fib)

  • config (Hash) (defaults to: {}) —

    Configuration to pass to each operation

  • num_workers (Integer) (defaults to: nil) —

    Max concurrent Ractors (default: optimal_workers)

Returns:

  • (Array) —

    Results from processing (in original order)



81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
 
# File 'lib/wp2txt/ractor_worker.rb', line 81

def parallel_process(items, operation:, config: {}, num_workers: nil)
  batch_size = num_workers || optimal_workers
  batch_size = [batch_size, 1].max

  # Fall back to sequential if Ractor not available or single item
  unless available? && items.size > 1
    return items.map { |item| process_single(item, operation, config) }
  end

  # Freeze config for sharing across Ractors
  frozen_config = deep_freeze(config.dup)

  # Process in batches to limit concurrent Ractors
  results = []
  items.each_slice(batch_size) do |batch|
    batch_results = process_batch(batch, operation, frozen_config)
    results.concat(batch_results)
  end

  results
rescue Ractor::Error => e
  warn "Ractor error (#{e.message}), falling back to sequential processing"
  items.map { |item| process_single(item, operation, config) }
end

.process_articles(pages, config:, strip_tmarker: false, num_workers: nil) ⇒ Array<String>

Process articles in parallel using Ractor (main entry point)

Parameters:

  • pages (Array<Array>) —

    Array of [title, text] pairs

  • config (Hash) —

    Configuration options for formatting

  • strip_tmarker (Boolean) (defaults to: false) —

    Whether to strip list markers

  • num_workers (Integer) (defaults to: nil) —

    Number of parallel Ractors (optional)

Returns:

  • (Array<String>) —

    Formatted article results



64
65
66
67
68
69
70
71
72
73
 
# File 'lib/wp2txt/ractor_worker.rb', line 64

def process_articles(pages, config:, strip_tmarker: false, num_workers: nil)
  items = pages.map { |title, text| [title, text, strip_tmarker] }

  parallel_process(
    items,
    operation: :process_article,
    config: config,
    num_workers: num_workers
  )
end

.process_batch(items, operation, frozen_config) ⇒ Array

Process a batch using map-join-value pattern (Ruby 4.0 API)

Parameters:

  • items (Array) —

    Items to process in this batch

  • operation (Symbol) —

    Operation to perform

  • frozen_config (Hash) —

    Frozen configuration hash

Returns:

  • (Array) —

    Results in original order



111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/wp2txt/ractor_worker.rb', line 111

def process_batch(items, operation, frozen_config)
  # Create one Ractor per item
  ractors = items.map.with_index do |item, idx|
    Ractor.new(item, frozen_config, operation, idx) do |it, cfg, op, i|
      result = begin
        case op
        when :process_article
          require_relative "utils"
          require_relative "regex"
          require_relative "article"
          require_relative "formatter"

          title, text, strip_tmarker = it
          formatter = Object.new
          formatter.extend(Wp2txt)
          formatter.extend(Wp2txt::Formatter)
          article = Wp2txt::Article.new(text, title, strip_tmarker)
          formatter.format_article(article, cfg)
        when :double
          it * 2
        when :fib
          fib = ->(n) { n <= 1 ? n : fib.call(n - 1) + fib.call(n - 2) }
          fib.call(it)
        else
          raise "Unknown operation: #{op}"
        end
      rescue StandardError
        nil # Return nil on error
      end
      [i, result] # Return index and result for ordering
    end
  end

  # Wait for all Ractors to complete and collect results
  collected = Array.new(items.size)
  ractors.each do |r|
    r.join
    idx, result = r.value
    collected[idx] = result
  end

  collected
end

.process_single(item, operation, config) ⇒ Object

Process a single item (for fallback/sequential processing)

Parameters:

  • item (Object) —

    Item to process

  • operation (Symbol) —

    Operation to perform

  • config (Hash) —

    Configuration options

Returns:

  • (Object) —

    Processing result



160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/wp2txt/ractor_worker.rb', line 160

def process_single(item, operation, config)
  case operation
  when :process_article
    require_relative "utils"
    require_relative "regex"
    require_relative "article"
    require_relative "formatter"

    title, text, strip_tmarker = item
    formatter = Object.new
    formatter.extend(Wp2txt)
    formatter.extend(Wp2txt::Formatter)
    article = Wp2txt::Article.new(text, title, strip_tmarker)
    formatter.format_article(article, config)
  when :double
    item * 2
  when :fib
    fib = ->(n) { n <= 1 ? n : fib.call(n - 1) + fib.call(n - 2) }
    fib.call(item)
  else
    raise "Unknown operation: #{operation}"
  end
end

.ruby_version_sufficient? ⇒ Boolean

Check if Ruby version meets minimum requirement

Returns:

  • (Boolean) —

    true if Ruby version is 4.0 or higher



54
55
56
 
# File 'lib/wp2txt/ractor_worker.rb', line 54

def ruby_version_sufficient?
  Gem::Version.new(RUBY_VERSION) >= Gem::Version.new(MINIMUM_RUBY_VERSION)
end