Class: Ignis::AI::BatchProcessor

Inherits:
Object
  • Object
show all
Defined in:
lib/nnw/ai/inference.rb

Overview

BatchProcessor — concurrent inference with dynamic batching.

Instance Method Summary collapse

Constructor Details

#initialize(model, tokenizer, max_batch_size: 8, max_wait_ms: 50) ⇒ BatchProcessor

Returns a new instance of BatchProcessor.

Parameters:

  • model (Transformer::Model)
  • tokenizer (Tokenizer)
  • max_batch_size (Integer) (defaults to: 8)
  • max_wait_ms (Integer) (defaults to: 50) —

    max milliseconds to wait for batch fill



156
157
158
159
160
161
162
 
# File 'lib/nnw/ai/inference.rb', line 156

def initialize(model, tokenizer, max_batch_size: 8, max_wait_ms: 50)
  @generator = TextGenerator.new(model, tokenizer)
  @max_batch_size = max_batch_size
  @max_wait_ms = max_wait_ms
  @queue = Queue.new
  @running = false
end

Instance Method Details

#start! ⇒ Thread

Start the batch processing loop in a background thread.

Returns:

  • (Thread) 


176
177
178
179
180
 
# File 'lib/nnw/ai/inference.rb', line 176

def start!
  @running = true
  @thread = Thread.new { batch_loop }
  @thread
end

#stop! ⇒ void

This method returns an undefined value.

Stop the processor.



184
185
186
187
 
# File 'lib/nnw/ai/inference.rb', line 184

def stop!
  @running = false
  @thread&.join(5)
end

#submit(prompt, **params) ⇒ String

Submit a request for processing.

Parameters:

  • prompt (String)
  • params (Hash) —

    generation parameters

Returns:

  • (String) —

    generated text (blocks until complete)



168
169
170
171
172
 
# File 'lib/nnw/ai/inference.rb', line 168

def submit(prompt, **params)
  result_queue = Queue.new
  @queue << { prompt: prompt, params: params, result: result_queue }
  result_queue.pop
end