Class: LLM::Context

Inherits:
Object
  • Object
show all
Includes:
Deserializer, Serializer
Defined in:
lib/llm/context.rb,
lib/llm/context/serializer.rb,
lib/llm/context/deserializer.rb

Overview

LLM::Context is the stateful execution boundary in llm.rb.

It holds the evolving runtime state for an LLM workflow: conversation history, tool calls and returns, schema and streaming configuration, accumulated usage, and request ownership for interruption.

This is broader than prompt context alone. A context is the object that lets one-off prompts, streaming turns, tool execution, persistence, retries, and serialized long-lived workflows all run through the same model.

A context can drive the chat completions API that all providers support or the Responses API on providers that expose it.

Examples:

#!/usr/bin/env ruby
require "llm"

llm = LLM.openai(key: ENV["KEY"])
ctx = LLM::Context.new(llm)

prompt = LLM::Prompt.new(llm) do
  system "Be concise and show your reasoning briefly."
  user "If a train goes 60 mph for 1.5 hours, how far does it travel?"
  user "Now double the speed for the same time."
end

ctx.talk(prompt)
ctx.messages.each { |m| puts "[#{m.role}] #{m.content}" }

Defined Under Namespace

Modules: Deserializer, Serializer

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Deserializer

#deserialize, #deserialize_message

Constructor Details

#initialize(llm, params = {}) ⇒ Context

Returns a new instance of Context.

Parameters:

  • llm (LLM::Provider) —

    A provider

  • params (Hash) (defaults to: {}) —

    The parameters to maintain throughout the conversation. Any parameter the provider supports can be included and not only those listed here.

Options Hash (params):

  • :mode (Symbol) —

    Defaults to :completions

  • :model (String) —

    Defaults to the provider’s default model

  • :tools (Array<LLM::Function>, nil) —

    Defaults to nil

  • :skills (Array<String>, nil) —

    Defaults to nil



76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/llm/context.rb', line 76

def initialize(llm, params = {})
  @llm = llm
  @mode = params.delete(:mode) || :completions
  @compactor = params.delete(:compactor)
  @guard = params.delete(:guard)
  @transformer = params.delete(:transformer)
  tools = [*params.delete(:tools), *load_skills(params.delete(:skills))]
  @params = {model: llm.default_model, schema: nil}.compact.merge!(params)
  @params[:tools] = tools unless tools.empty?
  @messages = LLM::Buffer.new(llm)
end

Instance Attribute Details

#llm ⇒ LLM::Provider (readonly)

Returns a provider

Returns:



51
52
53
 
# File 'lib/llm/context.rb', line 51

def llm
  @llm
end

#messages ⇒ LLM::Buffer<LLM::Message> (readonly)

Returns the accumulated message history for this context

Returns:



46
47
48
 
# File 'lib/llm/context.rb', line 46

def messages
  @messages
end

#mode ⇒ Symbol (readonly)

Returns the context mode

Returns:

  • (Symbol) 


56
57
58
 
# File 'lib/llm/context.rb', line 56

def mode
  @mode
end

Instance Method Details

#call(target) ⇒ Array<LLM::Function::Return>

Calls a named collection of work through the context.

This currently supports ‘:functions`, forwarding to `functions.call`.

Parameters:

  • target (Symbol) —

    The work collection to call

Returns:



250
251
252
253
254
255
 
# File 'lib/llm/context.rb', line 250

def call(target)
  case target
  when :functions then guarded_returns || functions.call
  else raise ArgumentError, "Unknown target: #{target.inspect}. Expected :functions"
  end
end

#compactor ⇒ LLM::Compactor

Returns a context compactor This feature is inspired by the compaction approach developed by General Intelligence Systems in [Brute](github.com/general-intelligence-systems/brute).

Returns:



94
95
96
97
 
# File 'lib/llm/context.rb', line 94

def compactor
  @compactor = LLM::Compactor.new(self, @compactor || {}) unless LLM::Compactor === @compactor
  @compactor
end

#compactor=(compactor) ⇒ LLM::Compactor, ...

Sets a context compactor or compactor config

Parameters:

Returns:



103
104
105
 
# File 'lib/llm/context.rb', line 103

def compactor=(compactor)
  @compactor = compactor
end

#context_window ⇒ Integer

Note:

This method returns 0 when the provider or model can’t be found within Registry.

Returns the model’s context window. The context window is the maximum amount of input and output tokens a model can consider in a single request.

Returns:

  • (Integer) 


451
452
453
454
455
456
457
458
 
# File 'lib/llm/context.rb', line 451

def context_window
  LLM
    .registry_for(llm)
    .limit(model:)
    .context
rescue LLM::NoSuchModelError, LLM::NoSuchRegistryError
  0
end

#cost ⇒ LLM::Cost

Returns an approximate cost for a given context based on both the provider, and model

Returns:

  • (LLM::Cost) —

    Returns an approximate cost for a given context based on both the provider, and model



434
435
436
437
438
439
440
441
 
# File 'lib/llm/context.rb', line 434

def cost
  return LLM::Cost.new(0, 0) unless usage
  cost = LLM.registry_for(llm).cost(model:)
  LLM::Cost.new(
    (cost.input.to_f / 1_000_000.0)  * usage.input_tokens,
    (cost.output.to_f / 1_000_000.0) * usage.output_tokens
  )
end

#functions ⇒ Array<LLM::Function>

Returns an array of functions that can be called

Returns:



229
230
231
232
233
234
235
236
237
238
239
240
 
# File 'lib/llm/context.rb', line 229

def functions
  return_ids = returns.map(&:id)
  @messages
    .select(&:assistant?)
    .flat_map do |msg|
      fns = msg.functions.select { _1.pending? && !return_ids.include?(_1.id) }
      fns.each do |fn|
        fn.tracer = tracer
        fn.model  = msg.model
      end
    end.extend(LLM::Function::Array)
end

#guard ⇒ #call?

Returns a guard, if configured.

Guards are context-level supervisors for agentic execution. A guard can inspect the runtime state and decide whether pending tool work should be blocked before the context keeps looping.

The built-in implementation is LLM::LoopGuard, which detects repeated tool-call patterns and turns them into in-band LLM::GuardError tool returns.

Returns:



119
120
121
122
123
124
 
# File 'lib/llm/context.rb', line 119

def guard
  return if @guard.nil? || @guard == false
  @guard = LLM::LoopGuard.new if @guard == true
  @guard = LLM::LoopGuard.new(@guard) if Hash === @guard
  @guard
end

#guard=(guard) ⇒ #call, ...

Sets a guard or guard config.

Guards must implement ‘call(ctx)` and return either `nil` or a warning string. Returning a warning tells the context to block pending tool work with guarded tool errors instead of continuing the loop.

Parameters:

  • guard (#call, Hash, Boolean, nil)

Returns:

  • (#call, Hash, Boolean, nil) 


135
136
137
 
# File 'lib/llm/context.rb', line 135

def guard=(guard)
  @guard = guard
end

#image_url(url) ⇒ LLM::Object

Recongize an object as a URL to an image

Parameters:

  • url (String) —

    The URL

Returns:



365
366
367
 
# File 'lib/llm/context.rb', line 365

def image_url(url)
  LLM::Object.from(value: url, kind: :image_url)
end

#inspect ⇒ String

Returns:

  • (String) 


220
221
222
223
224
 
# File 'lib/llm/context.rb', line 220

def inspect
  "#<#{self.class.name}:0x#{object_id.to_s(16)} " \
  "@llm=#{@llm.class}, @mode=#{@mode.inspect}, @params=#{@params.inspect}, " \
  "@messages=#{@messages.inspect}>"
end

#interrupt! ⇒ nil Also known as: cancel!

Interrupt the active request, if any. This is inspired by Go’s context cancellation model.

Returns:

  • (nil) 


315
316
317
318
 
# File 'lib/llm/context.rb', line 315

def interrupt!
  llm.interrupt!(@owner)
  queue&.interrupt!
end

#local_file(path) ⇒ LLM::Object

Recongize an object as a local file

Parameters:

  • path (String) —

    The path

Returns:



375
376
377
 
# File 'lib/llm/context.rb', line 375

def local_file(path)
  LLM::Object.from(value: LLM.File(path), kind: :local_file)
end

#model ⇒ String

Returns the model a Context is actively using

Returns:

  • (String) 


399
400
401
 
# File 'lib/llm/context.rb', line 399

def model
  messages.find(&:assistant?)&.model || @params[:model]
end

#params ⇒ Hash

Returns the default params for this context

Returns:

  • (Hash) 


61
62
63
 
# File 'lib/llm/context.rb', line 61

def params
  @params.dup
end

#prompt(&b) ⇒ LLM::Prompt Also known as: build_prompt

Build a role-aware prompt for a single request.

Prefer this method over #build_prompt. The older method name is kept for backward compatibility.

Examples:

prompt = ctx.prompt do
  system "Your task is to assist the user"
  user "Hello, can you assist me?"
end
ctx.talk(prompt)

Parameters:

  • b (Proc) —

    A block that composes messages. If it takes one argument, it receives the prompt object. Otherwise it runs in prompt context.

Returns:



354
355
356
 
# File 'lib/llm/context.rb', line 354

def prompt(&b)
  LLM::Prompt.new(@llm, &b)
end

#remote_file(res) ⇒ LLM::Object

Reconginize an object as a remote file

Parameters:

Returns:



385
386
387
 
# File 'lib/llm/context.rb', line 385

def remote_file(res)
  LLM::Object.from(value: res, kind: :remote_file)
end

#respond(prompt, params = {}) ⇒ LLM::Response

Note:

Not all LLM providers support this API

Interact with the context via the responses API. This method immediately sends a request to the LLM and returns the response.

Examples:

llm = LLM.openai(key: ENV["KEY"])
ctx = LLM::Context.new(llm)
res = ctx.respond("What is the capital of France?")
puts res.output_text

Parameters:

  • params (defaults to: {}) —

    The params, including optional :role (defaults to :user), :stream, :tools, :schema etc.

  • prompt (String) —

    The input prompt to be completed

Returns:

  • (LLM::Response) —

    Returns the LLM’s response for this turn.



203
204
205
206
207
208
209
210
211
212
213
214
215
216
 
# File 'lib/llm/context.rb', line 203

def respond(prompt, params = {})
  @owner = Fiber.current
  compactor.compact!(prompt) if compactor.compact?(prompt)
  params = @params.merge(params)
  prompt, params = transform(prompt, params)
  bind!(params[:stream], params[:model], params[:tools])
  res_id = params[:store] == false ? nil : @messages.find(&:assistant?)&.response&.response_id
  params = params.merge(previous_response_id: res_id, input: @messages.to_a).compact
  res = @llm.responses.create(prompt, params)
  role = params[:role] || @llm.user_role
  @messages.concat LLM::Prompt === prompt ? prompt.to_a : [LLM::Message.new(role, prompt)]
  @messages.concat [res.choices[-1]]
  res
end

#returns ⇒ Array<LLM::Function::Return>

Returns tool returns accumulated in this context

Returns:



275
276
277
278
279
280
281
282
283
 
# File 'lib/llm/context.rb', line 275

def returns
  @messages
    .select(&:tool_return?)
    .flat_map do |msg|
      LLM::Function::Return === msg.content ?
        [msg.content] :
        [*msg.content].grep(LLM::Function::Return)
    end
end

#serialize(path:) ⇒ void Also known as: save

This method returns an undefined value.

Save the current context state

Examples:

llm = LLM.openai(key: ENV["KEY"])
ctx = LLM::Context.new(llm)
ctx.talk "Hello"
ctx.save(path: "context.json")

Raises:

  • (SystemCallError) —

    Might raise a number of SystemCallError subclasses



425
426
427
 
# File 'lib/llm/context.rb', line 425

def serialize(path:)
  ::File.binwrite path, LLM.json.dump(to_h)
end

#spawn(function, strategy) ⇒ LLM::Function::Return, LLM::Function::Task

Spawns a function through the context.

When a guard is configured, this method can return an in-band guarded tool error instead of spawning work.

Parameters:

Returns:



266
267
268
269
270
 
# File 'lib/llm/context.rb', line 266

def spawn(function, strategy)
  warning = guard&.call(self)
  return guarded_return_for(function, warning) if warning
  function.spawn(strategy)
end

#talk(prompt, params = {}) ⇒ LLM::Response Also known as: chat

Interact with the context via the chat completions API. This method immediately sends a request to the LLM and returns the response.

Examples:

llm = LLM.openai(key: ENV["KEY"])
ctx = LLM::Context.new(llm)
res = ctx.talk("Hello, what is your name?")
puts res.messages[0].content

Parameters:

  • params (defaults to: {}) —

    The params, including optional :role (defaults to :user), :stream, :tools, :schema etc.

  • prompt (String) —

    The input prompt to be completed

Returns:

  • (LLM::Response) —

    Returns the LLM’s response for this turn.



173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
 
# File 'lib/llm/context.rb', line 173

def talk(prompt, params = {})
  return respond(prompt, params) if mode == :responses
  @owner = Fiber.current
  compactor.compact!(prompt) if compactor.compact?(prompt)
  params = params.merge(messages: @messages.to_a)
  params = @params.merge(params)
  prompt, params = transform(prompt, params)
  bind!(params[:stream], params[:model], params[:tools])
  res = @llm.complete(prompt, params)
  role = params[:role] || @llm.user_role
  role = @llm.tool_role if params[:role].nil? && [*prompt].grep(LLM::Function::Return).any?
  @messages.concat LLM::Prompt === prompt ? prompt.to_a : [LLM::Message.new(role, prompt)]
  @messages.concat [res.choices[-1]]
  res
end

#to_h ⇒ Hash

Returns:

  • (Hash) 


405
406
407
 
# File 'lib/llm/context.rb', line 405

def to_h
  {schema_version: 1, model:, messages: @messages.map { serialize_message(_1) }}
end

#to_json ⇒ String

Returns:

  • (String) 


411
412
413
 
# File 'lib/llm/context.rb', line 411

def to_json(...)
  to_h.to_json(...)
end

#tracer ⇒ LLM::Tracer

Returns an LLM tracer

Returns:



392
393
394
 
# File 'lib/llm/context.rb', line 392

def tracer
  @llm.tracer
end

#transformer ⇒ #call?

Returns a transformer, if configured.

Transformers can rewrite outgoing prompts and params before a request is sent to the provider.

Returns:



146
147
148
 
# File 'lib/llm/context.rb', line 146

def transformer
  @transformer
end

#transformer=(transformer) ⇒ #call?

Sets a transformer.

Transformers must implement ‘call(ctx, prompt, params)` and return a two-element array of `[prompt, params]`.

Parameters:

  • transformer (#call, nil)

Returns:



158
159
160
 
# File 'lib/llm/context.rb', line 158

def transformer=(transformer)
  @transformer = transformer
end

#usage ⇒ LLM::Object?

Note:

Returns token usage accumulated in this context This method returns token usage for the latest assistant message, and it returns nil for non-assistant messages.

Returns:



328
329
330
331
332
333
334
335
336
337
 
# File 'lib/llm/context.rb', line 328

def usage
  usage = @messages.find(&:assistant?)&.usage
  return unless usage
  LLM::Object.from(
    input_tokens: usage.input_tokens || 0,
    output_tokens: usage.output_tokens || 0,
    reasoning_tokens: usage.reasoning_tokens || 0,
    total_tokens: usage.total_tokens || 0
  )
end

#wait(strategy) ⇒ Array<LLM::Function::Return>

Waits for queued tool work to finish.

This prefers queued streamed tool work when the configured stream exposes a non-empty queue. Otherwise it falls back to waiting on the context’s pending functions directly.

Parameters:

  • strategy (Symbol, Array<Symbol>) —

    The concurrency strategy to use, or the possible concurrency strategies to wait on. For example, ‘[:thread, :ractor]` waits for any queued thread or ractor work, in that order.

Returns:



297
298
299
300
301
302
303
304
305
306
307
308
309
 
# File 'lib/llm/context.rb', line 297

def wait(strategy)
  stream = @params[:stream]
  if LLM::Stream === stream && !stream.queue.empty?
    @queue = stream.queue
    @queue.wait(strategy)
  else
    return guarded_returns if guarded_returns
    @queue = functions.spawn(strategy)
    @queue.wait
  end
ensure
  @queue = nil
end