Class: Terret::LLM::Service

Inherits:
Hames::Service
  • Object
show all
Defined in:
lib/terret/llm.rb

Overview

ctx.llm — the adapter seam. llm/stream is a waterfall wrapping every request: middleware may rewrite the request or replace the stream.

Instance Method Summary collapse

Instance Method Details

#reconfigure(config) ⇒ Object

The role map is the whole of this service's config, and layering replaces a row's config wholesale — so does this. Registered adapters are runtime state, not config, and survive untouched.



100
101
102
 
# File 'lib/terret/llm.rb', line 100

def reconfigure(config)
  @roles = (config[:roles] || {}).dup
end

#register_adapter(name, adapter) ⇒ Object 



104
105
106
107
 
# File 'lib/terret/llm.rb', line 104

def register_adapter(name, adapter)
  @adapters[name.to_s] = adapter
  -> { @adapters.delete(name.to_s) }
end

#resolve(role) ⇒ Object 



109
110
111
112
113
 
# File 'lib/terret/llm.rb', line 109

def resolve(role)
  spec = @roles.fetch(role) { raise KeyError, "no model role #{role.inspect}" }
  provider, model = spec.split("/", 2)
  [@adapters.fetch(provider), model]
end

#set_role(role, spec) ⇒ Object

§9.2 set_model lands here: repoint a role at a "provider/model" spec on the live service. Takes effect at the next step's resolve.

Raises:

  • (ArgumentError) 


117
118
119
120
121
122
123
124
125
126
127
 
# File 'lib/terret/llm.rb', line 117

def set_role(role, spec)
  raise ArgumentError, "role must be a name, got #{role.inspect}" unless role.respond_to?(:to_sym)

  provider, model = spec.to_s.split("/", 2)
  if provider.nil? || provider.empty? || model.nil? || model.empty?
    raise ArgumentError, "model spec must be provider/model, got #{spec.inspect}"
  end

  @adapters.fetch(provider) { raise ArgumentError, "unknown provider #{provider.inspect}" }
  @roles[role.to_sym] = spec.to_s
end

#start(_ctx) ⇒ Object 



92
93
94
95
 
# File 'lib/terret/llm.rb', line 92

def start(_ctx)
  @adapters = {}
  @roles    = (config[:roles] || {}).dup # :main => "fake/scripted"
end

#stream(ctx, role:, request:, &on_event) ⇒ Object

Streams the request through the llm/stream waterfall; the base of the waterfall invokes the resolved adapter. Yields StreamEvents, returns the final assistant Message.



132
133
134
135
136
137
138
 
# File 'lib/terret/llm.rb', line 132

def stream(ctx, role:, request:, &on_event)
  adapter, model = resolve(role)
  req = request.with(model: model)
  ctx.waterfall("llm/stream", req) do |final_req|
    adapter.stream(final_req, &on_event)
  end
end