Class: Btape::LLM::Client

Inherits:
Object
  • Object
show all
Defined in:
lib/btape/llm/client.rb

Overview

A chat client for an OpenAI-compatible server. Nothing here knows which one it is talking to: LM Studio, Ollama, llama.cpp's server and vLLM all answer /v1/chat/completions in the same shape, so pointing --llm-url at one of them is the whole of the configuration.

The defaults assume the model is running on this machine, where there is usually no key to send and no reason for the request to leave it.

Constant Summary collapse

DEFAULT_BASE_URL =

LM Studio's; Ollama serves the same API on 11434.

'http://localhost:1234/v1'
DEFAULT_TEMPERATURE = 
0.2
DEFAULT_TIMEOUT =

A local model on a CPU answers in tens of seconds rather than the hundreds of milliseconds a hosted one would take.

300
ERROR_BODY_LIMIT =

How much of a server's error body to quote back. Enough to name the problem, not so much that a stack trace fills the terminal.

500

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(base_url: nil, model: nil, api_key: nil, temperature: nil, timeout: nil) ⇒ Client

Returns a new instance of Client.



36
37
38
39
40
41
42
 
# File 'lib/btape/llm/client.rb', line 36

def initialize(base_url: nil, model: nil, api_key: nil, temperature: nil, timeout: nil)
  @base_url = (base_url || ENV.fetch('BTAPE_LLM_URL', DEFAULT_BASE_URL)).chomp('/')
  @model = model || ENV.fetch('BTAPE_LLM_MODEL', nil)
  @api_key = api_key || ENV.fetch('BTAPE_LLM_KEY', nil)
  @temperature = temperature || DEFAULT_TEMPERATURE
  @timeout = timeout || DEFAULT_TIMEOUT
end

Instance Attribute Details

#base_url ⇒ Object (readonly)

Returns the value of attribute base_url.



34
35
36
 
# File 'lib/btape/llm/client.rb', line 34

def base_url
  @base_url
end

Instance Method Details

#complete(messages) ⇒ Object

Sends the conversation and returns the reply's text.

Raises:



45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/btape/llm/client.rb', line 45

def complete(messages)
  payload = { model: model, messages: messages, temperature: @temperature, stream: false }
  body = post('/chat/completions', payload)
  content = body.dig('choices', 0, 'message', 'content')
  # Not every server answers with a String there: some hand back the
  # content as a list of parts, and a reasoning model may answer with
  # nothing but its thoughts. Both are this client's error to report,
  # rather than a NoMethodError from inside it.
  raise Error, "#{@base_url} answered without a message" unless content.is_a?(String) && !content.strip.empty?

  content
end

#model ⇒ Object

The model to ask for. A server hosting one model still wants to be told which, and the name differs with every download — so when nobody has said, the loaded one is asked for by name.



61
62
63
 
# File 'lib/btape/llm/client.rb', line 61

def model
  @model ||= first_loaded_model
end