Module: Vangrail::Providers::Llmlite

Defined in:
lib/vangrail/providers/llmlite.rb

Overview

llmlite: a local OpenAI-compatible proxy, and the default this gem builds around.

It is the right default for guardrails specifically. Rails run on every turn, so their latency and their failure modes are the application's; a local endpoint keeps both on this machine, needs no shared credential, and cannot bill anyone. It also means a laptop with the proxy running has working rails with nothing configured.

The proxy serves an instruct model, not a safety classifier, so model(:guard) is nil and the builder puts a policy rail on the input side. That is a real difference between endpoints, and it belongs here rather than in a rail deciding what it is talking to.

Constant Summary collapse

HOST = 
'127.0.0.1'
DEFAULT_PORT = 
8760
DEFAULT_KEY = 
'grok-inside'
DEFAULT_MODEL = 
'grok-4.5'

Class Method Summary collapse

Class Method Details

.base_url(env = ENV) ⇒ Object 



37
38
39
 
# File 'lib/vangrail/providers/llmlite.rb', line 37

def base_url(env = ENV)
  "http://#{host(env)}:#{port(env)}/v1"
end

.host(env = ENV) ⇒ Object 



33
34
35
 
# File 'lib/vangrail/providers/llmlite.rb', line 33

def host(env = ENV)
  env['LLMLITE_HOST'].to_s.strip.empty? ? HOST : env['LLMLITE_HOST'].strip
end

.key(env = ENV) ⇒ Object 



55
56
57
 
# File 'lib/vangrail/providers/llmlite.rb', line 55

def key(env = ENV)
  env['LLMLITE_API_KEY'] || DEFAULT_KEY
end

.listening?(env = ENV) ⇒ Boolean

A TCP connect, not a request: a proxy that is not running is the common case, and finding that out must cost microseconds rather than a timeout.

Returns:

  • (Boolean) 


43
44
45
46
47
48
49
 
# File 'lib/vangrail/providers/llmlite.rb', line 43

def listening?(env = ENV)
  socket = TCPSocket.new(host(env), port(env))
  socket.close
  true
rescue StandardError
  false
end

.model(env = ENV) ⇒ Object 



51
52
53
 
# File 'lib/vangrail/providers/llmlite.rb', line 51

def model(env = ENV)
  env['LLMLITE_MODEL'] || env['GROK_LLMLITE_MODEL'] || DEFAULT_MODEL
end

.port(env = ENV) ⇒ Object 



29
30
31
 
# File 'lib/vangrail/providers/llmlite.rb', line 29

def port(env = ENV)
  (env['LLMLITE_PORT'] || env['GROK_SHIM_PORT'] || DEFAULT_PORT).to_i
end

.provider(env = ENV) ⇒ Object 



59
60
61
62
63
64
65
66
67
68
 
# File 'lib/vangrail/providers/llmlite.rb', line 59

def provider(env = ENV)
  Provider.new(
    name: 'llmlite',
    base_url: base_url(env),
    models: { judge: model(env), guard: nil },
    key_resolver: -> { key(env) },
    local: true,
    probe: -> { listening?(env) }
  )
end