Module: Insika::Media::Output

Defined in:
lib/insika/media.rb

Overview

WS9 (saída): generated media. The OUTPUT shape is an additive part — { "type": "image"|"audio", "mime_type": …, "base64": …, "model": … } — that rides the turn's output_parts (terminal event + /v1/responses envelope), NEVER the answer text: the customer's channel consumes the bytes, the model's prose stays the answer.

The GENERATION SEAMS are injectable like the STT seam: each is a ->(content, config) { [ part_hash, usage_hash ] } (part_hash already carries its "type"), specs stub them, and the defaults hit the provider behind lazy requires: · image — RubyLLM.paint (the gem has vision AND painting), billed tokens merged into the turn's usage like any ask; · tts — RubyLLM still has NO speech API (as of 1.16.0), so the default is a thin POST to the OpenAI-compatible <base>/audio/speech endpoint (base + key from the provider config the chat uses — a deployment pointing OpenAI at a gateway keeps TTS pointing there). OpenAI's speech API reports no token usage; the part carries the model so the consumer can price it, and the turn counts the call.

Constant Summary collapse

DEFAULT_IMAGE_SIZE = 
"1024x1024"
DEFAULT_TTS_MODEL = 
"tts-1"
DEFAULT_TTS_VOICE = 
"alloy"
DEFAULT_TTS_FORMAT = 
"mp3"
MAX_EMBEDDED_BYTES =

Base64 inlines into the envelope — a cap so a pathological generation cannot blow up the SSE frame. A generated 1024x1024 PNG sits well under.

8 * 1024 * 1024

Class Method Summary collapse

Class Method Details

.defaults(context:) ⇒ Object

-> { image: seam, tts: seam } with the DEFAULTS bound to a context (the graph's RubyLLM::Context when it owns credentials — nil = the process-wide RubyLLM constant). Built lazily on first generation so the core loads without ruby_llm (load-guard).



175
176
177
178
179
180
 
# File 'lib/insika/media.rb', line 175

def defaults(context:)
  {
    image: ->(prompt, config) { generate_image(prompt, config: config, context: context) },
    tts: ->(text, config) { synthesize_speech(text, config: config, context: context) }
  }
end

.generate_image(prompt, config:, context:) ⇒ Object

-> [Part, usage]: paint via RubyLLM. usage is the provider's token counts ({ input_tokens:, output_tokens: } — merged into the turn's usage by the Executor); a provider without counts reports nothing.

Raises:



185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
 
# File 'lib/insika/media.rb', line 185

def generate_image(prompt, config:, context:)
  require "ruby_llm" # lazy — the core loads without it (load-guard)

  cfg = Insika::Coercion.deep_stringify(config || {})
  model = Insika::Coercion.presence(cfg["model"]) || image_model(context)
  api = context || RubyLLM
  image = api.paint(prompt.to_s, model: model, assume_model_exists: true,
                                  size: presence(cfg["size"]) || DEFAULT_IMAGE_SIZE)
  data = image.respond_to?(:data) ? image.data : nil
  raise Insika::MediaError, "image generation returned no embeddable data" if data.to_s.empty?

  enforce_embedded_size!(data, "generated image")
  mime = image.respond_to?(:mime_type) ? image.mime_type : nil
  model_id = image.respond_to?(:model_id) ? image.model_id : nil
  usage = image.respond_to?(:usage) ? token_usage(image.usage) : {}
  part = { "type" => "image", "mime_type" => presence(mime) || "image/png",
           "base64" => data, "model" => presence(model_id) }
  [part.compact, usage]
end

.synthesize_speech(text, config:, context:) ⇒ Object

-> [Part, {}]: synthesize speech via the OpenAI-compatible <base>/audio/speech endpoint. context supplies the base URL + key (the same config the chat uses — see speech_endpoint). The bytes embed base64 in the part; the usage is empty (no token counts on the speech API) and the part carries the model for consumer-side pricing.



210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
 
# File 'lib/insika/media.rb', line 210

def synthesize_speech(text, config:, context:)
  require "net/http"
  require "uri"
  require "json"
  require "base64"

  cfg = Insika::Coercion.deep_stringify(config || {})
  model = presence(cfg["model"]) || DEFAULT_TTS_MODEL
  voice = presence(cfg["voice"]) || DEFAULT_TTS_VOICE
  format = presence(cfg["format"]) || DEFAULT_TTS_FORMAT
  base, key = speech_endpoint(context)
  if key.to_s.empty?
    raise Insika::MediaError,
          "TTS needs an OpenAI API key (provider config) — set it on the " \
          "provider the agent uses, or inject a tts seam"
  end

  uri = URI.parse("#{base}/audio/speech")
  req = Net::HTTP::Post.new(uri)
  req["Authorization"] = "Bearer #{key}"
  req["Content-Type"] = "application/json"
  req.body = JSON.generate(model: model, voice: voice, input: text.to_s,
                           response_format: format)
  opts = { use_ssl: uri.scheme == "https", open_timeout: 30, read_timeout: 60 }
  bytes = Net::HTTP.start(uri.host, uri.port, opts) do |http|
    resp = http.request(req)
    raise Insika::MediaError, "TTS HTTP #{resp.code}" unless resp.is_a?(Net::HTTPSuccess)

    # stream into the cap — a rogue/broken endpoint must not grow the
    # process past MAX_EMBEDDED_BYTES before the refusal.
    buf = +"".b
    resp.read_body do |chunk|
      buf << chunk
      break if buf.bytesize > MAX_EMBEDDED_BYTES
    end
    buf
  end
  enforce_embedded_size!(bytes, "synthesized speech")
  part = { "type" => "audio", "mime_type" => mime_for(format), "base64" => Base64.strict_encode64(bytes), "model" => model }
  [part.compact, {}]
rescue URI::InvalidURIError
  raise Insika::MediaError, "invalid TTS endpoint"
end