Module: Insika::Media::Output

Defined in:
lib/insika/media.rb

Overview

WS9 (saída): generated media. The OUTPUT shape is an additive part — { "type": "image"|"audio", "mime_type": …, "base64": …, "model": … } — that rides the turn's output_parts (terminal event + /v1/responses envelope), NEVER the answer text: the customer's channel consumes the bytes, the model's prose stays the answer.

The GENERATION SEAMS are injectable like the STT seam: each is a ->(content, config) { [ part_hash, usage_hash ] } (part_hash already carries its "type"), specs stub them, and the defaults hit the provider behind lazy requires: · image — RubyLLM.paint (the gem has vision AND painting), billed tokens merged into the turn's usage like any ask; · tts — RubyLLM still has NO speech API (as of 1.16.0), so the default is a thin POST to the OpenAI-compatible <base>/audio/speech endpoint (base + key from the provider config the chat uses — a deployment pointing OpenAI at a gateway keeps TTS pointing there). OpenAI's speech API reports no token usage; the part carries the model so the consumer can price it, and the turn counts the call.

Constant Summary collapse

DEFAULT_IMAGE_SIZE = 
"1024x1024"
DEFAULT_TTS_MODEL = 
"tts-1"
DEFAULT_TTS_VOICE = 
"alloy"
DEFAULT_TTS_FORMAT = 
"mp3"
MAX_EMBEDDED_BYTES =

Base64 inlines into the envelope — a cap so a pathological generation cannot blow up the SSE frame. A generated 1024x1024 PNG sits well under.

8 * 1024 * 1024
MAX_SOURCE_IMAGES =

Edit sources on one paint(with:) call — a fitting room needs a handful of angles, not a gallery.

4

Class Method Summary collapse

Class Method Details

.defaults(context:) ⇒ Object

-> { image: seam, tts: seam } with the DEFAULTS bound to a context (the graph's RubyLLM::Context when it owns credentials — nil = the process-wide RubyLLM constant). Built lazily on first generation so the core loads without ruby_llm (load-guard).



227
228
229
230
231
232
 
# File 'lib/insika/media.rb', line 227

def defaults(context:)
  {
    image: ->(prompt, config) { generate_image(prompt, config: config, context: context) },
    tts: ->(text, config) { synthesize_speech(text, config: config, context: context) }
  }
end

.generate_image(prompt, config:, context:) ⇒ Object

-> [Part, usage]: paint via RubyLLM — text-to-image when the config carries no sources (byte-identical to before this call), image EDITING when it does: source_urls/source_attachments ride paint(with:), mask_url rides paint(mask:). usage is the provider's token counts ({ input_tokens:, output_tokens: } — merged into the turn's usage by the Executor); a provider without counts reports nothing.

Raises:



241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
 
# File 'lib/insika/media.rb', line 241

def generate_image(prompt, config:, context:)
  require "ruby_llm" # lazy — the core loads without it (load-guard)

  cfg = Insika::Coercion.deep_stringify(config || {})
  model = Insika::Coercion.presence(cfg["model"]) || image_model(context)
  # `assume_model_exists` is deliberately NOT passed: RubyLLM raises
  # ArgumentError when it's true without an explicit `provider` (see
  # ModelSelection#assume_model_exists?), and `model` here is a bare
  # ref like `utility_model` elsewhere — the registry resolves it.
  api = context || RubyLLM
  image = api.paint(prompt.to_s, model: model,
                                  size: presence(cfg["size"]) || DEFAULT_IMAGE_SIZE,
                                  with: source_attachments(cfg), mask: mask_attachment(cfg))
  data = image.respond_to?(:data) ? image.data : nil
  raise Insika::MediaError, "image generation returned no embeddable data" if data.to_s.empty?

  enforce_embedded_size!(data, "generated image")
  mime = image.respond_to?(:mime_type) ? image.mime_type : nil
  model_id = image.respond_to?(:model_id) ? image.model_id : nil
  usage = image.respond_to?(:usage) ? token_usage(image.usage) : {}
  part = { "type" => "image", "mime_type" => presence(mime) || "image/png",
           "base64" => data, "model" => presence(model_id) }
  [part.compact, usage]
end

.synthesize_speech(text, config:, context:) ⇒ Object

-> [Part, {}]: synthesize speech via the OpenAI-compatible <base>/audio/speech endpoint. context supplies the base URL + key (the same config the chat uses — see speech_endpoint). The bytes embed base64 in the part; the usage is empty (no token counts on the speech API) and the part carries the model for consumer-side pricing.



271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
 
# File 'lib/insika/media.rb', line 271

def synthesize_speech(text, config:, context:)
  require "net/http"
  require "uri"
  require "json"
  require "base64"

  cfg = Insika::Coercion.deep_stringify(config || {})
  model = presence(cfg["model"]) || DEFAULT_TTS_MODEL
  voice = presence(cfg["voice"]) || DEFAULT_TTS_VOICE
  format = presence(cfg["format"]) || DEFAULT_TTS_FORMAT
  base, key = speech_endpoint(context)
  if key.to_s.empty?
    raise Insika::MediaError,
          "TTS needs an OpenAI API key (provider config) — set it on the " \
          "provider the agent uses, or inject a tts seam"
  end

  uri = URI.parse("#{base}/audio/speech")
  req = Net::HTTP::Post.new(uri)
  req["Authorization"] = "Bearer #{key}"
  req["Content-Type"] = "application/json"
  req.body = JSON.generate(model: model, voice: voice, input: text.to_s,
                           response_format: format)
  opts = { use_ssl: uri.scheme == "https", open_timeout: 30, read_timeout: 60 }
  bytes = Net::HTTP.start(uri.host, uri.port, opts) do |http|
    resp = http.request(req)
    raise Insika::MediaError, "TTS HTTP #{resp.code}" unless resp.is_a?(Net::HTTPSuccess)

    # stream into the cap — a rogue/broken endpoint must not grow the
    # process past MAX_EMBEDDED_BYTES before the refusal.
    buf = +"".b
    resp.read_body do |chunk|
      buf << chunk
      break if buf.bytesize > MAX_EMBEDDED_BYTES
    end
    buf
  end
  enforce_embedded_size!(bytes, "synthesized speech")
  part = { "type" => "audio", "mime_type" => mime_for(format), "base64" => Base64.strict_encode64(bytes), "model" => model }
  [part.compact, {}]
rescue URI::InvalidURIError
  raise Insika::MediaError, "invalid TTS endpoint"
end