Class: Tep::Llm::OpenAI::ChatCompletionsStreamer

Inherits:
Streamer
  • Object
show all
Defined in:
lib/tep/openai_server.rb

Overview

Runs one streaming chat completion. Subclass of Tep::Streamer. Drives backend.chat_completion_stream through ChatStreamSink, writes the terminating data:, then emits the toy/v1 serving event (kind:eval, phase:serve, name:request) with sink.completion_count (mirrors CompletionsStreamer’s #128 shape).

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ ChatCompletionsStreamer

Returns a new instance of ChatCompletionsStreamer.



464
465
466
467
468
469
470
471
 
# File 'lib/tep/openai_server.rb', line 464

def initialize
  @req_ref       = Tep::Request.new
  @model         = ""
  @prompt_tokens = 0
  @t0            = 0
  @request_id    = ""
  @principal_id  = ""
end

Instance Attribute Details

#model ⇒ Object

Returns the value of attribute model.



461
462
463
 
# File 'lib/tep/openai_server.rb', line 461

def model
  @model
end

#principal_id ⇒ Object

Returns the value of attribute principal_id.



462
463
464
 
# File 'lib/tep/openai_server.rb', line 462

def principal_id
  @principal_id
end

#prompt_tokens ⇒ Object

Returns the value of attribute prompt_tokens.



461
462
463
 
# File 'lib/tep/openai_server.rb', line 461

def prompt_tokens
  @prompt_tokens
end

#req_ref ⇒ Object

Returns the value of attribute req_ref.



461
462
463
 
# File 'lib/tep/openai_server.rb', line 461

def req_ref
  @req_ref
end

#request_id ⇒ Object

Returns the value of attribute request_id.



462
463
464
 
# File 'lib/tep/openai_server.rb', line 462

def request_id
  @request_id
end

#t0 ⇒ Object

Returns the value of attribute t0.



462
463
464
 
# File 'lib/tep/openai_server.rb', line 462

def t0
  @t0
end

Instance Method Details

#pump(out) ⇒ Object 



473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
 
# File 'lib/tep/openai_server.rb', line 473

def pump(out)
  sink = Tep::Llm::OpenAI::ChatStreamSink.new
  sink.out   = out
  sink.model = @model
  sink.emit_role_prelude("assistant")
  Tep::APP.openai_backend.chat_completion_stream(@req_ref, sink)
  sink.emit_finish("stop")
  out.write("data: [DONE]\n\n")
  wall_us = (Time.now.to_i - @t0) * 1_000_000
  extra = "{" +
    Tep::Json.encode_pair_str("request_id", @request_id) + "," +
    Tep::Json.encode_pair_str("principal_id", @principal_id) +
  "}"
  Tep::APP.openai_events.inference(
    @model, @prompt_tokens, sink.completion_count, wall_us, extra)
  0
end