Class: RubyPi::Context::Compaction

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby_pi/context/compaction.rb

Overview

Manages context window size by summarizing older messages when the estimated token count exceeds a configurable threshold. The most recent N messages are always preserved to maintain conversational coherence.

Examples:

Configuring compaction

compaction = RubyPi::Context::Compaction.new(
  max_tokens: 8000,
  summary_model: model,
  preserve_last_n: 4
)
compacted = compaction.compact(messages, system_prompt)

Constant Summary collapse

CHARS_PER_TOKEN =

Average characters per token — a rough heuristic that avoids the need for provider-specific tokenizers. Errs on the conservative side.

4
SUMMARY_METADATA_LIMIT = 
16_384

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(max_tokens: 8000, summary_model:, preserve_last_n: 4) ⇒ Compaction

Creates a new Compaction instance.

Parameters:

  • max_tokens (Integer) (defaults to: 8000) —

    trigger compaction above this token estimate (default: 8000)

  • summary_model (RubyPi::LLM::BaseProvider) —

    model for summarization

  • preserve_last_n (Integer) (defaults to: 4) —

    always keep the last N messages (default: 4)



54
55
56
57
58
59
 
# File 'lib/ruby_pi/context/compaction.rb', line 54

def initialize(max_tokens: 8000, summary_model:, preserve_last_n: 4)
  @max_tokens = max_tokens
  @summary_model = summary_model
  @preserve_last_n = preserve_last_n
  @emitter = nil
end

Instance Attribute Details

#emitter ⇒ #emit?

Returns optional event emitter for :compaction events.

Returns:

  • (#emit, nil) —

    optional event emitter for :compaction events



45
46
47
 
# File 'lib/ruby_pi/context/compaction.rb', line 45

def emitter
  @emitter
end

#max_tokens ⇒ Integer (readonly)

Returns the token threshold above which compaction triggers.

Returns:

  • (Integer) —

    the token threshold above which compaction triggers



36
37
38
 
# File 'lib/ruby_pi/context/compaction.rb', line 36

def max_tokens
  @max_tokens
end

#preserve_last_n ⇒ Integer (readonly)

Returns number of recent messages always preserved.

Returns:

  • (Integer) —

    number of recent messages always preserved



42
43
44
 
# File 'lib/ruby_pi/context/compaction.rb', line 42

def preserve_last_n
  @preserve_last_n
end

#summary_model ⇒ RubyPi::LLM::BaseProvider (readonly)

Returns the model used to generate summaries.

Returns:



39
40
41
 
# File 'lib/ruby_pi/context/compaction.rb', line 39

def summary_model
  @summary_model
end

Instance Method Details

#build_compacted_history(summary, preserved) ⇒ Array<Hash>

Builds the compacted history: a summary message followed by the preserved tail.

The summary becomes the FIRST message of the compacted history, so it must satisfy the strictest provider constraints (Anthropic):

1. The summary role MUST NOT be :system — that would overwrite the
 real system prompt on Anthropic, which promotes the last :system
 message to the top-level `system:` parameter.
2. The first message MUST use role :user.
3. Consecutive same-role messages are rejected.

A :user summary satisfies (1) and (2). For (3): the orphan-strip above guarantees the first preserved message is :assistant, :user, or absent (never :tool). When it is :assistant or absent, a standalone :user summary alternates correctly. When it is :user, a separate :user summary would create two consecutive user messages, so we instead merge the summary text into that existing user message — keeping the first message a single :user message with no role collision.

Parameters:

  • summary (String) —

    the generated summary text

  • preserved (Array<Hash>) —

    the preserved tail of messages

Returns:

  • (Array<Hash>) —

    the compacted history



141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/ruby_pi/context/compaction.rb', line 141

def build_compacted_history(summary, preserved)
  summary_text = "[Conversation Summary]\n#{summary}"
  first_preserved = preserved.first

  if first_preserved && message_value(first_preserved, :role).to_s == "user"
    merged = first_preserved.dup
    content_key = first_preserved.key?(:content) ? :content : "content"
    merged[content_key] = "#{summary_text}\n\n#{message_value(first_preserved, :content)}"
    [merged] + preserved.drop(1)
  else
    [{ role: :user, content: summary_text }] + preserved
  end
end

#compact(messages, system_prompt) ⇒ Array<Hash>?

Compacts the message history if the estimated token count exceeds the threshold. Returns the compacted messages array, or nil if no compaction was needed.

The compaction process:

  1. Estimate total tokens for system_prompt + all messages.
  2. If under threshold, return nil (no compaction needed).
  3. Split messages into "droppable" (older) and "preserved" (recent).
  4. Summarize the droppable messages via the summary model.
  5. Return a new array: [summary_message] + preserved_messages.

Parameters:

  • messages (Array<Hash>) —

    the current conversation history

  • system_prompt (String) —

    the system prompt (included in estimate)

Returns:

  • (Array<Hash>, nil) —

    compacted messages, or nil if not needed



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/ruby_pi/context/compaction.rb', line 75

def compact(messages, system_prompt)
  total_tokens = estimate_tokens(system_prompt, messages)
  return nil if total_tokens <= @max_tokens

  # Split into messages to summarize and messages to keep
  preserved_count = [@preserve_last_n, messages.size].min
  droppable = messages[0...(messages.size - preserved_count)].dup
  preserved = messages[(messages.size - preserved_count)..].dup

  # If there's nothing to drop, we can't compact further
  return nil if droppable.empty?

  # Anthropic and OpenAI both require every tool_result / tool message
  # to reference a tool_use / tool_call from a preceding assistant
  # message. If we summarize the assistant turn that originated a tool
  # call but keep the matching tool_result, the API rejects the
  # request with "tool_result without preceding tool_use".
  #
  # When the boundary between droppable and preserved cuts mid-exchange,
  # preserved can start with one or more orphan :tool messages whose
  # matching assistant turn is in droppable. Strip those off the head of
  # preserved and move them into droppable so they are summarized away
  # rather than sent. Because the originating assistant message is older,
  # it is already in droppable, so the pair stays together there — there
  # is no mirror case to handle (once a tool result is moved across, its
  # assistant is never left stranded on the preserved side).
  while preserved.first && message_value(preserved.first, :role).to_s == "tool"
    droppable << preserved.shift
  end

  # The orphan-strip only moves messages INTO droppable, so droppable
  # cannot have shrunk; it is still non-empty here. preserved, however,
  # may now be empty (the whole window was tool results) — the summary
  # construction below handles that case.

  # Generate a summary of the dropped messages
  summary = summarize(droppable)

  # Emit compaction event if an emitter is available
  @emitter&.emit(:compaction, dropped_count: droppable.size, summary: summary)

  build_compacted_history(summary, preserved)
end

#estimate_tokens(system_prompt, messages) ⇒ Integer

Estimates the total token count for a system prompt and message array using the character-based heuristic.

Parameters:

  • system_prompt (String) —

    the system prompt text

  • messages (Array<Hash>) —

    conversation messages

Returns:

  • (Integer) —

    estimated token count



161
162
163
164
165
166
167
168
169
170
171
172
173
 
# File 'lib/ruby_pi/context/compaction.rb', line 161

def estimate_tokens(system_prompt, messages)
  total_chars = system_prompt.to_s.length

  messages.each do |msg|
    total_chars += message_value(msg, :content).to_s.length
    metadata = msg.reject { |key, _| key == :role || key == "role" || key == :content || key == "content" }
    total_chars += serialized_message(metadata).length unless metadata.empty?
    # Account for role and structural overhead (~10 tokens per message)
    total_chars += 40
  end

  (total_chars.to_f / CHARS_PER_TOKEN).ceil
end