Class: SkillBench::Services::ResponseCache

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/services/response_cache.rb

Overview

Content-addressed, in-memory cache for LLM responses.

The cache is opt-in and disabled by default. When enabled it lets repeated, identical LLM requests reuse a previously computed response instead of hitting the network again. The canonical example is compare, which runs the skill-less baseline twice with identical inputs.

The backing store is a process-lifetime Hash keyed by a stable SHA-256 digest of the request, so the same logical request always maps to the same entry regardless of hash-key ordering. Access to the store is serialized with a mutex so concurrent callers (e.g. Parallel-driven agents) cannot corrupt it or double-store a key.

Constant Summary collapse

ENV_FLAG =

Environment variable that opts caching on when set to a truthy value.

'SKILL_BENCH_CACHE'
TRUTHY_VALUES =

Raw env values treated as "on".

%w[1 true yes on].freeze
MUTEX =

Guards every read/write of the shared store. Concurrent agents/judges run on separate threads; without this, the membership check and the write in fetch could interleave and store a key more than once.

Mutex.new

Class Method Summary collapse

Class Method Details

.clear ⇒ void

This method returns an undefined value.

Removes every cached entry.



95
96
97
 
# File 'lib/skill_bench/services/response_cache.rb', line 95

def clear
  MUTEX.synchronize { store.clear }
end

.enabled? ⇒ Boolean

Whether response caching is currently enabled.

Enabled when ENV_FLAG is set to a truthy value (one of TRUTHY_VALUES); disabled when unset or set to anything else.

Returns:

  • (Boolean) —

    true when caching is on



39
40
41
42
 
# File 'lib/skill_bench/services/response_cache.rb', line 39

def enabled?
  raw = ENV.fetch(ENV_FLAG, '').to_s.strip.downcase
  TRUTHY_VALUES.include?(raw)
end

.fetch(key) { ... } ⇒ Object

Returns the cached value for a key, computing and storing it on a miss.

The value is computed outside the lock so requests for distinct keys run concurrently; the store read and the store write are each serialized by MUTEX, and a missing key is written exactly once (first writer wins).

Parameters:

  • key (String) —

    Cache key from key

Yields:

  • Computes the value to cache when the key is absent

Yield Returns:

  • (Object) —

    The value to cache

Returns:

  • (Object) —

    The cached value (existing on a hit, freshly stored on a miss)



84
85
86
87
88
89
90
 
# File 'lib/skill_bench/services/response_cache.rb', line 84

def fetch(key)
  hit = MUTEX.synchronize { store[key] }
  return hit unless hit.nil?

  value = yield
  MUTEX.synchronize { store[key] ||= value }
end

.key(provider:, model:, system_prompt:, messages:, tools: nil, temperature: nil, provider_config: {}) ⇒ String

Computes a stable content-addressed cache key for a request.

The inputs are assembled into a canonical structure (hash keys sorted and stringified recursively) and hashed, so semantically identical requests always produce the same digest. Request-affecting provider configuration (endpoint/base URL/etc.) is included so two providers that share a name but target different endpoints never collide.

Parameters:

  • provider (Symbol, String) —

    Resolved provider identifier

  • model (String, nil) —

    Model name

  • system_prompt (String) —

    System prompt

  • messages (Array<Hash>) —

    Conversation messages

  • tools (Array<Hash>, nil) (defaults to: nil) —

    Tool definitions, when present

  • temperature (Float, nil) (defaults to: nil) —

    Sampling temperature, when present

  • provider_config (Hash) (defaults to: {}) —

    Request-affecting provider settings such as base_url, request_path, endpoint, location, project_id, api_version

Returns:

  • (String) —

    Hex-encoded SHA-256 digest of the canonical request



61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/skill_bench/services/response_cache.rb', line 61

def key(provider:, model:, system_prompt:, messages:, tools: nil, temperature: nil, provider_config: {})
  payload = {
    provider: provider.to_s,
    model: model,
    system_prompt: system_prompt,
    messages: messages,
    tools: tools,
    temperature: temperature,
    provider_config: provider_config
  }
  Digest::SHA256.hexdigest(JSON.generate(canonicalize(payload)))
end