Class: RubynCode::Observability::SkillAnalytics

Inherits:
Object
  • Object
show all
Defined in:
lib/rubyn_code/observability/skill_analytics.rb

Overview

Tracks per-skill usage and ROI metrics. Records when skills are loaded, how long they stay in context, whether suggestions from them are accepted, and their token cost. Enables monthly pruning of low-usage skills.

Defined Under Namespace

Classes: Entry

Constant Summary collapse

TABLE_NAME = 
'skill_usage'

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(db: nil) ⇒ SkillAnalytics

Returns a new instance of SkillAnalytics.



21
22
23
24
 
# File 'lib/rubyn_code/observability/skill_analytics.rb', line 21

def initialize(db: nil)
  @db = db
  @entries = []
end

Instance Attribute Details

#entries ⇒ Object (readonly)

Returns the value of attribute entries.



19
20
21
 
# File 'lib/rubyn_code/observability/skill_analytics.rb', line 19

def entries
  @entries
end

Instance Method Details

#low_usage_skills(threshold: 0.05) ⇒ Object

Returns skills with usage rate below threshold (candidates for pruning).



58
59
60
61
62
63
64
65
66
 
# File 'lib/rubyn_code/observability/skill_analytics.rb', line 58

def low_usage_skills(threshold: 0.05)
  stats = usage_stats
  total = @entries.size.to_f
  return [] if total.zero?

  stats.select do |_, s|
    (s[:load_count] / total) < threshold
  end.keys
end

#record(skill_name:, loaded_at_turn:, last_referenced_turn: nil, tokens_cost: 0, accepted: nil) ⇒ Object

Record a skill usage event.



27
28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/rubyn_code/observability/skill_analytics.rb', line 27

def record(skill_name:, loaded_at_turn:, last_referenced_turn: nil, tokens_cost: 0, accepted: nil)
  entry = Entry.new(
    skill_name: skill_name.to_s,
    loaded_at_turn: loaded_at_turn,
    last_referenced_turn: last_referenced_turn || loaded_at_turn,
    tokens_cost: tokens_cost.to_i,
    accepted: accepted,
    session_id: nil
  )
  @entries << entry
  persist(entry) if @db
  entry
end

#report ⇒ Object

Format a report for the /cost command.



79
80
81
82
83
84
85
86
87
88
 
# File 'lib/rubyn_code/observability/skill_analytics.rb', line 79

def report
  stats = usage_stats
  return 'No skill usage data.' if stats.empty?

  lines = ['Skill Usage:']
  stats.each do |name, s|
    lines << "  #{name}: #{s[:load_count]}x loaded, #{s[:total_tokens]} tokens"
  end
  lines.join("\n")
end

#roi_ranking ⇒ Object

Returns skills sorted by ROI (accepted suggestions per token spent).



69
70
71
72
73
74
75
76
 
# File 'lib/rubyn_code/observability/skill_analytics.rb', line 69

def roi_ranking
  stats = usage_stats
  stats.sort_by do |_, s|
    tokens = s[:total_tokens]
    rate = s[:acceptance_rate] || 0
    tokens.positive? ? -(rate / tokens) : 0
  end.map(&:first)
end

#usage_stats ⇒ Object

Calculate usage statistics across all recorded entries.



42
43
44
45
46
47
48
49
50
51
52
53
54
55
 
# File 'lib/rubyn_code/observability/skill_analytics.rb', line 42

def usage_stats
  return {} if @entries.empty?

  by_skill = @entries.group_by(&:skill_name)
  by_skill.transform_values do |entries|
    {
      load_count: entries.size,
      total_tokens: entries.sum(&:tokens_cost),
      avg_tokens: (entries.sum(&:tokens_cost).to_f / entries.size).round(0),
      acceptance_rate: acceptance_rate(entries),
      avg_lifespan: avg_lifespan(entries)
    }
  end
end