Class: SkillBench::Models::Eval

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/models/eval.rb

Overview

Represents an evaluation scenario

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name:, path:, task: '', criteria: {}, source_code: '', metadata: {}) ⇒ Eval

Returns a new instance of Eval.

Parameters:

  • name (String) —

    Eval name

  • path (String) —

    Path to eval directory

  • task (String) (defaults to: '') —

    Task description from task.md

  • criteria (Hash) (defaults to: {}) —

    Criteria from criteria.json

  • source_code (String) (defaults to: '') —

    Source code to evaluate

  • metadata (Hash) (defaults to: {}) —

    Metadata from metadata.json



19
20
21
22
23
24
25
26
 
# File 'lib/skill_bench/models/eval.rb', line 19

def initialize(name:, path:, task: '', criteria: {}, source_code: '', metadata: {})
  @name = name
  @path = path
  @task = task
  @criteria = criteria
  @source_code = source_code
  @metadata = metadata
end

Instance Attribute Details

#criteria ⇒ Object (readonly)

Returns the value of attribute criteria.



11
12
13
 
# File 'lib/skill_bench/models/eval.rb', line 11

def criteria
  @criteria
end

#metadata ⇒ Object (readonly)

Returns the value of attribute metadata.



11
12
13
 
# File 'lib/skill_bench/models/eval.rb', line 11

def metadata
  @metadata
end

#name ⇒ Object (readonly)

Returns the value of attribute name.



11
12
13
 
# File 'lib/skill_bench/models/eval.rb', line 11

def name
  @name
end

#path ⇒ Object (readonly)

Returns the value of attribute path.



11
12
13
 
# File 'lib/skill_bench/models/eval.rb', line 11

def path
  @path
end

#source_code ⇒ Object (readonly)

Returns the value of attribute source_code.



11
12
13
 
# File 'lib/skill_bench/models/eval.rb', line 11

def source_code
  @source_code
end

#task ⇒ Object (readonly)

Returns the value of attribute task.



11
12
13
 
# File 'lib/skill_bench/models/eval.rb', line 11

def task
  @task
end

Class Method Details

.load(dir_path) ⇒ SkillBench::Models::Eval

Load an eval from a directory

Parameters:

  • dir_path (String) —

    Path to eval directory

Returns:

Raises:

  • (Errno::ENOENT) —

    if eval directory does not exist



32
33
34
35
36
37
38
39
40
41
42
 
# File 'lib/skill_bench/models/eval.rb', line 32

def self.load(dir_path)
  path = Pathname.new(dir_path)
  raise Errno::ENOENT, "Eval directory not found: #{dir_path}" unless path.exist?

  name = path.basename.to_s
  task = load_task(path)
  criteria = load_criteria(path)
  metadata = load_metadata(path)

  new(name: name, path: dir_path, task: task, criteria: criteria, metadata: metadata)
end