Class: Brute::Eval::Suite

Inherits:
Object
  • Object
show all
Defined in:
lib/brute/eval/suite.rb

Overview

Runs the cases against one agent and reports what happened.

Brute::Eval::Suite.new(agent: "agent.ru", cases: CASES).run
Brute::Eval::Suite.new(agent: -> { build_agent }, world: Room.new, cases: CASES).run

The agent is built fresh for every attempt -- from its .ru file, or from a block that answers a new pipeline -- so nothing carries from one case to the next but the world, which is laid out again first. A case with runs: above one is run that many times and passes only if every run did: the model is not deterministic, and a case that passes two times in three is a case that fails.

#run answers a process exit status, so an eval script ends exit(...).

Defined Under Namespace

Classes: Result

Instance Method Summary collapse

Constructor Details

#initialize(agent:, cases:, world: World.new, out: $stdout) ⇒ Suite

Returns a new instance of Suite.



26
27
28
29
30
31
 
# File 'lib/brute/eval/suite.rb', line 26

def initialize(agent:, cases:, world: World.new, out: $stdout)
  @agent = agent
  @cases = cases
  @world = world
  @out = out
end

Instance Method Details

#run ⇒ Object 



33
34
35
36
37
38
39
40
41
42
 
# File 'lib/brute/eval/suite.rb', line 33

def run
  results = @cases.flat_map { |kase| attempts(kase) }
  summarise(results)

  if results.all? { |result| result.verdict.passed? }
    0
  else
    1
  end
end