Class: SkillBench::Execution::Sandbox

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/execution/sandbox.rb

Overview

Manages isolated sandbox environments for running agent evaluations. Handles copying files, initializing git, and capturing diffs. Now supports Docker container isolation for secure command execution.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source_dir) ⇒ Sandbox

Returns a new instance of Sandbox.

Parameters:

  • source_dir (String, Pathname) —

    The directory to copy into the sandbox.



28
29
30
31
32
 
# File 'lib/skill_bench/execution/sandbox.rb', line 28

def initialize(source_dir)
  @source_dir = source_dir
  @path = nil
  @container_id = nil
end

Instance Attribute Details

#container_id ⇒ Object (readonly)

Returns the value of attribute container_id.



13
14
15
 
# File 'lib/skill_bench/execution/sandbox.rb', line 13

def container_id
  @container_id
end

#path ⇒ Object (readonly)

Returns the value of attribute path.



13
14
15
 
# File 'lib/skill_bench/execution/sandbox.rb', line 13

def path
  @path
end

Class Method Details

.capture_diff(sandbox_dir) ⇒ String

Captures the git diff of changes made within the sandbox.

Parameters:

  • sandbox_dir (String) —

    The path to the sandbox directory.

Returns:

  • (String) —

    The git diff, or a message indicating no changes.

Raises:

  • (SystemCallError) —

    when git commands fail.



61
62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/skill_bench/execution/sandbox.rb', line 61

def self.capture_diff(sandbox_dir)
  sandbox_path = File.realpath(sandbox_dir)
  tmp_prefix = File.realpath(Dir.tmpdir) + File::SEPARATOR
  raise "Sandbox directory #{sandbox_dir} is outside temp directory" unless sandbox_path.start_with?(tmp_prefix)

  return 'No code changes made.' unless File.directory?(File.join(sandbox_path, '.git'))

  raise "Failed to stage changes in #{sandbox_path}" unless system('git', 'add', '.', chdir: sandbox_path)

  diff, status = Open3.capture2('git', 'diff', '--cached', chdir: sandbox_path)
  raise "Failed to capture diff in #{sandbox_path}" unless status.success?

  diff.strip.empty? ? 'No code changes made.' : diff
end

.run(source_dir) {|sandbox| ... } ⇒ Object

Runs a block of code within a temporary, isolated sandbox directory. The sandbox is initialized as a git repository and optionally wrapped in a Docker container.

Parameters:

  • source_dir (String, Pathname) —

    The directory to copy into the sandbox.

Yield Parameters:

Returns:

  • (Object) —

    The result of the yielded block.

Raises:

  • (SystemCallError) —

    when file operations or directory creation fails.

  • (RuntimeError) —

    when Docker commands fail.



23
24
25
 
# File 'lib/skill_bench/execution/sandbox.rb', line 23

def self.run(source_dir, &)
  new(source_dir).run(&)
end

Instance Method Details

#run {|sandbox| ... } ⇒ Object

Executes the sandbox environment setup and yields the sandbox instance.

Yield Parameters:

Returns:

  • (Object) —

    The result of the yielded block.

Raises:

  • (SystemCallError) —

    when file operations or directory creation fails.

  • (RuntimeError) —

    when Docker commands fail.



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/skill_bench/execution/sandbox.rb', line 40

def run
  Dir.mktmpdir('evaluator_sandbox_') do |sandbox_dir|
    @path = sandbox_dir
    copy_source_files(sandbox_dir)

    setup_git

    start_container if docker_available?
    begin
      yield self
    ensure
      stop_container
    end
  end
end