Class: SkillBench::Execution::Sandbox

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/execution/sandbox.rb

Overview

Manages isolated sandbox environments for running agent evaluations. Handles copying files, initializing git, and capturing diffs.

NOTE: A Docker build context is packaged under execution/docker so gem installs can activate container isolation when a Docker daemon is present. When docker_available? is false (no context, no daemon, or docker missing), container_id stays nil and commands run on the host only if Config.allow_host_execution is enabled (fail closed by default).

Constant Summary collapse

GIT_HARDENING =

Global git options applied to every host-side invocation. They strip the repository's and user's ability to launch external programs during routine git operations on untrusted source:

- core.attributesFile=/dev/null  no user-level .gitattributes drivers
- core.fsmonitor=false           no fsmonitor hook program
- core.hooksPath=/dev/null       no git hooks (pre-commit, etc.)
- core.symlinks=false            symlinks treated as plain files

Combined with not copying the source .git, this neutralizes the .gitattributes/config diff & filter driver code-execution vector.

[
  '-c', 'core.attributesFile=/dev/null',
  '-c', 'core.fsmonitor=false',
  '-c', 'core.hooksPath=/dev/null',
  '-c', 'core.symlinks=false'
].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source_dir) ⇒ Sandbox

Returns a new instance of Sandbox.

Parameters:

  • source_dir (String, Pathname) —

    The directory to copy into the sandbox.



74
75
76
77
78
 
# File 'lib/skill_bench/execution/sandbox.rb', line 74

def initialize(source_dir)
  @source_dir = source_dir
  @path = nil
  @container_id = nil
end

Instance Attribute Details

#container_id ⇒ Object (readonly)

Returns the value of attribute container_id.



19
20
21
 
# File 'lib/skill_bench/execution/sandbox.rb', line 19

def container_id
  @container_id
end

#path ⇒ Object (readonly)

Returns the value of attribute path.



19
20
21
 
# File 'lib/skill_bench/execution/sandbox.rb', line 19

def path
  @path
end

Class Method Details

.capture_diff(sandbox_dir) ⇒ String

Captures the git diff of changes made within the sandbox.

Parameters:

  • sandbox_dir (String) —

    The path to the sandbox directory.

Returns:

  • (String) —

    The git diff, or a message indicating no changes.

Raises:

  • (SystemCallError) —

    when git commands fail.



107
108
109
110
111
112
113
114
115
116
117
118
119
120
 
# File 'lib/skill_bench/execution/sandbox.rb', line 107

def self.capture_diff(sandbox_dir)
  sandbox_path = File.realpath(sandbox_dir)
  tmp_prefix = File.realpath(Dir.tmpdir) + File::SEPARATOR
  raise "Sandbox directory #{sandbox_dir} is outside temp directory" unless sandbox_path.start_with?(tmp_prefix)

  return 'No code changes made.' unless File.directory?(File.join(sandbox_path, '.git'))

  raise "Failed to stage changes in #{sandbox_path}" unless system(*git_command('add', '.'), chdir: sandbox_path)

  diff, status = Open3.capture2(*git_command('diff', '--cached'), chdir: sandbox_path)
  raise "Failed to capture diff in #{sandbox_path}" unless status.success?

  diff.strip.empty? ? 'No code changes made.' : diff
end

.docker_context_path ⇒ String

Absolute path to the packaged Docker build context.

Returns:

  • (String) —

    path to lib/skill_bench/execution/docker



50
51
52
 
# File 'lib/skill_bench/execution/sandbox.rb', line 50

def self.docker_context_path
  Constants::Sandbox.docker_context_path
end

.git_command(*args) ⇒ Array<String>

Builds a hardened git argv: the binary, the hardening flags, then the given subcommand and arguments. Single source of truth so every git call in this file is invoked with the same protections.

Parameters:

  • args (Array<String>) —

    git subcommand and its arguments.

Returns:

  • (Array<String>) —

    full argv beginning with git and the flags.



43
44
45
 
# File 'lib/skill_bench/execution/sandbox.rb', line 43

def self.git_command(*args)
  ['git', *GIT_HARDENING, *args]
end

.image_ref ⇒ String

Versioned Docker image reference for this gem release.

Returns:

  • (String) —

    image:tag such as evaluator-sandbox:1.2.0



57
58
59
 
# File 'lib/skill_bench/execution/sandbox.rb', line 57

def self.image_ref
  Constants::Sandbox.image_ref
end

.run(source_dir) {|sandbox| ... } ⇒ Object

Runs a block of code within a temporary, isolated sandbox directory. The sandbox is initialized as a git repository and optionally wrapped in a Docker container.

Parameters:

  • source_dir (String, Pathname) —

    The directory to copy into the sandbox.

Yield Parameters:

Returns:

  • (Object) —

    The result of the yielded block.

Raises:

  • (SystemCallError) —

    when file operations or directory creation fails.

  • (RuntimeError) —

    when Docker commands fail.



69
70
71
 
# File 'lib/skill_bench/execution/sandbox.rb', line 69

def self.run(source_dir, &)
  new(source_dir).run(&)
end

Instance Method Details

#run {|sandbox| ... } ⇒ Object

Executes the sandbox environment setup and yields the sandbox instance.

Yield Parameters:

Returns:

  • (Object) —

    The result of the yielded block.

Raises:

  • (SystemCallError) —

    when file operations or directory creation fails.

  • (RuntimeError) —

    when Docker commands fail.



86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'lib/skill_bench/execution/sandbox.rb', line 86

def run
  Dir.mktmpdir('evaluator_sandbox_') do |sandbox_dir|
    @path = sandbox_dir
    copy_source_files(sandbox_dir)

    setup_git

    start_container if docker_available?
    begin
      yield self
    ensure
      stop_container
    end
  end
end