Class: Daytona::ComputerUse::Accessibility

Inherits:
Object
  • Object
show all
Includes:
Instrumentation
Defined in:
lib/daytona/computer_use.rb

Overview

Accessibility operations for computer use functionality.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Instrumentation

included

Constructor Details

#initialize(sandbox_id:, toolbox_api:, otel_state: nil) ⇒ Accessibility

Returns a new instance of Accessibility.

Parameters:

  • sandbox_id (String) —

    The ID of the sandbox

  • toolbox_api (DaytonaToolboxApiClient::ComputerUseApi) —

    API client for sandbox operations

  • otel_state (Daytona::OtelState, nil) (defaults to: nil) 


425
426
427
428
429
 
# File 'lib/daytona/computer_use.rb', line 425

def initialize(sandbox_id:, toolbox_api:, otel_state: nil)
  @sandbox_id = sandbox_id
  @toolbox_api = toolbox_api
  @otel_state = otel_state
end

Instance Attribute Details

#sandbox_id ⇒ String (readonly)

Returns The ID of the sandbox.

Returns:

  • (String) —

    The ID of the sandbox



417
418
419
 
# File 'lib/daytona/computer_use.rb', line 417

def sandbox_id
  @sandbox_id
end

#toolbox_api ⇒ DaytonaToolboxApiClient::ComputerUseApi (readonly)

Returns API client for sandbox operations.

Returns:

  • (DaytonaToolboxApiClient::ComputerUseApi) —

    API client for sandbox operations



420
421
422
 
# File 'lib/daytona/computer_use.rb', line 420

def toolbox_api
  @toolbox_api
end

Instance Method Details

#find_nodes(scope: nil, pid: nil, role: nil, name: nil, name_match: nil, states: nil, limit: nil) ⇒ DaytonaToolboxApiClient::AccessibilityNodesResponse

Finds AT-SPI accessibility nodes matching the provided filters.

Examples:

buttons = sandbox.computer_use.accessibility.find_nodes(
  scope: "all",
  role: "button",
  name: "Submit",
  name_match: "substring"
)
puts buttons.matches.length

Parameters:

  • scope (String, nil) (defaults to: nil) —

    Search scope: “focused”, “pid”, or “all”

  • pid (Integer, nil) (defaults to: nil) —

    Process ID when scope is “pid”

  • role (String, nil) (defaults to: nil) —

    Accessibility role to match, such as “button”

  • name (String, nil) (defaults to: nil) —

    Accessible name to match

  • name_match (String, nil) (defaults to: nil) —

    Name match mode, such as “exact” or “substring”

  • states (Array<String>, nil) (defaults to: nil) —

    Required accessibility states

  • limit (Integer, nil) (defaults to: nil) —

    Maximum number of matches

Returns:

  • (DaytonaToolboxApiClient::AccessibilityNodesResponse) —

    Matching accessibility nodes

Raises:



473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
 
# File 'lib/daytona/computer_use.rb', line 473

def find_nodes(scope: nil, pid: nil, role: nil, name: nil, name_match: nil, states: nil, limit: nil)
  attrs = {}
  attrs[:scope] = scope unless scope.nil?
  attrs[:pid] = pid unless pid.nil?
  attrs[:role] = role unless role.nil?
  attrs[:name] = name unless name.nil?
  attrs[:name_match] = name_match unless name_match.nil?
  attrs[:states] = states unless states.nil?
  attrs[:limit] = limit unless limit.nil?

  request = DaytonaToolboxApiClient::FindAccessibilityNodesRequest.new(attrs)
  toolbox_api.find_accessibility_nodes(request)
rescue StandardError => e
  raise Sdk::Error, "Failed to find accessibility nodes: #{e.message}"
end

#focus_node(id:) ⇒ Object

Focuses an AT-SPI accessibility node.

Examples:

sandbox.computer_use.accessibility.focus_node(id: node.id)

Parameters:

  • id (String) —

    Accessibility node ID returned by get_tree or find_nodes

Raises:



496
497
498
499
500
501
 
# File 'lib/daytona/computer_use.rb', line 496

def focus_node(id:)
  request = DaytonaToolboxApiClient::AccessibilityNodeRequest.new(id:)
  toolbox_api.focus_accessibility_node(request)
rescue StandardError => e
  raise Sdk::Error, "Failed to focus accessibility node: #{e.message}"
end

#get_tree(scope: nil, pid: nil, max_depth: nil) ⇒ DaytonaToolboxApiClient::AccessibilityTreeResponse

Fetches the AT-SPI accessibility tree.

Examples:

tree = sandbox.computer_use.accessibility.get_tree(scope: "all", max_depth: 3)
puts tree.root.name

Parameters:

  • scope (String, nil) (defaults to: nil) —

    Tree scope to inspect: “focused”, “pid”, or “all”

  • pid (Integer, nil) (defaults to: nil) —

    Process ID when scope is “pid”

  • max_depth (Integer, nil) (defaults to: nil) —

    Maximum depth to descend; 0 returns only the root

Returns:

  • (DaytonaToolboxApiClient::AccessibilityTreeResponse) —

    Accessibility tree response

Raises:



442
443
444
445
446
447
448
449
450
451
 
# File 'lib/daytona/computer_use.rb', line 442

def get_tree(scope: nil, pid: nil, max_depth: nil)
  opts = {}
  opts[:scope] = scope unless scope.nil?
  opts[:pid] = pid unless pid.nil?
  opts[:max_depth] = max_depth unless max_depth.nil?

  toolbox_api.get_accessibility_tree(opts)
rescue StandardError => e
  raise Sdk::Error, "Failed to get accessibility tree: #{e.message}"
end

#invoke_node(id:, action: nil) ⇒ Object

Invokes an AT-SPI accessibility node action.

Examples:

sandbox.computer_use.accessibility.invoke_node(id: node.id, action: "click")

Parameters:

  • id (String) —

    Accessibility node ID returned by get_tree or find_nodes

  • action (String, nil) (defaults to: nil) —

    Action name to invoke, or nil for the primary action

Raises:



511
512
513
514
515
516
517
518
519
 
# File 'lib/daytona/computer_use.rb', line 511

def invoke_node(id:, action: nil)
  attrs = { id: }
  attrs[:action] = action unless action.nil?

  request = DaytonaToolboxApiClient::AccessibilityInvokeRequest.new(attrs)
  toolbox_api.invoke_accessibility_node(request)
rescue StandardError => e
  raise Sdk::Error, "Failed to invoke accessibility node: #{e.message}"
end

#set_node_value(id:, value:) ⇒ Object

Sets an AT-SPI accessibility node value.

Examples:

sandbox.computer_use.accessibility.set_node_value(id: node.id, value: "hello")

Parameters:

  • id (String) —

    Accessibility node ID returned by get_tree or find_nodes

  • value (String) —

    Value to write to the node

Raises:



529
530
531
532
533
534
 
# File 'lib/daytona/computer_use.rb', line 529

def set_node_value(id:, value:)
  request = DaytonaToolboxApiClient::AccessibilitySetValueRequest.new(id:, value:)
  toolbox_api.set_accessibility_node_value(request)
rescue StandardError => e
  raise Sdk::Error, "Failed to set accessibility node value: #{e.message}"
end