Module: SparkConnect::PlanBuilder

Defined in:
lib/spark_connect/plan.rb

Overview

Low-level helpers for assembling the protobuf logical plan that the client sends to the server. DataFrame and SparkSession build relations through these helpers so that every relation carries a unique ‘plan_id` (used by the server to resolve columns to a specific subtree, e.g. for self-joins).

Constant Summary collapse

Proto = 
SparkConnect::Proto

Class Method Summary collapse

Class Method Details

.command_plan(command) ⇒ Spark::Connect::Plan

Wrap a command as an executable Spark::Connect::Plan.

Parameters:

Returns:



35
36
37
 
# File 'lib/spark_connect/plan.rb', line 35

def command_plan(command)
  Proto::Plan.new(command: command)
end

.relation(id_source, **rel) ⇒ Spark::Connect::Relation

Wrap a ‘rel_type` oneof keyword into a Spark::Connect::Relation, attaching a fresh `plan_id` from `id_source`.

Parameters:

  • id_source (#next_plan_id) —

    usually a SparkSession.

  • rel (Hash) —

    exactly one ‘rel_type` keyword, e.g. `project:`.

Returns:



19
20
21
 
# File 'lib/spark_connect/plan.rb', line 19

def relation(id_source, **rel)
  Proto::Relation.new(common: Proto::RelationCommon.new(plan_id: id_source.next_plan_id), **rel)
end

.root_plan(relation) ⇒ Spark::Connect::Plan

Wrap a relation as the root of an executable Spark::Connect::Plan.

Parameters:

Returns:



27
28
29
 
# File 'lib/spark_connect/plan.rb', line 27

def root_plan(relation)
  Proto::Plan.new(root: relation)
end