Class: SparkConnect::WindowSpec

Inherits:
Object
  • Object
show all
Defined in:
lib/spark_connect/window.rb

Overview

Defines the partitioning, ordering, and frame for a window aggregation. Build one with the Window factory and attach it to an analytic column via Column#over.

Examples:

w = SparkConnect::Window.partition_by("dept").order_by(F.col("salary").desc)
df.with_column("rank", F.rank.over(w))

Constant Summary collapse

Proto = 
SparkConnect::Proto
UNBOUNDED_PRECEDING =

Boundary sentinels (matching Spark’s ‘Window.unboundedPreceding`, etc.).

-(2**63)
UNBOUNDED_FOLLOWING = 
(2**63) - 1
CURRENT_ROW = 
0

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(partition_spec: [], order_spec: [], frame_spec: nil) ⇒ WindowSpec

Returns a new instance of WindowSpec.



26
27
28
29
30
 
# File 'lib/spark_connect/window.rb', line 26

def initialize(partition_spec: [], order_spec: [], frame_spec: nil)
  @partition_spec = partition_spec
  @order_spec = order_spec
  @frame_spec = frame_spec
end

Instance Attribute Details

#frame_spec ⇒ Spark::Connect::Expression::Window::WindowFrame? (readonly)

Returns:

  • (Spark::Connect::Expression::Window::WindowFrame, nil) 


24
25
26
 
# File 'lib/spark_connect/window.rb', line 24

def frame_spec
  @frame_spec
end

#order_spec ⇒ Array<Spark::Connect::Expression::SortOrder> (readonly)

Returns:

  • (Array<Spark::Connect::Expression::SortOrder>) 


22
23
24
 
# File 'lib/spark_connect/window.rb', line 22

def order_spec
  @order_spec
end

#partition_spec ⇒ Array<Spark::Connect::Expression> (readonly)

Returns:



20
21
22
 
# File 'lib/spark_connect/window.rb', line 20

def partition_spec
  @partition_spec
end

Instance Method Details

#order_by(*cols) ⇒ WindowSpec

Returns a copy ordered by the given columns.

Returns:

  • (WindowSpec) —

    a copy ordered by the given columns.



38
39
40
 
# File 'lib/spark_connect/window.rb', line 38

def order_by(*cols)
  copy(order_spec: to_sort_orders(cols))
end

#partition_by(*cols) ⇒ WindowSpec

Returns a copy partitioned by the given columns.

Returns:

  • (WindowSpec) —

    a copy partitioned by the given columns.



33
34
35
 
# File 'lib/spark_connect/window.rb', line 33

def partition_by(*cols)
  copy(partition_spec: to_exprs(cols))
end

#range_between(start_, end_) ⇒ WindowSpec

Range-based frame between ‘start` and `end` (value offsets over the ordering).

Returns:



50
51
52
 
# File 'lib/spark_connect/window.rb', line 50

def range_between(start_, end_)
  copy(frame_spec: frame(:FRAME_TYPE_RANGE, start_, end_))
end

#rows_between(start_, end_) ⇒ WindowSpec

Row-based frame between ‘start` and `end` (offsets relative to the current row).

Returns:



44
45
46
 
# File 'lib/spark_connect/window.rb', line 44

def rows_between(start_, end_)
  copy(frame_spec: frame(:FRAME_TYPE_ROW, start_, end_))
end