Class: CAFrame

Inherits:
Object
  • Object
show all
Defined in:
lib/carray/frame/join.rb,
lib/carray/frame/io.rb,
lib/carray/frame/sort.rb,
lib/carray/frame/frame.rb,
lib/carray/frame/group.rb,
lib/carray/frame/verbs.rb,
lib/carray/frame/concat.rb,
lib/carray/frame/convert.rb,
lib/carray/frame/records.rb,
lib/carray/frame/csv_parser.rb

Overview

CAFrame record (row-oriented) input (memo §11.2, §11.5).

from_records takes an Array of row Hashes -- the shape JSON.parse yields for a JSON array of objects -- and arranges it into columns. Unlike from_csv, the cell values are already typed Ruby objects (Float / Integer / DateTime / String), so a homogeneous column is built at its native leaf type rather than left as strings ("arrange by the value's own type", not string inference -- §4.2 stays intact: no date-like string is parsed, mixed columns stay object).

An Array-valued cell that is the same length across every record becomes an N-D column (§11.5): { "temp" => [min, mean, max] } over N records is one (N, 3) column. Ragged or non-numeric arrays fall back to an object column.

Missing keys and explicit nils become UNDEF for numeric columns (mask, not a float promotion): an int column with a hole stays int + UNDEF.

Defined Under Namespace

Modules: CSVParser Classes: CSVReader

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(columns = {}, **opts) ⇒ CAFrame

Build a frame from a Hash of +name => column+. Columns may be CArrays or anything that answers +to_ca+ (Ruby Array, lazy view). All columns must share axis-0 length N.

The column hash may be passed either explicitly (+CAFrame.new(hash, axis_name: ...)+) or as bare inline pairs (+CAFrame.new("a" => x, "b" => y)+). +:axis_name+ / +:index+ are control options; any remaining (string-keyed) options are treated as columns.

Column names are normalized to Strings, so a Symbol key in an explicit column hash is stringified. The Symbol rejection below applies only to the keyword channel, which doubles as the +:axis_name+ / +:index+ control-option channel: a stray Symbol there is a mistyped control option, not a column.



35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/carray/frame/frame.rb', line 35

def initialize(columns = {}, **opts)
  axis_name = opts.delete(:axis_name)
  index     = opts.delete(:index)
  stray = opts.keys.reject { |k| k.is_a?(String) }
  unless stray.empty?
    raise ArgumentError,
          "column keys must be Strings (Symbols are reserved); got #{stray.inspect}"
  end
  columns = columns.merge(opts) unless opts.empty?

  @columns   = {}
  @axis_name = axis_name || DEFAULT_AXIS_NAME
  @index     = nil

  n = nil
  columns.each do |name, col|
    key = name.to_s
    ca  = coerce_column(col)
    len = ca.shape[0]
    if n.nil?
      n = len
    elsif len != n
      raise ArgumentError,
            "column #{key.inspect} has axis-0 length #{len}, expected #{n}"
    end
    @columns[key] = ca
  end
  @nrow = n || 0

  if index
    idx = coerce_column(index)
    unless idx.ndim == 1
      raise ArgumentError, "index must be a 1-D column (got ndim #{idx.ndim})"
    end
    if n && idx.shape[0] != n
      raise ArgumentError,
            "index length #{idx.shape[0]} does not match nrow #{n}"
    end
    @index = idx
    @nrow  = idx.shape[0] if n.nil?
  end

  # With an index, the row axis is named after it; a column of the same name
  # would shadow the index in +row+ and +reset_index+. Reject the collision at
  # construction so those paths never silently drop one for the other.
  if @index && @columns.key?(@axis_name)
    raise ArgumentError,
          "axis_name #{@axis_name.inspect} collides with a column of the same name"
  end
end

Instance Attribute Details

#axis_name ⇒ Object (readonly)

Row-axis name.



427
428
429
 
# File 'lib/carray/frame/frame.rb', line 427

def axis_name
  @axis_name
end

#index ⇒ Object (readonly)

Index column (CArray) or nil.



430
431
432
 
# File 'lib/carray/frame/frame.rb', line 430

def index
  @index
end

#nrow ⇒ Object (readonly)

Number of rows (axis-0 length N).



424
425
426
 
# File 'lib/carray/frame/frame.rb', line 424

def nrow
  @nrow
end

Class Method Details

.concatenate(*frames) ⇒ Object

Concatenate frames along the row axis, eagerly. Each output column is +CArray.concatenate+ of that column across the input frames, so per-column dtypes auto-promote to a common type. The result is a fresh, independent frame — writes to it do not propagate back to the input frames.

For a view frame that shares storage with the inputs (strict same dtype per column, chain composability preserved) use meld.

Column matching, index handling, and column-set / index-mix rules match meld.

CAFrame.concatenate(jan, feb, mar) # eager, independent result CAFrame.concatenate([jan, feb, mar]) # an Array is accepted too



62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/carray/frame/concat.rb', line 62

def self.concatenate(*frames)
  frames = frames.flatten
  check_concat_inputs(frames, verb: "concatenate")
  first = frames.first
  names = first.variable_names
  check_column_sets(frames, names, verb: "concatenate")
  cols = {}
  names.each do |name|
    cols[name] = CArray.concatenate(frames.map { |f| f[name] })
  end
  new(cols, axis_name: first.axis_name, index: concatenate_index(frames))
end

.from_csv(path, types: nil, sep: ",", quote: '"', strip: false, encoding: "bom|utf-8", parser: nil, &block) ⇒ Object

Read a CSV into a frame. The header row supplies column names (Strings, §3.7); every column is built raw as an object CArray of the cell strings (§4.2 -- read and arrange only, no type-inference engine). Casting is a separate step: pass +types:+ ({ "temp" => :float64 }, or the array-key / reverse forms of +cast+) to cast named columns on load, or call +cast+ later. Broken cells fail to_type and become UNDEF automatically (parse-mask, §6-2).

Parsing uses the built-in fast tokenizer (CSVParser). Options: sep: field separator (default ",") quote: quote character (default '"') strip: trim spaces from unquoted fields (default false, RFC spacing) +encoding+: IO open-mode encoding (default "bom|utf-8", strips a BOM) parser: a callable path -> [headers, rows] to inject another parser (e.g. the stdlib +csv+, or a typed-table source); when given, sep/quote/strip/encoding and any block are that parser's concern.

A block gives reading control for files with preamble lines, a units row, or no header (memo §11.2), using +skip+ / +header+ / +column_names+ / +body+ (see CSVReader). Without a block the default is +header+ then +body+.

CAFrame.from_csv("obs.csv") do skip 2; header; skip 1; body end

Columns are handed to the frame as CABlock views over one backing object array (§3.6 view-by-default); casting a column materializes it, and +copy+ gives an independent frame.



34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/carray/frame/io.rb', line 34

def self.from_csv(path, types: nil,
                  sep: ",", quote: '"', strip: false,
                  encoding: "bom|utf-8", parser: nil, &block)
  names, rows =
    if parser
      parser.call(path)
    else
      File.open(path, "r:#{encoding}") do |io|
        reader = CSVReader.new(io, sep: sep, quote: quote, strip: strip)
        if block
          block.arity == 1 ? block.call(reader) : reader.instance_exec(&block)
        else
          reader.header
          reader.body
        end
        reader.result
      end
    end

  frame = build_frame(names, rows)
  frame.cast(types) if types
  frame
end

.from_records(records, types: nil) ⇒ Object

Build a frame from an Array of row Hashes. Column set is the union of keys in first-appearance order; keys are stringified. +types:+ casts named columns afterward (same map / array-key forms as +cast+).



21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/carray/frame/records.rb', line 21

def self.from_records(records, types: nil)
  unless records.is_a?(Array) && records.all? { |r| r.is_a?(Hash) }
    raise ArgumentError, "from_records expects an Array of Hashes"
  end
  return new if records.empty?

  keys = record_key_union(records)
  n = records.size
  cols = {}
  keys.each do |key|
    values = records.map { |r| r[key] }
    cols[key.to_s] = build_record_column(values, n)
  end

  frame = new(cols)
  frame.cast(types) if types
  frame
end

.meld(*frames) ⇒ Object

Weld frames along the row axis, view-style. Each output column is +CArray.meld+ of that column across the input frames, so the result is a view frame that shares storage with the inputs (chain composability preserved: writes to the result flow back to whichever input frame owns the target segment, and vice versa).

Per-column +data_type+ must match across frames (CArray.meld is a view constructor that refuses to auto-cast — silent promotion here would hide schema drift). For an eager, auto-casting alternative use concatenate.

Column matching is by name (output order follows the first frame); every frame must carry the same column-name set. N-D columns carry their trailing dimensions along; masks are preserved.

The index is welded too when every frame has one (their +axis_name+ must agree); if none do, the result has no index; a mix raises. Column-set mismatch raises — a union-with-UNDEF mode is a possible future opt-in, kept out here to stay explicit (memo §4.2).

CAFrame.meld(jan, feb, mar) # view over three months CAFrame.meld([jan, feb, mar]) # an Array is accepted too



36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/carray/frame/concat.rb', line 36

def self.meld(*frames)
  frames = frames.flatten
  check_concat_inputs(frames, verb: "meld")
  first = frames.first
  names = first.variable_names
  check_column_sets(frames, names, verb: "meld")
  cols = {}
  names.each do |name|
    cols[name] = CArray.meld(frames.map { |f| f[name] }, axis: 0)
  end
  new(cols, axis_name: first.axis_name, index: meld_index(frames))
end

.paste(*frames, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

Paste N frames side by side by row position (symmetric N-ary sibling of the instance +#paste+; see the instance verb's docstring for the pair semantics). All frames must have the same +nrow+. The result's +axis_name+ and +index+ come from the first frame (peers, but a base is needed for the index). Column names that appear in more than one frame are colliding — pass +suffixes:+ as an Array of exactly K strings (one per input frame) to disambiguate them; otherwise the collision raises.

CAFrame.paste(obs, fcst) CAFrame.paste(obs, ecmwf, gfs, suffixes: ["_obs", "_ecmwf", "_gfs"]) CAFrame.paste([obs, fcst]) # Array is accepted too

Raises:

  • (ArgumentError) 


121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
 
# File 'lib/carray/frame/join.rb', line 121

def self.paste(*frames, suffixes: DEFAULT_JOIN_SUFFIXES)
  frames = frames.flatten
  raise ArgumentError, "paste requires at least one frame" if frames.empty?
  unless frames.all? { |f| f.is_a?(CAFrame) }
    raise ArgumentError, "paste expects CAFrame arguments"
  end
  first = frames.first
  nrow  = first.nrow
  frames.each_with_index do |f, i|
    next if i.zero?
    unless f.nrow == nrow
      raise ArgumentError,
            "paste: row-count mismatch (frame #{i}: #{f.nrow} vs #{nrow}); " \
            "use join for key alignment"
    end
  end

  # Detect columns that appear in 2+ frames (collisions across the K inputs).
  name_count = Hash.new(0)
  frames.each { |f| f.variable_names.each { |name| name_count[name] += 1 } }
  collisions = name_count.select { |_, n| n > 1 }.keys

  if collisions.any?
    if suffixes == false
      raise ArgumentError,
            "paste column name collision: #{collisions.inspect} " \
            "(pass suffixes: [\"_a\", \"_b\", ...] with one entry per frame, " \
            "or rename first)"
    end
    # For the 2-frame case DEFAULT_JOIN_SUFFIXES ("_left", "_right") works
    # as-is; for K > 2 the caller must pass a K-length Array (or use the
    # pair-only default indirectly via the instance method).
    unless suffixes.is_a?(Array) && suffixes.size == frames.size &&
           suffixes.all? { |s| s.is_a?(String) }
      raise ArgumentError,
            "paste: suffixes must be an Array of #{frames.size} Strings " \
            "(one per frame), got #{suffixes.inspect}"
    end
  end

  cols = {}
  frames.each_with_index do |f, i|
    f.variable_names.each do |name|
      key = collisions.include?(name) ? "#{name}#{suffixes[i]}" : name
      if cols.key?(key)
        raise ArgumentError,
              "paste: column name #{key.inspect} produced twice " \
              "(collision after suffix from frame #{i})"
      end
      cols[key] = f[name]
    end
  end
  new(cols, axis_name: first.axis_name, index: first.index)
end

Instance Method Details

#[](*keys) ⇒ Object

String -> a column (raw CArray, the escape unit) String x2+ -> Array (the escape unit, plural) Integer -> a row (Ruby Hash) Range(int)-> positional row slice (view-frame) boolean CA-> row filter (view-frame) integer CA-> row gather (view-frame)

String keys escape: df[...] hands back the raw column(s), never a frame. One name collapses to a bare CArray; several give an Array of them (the "single collapses, plural is an array" rule of ca[i] vs ca[i..j]), so +t, rh = df["temp", "rh"]+ destructures. A column-subset frame comes from +select+ (memo §13.2).



100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/carray/frame/frame.rb', line 100

def [](*keys)
  if keys.size > 1
    unless keys.all? { |k| k.is_a?(String) }
      raise ArgumentError,
            "multi-key df[...] escapes columns; every key must be a String " \
            "(use df.select(...) for a subset frame)"
    end
    return keys.map { |k| @columns.fetch(k) { raise KeyError, "no column #{k.inspect}" } }
  end

  key = keys.first
  case key
  when String
    @columns.fetch(key) { raise KeyError, "no column #{key.inspect}" }
  when Integer
    row(key)
  when Range
    unless positional_range?(key)
      raise ArgumentError,
            "df[range] takes positional (integer) ranges only; " \
            "use filter { |f| f.index ... } for label ranges"
    end
    select_rows(key)
  when CArray
    case key.data_type
    when :boolean
      select_rows(key)
    when *INTEGER_TYPES
      select_rows(key)
    else
      raise ArgumentError,
            "CArray df[] key must be boolean or integer (got #{key.data_type})"
    end
  when Symbol
    raise NotImplementedError, "df[symbol] is reserved for predicate keys"
  else
    raise ArgumentError, "unsupported df[] key: #{key.class}"
  end
end

#[]=(*keys, value) ⇒ Object

--- df[...] = value : row-level assignment ---------------------------

The RHS value decides the operation; the key selects the rows:

df[sel] = UNDEF -> mask the selected rows across every column, in place. Shape is unchanged and the write goes through to the column storage; the index stays, so the rows remain identifiable. df[sel] = nil -> delete the selected rows. The frame shrinks and the survivors keep their order. df[sel] = other -> splice: replace the selected (contiguous) rows with +other+'s rows. +other+ may carry any number of rows, so the row count changes (Ruby Array#[]= splice semantics); its column set must match exactly.

+sel+ is a row-axis indexer key, classified exactly as CArray classifies a 1-D column key (docs/topics/Indexer_decision_tree.md): a slice (BLOCK — Range / ArithmeticSequence / [start, count, step]), a boolean CArray (SELECT), an integer CArray (GRID), or an Integer (a single row). Mask and delete take any of them and are forwarded to the column indexer, so their errors match the rest of CArray. Splice reuses the same classifier (CArray.scan_index) but needs a contiguous slice, so it takes an Integer or a step-1 BLOCK (Range, step-1 ArithmeticSequence, or [start, count]); a strided or scattered selector raises.

Delete and splice rebuild the column set, so an alias previously taken from this frame no longer tracks the frame's new column identity (@columns is rebound to a fresh Hash). Reads through the old alias see the ORIGINAL column's data (unchanged by splice construction), matching the pre-splice snapshot. Writes flow along the CAMeld chain:

  • Writes reaching the HEAD or TAIL segment (the parts of the original column that survived the splice) land back in that column's storage. External aliases into the original column see those writes — this is the chain composability that CAMeld deliberately preserves: a reference stays connected to what it references.
  • Writes reaching the spliced MIDDLE segment (the rows from +other+) do NOT reach +other+'s storage: +other+ is snapshotted via +.copy+ at splice time so it behaves like a value that was handed over. +df1[...] = df2+ followed by writes to df1's spliced rows leaves df2 untouched, which matches the usual "assignment" intuition for the RHS of +[]=+.

Callers who want write isolation on the head/tail segments too can materialise explicitly with +col_view.copy+ before the splice, or +df["c"] = df["c"].copy+ afterwards. The shape-preserving +UNDEF+ write path always propagates to derived views (unchanged from previous behaviour).



188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
 
# File 'lib/carray/frame/frame.rb', line 188

def []=(*keys, value)
  if keys.size != 1
    raise ArgumentError,
          "df[...] = takes one key (a column name or a row selector); " \
          "got #{keys.size}"
  end
  selector = keys.first

  # The key picks the axis, exactly as it does when reading (§13.2): a
  # String names a column, anything else selects rows.
  if selector.is_a?(String)
    column_assign(selector, value)
  elsif UNDEF.equal?(value)
    mask_rows(selector)
  elsif value.nil?
    delete_rows(selector)
  elsif value.is_a?(CAFrame)
    splice_rows(selector, value)
  else
    raise ArgumentError,
          "df[...] = expects UNDEF (mask rows), nil (delete rows), or a " \
          "CAFrame (splice rows); got #{value.class}"
  end
  value
end

#align(key, reference) ⇒ Object

Conform every variable to an externally supplied reference key set (the asymmetric sibling of +join+; pandas +reindex+). Given a +reference+ array of key values, each column is gathered onto it by exact key match (+locate_addr+): the aligned key column (or index) becomes the reference itself, and a reference key absent from the source comes back UNDEF in every other column.

CAFrame measures no interval and generates nothing -- the caller owns the reference. This is the primitive for reindexing to a caller-built axis (e.g. a complete 10-minute time grid): supply that axis as +reference+ and the gaps fill with UNDEF rows carrying only the reference key.

df.align("time", reftime) # reftime = a CArray of reference key values

+key+ may be a column name or the index axis name. +reference+ is a CArray (or Array); its keys are matched against the source key by value, so their data types must be comparable (a DateTime object key matches by eql?/hash).



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/carray/frame/join.rb', line 76

def align(key, reference)
  key = key.to_s
  ref = reference.is_a?(CArray) ? reference : reference.to_ca
  on_index = !@columns.key?(key) && @axis_name == key && @index
  addr = ref.locate_addr(on_index ? @index : self[key])

  cols = {}
  variable_names.each { |name| cols[name] = project_rows(self[name], addr) }
  if on_index
    CAFrame.new(cols, axis_name: key, index: ref)
  else
    cols[key] = ref
    CAFrame.new(cols, axis_name: @axis_name,
                      index: @index && project_rows(@index, addr))
  end
end

#append(name, col) ⇒ Object

Add (or replace) a column, returning a new frame — the column set changes, so the result is a different table (memo §3.8). The column is shared by reference; the envelope is cheap. Chain or reassign: +df = df.append(...)+.



12
13
14
15
16
 
# File 'lib/carray/frame/verbs.rb', line 12

def append(name, col)
  key = name.to_s
  ca  = coerce_column(col)
  rebuild(@columns.merge(key => ca))
end

#at(label) ⇒ Object

The single row whose index label equals +label+, as a Ruby Hash (memo §13.2b). +label+ is matched exactly against the index (+index.eq+), so any orderable / object / datetime / categorical index works. The return type is a row Hash and stays that way: zero matches raise KeyError, and duplicate labels raise (go through +filter { |f| f.index.eq(label) }+ for the multi-row, frame-returning path). Positional access is +df[i]+.

Raises:

  • (ArgumentError) 


371
372
373
374
375
376
377
378
379
380
381
382
383
384
 
# File 'lib/carray/frame/frame.rb', line 371

def at(label)
  raise ArgumentError, "at requires an index (set one with set_index)" unless @index
  pos = @index.eq(label).where
  case pos.elements
  when 0
    raise KeyError, "no row with index label #{label.inspect}"
  when 1
    row(pos[0])
  else
    raise ArgumentError,
          "index label #{label.inspect} matches #{pos.elements} rows; " \
          "use filter { |f| f.index.eq(label) } for duplicate labels"
  end
end

#cast(name_or_map, type = nil) ⇒ Object

Cast columns to a data type and rebind them (memo §11.4). Uses to_type, so parse failures on string columns become UNDEF (parse-mask, §6-2). Three call shapes, disambiguated by the fact that column names are always Strings and types always Symbols (§3.7):

cast("temp", :float64) # one column (chains) cast("temp" => :float64, "rh" => :int32) # name => type map cast(["temp", "rh"] => :float64) # names sharing one type

A map key may be a single name or an Array of names; the value is the target type. Returns self so calls chain.



69
70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/carray/frame/verbs.rb', line 69

def cast(name_or_map, type = nil)
  if name_or_map.is_a?(Hash)
    unless type.nil?
      raise ArgumentError, "cast(map) takes no positional type argument"
    end
    name_or_map.each do |names, t|
      Array(names).each { |name| cast_one(name, t) }
    end
  else
    cast_one(name_or_map, type)
  end
  self
end

#copy ⇒ Object

An independent frame — every column and the index materialized (§3.6).



387
388
389
390
391
 
# File 'lib/carray/frame/frame.rb', line 387

def copy
  cols = {}
  @columns.each { |k, v| cols[k] = v.copy }
  CAFrame.new(cols, axis_name: @axis_name, index: @index && @index.copy)
end

#data_types ⇒ Object

name => data_type, freshly derived.



433
434
435
436
437
 
# File 'lib/carray/frame/frame.rb', line 433

def data_types
  h = {}
  @columns.each { |k, v| h[k] = v.data_type }
  h
end

#drop(*names) ⇒ Object

Remove one or more columns, returning a new frame (column set changes, memo §3.8). Column data is untouched and shared with the original.



20
21
22
23
24
25
26
27
28
 
# File 'lib/carray/frame/verbs.rb', line 20

def drop(*names)
  cols = @columns.dup
  names.each do |name|
    key = name.to_s
    raise KeyError, "no column #{key.inspect}" unless cols.key?(key)
    cols.delete(key)
  end
  rebuild(cols)
end

#each_row ⇒ Object

Iterate rows as Ruby Hashes (memo §11.11). This is an escape path — the primary idiom is column-vectorized work (§4.3); each_row is for touching heterogeneous / Face-carrying rows now and then, not hot loops. Without a block, returns an Enumerator.



8
9
10
11
12
 
# File 'lib/carray/frame/convert.rb', line 8

def each_row
  return enum_for(:each_row) unless block_given?
  nrow.times { |i| yield row(i) }
  self
end

#fill(name, method_or_value) ⇒ Object

Fill masked cells of a column (memo §6, §11.8). A Symbol selects a scan method; anything else is a constant fill value. The fill is write-through: it edits the live shared column rather than rebinding a filled copy (memo §3.8). Returns self so calls chain.

fill("temp", :ffill) # forward hold (carry last valid value) fill("temp", :bfill) # backward hold (carry next valid value) fill("temp", :linear) # linear interpolation; x = this frame's index # coordinate when present, else the cell position fill("temp", 0.0) # constant fill

:ffill / :bfill work for any writable column data_type (numeric, time, object, fixlen, …); :linear needs a numeric or time (CATime / CATimedelta) column. A categorical column's codes are read-only (memo §13.4), so any fill on it raises — rebind a filled copy instead: df.append(name, df[name].strip_mask(method: :forward)).

Raises:

  • (KeyError) 


174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
 
# File 'lib/carray/frame/verbs.rb', line 174

def fill(name, method_or_value)
  key = name.to_s
  raise KeyError, "no column #{key.inspect}" unless @columns.key?(key)
  col = @columns[key]
  case method_or_value
  when :ffill  then col.unmask(method: :forward)
  when :bfill  then col.unmask(method: :backward)
  when :linear then fill_linear_column(key, col)
  when Symbol
    raise ArgumentError,
          "fill: unknown method #{method_or_value.inspect} " \
          "(:ffill | :bfill | :linear, or a constant fill value)"
  else
    col.unmask(method_or_value)   # constant fill, in place
  end
  self
end

#filter(keep_masked: false) ⇒ Object

Filter rows with a block that receives the frame and returns a boolean column (memo §15). The block builds the mask from +f["col"]+ / +f.index+. Keep the rows where the block's boolean selector is true.

A masked (UNDEF) selector cell means the row's membership is genuinely undetermined (e.g. the predicate read a masked input). By default such rows are silently dropped, exactly as a false cell would be. Pass +keep_masked: true+ to carry the UNDEF forward instead: the undetermined rows survive into the result with their data cells masked (and their index value kept, so the row stays identifiable), leaving a later, better-informed pass to re-judge them. Definitely-true rows carry their values through unchanged in both modes.



310
311
312
313
314
315
316
317
318
319
320
321
 
# File 'lib/carray/frame/frame.rb', line 310

def filter(keep_masked: false)
  mask = yield(self)
  unless mask.is_a?(CArray) && mask.data_type == :boolean
    raise ArgumentError,
          "filter block must return a boolean CArray (got #{mask.class})"
  end
  unless mask.shape[0] == @nrow
    raise ArgumentError,
          "filter mask has axis-0 length #{mask.shape[0]}, expected nrow #{@nrow}"
  end
  select_rows(mask, keep_masked: keep_masked)
end

#group_by(*keys) ⇒ Object

Group rows by one or more keys. Each key is a column name (String) or an external length-N CArray. Returns a GroupedFrame.

Raises:

  • (ArgumentError) 


12
13
14
15
16
17
 
# File 'lib/carray/frame/group.rb', line 12

def group_by(*keys)
  raise ArgumentError, "group_by needs at least one key" if keys.empty?
  cat  = grouping_categorical(keys)
  axis = keys.size == 1 && keys.first.is_a?(String) ? keys.first : "group"
  GroupedFrame.new(self, cat, axis)
end

#head(n = 5) ⇒ Object

The first +n+ rows as a positional view-frame (memo §11.3). +n+ larger than +nrow+ yields the whole frame; +n+ of 0 yields an empty frame.

Raises:

  • (ArgumentError) 


353
354
355
356
 
# File 'lib/carray/frame/frame.rb', line 353

def head(n = 5)
  raise ArgumentError, "head count must be non-negative (got #{n})" if n < 0
  row_span(0, [n, @nrow].min)
end

#inspect ⇒ String

Returns:

  • (String) 


440
441
442
443
444
 
# File 'lib/carray/frame/frame.rb', line 440

def inspect
  parts = @columns.map { |k, v| "#{k}:#{v.data_type}#{v.ndim > 1 ? v.shape[1..].inspect : ''}" }
  idx = @index ? " index=#{@axis_name.inspect}" : ""
  "#<CAFrame nrow=#{@nrow} vars=[#{parts.join(', ')}]#{idx}>"
end

#join(other, on:, how: :left, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

Join +other+ on a shared key column.

how: :left (default) — keep all left rows; right columns gathered per left row (locate_addr), misses become UNDEF. how: :inner/:outer/:right — set-align both key sets (align_addr) and gather both sides; the aligned key values form the +on+ column.

A non-key column present on both sides collides. The key column (+on+) is kept once, never suffixed. By default the collision is resolved by suffixing both sides (+suffixes: ["_left", "_right"]+); pass a 2-element array to pick meaningful names up front (+["_obs", "_fcst"]+), or +suffixes: false+ to raise instead. Rename afterward with +rename+ if needed (§12-C).



29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/carray/frame/join.rb', line 29

def join(other, on:, how: :left, suffixes: DEFAULT_JOIN_SUFFIXES)
  on   = on.to_s
  lkey = self[on]
  rkey = other[on]
  plan = join_name_plan(other, on, suffixes)

  case how
  when :left
    join_left(other, on, lkey, rkey, plan)
  when :inner, :outer, :right
    join_align(other, on, lkey, rkey, how, plan)
  else
    raise ArgumentError, "unknown join mode #{how.inspect}"
  end
end

#join_asof(other, on:, direction: :floor, tolerance: nil, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

As-of (nearest-key) left join for irregular series (memo §11.7). Each left row is matched to the nearest +other+ row by +on+ via locate_nearest_addr (the per-row, row-preserving counterpart of align_nearest_addr); +direction:+ follows CArray (:floor = most recent at-or-before, :ceil = next, :round = nearest), and rows with no match in range or beyond +tolerance:+ come back UNDEF. Same wiring as the left join, only the address primitive differs.



52
53
54
55
56
57
 
# File 'lib/carray/frame/join.rb', line 52

def join_asof(other, on:, direction: :floor, tolerance: nil, suffixes: DEFAULT_JOIN_SUFFIXES)
  on   = on.to_s
  plan = join_name_plan(other, on, suffixes)
  addr = self[on].locate_nearest_addr(other[on], direction: direction, tolerance: tolerance)
  join_by_addr(other, on, addr, plan)
end

#mask_eq(name, value) ⇒ Object

Mask cells of a column that equal +value+ (sentinel -> mask, memo §11.4). In-place on the column (write-through, §4.3): numeric/object columns mask in place; a categorical column's codes are read-only (§13.4) so this raises — recode by rebuilding and rebinding instead.



53
54
55
56
 
# File 'lib/carray/frame/verbs.rb', line 53

def mask_eq(name, value)
  self[name][:eq, value] = UNDEF
  self
end

#nvar ⇒ Object

Number of variables (columns).



419
420
421
 
# File 'lib/carray/frame/frame.rb', line 419

def nvar
  @columns.size
end

#parse_to_time(name, format = nil, unit: :s) ⇒ Object

Parse a string column into a time column and rebind it (memo §11.2). The column must be string-bearing (an object CArray of Strings, or a CAString / CAConstString / CAFixlenString) — this is the "text -> time" mode, distinct from the integer-serial mode of +to_time+. Delegates to +CArray.time(col, on_error: :mask)+: +format+ picks strptime parsing (auto-detect when nil), +unit+ the storage resolution; masked / nil and unparseable cells become UNDEF (bulk column parse tolerates bad cells). Make it the index with +set_index+ afterward.

df.parse_to_time("time").set_index("time")



122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/carray/frame/verbs.rb', line 122

def parse_to_time(name, format = nil, unit: :s)
  key = name.to_s
  col = @columns.fetch(key) { raise KeyError, "no column #{key.inspect}" }
  unless string_column?(col)
    raise ArgumentError,
          "parse_to_time needs a string column (object / CAString / " \
          "CAConstString / CAFixlenString); #{key.inspect} is #{col.data_type}"
  end
  @columns[key] = CArray.time(col, format: format, unit: unit, on_error: :mask)
  self
end

#paste(other, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

Paste +other+'s variables beside this frame's, matched by row position (memo §12-C) — a keyless column merge (the column-direction counterpart of the row-stacking +concat+, and the positional counterpart of the key-aligned +join+; named after the UNIX +paste+). Both frames must have the same +nrow+; rows are assumed to already correspond (no key alignment, consistent with the no-implicit-align stance). Colliding column names are disambiguated by the same policy as +join+ (default suffix +_left+/+_right+, +suffixes:+ to override, +suffixes: false+ to raise). This frame's index is kept; +other+'s index, if any, is not carried (only its columns are pasted). Returns a new frame.

obs.paste(fcst) # side by side, same rows obs.paste(fcst, suffixes: ["_obs", "_fcst"])



106
107
108
 
# File 'lib/carray/frame/join.rb', line 106

def paste(other, suffixes: DEFAULT_JOIN_SUFFIXES)
  CAFrame.paste(self, other, suffixes: suffixes)
end

#promote(type = nil) ⇒ Object

Bring every column to one common data type and rebind them (memo §11.4). The frame-level counterpart of +CArray.promote_list+: where +cast+ forces named columns to a type, +promote+ widens the whole table until it has a single type. Returns self so calls chain (§3.8 -- a type-interpretation change), and like +cast+ it allocates fresh columns, so a parent frame sharing the old ones is untouched. The index is not a column and is left alone.

df.promote # the common type CArray.result_type would pick df.promote(:object) # force the widest type: anything fits

Without an argument the common type is whatever +promote_list+ picks -- the same decision +to_ca+ / +CArray.stack+ make internally, so +df.promote+ is exactly "make this frame stackable". Columns that are already uniform (including a uniformly Face-typed frame) are left as they are.

With an argument the type must be a widening for every column: promoting is not the place to lose values, so a narrowing target raises and points at +cast+, which is the verb that forces. +:object+ is the widest type and therefore always accepted -- it is the way a frame mixing text, Face-typed and numeric columns becomes a single-type table (and so the way +to_ca+ can hand back a matrix for it).



106
107
108
109
110
 
# File 'lib/carray/frame/verbs.rb', line 106

def promote(type = nil)
  return self if @columns.empty?
  type.nil? ? promote_to_common : promote_to_type(type)
  self
end

#rename(mapping) ⇒ Object

Rename columns, returning a new frame (column names change, memo §3.8). Column order is preserved (memo §13.2 "column order = insertion order"); columns are shared by reference.



33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/carray/frame/verbs.rb', line 33

def rename(mapping)
  norm = {}
  mapping.each do |old, new|
    o = old.to_s
    n = new.to_s
    raise KeyError, "no column #{o.inspect}" unless @columns.key?(o)
    if n != o && @columns.key?(n)
      raise ArgumentError, "rename target #{n.inspect} already exists"
    end
    norm[o] = n
  end
  rebuilt = {}
  @columns.each { |k, v| rebuilt[norm[k] || k] = v }
  rebuild(rebuilt)
end

#reset_index ⇒ Object

Drop the index back into an ordinary column (the inverse of set_index). Also an index-role change, so it mutates self (memo §3.8). The former index becomes the first column, named after the row axis.



343
344
345
346
347
348
349
 
# File 'lib/carray/frame/frame.rb', line 343

def reset_index
  return self unless @index
  @columns   = { @axis_name => @index }.merge(@columns)
  @index     = nil
  @axis_name = DEFAULT_AXIS_NAME
  self
end

#select(*names) ⇒ Object

Column projection (memo §13.2): a view-frame holding the named columns as aliases (zero-copy, sharing storage with the parent frame). Distinct from +df[...]+, which escapes columns to raw CArrays, and from +filter+, which selects rows. +select+ takes no block — row conditions go through +filter+ (old carray-dataframe fused the two into +select(cols) { cond }+; here they are orthogonal and compose by chaining: +df.select(...).filter { ... }+).

Raises:

  • (ArgumentError) 


287
288
289
290
291
292
293
294
295
296
 
# File 'lib/carray/frame/frame.rb', line 287

def select(*names)
  raise ArgumentError, "select requires at least one column name" if names.empty?
  cols = {}
  names.each do |name|
    key = name.to_s
    raise KeyError, "no column #{key.inspect}" unless @columns.key?(key)
    cols[key] = @columns[key]
  end
  rebuild(cols)
end

#set_index(name) ⇒ Object

Move a column to the index and name the row axis after it (memo §3). An index-role change: the data is unchanged, so this mutates self and returns it (memo §3.8). Any existing index is replaced.

Raises:

  • (KeyError) 


326
327
328
329
330
331
332
333
334
335
336
337
338
 
# File 'lib/carray/frame/frame.rb', line 326

def set_index(name)
  key = name.to_s
  raise KeyError, "no column #{key.inspect}" unless @columns.key?(key)
  idx = @columns[key]
  unless idx.ndim == 1
    raise ArgumentError, "index must be a 1-D column (got ndim #{idx.ndim})"
  end
  @columns.delete(key)
  @index     = idx
  @axis_name = key
  @nrow      = idx.shape[0]
  self
end

#sort_by(masked_position: :last) ⇒ Object

Sort rows by key columns the block builds (memo §4.3 escape). The block receives the frame and returns a key CArray, or an Array of them for a multi-key lexicographic sort; rows are sorted ascending by +CArray.sort_addr+ over those keys and gathered into a view-frame. This is the sort sibling of +filter+: where +sort_by_key+ takes plain column names, the block form lets you build any key with CArray ops -- a descending key is its dense descending rank, derived or composite keys are ordinary column math. +masked_position:+ places masked key rows first or last (default last).

df.sort_by { |f| (f["temp"] - target).abs } # nearest-to-target first df.sort_by { |f| f["temp"].order(descending: true, method: :dense) }

Raises:

  • (ArgumentError) 


60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/carray/frame/sort.rb', line 60

def sort_by(masked_position: :last)
  raise ArgumentError, "sort_by requires a block" unless block_given?
  keys = yield(self)
  keys = [keys] unless keys.is_a?(Array)
  raise ArgumentError, "sort_by block must return a key CArray or an Array of them" if keys.empty?
  keys.each_with_index do |k, i|
    unless k.is_a?(CArray)
      raise ArgumentError, "sort_by key #{i} must be a CArray (got #{k.class})"
    end
    unless k.ndim == 1 && k.shape[0] == nrow
      raise ArgumentError,
            "sort_by key #{i} must be a 1-D column of length #{nrow} (got shape #{k.shape.inspect})"
    end
  end
  select_rows(CArray.sort_addr(*keys, masked_position: masked_position))
end

#sort_by_key(*specs, order: :asc, masked_position: :last) ⇒ Object

Sort rows by one or more key columns, lexicographic (the first key is primary), returning a view-frame. Delegates to +CArray.sort_addr+ for the row permutation and gathers every column and the index by it.

Each key is a column name (or the index axis name), or a [name, :asc | :desc] pair for a per-key direction. The +order:+ keyword is the direction for bare-name keys (default +:asc+). +masked_position:+ sends masked key rows to the +:last+ (default) or +:first+ end.

df.sort_by_key("temp") # one column, ascending df.sort_by_key("station", "temp") # lexicographic df.sort_by_key("temp", order: :desc) # all keys descending df.sort_by_key(["station", :asc], ["temp", :desc]) # per-key direction df.sort_by_key("temp", masked_position: :first) # masked rows first df.sort_by_key("time") # by the index

Raises:

  • (ArgumentError) 


38
39
40
41
42
43
44
45
46
47
 
# File 'lib/carray/frame/sort.rb', line 38

def sort_by_key(*specs, order: :asc, masked_position: :last)
  raise ArgumentError, "sort_by_key requires at least one key" if specs.empty?
  validate_sort_order(order)
  keys = specs.map do |spec|
    name, dir = parse_sort_key_spec(spec, order)
    col = sort_key_column(name)
    dir == :desc ? col.order(descending: true, method: :dense) : col
  end
  select_rows(CArray.sort_addr(*keys, masked_position: masked_position))
end

#tail(n = 5) ⇒ Object

The last +n+ rows as a positional view-frame (memo §11.3). +n+ larger than +nrow+ yields the whole frame; +n+ of 0 yields an empty frame.

Raises:

  • (ArgumentError) 


360
361
362
363
 
# File 'lib/carray/frame/frame.rb', line 360

def tail(n = 5)
  raise ArgumentError, "tail count must be non-negative (got #{n})" if n < 0
  row_span(@nrow - [n, @nrow].min, @nrow)
end

#to_ca(writable: false) ⇒ Object

Hand the frame over as a CArray with the minimum work (memo §11.9): a 2-D view of shape (nrow, nvar), one column per variable in column order (§12-C). Nothing is materialised — the result is a +CAStack+ over the stored columns, so reads gather from them and writes flow back (§3.6). Call +copy+ on it for an independent, owned matrix; +CArray.tabulate+ is the eager sibling that builds one directly.

Only same-shape scalar (1-D) columns qualify. An N-D column has no single matrix form and raises — escape it per column with +df["name"]+. A mixed dtype set is promoted to a common type (+result_type+, §12-F) through lazy cast lanes, so the promotion costs no buffer either.

+writable: true+ demands a result whose writes reach this frame's own columns (the 3.0 +to_ca+ contract). That holds only when every column enters the stack unchanged, so it is refused when a column is read-only or when the common type promotes it: the promoted column is stacked through a cast lane, and a write there is no longer the value the caller handed over. Cast the frame first (+df.cast+) when the promoted matrix is what should be written to.

Raises:

  • (ArgumentError) 


44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/carray/frame/convert.rb', line 44

def to_ca(writable: false)
  cols = @columns.values
  raise ArgumentError, "frame has no columns to stack into a matrix" if cols.empty?
  nd = @columns.find { |_, c| c.ndim != 1 }
  if nd
    raise ArgumentError,
          "to_ca needs all-scalar (1-D) columns; #{nd.first.inspect} is " \
          "#{nd.last.ndim}-D — escape per column with df[name]"
  end
  refuse_unshared_columns(cols) if writable
  begin
    CArray.stack(cols, axis: 1)
  rescue ArgumentError, RuntimeError => e
    # The columns have no common type (a text or Face column beside a
    # numeric one). Point at the frame-level verb that gives them one.
    raise e.class,
          "#{e.message} -- df.promote(:object) brings every column to its " \
          "surface values, and that frame stacks"
  end
end

#to_csv(path = nil, sep: ",", quote: '"', header: true, index: true) ⇒ Object

Build a frame from parsed [names, rows]. When names is nil (headerless and no column_names) positional names "c0".."cN" are generated from the widest row. Rows are squared off to the column count (short rows padded with nil, over-long rows raise), one 2-D object array is bulk-filled, and each column is a view into it (§3.6). Write the frame as CSV. CSV is a flat table of scalar cells, so this is the text form of the same all-scalar subset +to_ca+ requires (§11.9): every column must be 1-D. Unlike +to_ca+ it does not promote to a common dtype -- each column is formatted to text independently, so mixed dtypes (numbers, strings, datetime / categorical Faces) sit side by side. An N-D column has no flat CSV cell and raises; export it per column, or use +to_records+ + JSON for the structured shape (memo §11.9, the N-D escape).

With +path+, writes the file and returns self; without it, returns the CSV String. The index (if any) is written as the first column under +axis_name+ unless +index: false+. A masked cell (UNDEF) becomes an empty field, which +from_csv+ reads back as UNDEF (parse-mask, §6-2) -- so mask round-trips. A genuine empty string is written quoted (+""+) to stay distinct from missing, matching the tokenizer's own unquoted-empty vs quoted-empty distinction.

df.to_csv("out.csv") # write file csv = df.to_csv # get a String

Options: +sep+ / +quote+ mirror +from_csv+; +header+ writes the name row (default true); +index+ writes the index column (default true).



83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/carray/frame/io.rb', line 83

def to_csv(path = nil, sep: ",", quote: '"', header: true, index: true)
  nd = @columns.find { |_, c| c.ndim != 1 }
  if nd
    raise ArgumentError,
          "to_csv needs all-scalar (1-D) columns; #{nd.first.inspect} is " \
          "#{nd.last.ndim}-D — export it per column or via to_records + JSON"
  end

  names     = []
  formatted = []
  if index && @index
    names     << @axis_name
    formatted << format_csv_column(@index)
  end
  @columns.each do |name, col|
    names     << name
    formatted << format_csv_column(col)
  end

  out = +""
  if header
    out << names.map { |t| quote_csv_field(t, sep, quote) }.join(sep) << "\n"
  end
  @nrow.times do |i|
    out << formatted.map { |fcol| quote_csv_field(fcol[i], sep, quote) }.join(sep) << "\n"
  end

  if path
    File.write(path, out)
    self
  else
    out
  end
end

#to_records ⇒ Object

Export rows as an Array of plain Ruby Hashes (memo §11.11) — the inverse of +from_records+ and the shape +JSON.generate+ wants. Each row is one Hash of column name => value; a scalar cell is a Ruby value, an N-D cell is a Ruby Array, and a masked cell (UNDEF) is +nil+. Normalizing UNDEF -> nil and CArray -> Array (unlike +each_row+, which yields the raw view with UNDEF / CArray slices) is what makes it round-trip: +from_records(df.to_records)+ rebuilds the same typing, and the result is JSON-serializable.



21
22
23
 
# File 'lib/carray/frame/convert.rb', line 21

def to_records
  each_row.map { |row| row.transform_values { |v| record_value(v) } }
end

#to_time(name, unit: :s, epoch: nil) ⇒ Object

Reinterpret an integer column as time serial counts and rebind it (memo §11.2). This is the "serial -> time" mode: each value is a count of +unit+ resolution since +epoch+ (default the Unix epoch, 1970-01-01 UTC). +epoch+ takes any time literal (String / Time / Integer), so a column measured from another origin — a netCDF "hours since 1990-01-01" axis, an Excel serial date (epoch "1899-12-30", unit :D) — converts directly.

A float column is accepted only when every value is whole (no fractional part); a fractional serial has sub-unit precision that a finer +unit+ should carry, so it raises rather than silently truncate. Make it the index with +set_index+ afterward.

df.to_time("time", unit: :h, epoch: "1990-01-01").set_index("time")



147
148
149
150
151
152
153
154
155
156
 
# File 'lib/carray/frame/verbs.rb', line 147

def to_time(name, unit: :s, epoch: nil)
  key = name.to_s
  col = @columns.fetch(key) { raise KeyError, "no column #{key.inspect}" }
  raw = integer_serial_column(col, key)
  if epoch
    raw = raw + CArray.time(epoch, unit: unit).ticks[0]
  end
  @columns[key] = raw.time(unit: unit)
  self
end

#variable_names ⇒ Object

Variable (column) names in column order. Returns +Array+.



407
408
409
 
# File 'lib/carray/frame/frame.rb', line 407

def variable_names
  @columns.keys
end

#variables ⇒ Object

Variables (raw column CArrays) in column order. Returns +Array+. Equivalent to +df[*variable_names]+ but built directly from the columns Hash.



414
415
416
 
# File 'lib/carray/frame/frame.rb', line 414

def variables
  @columns.values
end