Class: DaruLite::DataFrame

Inherits:
Object show all
Extended by:
Gem::Deprecate
Includes:
Aggregatable, Calculatable, Convertible, Duplicatable, Fetchable, Filterable, IOAble, Indexable, Iterable, Joinable, Missable, Pivotable, Queryable, Setable, Sortable, Maths::Arithmetic::DataFrame, Maths::Statistics::DataFrame
Defined in:
lib/daru_lite/dataframe.rb,
lib/daru_lite/data_frame/setable.rb,
lib/daru_lite/data_frame/i_o_able.rb,
lib/daru_lite/data_frame/iterable.rb,
lib/daru_lite/data_frame/joinable.rb,
lib/daru_lite/data_frame/missable.rb,
lib/daru_lite/data_frame/sortable.rb,
lib/daru_lite/data_frame/fetchable.rb,
lib/daru_lite/data_frame/indexable.rb,
lib/daru_lite/data_frame/pivotable.rb,
lib/daru_lite/data_frame/queryable.rb,
lib/daru_lite/extensions/which_dsl.rb,
lib/daru_lite/data_frame/filterable.rb,
lib/daru_lite/data_frame/convertible.rb,
lib/daru_lite/data_frame/aggregatable.rb,
lib/daru_lite/data_frame/calculatable.rb,
lib/daru_lite/data_frame/duplicatable.rb

Overview

rubocop:disable Metrics/ClassLength

Defined Under Namespace

Modules: Aggregatable, Calculatable, Convertible, Duplicatable, Fetchable, Filterable, IOAble, Indexable, Iterable, Joinable, Missable, Pivotable, Queryable, Setable, Sortable

Constant Summary collapse

AXES = 
%i[row vector].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Maths::Statistics::DataFrame

#acf, #correlation, #count, #covariance, #cumsum, #describe, #ema, #max, #mean, #median, #min, #mode, #percent_change, #product, #range, #rolling_count, #rolling_max, #rolling_mean, #rolling_median, #rolling_min, #rolling_std, #rolling_variance, #standardize, #std, #sum, #variance_sample

Methods included from Maths::Arithmetic::DataFrame

#%, #*, #**, #+, #-, #/, #exp, #round, #sqrt

Methods included from Queryable

#all?, #any?, #has_vector?, #include_values?

Methods included from Sortable

#order=, #rotate_vectors, #sort, #sort!

Methods included from Setable

#[]=, #add_row, #add_vector, #insert_vector, #set_at, #set_row_at

Methods included from Pivotable

#pivot_table

Methods included from Missable

#has_missing_data?, #missing_values_rows, #rolling_fillna, #rolling_fillna!

Methods included from Joinable

#concat, #join, #merge, #one_to_many, #union

Methods included from IOAble

#_dump, included, #save, #write_csv, #write_excel, #write_sql

Methods included from Iterable

#apply_method, #collect, #collect_matrix, #collect_row_with_index, #collect_rows, #collect_vector_with_index, #collect_vectors, #each, #each_index, #each_row, #each_row_with_index, #each_vector, #each_vector_with_index, #map, #map!, #map_rows, #map_rows!, #map_rows_with_index, #map_vectors, #map_vectors!, #map_vectors_with_index, #recode, #recode_rows, #recode_vectors, #replace_values, #verify

Methods included from Indexable

#index=, #reindex, #reindex_vectors, #reset_index, #set_index, #vectors=

Methods included from Filterable

#filter, #filter_rows, #filter_vector, #filter_vectors, #keep_row_if, #keep_vector_if, #reject_values, #uniq, #where

Methods included from Fetchable

#[], #access_row_tuples_by_indexs, #at, #get_sub_dataframe, #get_vector_anyways, #head, #numeric_vector_names, #numeric_vectors, #only_numerics, #row_at, #split_by_category, #tail

Methods included from Duplicatable

#clone, #clone_only_valid, #clone_structure, #dup, #dup_only_valid

Methods included from Convertible

#create_sql, #to_a, #to_df, #to_h, #to_html, #to_html_tbody, #to_html_thead, #to_json, #to_matrix, #to_s

Methods included from Calculatable

#compute, #summary, #vector_by_calculation, #vector_count_characters, #vector_mean, #vector_sum

Methods included from Aggregatable

#aggregate, #group_by, #group_by_and_aggregate

Constructor Details

#initialize(source = {}, opts = {}) ⇒ DataFrame

DataFrame basically consists of an Array of Vector objects. These objects are indexed by row and column by vectors and index Index objects.

Arguments

  • source - Source from the DataFrame is to be initialized. Can be a Hash

of names and vectors (array or DaruLite::Vector), an array of arrays or array of DaruLite::Vectors.

Options

:order - An Array/DaruLite::Index/DaruLite::MultiIndex containing the order in which Vectors should appear in the DataFrame.

:index - An Array/DaruLite::Index/DaruLite::MultiIndex containing the order in which rows of the DataFrame will be named.

:name - A name for the DataFrame.

:clone - Specify as true or false. When set to false, and Vector objects are passed for the source, the Vector objects will not duplicated when creating the DataFrame. Will have no effect if Array is passed in the source, or if the passed DaruLite::Vectors have different indexes. Default to true.

Usage

df = DaruLite::DataFrame.new
# =>
# <DaruLite::DataFrame(0x0)>
# Creates an empty DataFrame with no rows or columns.

df = DaruLite::DataFrame.new({}, order: [:a, :b])
#<DaruLite::DataFrame(0x2)>
  a   b
# Creates a DataFrame with no rows and columns :a and :b

df = DaruLite::DataFrame.new({a: [1,2,3,4], b: [6,7,8,9]}, order: [:b, :a],
  index: [:a, :b, :c, :d], name: :spider_man)

# =>
# <DaruLite::DataFrame:80766980 @name = spider_man @size = 4>
#             b          a
#  a          6          1
#  b          7          2
#  c          8          3
#  d          9          4

df = DaruLite::DataFrame.new([[1,2,3,4],[6,7,8,9]], name: :bat_man)

# =>
# #<DaruLite::DataFrame: bat_man (4x2)>
#             0          1
#  0          1          6
#  1          2          7
#  2          3          8
#  3          4          9

# Dataframe having Index name

df = DaruLite::DataFrame.new({a: [1,2,3,4], b: [6,7,8,9]}, order: [:b, :a],
  index: DaruLite::Index.new([:a, :b, :c, :d], name: 'idx_name'),
  name: :spider_man)

# =>
# <DaruLite::DataFrame:80766980 @name = spider_man @size = 4>
# idx_name            b          a
#        a          6          1
#        b          7          2
#        c          8          3
#        d          9          4

idx = DaruLite::Index.new [100, 99, 101, 1, 2], name: "s1"
=> #<DaruLite::Index(5): s1 {100, 99, 101, 1, 2}>

df = DaruLite::DataFrame.new({b: [11,12,13,14,15], a: [1,2,3,4,5],
  c: [11,22,33,44,55]},
  order: [:a, :b, :c],
  index: idx)
 # =>
 #<DaruLite::DataFrame(5x3)>
 #   s1   a   b   c
 #  100   1  11  11
 #   99   2  12  22
 #  101   3  13  33
 #    1   4  14  44
 #    2   5  15  55



229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
 
# File 'lib/daru_lite/dataframe.rb', line 229

def initialize(source = {}, opts = {})
  vectors = opts[:order]
  index = opts[:index] # FIXME: just keyword arges after Ruby 2.1
  @data = []
  @name = opts[:name]

  case source
  when [], {}
    create_empty_vectors(vectors, index)
  when Array
    initialize_from_array source, vectors, index, opts
  when Hash
    initialize_from_hash source, vectors, index, opts
  end

  set_size
  validate
  update
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args) ⇒ Object 



484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
 
# File 'lib/daru_lite/dataframe.rb', line 484

def method_missing(name, *args, &)
  stringified_name = name.to_s

  if /^([^=]+)=/.match?(stringified_name)
    name = stringified_name[/^([^=]+)=/].delete('=')
    name = name.to_sym unless has_vector?(name)
    insert_or_modify_vector [name], args[0]
  elsif has_vector?(name)
    self[name]
  elsif has_vector?(stringified_name)
    self[stringified_name]
  else
    super
  end
end

Instance Attribute Details

#data ⇒ Object (readonly)

TOREMOVE



129
130
131
 
# File 'lib/daru_lite/dataframe.rb', line 129

def data
  @data
end

#index ⇒ Object (readonly)

The index of the rows of the DataFrame



132
133
134
 
# File 'lib/daru_lite/dataframe.rb', line 132

def index
  @index
end

#name ⇒ Object (readonly)

The name of the DataFrame



135
136
137
 
# File 'lib/daru_lite/dataframe.rb', line 135

def name
  @name
end

#size ⇒ Object (readonly)

The number of rows present in the DataFrame



138
139
140
 
# File 'lib/daru_lite/dataframe.rb', line 138

def size
  @size
end

#vectors ⇒ Object (readonly)

The vectors (columns) index of the DataFrame



127
128
129
 
# File 'lib/daru_lite/dataframe.rb', line 127

def vectors
  @vectors
end

Class Method Details

.crosstab_by_assignation(rows, columns, values) ⇒ Object

Generates a new dataset, using three vectors

  • Rows

  • Columns

  • Values

For example, you have these values

x   y   v
a   a   0
a   b   1
b   a   1
b   b   0

You obtain

id  a   b
 a  0   1
 b  1   0

Useful to process outputs from databases



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'lib/daru_lite/dataframe.rb', line 84

def crosstab_by_assignation(rows, columns, values)
  raise 'Three vectors should be equal size' if
    rows.size != columns.size || rows.size != values.size

  row_index = rows.uniq.to_a
  data = Hash.new do |h, col|
    h[col] = row_index.map { |r| [r, nil] }.to_h
  end
  columns.zip(rows, values).each { |c, r, v| data[c][r] = v }

  # FIXME: in fact, WITHOUT this line you'll obtain more "right"
  # data: with vectors having "rows" as an index...
  data = data.transform_values(&:values)
  data[:_id] = row_index

  DataFrame.new(data)
end

.rows(source, opts = {}) ⇒ Object

Create DataFrame by specifying rows as an Array of Arrays or Array of DaruLite::Vector objects.

Raises:



50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/daru_lite/dataframe.rb', line 50

def rows(source, opts = {})
  raise SizeError, 'All vectors must have same length' \
    unless source.all? { |v| v.size == source.first.size }

  opts[:order] ||= guess_order(source)

  if ArrayHelper.array_of?(source, Array) || source.empty?
    DataFrame.new(source.transpose, opts)
  elsif ArrayHelper.array_of?(source, Vector)
    from_vector_rows(source, opts)
  else
    raise ArgumentError, "Can't create DataFrame from #{source}"
  end
end

Instance Method Details

#==(other) ⇒ Object 



459
460
461
462
463
464
465
 
# File 'lib/daru_lite/dataframe.rb', line 459

def ==(other)
  self.class == other.class   &&
    @size    == other.size    &&
    @index   == other.index   &&
    @vectors == other.vectors &&
    @vectors.to_a.all? { |v| self[v] == other[v] }
end

#add_level_to_vectors(top_level_label) ⇒ Object

Converts the vectors to a DaruLite::MultiIndex. The argument passed is used as the MultiIndex’s top level



400
401
402
403
 
# File 'lib/daru_lite/dataframe.rb', line 400

def add_level_to_vectors(top_level_label)
  tuples = vectors.map { |label| [top_level_label, *label] }
  self.vectors = DaruLite::MultiIndex.from_tuples(tuples)
end

#add_vectors_by_split(name, join = '-', sep = DaruLite::SPLIT_TOKEN) ⇒ Object 



337
338
339
340
341
 
# File 'lib/daru_lite/dataframe.rb', line 337

def add_vectors_by_split(name, join = '-', sep = DaruLite::SPLIT_TOKEN)
  self[name]
    .split_by_separator(sep)
    .each { |k, v| self[:"#{name}#{join}#{k}"] = v }
end

#add_vectors_by_split_recode(nm, join = '-', sep = DaruLite::SPLIT_TOKEN) ⇒ Object 



405
406
407
408
409
410
411
412
 
# File 'lib/daru_lite/dataframe.rb', line 405

def add_vectors_by_split_recode(nm, join = '-', sep = DaruLite::SPLIT_TOKEN)
  self[nm]
    .split_by_separator(sep)
    .each_with_index do |(k, v), i|
      v.rename "#{nm}:#{k}"
      self[:"#{nm}#{join}#{i + 1}"] = v
    end
end

#bootstrap(n = nil) ⇒ DaruLite::DataFrame

Creates a DataFrame with the random data, of n size. If n not given, uses original number of rows.

Returns:



305
306
307
308
309
310
311
312
313
 
# File 'lib/daru_lite/dataframe.rb', line 305

def bootstrap(n = nil)
  n ||= nrows
  DaruLite::DataFrame.new({}, order: @vectors).tap do |df_boot|
    n.times do
      df_boot.add_row(row[rand(n)])
    end
    df_boot.update
  end
end

#delete_at_position(position) ⇒ Object

Delete a row based on its position More robust than #delete_row when working with a CategoricalIndex or when the Index includes integers

Raises:

  • (IndexError) 


292
293
294
295
296
297
298
299
 
# File 'lib/daru_lite/dataframe.rb', line 292

def delete_at_position(position)
  raise IndexError, "Position #{position} does not exist." unless position < size

  @index = @index.delete_at(position)
  each_vector { |vector| vector.delete_at_position(position) }

  set_size
end

#delete_row(index) ⇒ Object

Delete a row

Raises:

  • (IndexError) 


276
277
278
279
280
281
282
283
284
285
286
287
 
# File 'lib/daru_lite/dataframe.rb', line 276

def delete_row(index)
  idx = named_index_for index

  raise IndexError, "Index #{index} does not exist." unless @index.include? idx

  @index = DaruLite::Index.new(@index.to_a - [idx])
  each_vector do |vector|
    vector.delete_at idx
  end

  set_size
end

#delete_vector(vector) ⇒ Object

Delete a vector

Raises:

  • (IndexError) 


259
260
261
262
263
264
265
266
 
# File 'lib/daru_lite/dataframe.rb', line 259

def delete_vector(vector)
  raise IndexError, "Vector #{vector} does not exist." unless @vectors.include?(vector)

  @data.delete_at @vectors[vector]
  @vectors = DaruLite::Index.new @vectors.to_a - [vector]

  self
end

#delete_vectors(*vectors) ⇒ Object

Deletes a list of vectors



269
270
271
272
273
 
# File 'lib/daru_lite/dataframe.rb', line 269

def delete_vectors(*vectors)
  Array(vectors).each { |vec| delete_vector vec }

  self
end

#inspect(spacing = DaruLite.spacing, threshold = DaruLite.max_rows) ⇒ Object

Pretty print in a nice table format for the command line (irb/pry/iruby)



442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
 
# File 'lib/daru_lite/dataframe.rb', line 442

def inspect(spacing = DaruLite.spacing, threshold = DaruLite.max_rows)
  name_part = @name ? ": #{@name} " : ''
  spacing = [
    headers.to_a.map { |header| header.try(:length) || header.to_s.length }.max,
    spacing
  ].max

  "#<#{self.class}#{name_part}(#{nrows}x#{ncols})>#{$INPUT_RECORD_SEPARATOR}" +
    Formatters::Table.format(
      each_row.lazy,
      row_headers: row_headers,
      headers: headers,
      threshold: threshold,
      spacing: spacing
    )
end

#interact_code(vector_names, full) ⇒ Object 



504
505
506
507
508
509
510
511
512
 
# File 'lib/daru_lite/dataframe.rb', line 504

def interact_code(vector_names, full)
  dfs = vector_names.zip(full).map do |vec_name, f|
    self[vec_name].contrast_code(full: f).each.to_a
  end

  all_vectors = recursive_product(dfs)
  DaruLite::DataFrame.new all_vectors,
                          order: all_vectors.map(&:name)
end

#ncols ⇒ Object

The number of vectors



354
355
356
 
# File 'lib/daru_lite/dataframe.rb', line 354

def ncols
  @vectors.size
end

#nest(*tree_keys, &block) ⇒ Object

Return a nested hash using vector names as keys and an array constructed of hashes with other values. If block provided, is used to provide the values, with parameters row of dataset, current last hash on hierarchy and name of the key to include



319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
 
# File 'lib/daru_lite/dataframe.rb', line 319

def nest(*tree_keys, &block)
  tree_keys = tree_keys[0] if tree_keys[0].is_a? Array

  each_row.with_object({}) do |row, current|
    # Create tree
    *keys, last = tree_keys
    current = keys.inject(current) { |c, f| c[row[f]] ||= {} }
    name = row[last]

    if block
      current[name] = yield(row, current, name)
    else
      current[name] ||= []
      current[name].push(row.to_h.delete_if { |key, _value| tree_keys.include? key })
    end
  end
end

#nrows ⇒ Object

The number of rows



349
350
351
 
# File 'lib/daru_lite/dataframe.rb', line 349

def nrows
  @index.size
end

#rename(new_name) ⇒ Object Also known as: name=

Rename the DataFrame.



424
425
426
427
 
# File 'lib/daru_lite/dataframe.rb', line 424

def rename(new_name)
  @name = new_name
  self
end

#rename_vectors(name_map) ⇒ Object

Renames the vectors

Arguments

  • name_map - A hash where the keys are the exising vector names and

    the values are the new names.  If a vector is renamed
to a vector name that is already in use, the existing
one is overwritten.

Usage

df = DaruLite::DataFrame.new({ a: [1,2,3,4], b: [:a,:b,:c,:d], c: [11,22,33,44] })
df.rename_vectors :a => :alpha, :c => :gamma
df.vectors.to_a #=> [:alpha, :b, :gamma]



372
373
374
375
376
377
378
 
# File 'lib/daru_lite/dataframe.rb', line 372

def rename_vectors(name_map)
  existing_targets = name_map.reject { |k, v| k == v }.values & vectors.to_a
  delete_vectors(*existing_targets)

  new_names = vectors.to_a.map { |v| name_map[v] || v }
  self.vectors = DaruLite::Index.new new_names
end

#rename_vectors!(name_map) ⇒ Object

Renames the vectors and returns itself

Arguments

  • name_map - A hash where the keys are the exising vector names and

    the values are the new names.  If a vector is renamed
to a vector name that is already in use, the existing
one is overwritten.

Usage

df = DaruLite::DataFrame.new({ a: [1,2,3,4], b: [:a,:b,:c,:d], c: [11,22,33,44] })
df.rename_vectors! :a => :alpha, :c => :gamma # df



393
394
395
396
 
# File 'lib/daru_lite/dataframe.rb', line 393

def rename_vectors!(name_map)
  rename_vectors(name_map)
  self
end

#respond_to_missing?(name, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


500
501
502
 
# File 'lib/daru_lite/dataframe.rb', line 500

def respond_to_missing?(name, include_private = false)
  name.to_s.end_with?('=') || has_vector?(name) || super
end

#row ⇒ Object

Access a row or set/create a row. Refer #[] and #[]= docs for details.

Usage

df.row[:a] # access row named ':a'
df.row[:b] = [1,2,3] # set row ':b' to [1,2,3]



254
255
256
 
# File 'lib/daru_lite/dataframe.rb', line 254

def row
  DaruLite::Accessors::DataFrameByRow.new(self)
end

#shape ⇒ Object

Return the number of rows and columns of the DataFrame in an Array.



344
345
346
 
# File 'lib/daru_lite/dataframe.rb', line 344

def shape
  [nrows, ncols]
end

#to_category(*names) ⇒ DaruLite::DataFrame

Converts the specified non category type vectors to category type vectors

Examples:

df = DaruLite::DataFrame.new({
  a: [1, 2, 3],
  b: ['a', 'a', 'b']
})
df.to_category :b
df[:b].type
# => :category

Parameters:

  • names (Array) —

    of non category type vectors to be converted

Returns:

  • (DaruLite::DataFrame) —

    data frame in which specified vectors have been converted to category type



479
480
481
482
 
# File 'lib/daru_lite/dataframe.rb', line 479

def to_category(*names)
  names.each { |n| self[n] = self[n].to_category }
  self
end

#transpose ⇒ Object

Transpose a DataFrame, tranposing elements and row, column indexing.



431
432
433
434
435
436
437
438
439
 
# File 'lib/daru_lite/dataframe.rb', line 431

def transpose
  DaruLite::DataFrame.new(
    each_vector.map(&:to_a).transpose,
    index: @vectors,
    order: @index,
    dtype: @dtype,
    name: @name
  )
end

#update ⇒ Object

Method for updating the metadata (i.e. missing value positions) of the after assingment/deletion etc. are complete. This is provided so that time is not wasted in creating the metadata for the vector each time assignment/deletion of elements is done. Updating data this way is called lazy loading. To set or unset lazy loading, see the .lazy_update= method.



419
420
421
 
# File 'lib/daru_lite/dataframe.rb', line 419

def update
  @data.each(&:update) if DaruLite.lazy_update
end

#which ⇒ Object

a simple query DSL for accessing where(), inspired by gem “squeel” e.g.: df.which{ ‘FamilySize` == `FamilySize`.max } equals df.where( df.eq( df.max ) )

e.g.: df.which{ (‘NameTitle` == ’Dr’) & (‘Sex` == ’female’) } equals df.where( df.eq(‘Dr’) & df.eq(‘female’) )



15
16
17
 
# File 'lib/daru_lite/extensions/which_dsl.rb', line 15

def which(&)
  WhichQuery.new(self, &).exec
end