Class: DaruLite::DataFrame

Inherits:
Object show all
Extended by:
Gem::Deprecate
Includes:
Aggregatable, Calculatable, Convertible, Duplicatable, Fetchable, Filterable, IOAble, Indexable, Iterable, Joinable, Missable, Pivotable, Queryable, Setable, Sortable, Maths::Arithmetic::DataFrame, Maths::Statistics::DataFrame
Defined in:
lib/daru_lite/dataframe.rb,
lib/daru_lite/data_frame/setable.rb,
lib/daru_lite/data_frame/i_o_able.rb,
lib/daru_lite/data_frame/iterable.rb,
lib/daru_lite/data_frame/joinable.rb,
lib/daru_lite/data_frame/missable.rb,
lib/daru_lite/data_frame/sortable.rb,
lib/daru_lite/data_frame/fetchable.rb,
lib/daru_lite/data_frame/indexable.rb,
lib/daru_lite/data_frame/pivotable.rb,
lib/daru_lite/data_frame/queryable.rb,
lib/daru_lite/extensions/which_dsl.rb,
lib/daru_lite/data_frame/filterable.rb,
lib/daru_lite/data_frame/convertible.rb,
lib/daru_lite/data_frame/aggregatable.rb,
lib/daru_lite/data_frame/calculatable.rb,
lib/daru_lite/data_frame/duplicatable.rb

Overview

rubocop:disable Metrics/ClassLength

Defined Under Namespace

Modules: Aggregatable, Calculatable, Convertible, Duplicatable, Fetchable, Filterable, IOAble, Indexable, Iterable, Joinable, Missable, Pivotable, Queryable, Setable, Sortable

Constant Summary collapse

AXES = 
%i[row vector].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Maths::Statistics::DataFrame

#acf, #correlation, #count, #covariance, #cumsum, #describe, #ema, #max, #mean, #median, #min, #mode, #percent_change, #product, #range, #rolling_count, #rolling_max, #rolling_mean, #rolling_median, #rolling_min, #rolling_std, #rolling_variance, #standardize, #std, #sum, #variance_sample

Methods included from Maths::Arithmetic::DataFrame

#%, #*, #**, #+, #-, #/, #exp, #round, #sqrt

Methods included from Queryable

#all?, #any?, #has_vector?, #include_values?

Methods included from Sortable

#order=, #rotate_vectors, #sort, #sort!

Methods included from Setable

#[]=, #add_row, #add_vector, #insert_vector, #set_at, #set_row_at

Methods included from Pivotable

#pivot_table

Methods included from Missable

#has_missing_data?, #missing_values_rows, #rolling_fillna, #rolling_fillna!

Methods included from Joinable

#concat, #join, #merge, #one_to_many, #union

Methods included from IOAble

#_dump, included, #save, #write_csv, #write_excel, #write_sql

Methods included from Iterable

#apply_method, #collect, #collect_matrix, #collect_row_with_index, #collect_rows, #collect_vector_with_index, #collect_vectors, #each, #each_index, #each_row, #each_row_with_index, #each_vector, #each_vector_with_index, #map, #map!, #map_rows, #map_rows!, #map_rows_with_index, #map_vectors, #map_vectors!, #map_vectors_with_index, #recode, #recode_rows, #recode_vectors, #replace_values, #verify

Methods included from Indexable

#index=, #reindex, #reindex_vectors, #reset_index, #set_index, #vectors=

Methods included from Filterable

#filter, #filter_rows, #filter_vector, #filter_vectors, #keep_row_if, #keep_vector_if, #reject_values, #uniq, #where

Methods included from Fetchable

#[], #access_row_tuples_by_indexs, #at, #get_sub_dataframe, #get_vector_anyways, #head, #numeric_vector_names, #numeric_vectors, #only_numerics, #row_at, #split_by_category, #tail

Methods included from Duplicatable

#clone, #clone_only_valid, #clone_structure, #dup, #dup_only_valid

Methods included from Convertible

#create_sql, #to_a, #to_df, #to_h, #to_html, #to_html_tbody, #to_html_thead, #to_json, #to_matrix, #to_s

Methods included from Calculatable

#compute, #summary, #vector_by_calculation, #vector_count_characters, #vector_mean, #vector_sum

Methods included from Aggregatable

#aggregate, #group_by, #group_by_and_aggregate

Constructor Details

#initialize(source = {}, opts = {}) ⇒ DataFrame

DataFrame basically consists of an Array of Vector objects. These objects are indexed by row and column by vectors and index Index objects.

Arguments

  • source - Source from the DataFrame is to be initialized. Can be a Hash of names and vectors (array or DaruLite::Vector), an array of arrays or array of DaruLite::Vectors.

Options

:order - An Array/DaruLite::Index/DaruLite::MultiIndex containing the order in which Vectors should appear in the DataFrame.

:index - An Array/DaruLite::Index/DaruLite::MultiIndex containing the order in which rows of the DataFrame will be named.

:name - A name for the DataFrame.

:clone - Specify as true or false. When set to false, and Vector objects are passed for the source, the Vector objects will not duplicated when creating the DataFrame. Will have no effect if Array is passed in the source, or if the passed DaruLite::Vectors have different indexes. Default to true.

Usage

df = DaruLite::DataFrame.new
# =>
# <DaruLite::DataFrame(0x0)>
# Creates an empty DataFrame with no rows or columns.

df = DaruLite::DataFrame.new({}, order: [:a, :b])
#<DaruLite::DataFrame(0x2)>
a   b
# Creates a DataFrame with no rows and columns :a and :b

df = DaruLite::DataFrame.new({a: [1,2,3,4], b: [6,7,8,9]}, order: [:b, :a],
index: [:a, :b, :c, :d], name: :spider_man)

# =>
# <DaruLite::DataFrame:80766980 @name = spider_man @size = 4>
#             b          a
#  a          6          1
#  b          7          2
#  c          8          3
#  d          9          4

df = DaruLite::DataFrame.new([[1,2,3,4],[6,7,8,9]], name: :bat_man)

# =>
# #<DaruLite::DataFrame: bat_man (4x2)>
#             0          1
#  0          1          6
#  1          2          7
#  2          3          8
#  3          4          9

# Dataframe having Index name

df = DaruLite::DataFrame.new({a: [1,2,3,4], b: [6,7,8,9]}, order: [:b, :a],
index: DaruLite::Index.new([:a, :b, :c, :d], name: 'idx_name'),
name: :spider_man)

# =>
# <DaruLite::DataFrame:80766980 @name = spider_man @size = 4>
# idx_name            b          a
#        a          6          1
#        b          7          2
#        c          8          3
#        d          9          4


idx = DaruLite::Index.new [100, 99, 101, 1, 2], name: "s1"
=> #<DaruLite::Index(5): s1 {100, 99, 101, 1, 2}>

df = DaruLite::DataFrame.new({b: [11,12,13,14,15], a: [1,2,3,4,5],
c: [11,22,33,44,55]},
order: [:a, :b, :c],
index: idx)
# =>
#<DaruLite::DataFrame(5x3)>
#   s1   a   b   c
#  100   1  11  11
#   99   2  12  22
#  101   3  13  33
#    1   4  14  44
#    2   5  15  55


237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
 
# File 'lib/daru_lite/dataframe.rb', line 237

def initialize(source = {}, opts = {})
  vectors = opts[:order]
  index = opts[:index] # FIXME: just keyword arges after Ruby 2.1
  @data = []
  @name = opts[:name]

  case source
  when [], {}
    create_empty_vectors(vectors, index)
  when Array
    initialize_from_array source, vectors, index, opts
  when Hash
    initialize_from_hash source, vectors, index, opts
  end

  set_size
  validate
  update
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args) ⇒ Object 



498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
 
# File 'lib/daru_lite/dataframe.rb', line 498

def method_missing(name, *args, &)
  stringified_name = name.to_s

  if /^([^=]+)=/.match?(stringified_name)
    name = stringified_name[/^([^=]+)=/].delete('=')
    name = name.to_sym unless has_vector?(name)
    insert_or_modify_vector [name], args[0]
  elsif has_vector?(name)
    self[name]
  elsif has_vector?(stringified_name)
    self[stringified_name]
  else
    super
  end
end

Instance Attribute Details

#data ⇒ Object (readonly)

TOREMOVE



137
138
139
 
# File 'lib/daru_lite/dataframe.rb', line 137

def data
  @data
end

#index ⇒ Object (readonly)

The index of the rows of the DataFrame



140
141
142
 
# File 'lib/daru_lite/dataframe.rb', line 140

def index
  @index
end

#name ⇒ Object (readonly)

The name of the DataFrame



143
144
145
 
# File 'lib/daru_lite/dataframe.rb', line 143

def name
  @name
end

#size ⇒ Object (readonly)

The number of rows present in the DataFrame



146
147
148
 
# File 'lib/daru_lite/dataframe.rb', line 146

def size
  @size
end

#vectors ⇒ Object (readonly)

The vectors (columns) index of the DataFrame



135
136
137
 
# File 'lib/daru_lite/dataframe.rb', line 135

def vectors
  @vectors
end

Class Method Details

.crosstab_by_assignation(rows, columns, values) ⇒ Object

Generates a new dataset, using three vectors

  • Rows
  • Columns
  • Values

For example, you have these values

x   y   v
a   a   0
a   b   1
b   a   1
b   b   0

You obtain id a b a 0 1 b 1 0

Useful to process outputs from databases



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/daru_lite/dataframe.rb', line 84

def crosstab_by_assignation(rows, columns, values)
  raise 'Three vectors should be equal size' if
    rows.size != columns.size || rows.size != values.size

  row_index = rows.uniq.to_a
  data = Hash.new do |h, col|
    h[col] = row_index.map { |r| [r, nil] }.to_h
  end
  validate_no_duplicate_pairs(rows, columns)
  columns.zip(rows, values).each { |c, r, v| data[c][r] = v }

  # FIXME: in fact, WITHOUT this line you'll obtain more "right"
  # data: with vectors having "rows" as an index...
  data = data.transform_values(&:values)
  data[:_id] = row_index

  DataFrame.new(data)
end

.rows(source, opts = {}) ⇒ Object

Create DataFrame by specifying rows as an Array of Arrays or Array of DaruLite::Vector objects.

Raises:



50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/daru_lite/dataframe.rb', line 50

def rows(source, opts = {})
  raise SizeError, 'All vectors must have same length' \
    unless source.all? { |v| v.size == source.first.size }

  opts[:order] ||= guess_order(source)

  if ArrayHelper.array_of?(source, Array) || source.empty?
    DataFrame.new(source.transpose, opts)
  elsif ArrayHelper.array_of?(source, Vector)
    from_vector_rows(source, opts)
  else
    raise ArgumentError, "Can't create DataFrame from #{source}"
  end
end

Instance Method Details

#==(other) ⇒ Object 



471
472
473
474
475
476
477
478
479
 
# File 'lib/daru_lite/dataframe.rb', line 471

def ==(other)
  self.class == other.class   &&
    @size    == other.size    &&
    @index   == other.index   &&
    @vectors == other.vectors &&
    # Compare columns by position: name-based access is ambiguous (and recurses infinitely)
    # when the vectors are a MultiIndex with duplicate tuples.
    @data    == other.data
end

#add_level_to_vectors(top_level_label) ⇒ Object

Converts the vectors to a DaruLite::MultiIndex. The argument passed is used as the MultiIndex's top level



412
413
414
415
 
# File 'lib/daru_lite/dataframe.rb', line 412

def add_level_to_vectors(top_level_label)
  tuples = vectors.map { |label| [top_level_label, *label] }
  self.vectors = DaruLite::MultiIndex.from_tuples(tuples)
end

#add_vectors_by_split(name, join = '-', sep = DaruLite::SPLIT_TOKEN) ⇒ Object 



349
350
351
352
353
 
# File 'lib/daru_lite/dataframe.rb', line 349

def add_vectors_by_split(name, join = '-', sep = DaruLite::SPLIT_TOKEN)
  self[name]
    .split_by_separator(sep)
    .each { |k, v| self[:"#{name}#{join}#{k}"] = v }
end

#add_vectors_by_split_recode(nm, join = '-', sep = DaruLite::SPLIT_TOKEN) ⇒ Object 



417
418
419
420
421
422
423
424
 
# File 'lib/daru_lite/dataframe.rb', line 417

def add_vectors_by_split_recode(nm, join = '-', sep = DaruLite::SPLIT_TOKEN)
  self[nm]
    .split_by_separator(sep)
    .each_with_index do |(k, v), i|
      v.rename "#{nm}:#{k}"
      self[:"#{nm}#{join}#{i + 1}"] = v
    end
end

#bootstrap(n = nil) ⇒ DaruLite::DataFrame

Creates a DataFrame with the random data, of n size. If n not given, uses original number of rows.

Returns:



317
318
319
320
321
322
323
324
325
 
# File 'lib/daru_lite/dataframe.rb', line 317

def bootstrap(n = nil)
  n ||= nrows
  DaruLite::DataFrame.new({}, order: @vectors).tap do |df_boot|
    n.times do
      df_boot.add_row(row[rand(n)])
    end
    df_boot.update
  end
end

#delete_at_position(position) ⇒ Object

Delete a row based on its position More robust than #delete_row when working with a CategoricalIndex or when the Index includes integers

Raises:

  • (IndexError) 


304
305
306
307
308
309
310
311
 
# File 'lib/daru_lite/dataframe.rb', line 304

def delete_at_position(position)
  raise IndexError, "Position #{position} does not exist." unless position < size

  @index = @index.delete_at(position)
  each_vector { |vector| vector.delete_at_position(position) }

  set_size
end

#delete_row(index) ⇒ Object

Delete a row

Raises:

  • (IndexError) 


288
289
290
291
292
293
294
295
296
297
298
299
 
# File 'lib/daru_lite/dataframe.rb', line 288

def delete_row(index)
  idx = named_index_for index

  raise IndexError, "Index #{index} does not exist." unless @index.include? idx

  @index = DaruLite::Index.new(@index.to_a - [idx])
  each_vector do |vector|
    vector.delete_at idx
  end

  set_size
end

#delete_vector(vector) ⇒ Object

Delete a vector

Raises:

  • (IndexError) 


267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/daru_lite/dataframe.rb', line 267

def delete_vector(vector)
  raise IndexError, "Vector #{vector} does not exist." unless @vectors.include?(vector)

  # Delete the first matching column by position: a MultiIndex with duplicate tuples resolves a
  # name to several positions, and removing every duplicate name would desync @vectors from @data.
  # Reuse the index's own #delete_at, which preserves the index type.
  position = @vectors.is_a?(MultiIndex) ? @vectors.positions_for(vector).first : @vectors[vector]
  @data.delete_at(position)
  @vectors = @vectors.delete_at(position)

  self
end

#delete_vectors(*vectors) ⇒ Object

Deletes a list of vectors



281
282
283
284
285
 
# File 'lib/daru_lite/dataframe.rb', line 281

def delete_vectors(*vectors)
  Array(vectors).each { |vec| delete_vector vec }

  self
end

#inspect(spacing = DaruLite.spacing, threshold = DaruLite.max_rows) ⇒ Object

Pretty print in a nice table format for the command line (irb/pry/iruby)



454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
 
# File 'lib/daru_lite/dataframe.rb', line 454

def inspect(spacing = DaruLite.spacing, threshold = DaruLite.max_rows)
  name_part = @name ? ": #{@name} " : ''
  spacing = [
    headers.to_a.map { |header| header.try(:length) || header.to_s.length }.max,
    spacing
  ].max

  "#<#{self.class}#{name_part}(#{nrows}x#{ncols})>#{$INPUT_RECORD_SEPARATOR}" +
    Formatters::Table.format(
      each_row.lazy,
      row_headers: row_headers,
      headers: headers,
      threshold: threshold,
      spacing: spacing
    )
end

#interact_code(vector_names, full) ⇒ Object 



518
519
520
521
522
523
524
525
526
 
# File 'lib/daru_lite/dataframe.rb', line 518

def interact_code(vector_names, full)
  dfs = vector_names.zip(full).map do |vec_name, f|
    self[vec_name].contrast_code(full: f).each.to_a
  end

  all_vectors = recursive_product(dfs)
  DaruLite::DataFrame.new all_vectors,
                          order: all_vectors.map(&:name)
end

#ncols ⇒ Object

The number of vectors



366
367
368
 
# File 'lib/daru_lite/dataframe.rb', line 366

def ncols
  @vectors.size
end

#nest(*tree_keys, &block) ⇒ Object

Return a nested hash using vector names as keys and an array constructed of hashes with other values. If block provided, is used to provide the values, with parameters row of dataset, current last hash on hierarchy and name of the key to include



331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
 
# File 'lib/daru_lite/dataframe.rb', line 331

def nest(*tree_keys, &block)
  tree_keys = tree_keys[0] if tree_keys[0].is_a? Array

  each_row.with_object({}) do |row, current|
    # Create tree
    *keys, last = tree_keys
    current = keys.inject(current) { |c, f| c[row[f]] ||= {} }
    name = row[last]

    if block
      current[name] = yield(row, current, name)
    else
      current[name] ||= []
      current[name].push(row.to_h.delete_if { |key, _value| tree_keys.include? key })
    end
  end
end

#nrows ⇒ Object

The number of rows



361
362
363
 
# File 'lib/daru_lite/dataframe.rb', line 361

def nrows
  @index.size
end

#rename(new_name) ⇒ Object Also known as: name=

Rename the DataFrame.



436
437
438
439
 
# File 'lib/daru_lite/dataframe.rb', line 436

def rename(new_name)
  @name = new_name
  self
end

#rename_vectors(name_map) ⇒ Object

Renames the vectors

Arguments

  • name_map - A hash where the keys are the exising vector names and the values are the new names. If a vector is renamed to a vector name that is already in use, the existing one is overwritten.

Usage

df = DaruLite::DataFrame.new({ a: [1,2,3,4], b: [:a,:b,:c,:d], c: [11,22,33,44] })
df.rename_vectors :a => :alpha, :c => :gamma
df.vectors.to_a #=> [:alpha, :b, :gamma]


384
385
386
387
388
389
390
 
# File 'lib/daru_lite/dataframe.rb', line 384

def rename_vectors(name_map)
  existing_targets = name_map.reject { |k, v| k == v }.values & vectors.to_a
  delete_vectors(*existing_targets)

  new_names = vectors.to_a.map { |v| name_map[v] || v }
  self.vectors = DaruLite::Index.new new_names
end

#rename_vectors!(name_map) ⇒ Object

Renames the vectors and returns itself

Arguments

  • name_map - A hash where the keys are the exising vector names and the values are the new names. If a vector is renamed to a vector name that is already in use, the existing one is overwritten.

Usage

df = DaruLite::DataFrame.new({ a: [1,2,3,4], b: [:a,:b,:c,:d], c: [11,22,33,44] })
df.rename_vectors! :a => :alpha, :c => :gamma # df


405
406
407
408
 
# File 'lib/daru_lite/dataframe.rb', line 405

def rename_vectors!(name_map)
  rename_vectors(name_map)
  self
end

#respond_to_missing?(name, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


514
515
516
 
# File 'lib/daru_lite/dataframe.rb', line 514

def respond_to_missing?(name, include_private = false)
  name.to_s.end_with?('=') || has_vector?(name) || super
end

#row ⇒ Object

Access a row or set/create a row. Refer #[] and #[]= docs for details.

Usage

df.row # access row named ':a' df.row = [1,2,3] # set row ':b' to [1,2,3]



262
263
264
 
# File 'lib/daru_lite/dataframe.rb', line 262

def row
  DaruLite::Accessors::DataFrameByRow.new(self)
end

#shape ⇒ Object

Return the number of rows and columns of the DataFrame in an Array.



356
357
358
 
# File 'lib/daru_lite/dataframe.rb', line 356

def shape
  [nrows, ncols]
end

#to_category(*names) ⇒ DaruLite::DataFrame

Converts the specified non category type vectors to category type vectors

Examples:

df = DaruLite::DataFrame.new({
  a: [1, 2, 3],
  b: ['a', 'a', 'b']
})
df.to_category :b
df[:b].type
# => :category

Parameters:

  • names (Array) —

    of non category type vectors to be converted

Returns:

  • (DaruLite::DataFrame) —

    data frame in which specified vectors have been converted to category type



493
494
495
496
 
# File 'lib/daru_lite/dataframe.rb', line 493

def to_category(*names)
  names.each { |n| self[n] = self[n].to_category }
  self
end

#transpose ⇒ Object

Transpose a DataFrame, tranposing elements and row, column indexing.



443
444
445
446
447
448
449
450
451
 
# File 'lib/daru_lite/dataframe.rb', line 443

def transpose
  DaruLite::DataFrame.new(
    each_vector.map(&:to_a).transpose,
    index: @vectors,
    order: @index,
    dtype: @dtype,
    name: @name
  )
end

#update ⇒ Object

Method for updating the metadata (i.e. missing value positions) of the after assingment/deletion etc. are complete. This is provided so that time is not wasted in creating the metadata for the vector each time assignment/deletion of elements is done. Updating data this way is called lazy loading. To set or unset lazy loading, see the .lazy_update= method.



431
432
433
 
# File 'lib/daru_lite/dataframe.rb', line 431

def update
  @data.each(&:update) if DaruLite.lazy_update
end

#which ⇒ Object

a simple query DSL for accessing where(), inspired by gem "squeel" e.g.: df.which{ FamilySize == FamilySize.max } equals df.where( df.eq( df.max ) )

e.g.: df.which{ (NameTitle == 'Dr') & (Sex == 'female') } equals df.where( df.eq('Dr') & df.eq('female') )



15
16
17
 
# File 'lib/daru_lite/extensions/which_dsl.rb', line 15

def which(&)
  WhichQuery.new(self, &).exec
end