Class: ElasticGraph::DatastoreCore::IndexDefinition::RolloverIndexTemplate

Inherits:
Object
  • Object
show all
Includes:
Base
Defined in:
lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb

Instance Method Summary collapse

Methods included from Base

#accessible_cluster_names_to_index_into, #accessible_from_queries?, #all_accessible_cluster_names, #cluster_to_query, #clusters_to_index_into, #flattened_env_setting_overrides, #has_custom_routing?, #ignored_values_for_routing, #known_related_query_rollover_indices, #list_counts_field_paths_for_source, #max_result_window, #routing_value_for_prepared_record, #searches_could_hit_incomplete_docs?, #to_s

Instance Method Details

#delete_from_datastore(datastore_client) ⇒ Object

We need to delete both the template and the actual indices for rollover indices



46
47
48
49
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 46

def delete_from_datastore(datastore_client)
  datastore_client.delete_index_template(name)
  datastore_client.delete_indices(index_expression_for_search)
end

#index_expression_for_search ⇒ Object

Two underscores used to avoid collisions with other types (e.g. payments_2020 and payments_xyz_2020), though regardless shouldn’t happen if types follow naming conventions.



63
64
65
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 63

def index_expression_for_search
  index_name_with_suffix("*")
end

#index_name_for_writes(record, timestamp_field_path: nil) ⇒ Object

Returns an index name to use for write operations. The index_definition selection is a function of the index_definition’s rollover configuration and the record’s timestamp.



69
70
71
72
73
74
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 69

def index_name_for_writes(record, timestamp_field_path: nil)
  index_name_with_suffix(rollover_index_suffix_for_record(
    record,
    timestamp_field_path: timestamp_field_path || self.timestamp_field_path
  ))
end

#mappings_in_datastore(datastore_client) ⇒ Object 



39
40
41
42
43
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 39

def mappings_in_datastore(datastore_client)
  IndexConfigNormalizer.normalize_mappings(
    datastore_client.get_index_template(name).dig("template", "mappings") || {}
  )
end

#related_rollover_index_for_timestamp(timestamp, setting_overrides = {}) ⇒ Object

Gets a single related ‘RolloverIndex` for a given timestamp.



121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 121

def related_rollover_index_for_timestamp(timestamp, setting_overrides = {})
  # @type var record: ::Hash[::String, untyped]
  # We need to use `__skip__` here because `inner_value` has different types on different
  # block iterations: initially, it's a string, then it becomes a hash. Steep has trouble
  # with this but it works fine.
  __skip__ = record = timestamp_field_path.split(".").reverse.reduce(timestamp) do |inner_value, field_name|
    {field_name => inner_value}
  end

  concrete_rollover_index_for(index_name_for_writes(record), setting_overrides)
end

#related_rollover_indices(datastore_client, only_if_exists: false) ⇒ Object

Returns a list of indices related to this template. This includes both indices that are specified in our configuration settings (e.g. via ‘setting_overrides_by_timestamp` and `custom_time_sets`) and also indices that have been auto-created from the template.

Note that there can be discrepancies between the configuration settings and the indices in the datastore. Sometimes this is planned/expected (e.g. such as when invoking ‘elasticgraph-admin` to configure an index newly defined in configuration) and in other cases it’s not.

The ‘only_if_exists` argument controls how a discrepancy is treated.

  • When ‘false` (the default), indices that are defined in config but do not exist in the datastore are still returned. This is generally what we want for indexing and cluster administration.

  • When ‘true`, any indices in our configuration that do not exist are ignored, and not included in the returned list. This is appropriate for searching the datastore: if we attempt to exclude an index which is defined in config but does not exist (e.g. via `-[index_name]` in the search index expression), the datastore will return an error, but we can safely ignore the index. Likewise, if we have an index in the datastore which we cannot infer a timestamp range, we need to ignore it to avoid getting errors. Ignoring an index is safe when searching because our search logic uses a wildcard to match all indices with the same prefix, and then excludes certain known indices that it can safely exclude based on their timestamp range. Ignored indices which exist will still be searched.

In addition, any indices which exist, but which are not controlled by our current configuration, are ignored. Examples:

- An index with a custom suffix (e.g. `__before_2019`) which has no corresponding configuration. We have no way to guess
  what the timestamp range is for such an index, and we want to completely ignore it.
- An index with for a different rollover frequency than our current configuration. For example, a `__2019-03` index,
  which must rollover monthly, would be ignored if our current rollover frequency is yearly or daily.

These latter cases are quite rare but can happen when we are dealing with indices defined before an update to our configuration. Our searches will continue to search these indices so long as their name matches the pattern, and we otherwise want to ignore these indices (e.g. we don’t want admin to attempt to configure them, or want our indexer to attempt to write to them).



107
108
109
110
111
112
113
114
115
116
117
118
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 107

def related_rollover_indices(datastore_client, only_if_exists: false)
  config_indices_by_name = rollover_indices_to_pre_create.to_h { |i| [i.name, i] }

  db_indices_by_name = datastore_client.list_indices_matching(index_expression_for_search).filter_map do |name|
    index = concrete_rollover_index_for(name, {}, config_indices_by_name[name]&.time_set)
    [name, index] if index
  end.to_h

  config_indices_by_name = config_indices_by_name.slice(*db_indices_by_name.keys) if only_if_exists

  db_indices_by_name.merge(config_indices_by_name).values
end

#rollover_index_template? ⇒ Boolean

Indicates if this is a rollover index definition.

Use of this is considered a mild code smell. When feasible, it’s generally better to implement a new polymorphic API on the IndexDefinition interface, rather then branching on the value of this predicate.

Returns:

  • (Boolean) 


56
57
58
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 56

def rollover_index_template?
  true
end