Class: Karafka::Core::Monitoring::StatisticsDecorator

Inherits:
Object
  • Object
show all
Includes:
Helpers::Time
Defined in:
lib/karafka/core/monitoring/statistics_decorator.rb

Overview

Many of the librdkafka statistics are absolute values instead of a gauge. This means, that for example number of messages sent is an absolute growing value instead of being a value of messages sent from the last statistics report. This decorator calculates the diff against previously emitted stats, so we get also the diff together with the original values

It adds two extra values to numerics:

- KEY_d - delta of the previous value and current
- KEY_fd - freeze duration - describes how long the delta remains unchanged (zero)
           and can be useful for detecting values that "hang" for extended period of time
           and do not have any change (delta always zero). This value is in ms for the
           consistency with other time operators we use.

Instance Method Summary collapse

Methods included from Helpers::Time

#float_now, #monotonic_now

Constructor Details

#initialize(excluded_keys: [], only_keys: []) ⇒ StatisticsDecorator

Returns a new instance of StatisticsDecorator.

Parameters:

  • excluded_keys (Array<String>) (defaults to: []) —

    list of key names to skip entirely during decoration. Excluded keys are not recursed into and not decorated with delta/freeze duration suffixes. This is useful for skipping large subtrees of the librdkafka statistics that are not consumed by the application (e.g. broker toppars, window stats like int_latency, outbuf_latency, throttle, batchsize, batchcnt, req).

  • only_keys (Array<String>) (defaults to: []) —

    when set, only these numeric keys will be decorated with delta/freeze duration suffixes. Hash children are still recursed into for structural navigation, but only the listed keys receive _d and _fd computation. This drastically reduces work at the partition level where most cost occurs. When empty (default), all numeric keys are decorated.



36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
 
# File 'lib/karafka/core/monitoring/statistics_decorator.rb', line 36

def initialize(excluded_keys: [], only_keys: [])
  @previous = EMPTY_HASH
  # Operate on ms precision only
  @previous_at = monotonic_now.round
  # Cache for memoized suffix keys to avoid repeated string allocations
  @suffix_keys_cache = {}
  # Frozen hash for O(1) key exclusion lookup, nil when empty to avoid per-key
  # lookups in the hot loop when no exclusions are configured
  @excluded_keys = unless excluded_keys.empty?
    excluded_keys.each_with_object({}) { |k, h| h[k] = true }.freeze
  end
  # Frozen array for direct-access decoration, nil when empty to use full decoration
  @only_keys = unless only_keys.empty?
    only_keys.freeze
  end
end

Instance Method Details

#call(emitted_stats) ⇒ Hash

Note:

We modify the emitted statistics, instead of creating new. Since we don’t expose any API to get raw data, users can just assume that the result of this decoration is the proper raw stats that they can use

Returns emitted statistics extended with the diff data.

Parameters:

  • emitted_stats (Hash) —

    original emitted statistics

Returns:

  • (Hash) —

    emitted statistics extended with the diff data



58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/karafka/core/monitoring/statistics_decorator.rb', line 58

def call(emitted_stats)
  current_at = monotonic_now.round
  change_d = current_at - @previous_at

  diff(
    @previous,
    emitted_stats,
    change_d
  )

  @previous = emitted_stats
  @previous_at = current_at

  emitted_stats.freeze
end