Class: Karafka::Web::Tracking::Consumers::Sampler::Metrics::Os

Inherits:
Base
  • Object
show all
Defined in:
lib/karafka/web/tracking/consumers/sampler/metrics/os.rb

Overview

Collects OS-level metrics from /proc filesystem and system commands Used when running directly on a host OS (not in containers)

Direct Known Subclasses

Container

Instance Method Summary collapse

Constructor Details

#initialize(shell) ⇒ Os

Returns a new instance of Os.

Parameters:

  • shell (MemoizedShell) —

    shell executor for running system commands



13
14
15
16
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 13

def initialize(shell)
  super()
  @shell = shell
end

Instance Method Details

#cpu_usage ⇒ Array<Float>

Returns load averages for last 1, 5 and 15 minutes.

Returns:

  • (Array<Float>) —

    load averages for last 1, 5 and 15 minutes



92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 92

def cpu_usage
  case RUBY_PLATFORM
  when /linux/
    File
      .read("/proc/loadavg")
      .split
      .first(3)
      .map(&:to_f)
  when /darwin|bsd/
    shell
      .call("w | head -1")
      .strip
      .split
      .map(&:to_f)
      .last(3)
  else
    [-1, -1, -1]
  end
end

#cpus ⇒ Integer

Returns CPU count.

Returns:

  • (Integer) —

    CPU count



113
114
115
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 113

def cpus
  @cpus ||= Etc.nprocessors
end

#memory_size ⇒ Integer

Note:

This is a STATIC value (system RAM capacity), memoized for performance

Note:

Used in Web UI to show “OS memory available” metric

This is the total physical memory available to the system/container. On Linux: reads MemTotal from /proc/meminfo On macOS: uses sysctl hw.memsize In containers: Container class overrides this to return cgroup memory limit

Returns:

  • (Integer) —

    total amount of available memory in kilobytes



69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 69

def memory_size
  return @memory_size if instance_variable_defined?(:@memory_size)

  @memory_size = case RUBY_PLATFORM
  when /linux/
    mem_info = File.read("/proc/meminfo")
    mem_total_line = mem_info.match(/MemTotal:\s*(?<total>\d+)/)
    mem_total_line["total"].to_i
  when /darwin|bsd/
    shell
      .call("sysctl -a")
      .split("\n")
      .find { |line| line.start_with?("hw.memsize:") }
      .to_s
      .split
      .last
      .to_i
  else
    0
  end
end

#memory_threads_ps ⇒ Array<Array<Integer, Integer, Integer>>, false

Note:

Sampler calls this once per sample cycle (every ~5 seconds) and caches the result in @memory_threads_ps to ensure consistent data within a single sample snapshot

Note:

The cache is refreshed on EVERY sample cycle, so data stays current

Note:

On Linux, thread count is only extracted for the current process to optimize performance

Loads process information for all running processes This method reads information about ALL processes on the system (or in the container). The data is used by multiple metrics:

  • memory_total_usage: sums RSS across all processes

  • threads: extracts thread count for current process

Format of each array element: [memory_in_kb, thread_count, process_id]

  • memory_in_kb: RSS (Resident Set Size) in kilobytes

  • thread_count: Number of threads (only populated for current process, 0 for others)

  • process_id: Process ID

Platform behavior:

  • Linux: Reads /proc/*/statm for ALL processes on host/container

  • macOS: Uses ‘ps -A` to get all processes

  • Containers: Due to PID namespaces, only sees processes within the container

Returns:

  • (Array<Array<Integer, Integer, Integer>>, false) —

    array of

    rss_kb, threads, pid

    for each process, or false if unavailable



153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 153

def memory_threads_ps
  case RUBY_PLATFORM
  when /linux/
    page_size = Helpers::Sysconf.page_size
    current_pid = ::Process.pid

    # Read all processes from /proc
    Dir.glob("/proc/[0-9]*/statm").filter_map do |statm_file|
      pid = statm_file.match(%r{/proc/(\d+)/statm})[1].to_i
      status_file = "/proc/#{pid}/status"

      # Extract RSS from /proc/<pid>/statm (second field)
      rss_pages = begin
        File.read(statm_file).split[1].to_i
      rescue
        next # Process may have exited
      end

      # Extract thread count from /proc/<pid>/status (only for current process)
      thcount = if pid == current_pid
        begin
          File.read(status_file)[/^Threads:\s+(\d+)/, 1].to_i
        rescue
          0
        end
      else
        0
      end

      # Convert RSS from pages to kilobytes
      rss_kb = (rss_pages * page_size) / 1024

      [rss_kb, thcount, pid]
    end
  # thcount is not available on macos ps
  # because of that we inject 0 as threads count similar to how we do on windows
  when /darwin|bsd/
    shell
      .call("ps -A -o rss=,pid=")
      .split("\n")
      .map { |row| row.strip.split.map(&:to_i) }
      .map { |row| [row.first, 0, row.last] }
  else
    false
  end
end

#memory_total_usage(memory_threads_ps) ⇒ Integer

Note:

This is DIFFERENT from memory_usage which only shows current process memory

Note:

Used in Web UI to show “OS memory used” metric

This represents system-wide (or container-wide) memory usage by summing RSS across all processes. On bare metal: sums memory for all processes on the host In containers: sums memory for all processes within the container

(due to PID namespace)

Parameters:

  • memory_threads_ps (Array, false) —

    parsed ps/proc output for all processes

Returns:

  • (Integer) —

    total memory used by all processes in the system (or container)



56
57
58
59
60
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 56

def memory_total_usage(memory_threads_ps)
  return 0 unless memory_threads_ps

  memory_threads_ps.sum(&:first)
end

#memory_usage ⇒ Integer

Note:

This represents ONLY the current Karafka process memory usage

This is the amount of physical memory currently used by the Karafka process. On Linux: reads VmRSS from /proc/pid/status On macOS: uses ps command to get RSS for current process

Returns:

  • (Integer) —

    memory used by this process in kilobytes (RSS - Resident Set Size)



23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 23

def memory_usage
  pid = ::Process.pid

  case RUBY_PLATFORM
  # Reading this that way is cheaper than running a shell command
  when /linux/
    File.readlines("/proc/#{pid}/status").each do |line|
      next unless line.start_with?("VmRSS:")

      break line.split[1].to_i
    end
  when /darwin|bsd/
    shell
      .call("ps -o pid,rss -p #{pid}")
      .lines
      .last
      .split
      .last
      .to_i
  else
    0
  end
end

#threads(memory_threads_ps) ⇒ Integer

Note:

This returns total number of threads from the OS perspective including native extensions threads, etc.

Returns number of process threads.

Parameters:

  • memory_threads_ps (Array, false) —

    parsed ps output

Returns:

  • (Integer) —

    number of process threads



121
122
123
124
125
 
# File 'lib/karafka/web/tracking/consumers/sampler/metrics/os.rb', line 121

def threads(memory_threads_ps)
  return 0 unless memory_threads_ps

  memory_threads_ps.find { |row| row.last == ::Process.pid }[1]
end