Class: Yobi::ResticOutput

Inherits:
Object
  • Object
show all
Defined in:
lib/yobi/restic_output.rb,
sig/yobi.rbs

Overview

Wraps the tempfile a repository operation's stdout+stderr are both written to, with random access so a caller asking only for the last line (see #last_line) doesn't have to read the entire output.

Defined Under Namespace

Classes: LazyList

Constant Summary collapse

DEFAULT_TAIL_CHUNK_SIZE =

Default read window size, in bytes, for #last_line.

Returns:

  • (Integer) 
4096

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(tail_chunk_size: DEFAULT_TAIL_CHUNK_SIZE, transform: nil) ⇒ ResticOutput

Returns a new instance of ResticOutput.

Parameters:

  • tail_chunk_size (Integer) (defaults to: DEFAULT_TAIL_CHUNK_SIZE) —

    read window size for #last_line

  • transform (Proc, nil) (defaults to: nil) —

    used by #write_and_parse during a streaming run to transform each message, and as #messages' default transform afterward, when no explicit one is given

  • tail_chunk_size: (Integer) (defaults to: DEFAULT_TAIL_CHUNK_SIZE)
  • transform: (^(Hash[String, untyped]) -> untyped) (defaults to: nil) 


22
23
24
25
26
27
28
29
 
# File 'lib/yobi/restic_output.rb', line 22

def initialize(tail_chunk_size: DEFAULT_TAIL_CHUNK_SIZE, transform: nil)
  @file = Tempfile.new("yobi-restic-output")
  @file.unlink
  @tail_chunk_size = tail_chunk_size
  @transform = transform
  @index = nil
  @stderr_offsets = []
end

Instance Attribute Details

#file ⇒ File (readonly)

Returns a fresh, already-unlinked tempfile of its own.

Returns:

  • (File) —

    a fresh, already-unlinked tempfile of its own



17
18
19
 
# File 'lib/yobi/restic_output.rb', line 17

def file
  @file
end

Instance Method Details

#each_line ⇒ void #each_line ⇒ Enumerator[String, void]

Returns if no block is given.

Overloads:

  • #each_line ⇒ void

    This method returns an undefined value.

  • #each_line ⇒ Enumerator[String, void]

    Returns:

    • (Enumerator[String, void])

Yields:

Yield Parameters:

  • line (String)

Yield Returns:

  • (void)

Returns:

  • (Enumerator) —

    if no block is given



71
72
73
74
75
76
 
# File 'lib/yobi/restic_output.rb', line 71

def each_line(&block)
  return enum_for(:each_line) unless block_given?

  @file.rewind
  @file.each_line(&block)
end

#index ⇒ Hash{String => Array<Integer>}

Byte offsets of every line, grouped by message_type. Built once and memoized.

Returns:

  • (Hash{String => Array<Integer>}) 


35
36
37
 
# File 'lib/yobi/restic_output.rb', line 35

def index
  @index ||= build_index
end

#inspect ⇒ String

Returns:

  • (String) 


144
145
146
 
# File 'lib/yobi/restic_output.rb', line 144

def inspect
  "#<#{self.class} #{@file.size} bytes>"
end

#last_line ⇒ String?

The last non-blank line, found by seeking backward from the end of the file.

Returns:

  • (String, nil) 


50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/yobi/restic_output.rb', line 50

def last_line
  size = @file.size
  return nil if size.zero?

  window = @tail_chunk_size
  loop do
    read_size = [window, size].min
    @file.seek(size - read_size)
    lines = @file.read(read_size).each_line.to_a
    lines.shift if read_size < size

    line = lines.reverse_each.find { |candidate| !candidate.strip.empty? }
    return line.strip if line
    return nil if read_size == size

    window *= 2
  end
end

#messages ⇒ void #messages(message_type) ⇒ Enumerator[Hash[String, untyped], void]

Every message recorded in #index, restricted to one message_type if given, in file order. Without an explicit transform block, uses the transform: given to #initialize (if any); with neither, just yields/returns the raw parsed Hash. Whenever a transform (explicit or the stored one) applies, wraps the result in a LazyList instead of yielding.

Overloads:

  • #messages ⇒ void

    This method returns an undefined value.

  • #messages(message_type) ⇒ Enumerator[Hash[String, untyped], void]

    Parameters:

    • message_type (String, nil)

    Returns:

    • (Enumerator[Hash[String, untyped], void])

Parameters:

  • message_type (String, nil) (defaults to: nil) —

    every message, if omitted

Yield Parameters:

  • message (Hash) —

    the raw parsed message, to transform

Returns:



89
90
91
92
93
94
95
 
# File 'lib/yobi/restic_output.rb', line 89

def messages(message_type = nil, &transform)
  transform ||= @transform
  return raw_messages(message_type) unless transform

  offsets = message_type ? index[message_type] : index.values.flatten.sort
  LazyList.new(self, offsets, transform)
end

#read_line_at(offset) ⇒ String

Returns the line starting at that offset.

Parameters:

  • offset (Integer) —

    a byte offset, as recorded by #index

Returns:

  • (String) —

    the line starting at that offset



41
42
43
44
 
# File 'lib/yobi/restic_output.rb', line 41

def read_line_at(offset)
  @file.seek(offset)
  @file.gets
end

#stderr_lines ⇒ void #stderr_lines ⇒ Enumerator[String, void]

Every line known to have arrived on stderr rather than stdout during a streaming run (see #write_and_parse), in file order.

Overloads:

  • #stderr_lines ⇒ void

    This method returns an undefined value.

  • #stderr_lines ⇒ Enumerator[String, void]

    Returns:

    • (Enumerator[String, void])

Yields:

Yield Parameters:

  • line (String)

Yield Returns:

  • (void)

Returns:

  • (Enumerator) —

    if no block is given



102
103
104
105
106
 
# File 'lib/yobi/restic_output.rb', line 102

def stderr_lines
  return enum_for(:stderr_lines) unless block_given?

  @stderr_offsets.each { |offset| yield read_line_at(offset) }
end

#to_s ⇒ String

Returns the full captured output.

Returns:

  • (String) —

    the full captured output



138
139
140
141
 
# File 'lib/yobi/restic_output.rb', line 138

def to_s
  @file.rewind
  @file.read
end

#write_and_parse(line, origin) ⇒ Object?

Called once per line read from a live streaming run's stdout/stderr. Writes it to the file, parses it, and - if it's a JSON message - indexes it by message_type and returns it transformed via the transform: given to #initialize (or the raw parsed Hash, if none was given). A non-JSON line's offset is recorded in #stderr_lines instead, if origin is :stderr; either way, returns nil.

Parameters:

  • line (String)
  • origin (:stdout, :stderr)

Returns:

  • (Object, nil) 


118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
 
# File 'lib/yobi/restic_output.rb', line 118

def write_and_parse(line, origin)
  offset = @file.pos
  @file.write(line)

  parsed = begin
    JSON.parse(line)
  rescue JSON::ParserError
    nil
  end

  if parsed
    (@index ||= Hash.new { |hash, message_type| hash[message_type] = [] })[parsed["message_type"]] << offset if parsed["message_type"]
    @transform ? @transform.call(parsed) : parsed
  elsif origin == :stderr
    @stderr_offsets << offset
    nil
  end
end