Class: Mbeditor::DuplicateContentScanner

Inherits:
Object
  • Object
show all
Defined in:
app/services/mbeditor/duplicate_content_scanner.rb

Overview

Finds files that look like they were written back to disk with their own content appended to themselves.

The corruption this detects came from the collaborative buffer: two clients that each found an empty shared document seeded it from disk, and Yjs merged both inserts at offset 0 into two concatenated copies. That bug is fixed (CollaborationDocStore.claim_seed), but files saved while it was live are still on disk, so there has to be a way to find them.

Two signals, cheapest first:

* :exact          — the file is byte-for-byte X + X. What a duplicated save
                  produces before anyone edits it again.
* :repeated_block — the first ANCHOR_LINES lines of the file occur again
                  later, verbatim. Survives edits made after the
                  corruption, which is the common case for a file noticed
                  days later.

Report-only. Nothing here writes.

Defined Under Namespace

Classes: Finding

Constant Summary collapse

ANCHOR_LINES = 
10
MIN_LINES =

below this, a repeat is unremarkable

20
MAX_FILE_SIZE = 
5 * 1024 * 1024

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(root, excluded: Mbeditor.configuration.excluded_paths) ⇒ DuplicateContentScanner

Returns a new instance of DuplicateContentScanner.



64
65
66
67
 
# File 'app/services/mbeditor/duplicate_content_scanner.rb', line 64

def initialize(root, excluded: Mbeditor.configuration.excluded_paths)
  @root = File.expand_path(root.to_s)
  @matcher = ExclusionMatcher.new(excluded, root: @root)
end

Class Method Details

.check(content) ⇒ Object 



32
33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'app/services/mbeditor/duplicate_content_scanner.rb', line 32

def self.check(content)
  return nil if content.nil? || content.empty?

  lines = content.lines
  return nil if lines.size < MIN_LINES

  half = content.bytesize / 2
  if content.bytesize.even? && content.byteslice(0, half) == content.byteslice(half, half)
    return { reason: :exact, line: (lines.size / 2) + 1 }
  end

  at = repeated_anchor_line(lines)
  at ? { reason: :repeated_block, line: at } : nil
end

Instance Method Details

#call ⇒ Object 



69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'app/services/mbeditor/duplicate_content_scanner.rb', line 69

def call
  findings = []
  Find.find(@root) do |path|
    rel = path == @root ? "" : path.delete_prefix("#{@root}/")
    if File.directory?(path)
      Find.prune if !rel.empty? && @matcher.excluded?(rel)
      next
    end
    next if @matcher.excluded?(rel)
    next unless File.file?(path) && File.size(path).between?(1, MAX_FILE_SIZE)

    content = read_text(path)
    next unless content

    hit = self.class.check(content)
    next unless hit

    findings << Finding.new(path: rel, reason: hit[:reason], line: hit[:line], lines: content.lines.size)
  end
  findings.sort_by { |f| [f.reason == :exact ? 0 : 1, f.path] }
end