Class: Sfdown::Mapper

Inherits:
Object
  • Object
show all
Defined in:
lib/sfdown.rb

Overview

Stage 1: walk the directory tree from the root, building the in-memory Node tree. Network only, never touches the filesystem. Directory pages are fetched by a pool of concurrent workers that enqueue subdirectories as they find them; counters are mutex-guarded since the progress hook reads them.

Instance Method Summary collapse

Constructor Details

#initialize(project, http, parser, concurrent: 1, log: ->(m) { warn m }) ⇒ Mapper

log: callback for non-fatal diagnostics (defaults to stderr; the CLI routes it through StatusBar#log so the bar stays pinned at the bottom).



310
311
312
313
314
315
316
317
318
319
320
321
322
323
 
# File 'lib/sfdown.rb', line 310

def initialize(project, http, parser, concurrent: 1, log: ->(m) { warn m })
  @project = project
  @http = http
  @parser = parser
  @concurrent = concurrent
  @log = log
  @mutex = Mutex.new
  @queue = Queue.new
  @pending = 0
  @folders = 0
  @files = 0
  @total_size = 0
  @failures = 0
end

Instance Method Details

#failures ⇒ Object 



328
 
# File 'lib/sfdown.rb', line 328

def failures = @mutex.synchronize { @failures }

#files ⇒ Object 



326
 
# File 'lib/sfdown.rb', line 326

def files = @mutex.synchronize { @files }

#folders ⇒ Object 



325
 
# File 'lib/sfdown.rb', line 325

def folders = @mutex.synchronize { @folders }

#map(&on_page) ⇒ Object

Build and return the synthetic root Node with the whole tree mapped. Yields each directory Node right after its page is parsed (progress hook).



332
333
334
335
336
337
338
339
340
341
342
343
344
345
 
# File 'lib/sfdown.rb', line 332

def map(&on_page)
  @on_page = on_page
  root = Node.new(name: @project, type: :d, path: "", timestamp: nil, size: 0, downloads: 0, children: [])
  enqueue(root)
  Pool.run(@queue, @concurrent) do |node|
    visit(node)
  ensure
    finish
  end
  aggregate(root)
  # Root timestamp isn't provided by SF; use the newest child's.
  root.timestamp = root.children.map(&:timestamp).compact.max
  root
end

#total_size ⇒ Object 



327
 
# File 'lib/sfdown.rb', line 327

def total_size = @mutex.synchronize { @total_size }