Class: Sfdown::Downloader

Inherits:
Object
  • Object
show all
Defined in:
lib/sfdown.rb

Overview

Stage 2: create the local tree and download files with a pool of concurrent workers. Counters are mutex-guarded so the ticker thread can snapshot progress while the workers advance it.

Constant Summary collapse

ILLEGAL =

Windows-illegal chars (per path segment)

/[<>:"\\|?*\x00-\x1f]/.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(project, http, dest, concurrent: 1, link: false, log: ->(m) { warn m }) ⇒ Downloader

dest is the project root directory (/).



409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
 
# File 'lib/sfdown.rb', line 409

def initialize(project, http, dest, concurrent: 1, link: false, log: ->(m) { warn m })
  @project = project
  @http = http
  @dest = dest
  @concurrent = concurrent
  @link = link
  @log = log
  @mutex = Mutex.new
  @files = []
  @dupes = []
  @ok = Set.new
  @total_files = 0
  @total_bytes = 0
  @duped_bytes = 0
  @resumed_files = 0
  @resumed_bytes = 0
  @done_files = 0    # completed this session (fetched + cloned)
  @done_bytes = 0    # streamed this session
  @duped_files = 0
  @failures = 0
  @mismatches = 0
  @warned_links = false
  @current = ""
end

Instance Attribute Details

#duped_bytes ⇒ Object (readonly)

Returns the value of attribute duped_bytes.



406
407
408
 
# File 'lib/sfdown.rb', line 406

def duped_bytes
  @duped_bytes
end

#total_bytes ⇒ Object (readonly)

Returns the value of attribute total_bytes.



406
407
408
 
# File 'lib/sfdown.rb', line 406

def total_bytes
  @total_bytes
end

#total_files ⇒ Object (readonly)

Returns the value of attribute total_files.



406
407
408
 
# File 'lib/sfdown.rb', line 406

def total_files
  @total_files
end

Instance Method Details

#apply_folder_times(root) ⇒ Object

Apply folder timestamps last, deepest-first, writing files into a directory bumps its mtime, so folders must be stamped after their contents are final.



474
475
476
477
478
479
480
481
 
# File 'lib/sfdown.rb', line 474

def apply_folder_times(root)
  dirs = []
  each_node(root) { |n| dirs << n if n.dir? }
  dirs.sort_by { |n| -n.path.count("/") }.each do |n|
    path = local_path(n)
    File.utime(n.timestamp, n.timestamp, path) if n.timestamp && File.directory?(path)
  end
end

#bytes_done ⇒ Object 



437
 
# File 'lib/sfdown.rb', line 437

def bytes_done = @mutex.synchronize { @resumed_bytes + @done_bytes }

#current ⇒ Object 



444
 
# File 'lib/sfdown.rb', line 444

def current = @mutex.synchronize { @current }

#download_all ⇒ Object

The work list is fixed here (unlike stage 1), so the queue is filled and closed up front and workers just drain it. Duplicates come after, once every original is on disk.



467
468
469
470
 
# File 'lib/sfdown.rb', line 467

def download_all
  run_pool(@files)
  @dupes.each { |node, source| clone_one(node, source) }
end

#downloaded_bytes ⇒ Object

this session only; drives speed



438
 
# File 'lib/sfdown.rb', line 438

def downloaded_bytes = @mutex.synchronize { @done_bytes } # this session only; drives speed

#duped_files ⇒ Object 



441
 
# File 'lib/sfdown.rb', line 441

def duped_files = @mutex.synchronize { @duped_files }

#failures ⇒ Object 



442
 
# File 'lib/sfdown.rb', line 442

def failures = @mutex.synchronize { @failures }

#files_done ⇒ Object

Progress counters credit files already on disk from a previous run, so a resumed download shows e.g. 500/700 MB rather than restarting at 0.



436
 
# File 'lib/sfdown.rb', line 436

def files_done = @mutex.synchronize { @resumed_files + @done_files }

#mismatches ⇒ Object 



443
 
# File 'lib/sfdown.rb', line 443

def mismatches = @mutex.synchronize { @mismatches }

#prepare(root) ⇒ Object

Create every directory up front, then split the files into those already on disk (resume), those to fetch, and the duplicates cloned from them. total_bytes covers the resumed + fetched bytes, so a resumed run's bar starts partway to 100% instead of at zero.



450
451
452
453
454
455
456
457
458
459
460
461
462
 
# File 'lib/sfdown.rb', line 450

def prepare(root)
  FileUtils.mkdir_p(@dest)
  files = []
  each_node(root) do |n|
    FileUtils.mkdir_p(local_path(n)) if n.dir? && !n.path.empty?
    files << n if n.file?
  end
  @total_files = files.size
  plan(files)
  @total_bytes = @resumed_bytes + @files.sum(&:size)
  @duped_bytes = @dupes.sum { |node, _| node.size }
  log_resume if @resumed_files.positive?
end

#resumed_bytes ⇒ Object 



440
 
# File 'lib/sfdown.rb', line 440

def resumed_bytes = @mutex.synchronize { @resumed_bytes }

#resumed_files ⇒ Object 



439
 
# File 'lib/sfdown.rb', line 439

def resumed_files = @mutex.synchronize { @resumed_files }