Class: SmoScottishLidar::Downloader

Inherits:
Object
  • Object
show all
Defined in:
lib/smo_scottish_lidar/downloader.rb

Overview

Downloads LiDAR files from the Scottish Government S3 bucket. No external dependencies. Uses only Ruby stdlib.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(verbose: false) ⇒ Downloader

Returns a new instance of Downloader.



11
12
13
14
15
 
# File 'lib/smo_scottish_lidar/downloader.rb', line 11

def initialize(verbose: false)
  @client  = Client.new(verbose: verbose)
  @lister  = Lister.new(verbose: verbose)
  @verbose = verbose
end

Instance Attribute Details

#client ⇒ Object (readonly)

Returns the value of attribute client.



9
10
11
 
# File 'lib/smo_scottish_lidar/downloader.rb', line 9

def client
  @client
end

#lister ⇒ Object (readonly)

Returns the value of attribute lister.



9
10
11
 
# File 'lib/smo_scottish_lidar/downloader.rb', line 9

def lister
  @lister
end

Instance Method Details

#download(phase, type, destination:, grid_square: nil, resolution: nil, skip_existing: true, dry_run: false) ⇒ Hash

Download all files for a given phase/type, with optional filtering.

Parameters:

  • phase (String) —

    e.g. “phase-1”, “outer-hebrides”

  • type (String) —

    “dsm”, “dtm”, or “laz”

  • destination (String) —

    Local directory to save files into

  • grid_square (String, nil) (defaults to: nil) —

    OS grid square filter e.g. “NS”

  • resolution (String, nil) (defaults to: nil) —

    Outer Hebrides resolution e.g. “50cm”

  • skip_existing (Boolean) (defaults to: true) —

    Skip files that already exist locally (default: true)

  • dry_run (Boolean) (defaults to: false) —

    List what would be downloaded without downloading

Returns:

  • (Hash) —

    { downloaded: […], skipped: […], failed: […] }



27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/smo_scottish_lidar/downloader.rb', line 27

def download(phase, type,
             destination:,
             grid_square: nil,
             resolution: nil,
             skip_existing: true,
             dry_run: false)

  FileUtils.mkdir_p(destination) unless dry_run

  objects = lister.list(phase, type, grid_square: grid_square, resolution: resolution)

  if objects.empty?
    puts "No files matched your criteria."
    return { downloaded: [], skipped: [], failed: [] }
  end

  total_bytes = objects.sum { |o| o[:size] }
  puts "Found #{objects.size} file(s) (#{format_bytes(total_bytes)} total)"
  puts "Destination: #{destination}"
  puts "(Dry run - no files will be downloaded)" if dry_run
  puts

  results = { downloaded: [], skipped: [], failed: [] }

  objects.each_with_index do |obj, idx|
    local_path = File.join(destination, obj[:filename])
    label = "[#{idx + 1}/#{objects.size}] #{obj[:filename]} (#{format_bytes(obj[:size])})"

    if skip_existing && File.exist?(local_path) && File.size(local_path) == obj[:size]
      puts "SKIP  #{label}"
      results[:skipped] << obj[:filename]
      next
    end

    if dry_run
      puts "WOULD #{label}"
      results[:downloaded] << obj[:filename]
      next
    end

    print "GET   #{label} ... "
    $stdout.flush

    begin
      client.download_object(obj[:key], local_path) do |received, total|
        next unless @verbose && total

        pct = (received.to_f / total * 100).round(1)
        print "\rGET   #{label} ... #{pct}%"
        $stdout.flush
      end
      puts "OK"
      results[:downloaded] << obj[:filename]
    rescue StandardError => e
      puts "FAILED (#{e.message})"
      results[:failed] << { file: obj[:filename], error: e.message }
    end
  end

  print_summary(results)
  results
end

#download_file(phase, type, filename, destination:, resolution: nil) ⇒ Object

Convenience: download a single file by its S3 key or filename.

Parameters:

  • phase (String)
  • type (String)
  • filename (String) —

    Exact filename to download

  • destination (String) —

    Local directory

  • resolution (String, nil) (defaults to: nil) 


97
98
99
100
101
102
103
104
105
106
107
 
# File 'lib/smo_scottish_lidar/downloader.rb', line 97

def download_file(phase, type, filename, destination:, resolution: nil)
  prefix = SmoScottishLidar.prefix_for(phase, type, resolution: resolution)
  key    = "#{prefix}#{filename}"
  local  = File.join(destination, filename)

  FileUtils.mkdir_p(destination)
  puts "Downloading #{filename} ..."
  client.download_object(key, local)
  puts "Saved to #{local}"
  local
end