Module: OvertureMaps::Storage

Defined in:
lib/overture_maps/storage.rb

Overview

Anonymous HTTP access to the public Overture bucket (or a configured mirror): object listing via the S3 REST API's XML responses and streaming downloads. No AWS SDK, no credentials, no global state.

Defined Under Namespace

Classes: Error

Constant Summary collapse

MAX_REDIRECTS = 
5

Class Method Summary collapse

Class Method Details

.download(key, to:, expected_size: nil) ⇒ Object

Streams an object to a local file. Skips the download when the local file already has the expected size.



51
52
53
54
55
56
57
58
 
# File 'lib/overture_maps/storage.rb', line 51

def download(key, to:, expected_size: nil)
  if expected_size && File.exist?(to) && File.size(to) == expected_size
    return :skipped
  end

  download_url("#{base_url}/#{escape_key(key)}", to: to)
  :downloaded
end

.download_url(url, to:) ⇒ Object

Downloads any URL to a file, following redirects, writing through a temp file so failures never leave a truncated file at the target path.



62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/overture_maps/storage.rb', line 62

def download_url(url, to:)
  FileUtils.mkdir_p(File.dirname(to))
  tmp = "#{to}.part"

  fetch_streaming(url) do |response|
    File.open(tmp, "wb") do |file|
      response.read_body { |chunk| file.write(chunk) }
    end
  end

  FileUtils.mv(tmp, to)
  to
ensure
  FileUtils.rm_f(tmp) if tmp && File.exist?(tmp)
end

.get(url) ⇒ Object

GET returning the body as a string, following redirects.



79
80
81
82
83
 
# File 'lib/overture_maps/storage.rb', line 79

def get(url)
  body = nil
  fetch_streaming(url) { |response| body = response.body }
  body
end

.list(prefix:, delimiter: nil) ⇒ Object

Lists objects under a prefix. Returns { objects: [size:], prefixes: [String] }, following continuation tokens.



20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/overture_maps/storage.rb', line 20

def list(prefix:, delimiter: nil)
  objects = []
  prefixes = []
  token = nil

  loop do
    params = { "list-type" => "2", "prefix" => prefix }
    params["delimiter"] = delimiter if delimiter
    params["continuation-token"] = token if token

    doc = REXML::Document.new(get("#{base_url}/?#{URI.encode_www_form(params)}"))
    root = doc.root
    raise Error, "unexpected listing response" unless root&.name == "ListBucketResult"

    root.each_element("Contents") do |el|
      objects << {
        key: el.elements["Key"]&.text,
        size: el.elements["Size"]&.text.to_i
      }
    end
    root.each_element("CommonPrefixes/Prefix") { |el| prefixes << el.text }

    token = root.elements["NextContinuationToken"]&.text
    break unless token
  end

  { objects: objects, prefixes: prefixes }
end