Module: Unlocodes::Data::Fetcher

Defined in:
lib/unlocodes/data/fetcher.rb

Overview

Downloads the UNCEFACT UN/LOCODE vocabulary from the upstream GitLab repository and stores it as lib/unlocodes/data/locode.jsonld.

The upstream tag (e.g. "2025-1") points at a snapshot of the vocab-locode project whose vocab/unlocode.jsonld is the canonical full dataset. Override the source URL with the UNLOCODE_PATH env variable if a different edition's path layout applies.

After a successful fetch, the tag is written to SOURCE_TAG next to the data file so the gem (and the check-upstream workflow) always knows which edition is bundled.

Constant Summary collapse

UPSTREAM_HOST = 
'opensource.unicc.org'
UPSTREAM_TAG_PATH = 
'/un/unece/uncefact/vocab-locode/-/raw/%<tag>s'
DATA_DIR = 
File.expand_path(__dir__)
OUTPUT_PATH = 
File.join(DATA_DIR, 'locode.jsonld')
SOURCE_TAG_PATH = 
File.join(DATA_DIR, 'SOURCE_TAG')
CANDIDATE_PATHS = 
[
  'vocab/unlocode.jsonld',
  'vocab/unlocode-vocab.jsonld',
  'unlocode.jsonld',
  'locodes.jsonld'
].freeze

Class Method Summary collapse

Class Method Details

.call(tag:) ⇒ String

Returns the path written.

Parameters:

  • tag (String) —

    the upstream Git tag (e.g. "2025-1")

Returns:

  • (String) —

    the path written



37
38
39
40
41
42
43
 
# File 'lib/unlocodes/data/fetcher.rb', line 37

def call(tag:)
  uri = resolve_uri(tag)
  data = download(uri)
  write(data, tag: tag)
  warn "Fetched UN/LOCODE #{tag} (#{data.bytesize} bytes) -> #{OUTPUT_PATH}"
  OUTPUT_PATH
end

.download(uri) ⇒ Object 



66
67
68
69
70
71
72
73
 
# File 'lib/unlocodes/data/fetcher.rb', line 66

def download(uri)
  Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
    response = http.get(uri.request_uri)
    raise "Download failed: #{response.code} #{response.message}" unless response.is_a?(Net::HTTPSuccess)

    response.body
  end
end

.exists?(uri) ⇒ Boolean

Returns:

  • (Boolean) 


57
58
59
60
61
62
63
64
 
# File 'lib/unlocodes/data/fetcher.rb', line 57

def exists?(uri)
  Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
    response = http.head(uri.request_uri)
    response.is_a?(Net::HTTPSuccess)
  end
rescue StandardError
  false
end

.resolve_uri(tag) ⇒ Object 



45
46
47
48
49
50
51
52
53
54
55
 
# File 'lib/unlocodes/data/fetcher.rb', line 45

def resolve_uri(tag)
  return URI(ENV.fetch('UNLOCODE_PATH')) if ENV.key?('UNLOCODE_PATH')

  CANDIDATE_PATHS.each do |candidate|
    path = format("#{UPSTREAM_TAG_PATH}/#{candidate}", tag: tag)
    uri = URI::HTTPS.build(host: UPSTREAM_HOST, path: path)
    return uri if exists?(uri)
  end
  raise "Could not find UN/LOCODE data under tag #{tag.inspect}. " \
        'Set UNLOCODE_PATH to the full JSON-LD URL and retry.'
end

.write(data, tag:) ⇒ Object 



75
76
77
78
79
 
# File 'lib/unlocodes/data/fetcher.rb', line 75

def write(data, tag:)
  FileUtils.mkdir_p(DATA_DIR)
  File.write(OUTPUT_PATH, data)
  File.write(SOURCE_TAG_PATH, "#{tag}\n")
end