Module: Iata::Data::Fetcher

Defined in:
lib/iata/data/fetcher.rb

Overview

Downloads the IATA airport code list from Wikidata via SPARQL and writes it to lib/iata/data/airports.json as a single JSON object keyed by IATA code.

Data source: Wikidata property P238 ("IATA airport code"). Each result is an airport with its English label, ISO 3166-1 alpha-2 country code, and WGS-84 coordinates.

Constant Summary collapse

SPARQL_ENDPOINT = 
'https://query.wikidata.org/sparql'
USER_AGENT = 
'metanorma-iata-gem/0.1 (https://github.com/metanorma/iata)'
DATA_DIR = 
File.expand_path(__dir__)
OUTPUT_PATH = 
File.join(DATA_DIR, 'airports.json')
QUERY = 
<<~SPARQL
  SELECT ?iata ?name ?country ?countryIso2 ?coord ?wdId WHERE {
    ?airport wdt:P238 ?iata .
    ?airport wdt:P17 ?country .
    ?airport rdfs:label ?name . FILTER(LANG(?name) = "en")
    OPTIONAL { ?country wdt:P297 ?countryIso2 . }
    OPTIONAL { ?airport wdt:P625 ?coord . }
    BIND(STRAFTER(STR(?airport), "/entity/") AS ?wdId)
  }
  ORDER BY ?iata
SPARQL

Class Method Summary collapse

Class Method Details

.call ⇒ String

Returns the path written.

Returns:

  • (String) —

    the path written



37
38
39
40
41
42
43
 
# File 'lib/iata/data/fetcher.rb', line 37

def call
  results = query
  data = transform(results)
  write(data, fetched_at: Time.now.utc.iso8601, result_count: results.size)
  warn "Fetched #{data.size} IATA airports from Wikidata (#{File.size(OUTPUT_PATH)} bytes)"
  OUTPUT_PATH
end

.query ⇒ Array<Hash>

Returns raw SPARQL bindings, each as a hash of {value:, type:}.

Returns:

  • (Array<Hash>) —

    raw SPARQL bindings, each as a hash of {value:, type:}



46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# File 'lib/iata/data/fetcher.rb', line 46

def query
  uri = URI(SPARQL_ENDPOINT)
  uri.query = URI.encode_www_form(
    query: QUERY,
    format: 'json'
  )

  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = (uri.scheme == 'https')
  http.open_timeout = 30
  http.read_timeout = 180

  request = Net::HTTP::Get.new(uri.request_uri)
  request['User-Agent'] = USER_AGENT
  request['Accept'] = 'application/sparql-results+json'

  response = http.request(request)
  unless response.is_a?(Net::HTTPSuccess)
    raise "Wikidata SPARQL query failed: HTTP #{response.code} #{response.message}\n#{response.body[0..500]}"
  end

  JSON.parse(response.body, symbolize_names: true)[:results][:bindings]
end

.transform(bindings) ⇒ Hash<String, Hash>

Returns keyed by IATA code.

Parameters:

  • bindings (Array<Hash>) —

    SPARQL result bindings

Returns:

  • (Hash<String, Hash>) —

    keyed by IATA code



72
73
74
75
76
77
78
79
80
81
 
# File 'lib/iata/data/fetcher.rb', line 72

def transform(bindings)
  data = {}
  bindings.each do |row|
    code = row[:iata]&.dig(:value)
    next if code.nil?

    data[code] = build_entry(row)
  end
  data
end

.write(data, fetched_at:, result_count:) ⇒ Object 



83
84
85
86
87
88
89
90
91
92
93
94
 
# File 'lib/iata/data/fetcher.rb', line 83

def write(data, fetched_at:, result_count:)
  FileUtils.mkdir_p(DATA_DIR)
  payload = {
    '_meta' => {
      'fetched_at' => fetched_at,
      'source' => 'Wikidata (property P238)',
      'result_count' => result_count,
      'entry_count' => data.size
    }
  }.merge(data)
  File.write(OUTPUT_PATH, JSON.pretty_generate(payload))
end