Class: Aspera::Rest
- Inherits:
-
Object
- Object
- Aspera::Rest
- Defined in:
- lib/aspera/rest.rb
Overview
Make HTTP calls, equivalent to rest-client rest call errors are raised as exception RestCallError and error are analyzed in RestErrorAnalyzer
Direct Known Subclasses
Api::Alee, Api::AoC, Api::Ats, Api::Faspex, Api::Httpgw, Api::Node
Instance Attribute Summary collapse
-
#auth_params ⇒ Object
readonly
All original constructor parameters.
-
#base_url ⇒ Object
readonly
The root URL for the API.
-
#headers ⇒ Object
readonly
Base common headers of API.
Class Method Summary collapse
-
.basic_authorization(user, pass) ⇒ String
Basic auth token.
-
.build_uri(url, query) ⇒ Object
Build URI from URL and parameters and check it is
httporhttps. -
.h_to_query_array(query) ⇒ Object
Support array for query parameter, there is no standard.
-
.io_http_session(http_session) ⇒ Net::BufferedIO
get Net::HTTP underlying socket i/o little hack, handy because HTTP debug, proxy, etc...
-
.parse_header(header) ⇒ Hash
Parses an HTTP Content-Type header string into its media type and parameters according to RFC 9110 and RFC 6838.
-
.parse_link_header(link_header, rel: 'next') ⇒ String?
Parse Link header according to RFC 8288 to extract a specific relation.
-
.php_style(query) ⇒ Hash
Indicate that the given Hash query uses php style for array parameters.
-
.query_to_h(query) ⇒ Hash
Decode query string as Hash if parameter is only once, then it's a scalar if a parameter is several, then it's array if parameter has [] then it's an array, and [] is removed Support arrays in query string, e.g.
-
.remote_certificate_chain(url, as_string: true) ⇒ String
PEM certificates of remote server.
-
.start_http_session(base_url) ⇒ Net::HTTP
Start a HTTP/S session, also used for web sockets.
Instance Method Summary collapse
-
#call(operation:, subpath: nil, query: nil, content_type: nil, body: nil, headers: nil, save_to_file: nil, exception: true, ret: :data) ⇒ Array(Hash, Net::HTTPResponse), ...
HTTP/S REST call.
-
#cancel(subpath, **kwargs) ⇒ Object
Cancel:
CANCEL. -
#create(subpath, params, **kwargs) ⇒ Object
Create:
POST. -
#delete(subpath, params = nil, **kwargs) ⇒ Object
Delete:
DELETE. -
#initialize(base_url:, auth: {type: :none}, not_auth_codes: ['401'], redirect_max: 0, headers: {}) ⇒ Rest
constructor
Create a REST object for API calls HTTP sessions parameters can be modified using global parameters in RestParameters For example, TLS verification can be skipped.
-
#oauth ⇒ Object
The OAuth object (create, or cached if already created).
-
#params ⇒ Object
Creation parameters.
-
#read(subpath, query = nil, **kwargs) ⇒ Object
Read:
GET. -
#update(subpath, params, **kwargs) ⇒ Object
Update:
PUT.
Constructor Details
#initialize(base_url:, auth: {type: :none}, not_auth_codes: ['401'], redirect_max: 0, headers: {}) ⇒ Rest
Create a REST object for API calls HTTP sessions parameters can be modified using global parameters in RestParameters For example, TLS verification can be skipped.
306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 |
# File 'lib/aspera/rest.rb', line 306 def initialize( base_url:, auth: {type: :none}, not_auth_codes: ['401'], redirect_max: 0, headers: {} ) Aspera.assert_type(base_url, String) # base url with no trailing slashes (note: string may be frozen) @base_url = base_url.chomp('/') # remove trailing port if it is 443 and scheme is https @base_url = @base_url.gsub(/:443$/, '') if @base_url.start_with?('https://') @base_url = @base_url.gsub(/:80$/, '') if @base_url.start_with?('http://') Log.log.debug{"Rest.new(#{@base_url})"} # default is no auth @auth_params = auth Aspera.assert_type(@auth_params, Hash) Aspera.assert(@auth_params.key?(:type), 'no auth type defined') @not_auth_codes = not_auth_codes Aspera.assert_type(@not_auth_codes, Array) # persistent session @http_session = nil @redirect_max = redirect_max Aspera.assert_type(@redirect_max, Integer) @headers = headers.clone Aspera.assert_type(@headers, Hash) @headers['User-Agent'] ||= RestParameters.instance.user_agent # OAuth object (created on demand) @oauth = nil end |
Instance Attribute Details
#auth_params ⇒ Object (readonly)
All original constructor parameters
274 275 276 |
# File 'lib/aspera/rest.rb', line 274 def auth_params @auth_params end |
#base_url ⇒ Object (readonly)
The root URL for the API
277 278 279 |
# File 'lib/aspera/rest.rb', line 277 def base_url @base_url end |
#headers ⇒ Object (readonly)
Base common headers of API
280 281 282 |
# File 'lib/aspera/rest.rb', line 280 def headers @headers end |
Class Method Details
.basic_authorization(user, pass) ⇒ String
Returns Basic auth token.
81 |
# File 'lib/aspera/rest.rb', line 81 def (user, pass); return "Basic #{Base64.strict_encode64("#{user}:#{pass}")}"; end |
.build_uri(url, query) ⇒ Object
Build URI from URL and parameters and check it is http or https.
Check if php style is specified.
nil values in query result in key without value, e.g. ?a, while empty string values result in ?a=.
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 |
# File 'lib/aspera/rest.rb', line 97 def build_uri(url, query) uri = URI.parse(url) Aspera.assert_values(uri.scheme, %w[http https]){'URI scheme'} return uri if query.nil? || query.respond_to?(:empty?) && query.empty? Log.dump(:query, query) uri.query = case query when String query when Hash URI.encode_www_form(h_to_query_array(query)) when Array Aspera.assert(query.all?{ |i| i.is_a?(Array) && i.length.eql?(2)}, 'Query must be array of arrays of 2 elements') URI.encode_www_form(query) # remove nil values else Aspera.error_unexpected_value(query.class){'query type'} end.gsub('%5B%5D=', '[]=') # [] is allowed in url parameters uri end |
.h_to_query_array(query) ⇒ Object
121 122 123 124 125 126 127 128 129 130 131 132 133 134 |
# File 'lib/aspera/rest.rb', line 121 def h_to_query_array(query) Aspera.assert_type(query, Hash) suffix = query.delete(:x_array_php_style) ? '[]' : nil query.each_with_object([]) do |(k, v), query_array| case v when Array v.each do |e| query_array.push(["#{k}#{suffix}", e]) end else query_array.push([k, v]) end end end |
.io_http_session(http_session) ⇒ Net::BufferedIO
get Net::HTTP underlying socket i/o
little hack, handy because HTTP debug, proxy, etc... will be available
used implement web sockets after start_http_session
210 211 212 213 214 215 216 |
# File 'lib/aspera/rest.rb', line 210 def io_http_session(http_session) Aspera.assert_type(http_session, Net::HTTP) # Net::BufferedIO in net/protocol.rb result = http_session.instance_variable_get(:@socket) Aspera.assert(!result.nil?){"no socket for #{http_session}"} return result end |
.parse_header(header) ⇒ Hash
Parses an HTTP Content-Type header string into its media type and parameters according to RFC 9110 and RFC 6838. TODO: use gem: content_type
249 250 251 252 253 254 255 256 257 258 259 260 |
# File 'lib/aspera/rest.rb', line 249 def parse_header(header) parts = header.split(';').map(&:strip) media_type = parts.shift.downcase parameters = parts.filter_map do |param| key, value = param.split('=', 2) next unless key && value key = key.strip.downcase.to_sym value = value.strip.gsub(/\A"|"\z/, '') [key, value] end.to_h {type: media_type, parameters: parameters} end |
.parse_link_header(link_header, rel: 'next') ⇒ String?
Parse Link header according to RFC 8288 to extract a specific relation
162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 |
# File 'lib/aspera/rest.rb', line 162 def parse_link_header(link_header, rel: 'next') return if link_header.nil? || link_header.empty? # RFC 8288: Link header format is: <URI>; param1=value1; param2=value2, <URI2>; ... # We look for the link with the specified rel link_header.split(',').each do |link_part| link_part = link_part.strip # Extract URL between < and > url_match = link_part.match(/<([^>]+)>/) next unless url_match url = url_match[1] # Extract parameters after the URL params_str = link_part[url_match.end(0)..] # Check if this link has the specified rel (with or without quotes, case insensitive) next unless /;\s*rel\s*=\s*"?#{Regexp.escape(rel)}"?/i.match?(params_str) return url end # Fallback: if no rel found and looking for 'next', try the first link (backward compatibility) if rel.eql?('next') first_link = link_header.split(',').first&.strip if first_link url_match = first_link.match(/<([^>]+)>/) return url_match[1] if url_match end end nil end |
.php_style(query) ⇒ Hash
Indicate that the given Hash query uses php style for array parameters
86 87 88 89 90 |
# File 'lib/aspera/rest.rb', line 86 def php_style(query) Aspera.assert_type(query, Hash){'query'} query[:x_array_php_style] = true query end |
.query_to_h(query) ⇒ Hash
143 144 145 146 147 148 149 150 151 152 153 154 155 156 |
# File 'lib/aspera/rest.rb', line 143 def query_to_h(query) URI.decode_www_form(query).each_with_object({}) do |(key, value), h| if key.end_with?('[]') key = key[..-3] h[key] = [] unless h.key?(key) end if h.key?(key) h[key] = [h[key]] if !h[key].is_a?(Array) h[key].push(value) else h[key] = value end end end |
.remote_certificate_chain(url, as_string: true) ⇒ String
Returns PEM certificates of remote server.
219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 |
# File 'lib/aspera/rest.rb', line 219 def remote_certificate_chain(url, as_string: true) result = [] # initiate a session to retrieve remote certificate http_session = Rest.start_http_session(url) begin # retrieve underlying openssl socket result = Rest.io_http_session(http_session).io.peer_cert_chain rescue result = http_session.peer_cert ensure http_session.finish end result = result.map(&:to_pem).join("\n") if as_string return result end |
.start_http_session(base_url) ⇒ Net::HTTP
Start a HTTP/S session, also used for web sockets
192 193 194 195 196 197 198 199 200 201 202 203 |
# File 'lib/aspera/rest.rb', line 192 def start_http_session(base_url) uri = URI.parse(base_url) Aspera.assert_values(uri.scheme, %w[http https]){'URI scheme'} # This honors http_proxy env var http_session = Net::HTTP.new(uri.host, uri.port) http_session.use_ssl = uri.scheme.eql?('https') # Set http options in callback, such as timeout and cert. verification RestParameters.instance.session_cb&.call(http_session) # Manually start session for keep alive (if supported by server, else, session is closed every time) http_session.start return http_session end |
Instance Method Details
#call(operation:, subpath: nil, query: nil, content_type: nil, body: nil, headers: nil, save_to_file: nil, exception: true, ret: :data) ⇒ Array(Hash, Net::HTTPResponse), ...
HTTP/S REST call
362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 |
# File 'lib/aspera/rest.rb', line 362 def call( operation:, subpath: nil, query: nil, content_type: nil, body: nil, headers: nil, save_to_file: nil, exception: true, ret: :data ) subpath = subpath.to_s if subpath.is_a?(Symbol) subpath = '' if subpath.nil? Log.log.debug{"call #{operation} [#{subpath}]".red.bold.bg_green} Log.dump(:body, body, level: :trace1) Log.dump(:query, query, level: :trace1) Log.dump(:headers, headers, level: :trace1) Aspera.assert_type(subpath, String) # We must have a way to check return code Aspera.assert(exception || !ret.eql?(:data), 'ret: :data requires exception handler') if headers.nil? headers = @headers.clone else h = headers headers = @headers.clone headers.merge!(h) end Aspera.assert_type(headers, Hash) case @auth_params[:type] when :none # no auth when :basic Log.log.debug('using Basic auth') # done in build_req when :oauth2 headers['Authorization'] = oauth. unless headers.key?('Authorization') when :url query ||= {} @auth_params[:url_query].each do |key, value| query[key] = value end else Aspera.error_unexpected_value(@auth_params[:type]) end result_http = nil result_data = nil # initialize with number of initial retries allowed, nil gives zero tries_remain_redirect = @redirect_max # start a block to be able to retry the actual HTTP request in case of OAuth token expiration begin Log.log.debug("send request (retries=#{tries_remain_redirect})") # TODO: shall we percent encode subpath (spaces) test with access key delete with space in id # URI.escape() separator = ['', '/'].include?(subpath) ? '' : '/' uri = self.class.build_uri("#{@base_url}#{separator}#{subpath}", query) Log.log.debug{"URI=#{uri}"} begin # instantiate request object based on string name req = Net::HTTP.const_get(operation.capitalize).new(uri) rescue NameError raise "unsupported operation : #{operation}" end case content_type when nil # ignore when Mime::JSON req.body = JSON.generate(body) # , ascii_only: true req['Content-Type'] = Mime::JSON when Mime::WWW req.body = URI.encode_www_form(body) req['Content-Type'] = Mime::WWW when Mime::TEXT req.body = body req['Content-Type'] = Mime::TEXT else Aspera.error_unexpected_value(content_type){'body type'} end # set headers headers.each do |key, value| req[key] = value end # :type = :basic req.basic_auth(@auth_params[:username], @auth_params[:password]) if @auth_params[:type].eql?(:basic) Log.dump(:req_body, req.body, level: :trace1) # we try the call, and will retry on some error types error_tries ||= 1 + RestParameters.instance.retry_max result_mime = nil file_saved = false # make http request (pipelined) http_session.request(req) do |response| result_http = response result_mime = self.class.parse_header(result_http['Content-Type'] || Mime::TEXT)[:type] Log.log.debug{"response: code=#{result_http.code}, mime=#{result_mime}, content-type=#{response['Content-Type']}"} # JSON data needs to be parsed, in case it contains an error code if !save_to_file.nil? && result_http.code.to_s.start_with?('2') && !Mime.json?(result_mime) total_size = result_http['Content-Length']&.to_i Log.log.debug('before write file') target_file = save_to_file # override user's path to path in header if !response['Content-Disposition'].nil? disposition = self.class.parse_header(response['Content-Disposition']) target_file = File.join(File.dirname(target_file), disposition[:parameters][:filename]) if disposition[:parameters].key?(:filename) && !disposition[:parameters][:filename].eql?('.') end # download with temp filename target_file_tmp = "#{target_file}#{RestParameters.instance.download_partial_suffix}" Log.log.debug{"saving to: #{target_file}"} written_size = 0 session_id = SecureRandom.uuid.freeze RestParameters.instance.&.event(:session_start, session_id: session_id) RestParameters.instance.&.event(:session_size, session_id: session_id, info: total_size) if total_size FileUtils.mkdir_p(File.dirname(target_file_tmp)) limiter = TimerLimiter.new(0.5) File.open(target_file_tmp, 'wb') do |file| result_http.read_body do |fragment| file.write(fragment) written_size += fragment.length RestParameters.instance.&.event(:transfer, session_id: session_id, info: written_size) if limiter.trigger? end end RestParameters.instance.&.event(:session_end, session_id: session_id) RestParameters.instance.&.event(:end) # rename at the end File.rename(target_file_tmp, target_file) file_saved = true end end Log.log.debug{"result: code=#{result_http.code} mime=#{result_mime}"} # sometimes there is a UTF8 char (e.g. © ) # TODO : related to mime type encoding ? # result_http.body.force_encoding('UTF-8') if result_http.body.is_a?(String) # Log.log.debug{"result: body=#{result_http.body}"} result_data = result_http.body Log.dump(:result_data_raw, result_data, level: :trace1) # TODO: Remove next 2 lines when bug in async node api is fixed. (Aspera/core/issues/4490) node_api_bug = result_data&.index('}HTTP/1.1 400 Bad Request') if result_data.is_a?(String) result_data = result_data[0..node_api_bug] if node_api_bug result_data = JSON.parse(result_data) if Mime.json?(result_mime) && !result_data.nil? && !result_data.empty? Log.dump(:result_data, result_data) RestErrorAnalyzer.instance.raise_on_error(req, result_data, result_http) unless file_saved || save_to_file.nil? FileUtils.mkdir_p(File.dirname(save_to_file)) File.write(save_to_file, result_http.body, binmode: true) end rescue RestCallError => e do_retry = false # AoC have some timeout , like Connect to platform.bss.asperasoft.com:443 ... do_retry ||= true if e.response.body.include?('failed: connect timed out') && RestParameters.instance.retry_on_timeout # AoC sometimes not available do_retry ||= true if RestParameters.instance.retry_on_unavailable && UNAVAILABLE_CODES.include?(result_http.code.to_s) # possibility to retry anything if it fails do_retry ||= true if RestParameters.instance.retry_on_error # not authorized: oauth token expired if @not_auth_codes.include?(result_http.code.to_s) && @auth_params[:type].eql?(:oauth2) begin # try to use refresh token req['Authorization'] = oauth.(refresh: true) rescue RestCallError => e_tok e = e_tok Log.log.error('refresh failed'.bg_red) # regenerate a brand new token req['Authorization'] = oauth.(cache: false) end Log.log.debug{"using new token=#{headers['Authorization']}"} do_retry ||= true end if do_retry && (error_tries -= 1).positive? sleep(RestParameters.instance.retry_sleep) unless RestParameters.instance.retry_sleep.eql?(0) retry end # redirect ? (any code beginning with 3) if e.response.is_a?(Net::HTTPRedirection) && tries_remain_redirect.positive? tries_remain_redirect -= 1 current_uri = URI.parse(@base_url) new_url = e.response['Location'] # special case: relative redirect if URI.parse(new_url).host.nil? # we don't manage relative redirects with non-absolute path Aspera.assert(new_url.start_with?('/')){"redirect location is relative: #{new_url}, but does not start with /."} new_url = "#{current_uri.scheme}://#{current_uri.host}#{new_url}" end # forwards the request to the new location return self.class.new( base_url: new_url, redirect_max: tries_remain_redirect ).call( operation: operation, subpath: new_url.end_with?('/') ? '/' : nil, query: query, body: body, content_type: content_type, save_to_file: save_to_file, exception: exception, headers: headers, ret: ret ) end # raise exception if could not retry and not return error in result raise e if exception end Log.log.debug{"result=http:#{result_http}, data:#{result_data.class}"} return case ret when :data then result_data when :resp then result_http when :both then [result_data, result_http] else Aspera.error_unexpected_value(ret){'Type of result for REST'} end end |
#cancel(subpath, **kwargs) ⇒ Object
Cancel: CANCEL
605 606 607 608 609 |
# File 'lib/aspera/rest.rb', line 605 def cancel(subpath, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') return call(operation: 'CANCEL', subpath: subpath, **kwargs) end |
#create(subpath, params, **kwargs) ⇒ Object
Create: POST
575 576 577 578 579 580 |
# File 'lib/aspera/rest.rb', line 575 def create(subpath, params, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') kwargs[:content_type] = Mime::JSON unless kwargs.key?(:content_type) return call(operation: 'POST', subpath: subpath, body: params, **kwargs) end |
#delete(subpath, params = nil, **kwargs) ⇒ Object
Delete: DELETE
598 599 600 601 602 |
# File 'lib/aspera/rest.rb', line 598 def delete(subpath, params = nil, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') return call(operation: 'DELETE', subpath: subpath, query: params, **kwargs) end |
#oauth ⇒ Object
Returns the OAuth object (create, or cached if already created).
338 339 340 341 342 343 344 345 346 |
# File 'lib/aspera/rest.rb', line 338 def oauth if @oauth.nil? Aspera.assert(@auth_params[:type].eql?(:oauth2), 'no OAuth defined') oauth_parameters = @auth_params.reject{ |k, _v| k.eql?(:type)} Log.dump(:oauth_parameters, oauth_parameters) @oauth = OAuth::Factory.instance.create(**oauth_parameters) end return @oauth end |
#params ⇒ Object
Returns creation parameters.
283 284 285 286 287 288 289 290 291 |
# File 'lib/aspera/rest.rb', line 283 def params return { base_url: @base_url, # String auth: @auth_params, # Hash not_auth_codes: @not_auth_codes, # Array redirect_max: @redirect_max, # Integer headers: @headers # Hash } end |
#read(subpath, query = nil, **kwargs) ⇒ Object
Read: GET
583 584 585 586 587 |
# File 'lib/aspera/rest.rb', line 583 def read(subpath, query = nil, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') return call(operation: 'GET', subpath: subpath, query: query, **kwargs) end |
#update(subpath, params, **kwargs) ⇒ Object
Update: PUT
590 591 592 593 594 595 |
# File 'lib/aspera/rest.rb', line 590 def update(subpath, params, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') kwargs[:content_type] = Mime::JSON unless kwargs.key?(:content_type) return call(operation: 'PUT', subpath: subpath, body: params, **kwargs) end |