Class: Aspera::Rest
- Inherits:
-
Object
- Object
- Aspera::Rest
- Defined in:
- lib/aspera/rest.rb
Overview
Make HTTP calls, equivalent to rest-client rest call errors are raised as exception RestCallError and error are analyzed in RestErrorAnalyzer
Direct Known Subclasses
Api::Alee, Api::AoC, Api::Ats, Api::Faspex, Api::Httpgw, Api::Node
Instance Attribute Summary collapse
-
#auth_params ⇒ Object
readonly
All original constructor parameters.
-
#base_url ⇒ Object
readonly
The root URL for the API.
-
#headers ⇒ Object
readonly
Base common headers of API.
Class Method Summary collapse
-
.basic_authorization(user, pass) ⇒ String
Basic auth token.
-
.build_uri(url, query) ⇒ Object
Build URI from URL and parameters and check it is ‘http` or `https`.
-
.h_to_query_array(query) ⇒ Object
Support array for query parameter, there is no standard.
-
.io_http_session(http_session) ⇒ Net::BufferedIO
get Net::HTTP underlying socket i/o little hack, handy because HTTP debug, proxy, etc…
-
.parse_header(header) ⇒ Hash
Parses an HTTP Content-Type header string into its media type and parameters according to RFC 9110 and RFC 6838.
-
.parse_link_header(link_header, rel: 'next') ⇒ String?
Parse Link header according to RFC 8288 to extract a specific relation.
-
.php_style(query) ⇒ Hash
Indicate that the given Hash query uses php style for array parameters.
-
.query_to_h(query) ⇒ Hash
Decode query string as Hash if parameter is only once, then it’s a scalar if a parameter is several, then it’s array if parameter has [] then it’s an array, and [] is removed Support arrays in query string, e.g.
-
.remote_certificate_chain(url, as_string: true) ⇒ String
PEM certificates of remote server.
-
.start_http_session(base_url) ⇒ Net::HTTP
Start a HTTP/S session, also used for web sockets.
Instance Method Summary collapse
-
#call(operation:, subpath: nil, query: nil, content_type: nil, body: nil, headers: nil, save_to_file: nil, exception: true, ret: :data) ⇒ Array(Hash, Net::HTTPResponse), ...
HTTP/S REST call.
-
#cancel(subpath, **kwargs) ⇒ Object
Cancel: ‘CANCEL`.
-
#create(subpath, params, **kwargs) ⇒ Object
Create: ‘POST`.
-
#delete(subpath, params = nil, **kwargs) ⇒ Object
Delete: ‘DELETE`.
-
#initialize(base_url:, auth: {type: :none}, not_auth_codes: ['401'], redirect_max: 0, headers: {}) ⇒ Rest
constructor
Create a REST object for API calls HTTP sessions parameters can be modified using global parameters in RestParameters For example, TLS verification can be skipped.
-
#oauth ⇒ Object
The OAuth object (create, or cached if already created).
-
#params ⇒ Object
Creation parameters.
-
#read(subpath, query = nil, **kwargs) ⇒ Object
Read: ‘GET`.
-
#update(subpath, params, **kwargs) ⇒ Object
Update: ‘PUT`.
Constructor Details
#initialize(base_url:, auth: {type: :none}, not_auth_codes: ['401'], redirect_max: 0, headers: {}) ⇒ Rest
Create a REST object for API calls HTTP sessions parameters can be modified using global parameters in RestParameters For example, TLS verification can be skipped.
302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 |
# File 'lib/aspera/rest.rb', line 302 def initialize( base_url:, auth: {type: :none}, not_auth_codes: ['401'], redirect_max: 0, headers: {} ) Aspera.assert_type(base_url, String) # base url with no trailing slashes (note: string may be frozen) @base_url = base_url.chomp('/') # remove trailing port if it is 443 and scheme is https @base_url = @base_url.gsub(/:443$/, '') if @base_url.start_with?('https://') @base_url = @base_url.gsub(/:80$/, '') if @base_url.start_with?('http://') Log.log.debug{"Rest.new(#{@base_url})"} # default is no auth @auth_params = auth Aspera.assert_type(@auth_params, Hash) Aspera.assert(@auth_params.key?(:type)){'no auth type defined'} @not_auth_codes = not_auth_codes Aspera.assert_type(@not_auth_codes, Array) # persistent session @http_session = nil @redirect_max = redirect_max Aspera.assert_type(@redirect_max, Integer) @headers = headers.clone Aspera.assert_type(@headers, Hash) @headers['User-Agent'] ||= RestParameters.instance.user_agent # OAuth object (created on demand) @oauth = nil end |
Instance Attribute Details
#auth_params ⇒ Object (readonly)
All original constructor parameters
270 271 272 |
# File 'lib/aspera/rest.rb', line 270 def auth_params @auth_params end |
#base_url ⇒ Object (readonly)
The root URL for the API
273 274 275 |
# File 'lib/aspera/rest.rb', line 273 def base_url @base_url end |
#headers ⇒ Object (readonly)
Base common headers of API
276 277 278 |
# File 'lib/aspera/rest.rb', line 276 def headers @headers end |
Class Method Details
.basic_authorization(user, pass) ⇒ String
Returns Basic auth token.
77 |
# File 'lib/aspera/rest.rb', line 77 def (user, pass); return "Basic #{Base64.strict_encode64("#{user}:#{pass}")}"; end |
.build_uri(url, query) ⇒ Object
Build URI from URL and parameters and check it is ‘http` or `https`. Check if php style is specified. `nil` values in query result in key without value, e.g. `?a`, while empty string values result in `?a=`.
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
# File 'lib/aspera/rest.rb', line 93 def build_uri(url, query) uri = URI.parse(url) Aspera.assert_values(uri.scheme, %w[http https]){'URI scheme'} return uri if query.nil? || query.respond_to?(:empty?) && query.empty? Log.dump(:query, query) uri.query = case query when String query when Hash URI.encode_www_form(h_to_query_array(query)) when Array Aspera.assert(query.all?{ |i| i.is_a?(Array) && i.length.eql?(2)}){'Query must be array of arrays of 2 elements'} URI.encode_www_form(query) # remove nil values else Aspera.error_unexpected_value(query.class){'query type'} end.gsub('%5B%5D=', '[]=') # [] is allowed in url parameters uri end |
.h_to_query_array(query) ⇒ Object
Support array for query parameter, there is no standard. Either p=1&p=2 (default) or p[]=1&p=2 (if ‘:x_array_php_style` is set to true in query)
117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
# File 'lib/aspera/rest.rb', line 117 def h_to_query_array(query) Aspera.assert_type(query, Hash) suffix = query.delete(:x_array_php_style) ? '[]' : nil query.each_with_object([]) do |(k, v), query_array| case v when Array v.each do |e| query_array.push(["#{k}#{suffix}", e]) end else query_array.push([k, v]) end end end |
.io_http_session(http_session) ⇒ Net::BufferedIO
get Net::HTTP underlying socket i/o little hack, handy because HTTP debug, proxy, etc… will be available used implement web sockets after ‘start_http_session`
206 207 208 209 210 211 212 |
# File 'lib/aspera/rest.rb', line 206 def io_http_session(http_session) Aspera.assert_type(http_session, Net::HTTP) # Net::BufferedIO in net/protocol.rb result = http_session.instance_variable_get(:@socket) Aspera.assert(!result.nil?){"no socket for #{http_session}"} return result end |
.parse_header(header) ⇒ Hash
Parses an HTTP Content-Type header string into its media type and parameters according to RFC 9110 and RFC 6838. TODO: use gem: content_type
245 246 247 248 249 250 251 252 253 254 255 256 |
# File 'lib/aspera/rest.rb', line 245 def parse_header(header) parts = header.split(';').map(&:strip) media_type = parts.shift.downcase parameters = parts.filter_map do |param| key, value = param.split('=', 2) next unless key && value key = key.strip.downcase.to_sym value = value.strip.gsub(/\A"|"\z/, '') [key, value] end.to_h {type: media_type, parameters: parameters} end |
.parse_link_header(link_header, rel: 'next') ⇒ String?
Parse Link header according to RFC 8288 to extract a specific relation
158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 |
# File 'lib/aspera/rest.rb', line 158 def parse_link_header(link_header, rel: 'next') return if link_header.nil? || link_header.empty? # RFC 8288: Link header format is: <URI>; param1=value1; param2=value2, <URI2>; ... # We look for the link with the specified rel link_header.split(',').each do |link_part| link_part = link_part.strip # Extract URL between < and > url_match = link_part.match(/<([^>]+)>/) next unless url_match url = url_match[1] # Extract parameters after the URL params_str = link_part[url_match.end(0)..-1] # Check if this link has the specified rel (with or without quotes, case insensitive) next unless /;\s*rel\s*=\s*"?#{Regexp.escape(rel)}"?/i.match?(params_str) return url end # Fallback: if no rel found and looking for 'next', try the first link (backward compatibility) if rel.eql?('next') first_link = link_header.split(',').first&.strip if first_link url_match = first_link.match(/<([^>]+)>/) return url_match[1] if url_match end end nil end |
.php_style(query) ⇒ Hash
Indicate that the given Hash query uses php style for array parameters
82 83 84 85 86 |
# File 'lib/aspera/rest.rb', line 82 def php_style(query) Aspera.assert_type(query, Hash){'query'} query[:x_array_php_style] = true query end |
.query_to_h(query) ⇒ Hash
Decode query string as Hash if parameter is only once, then it’s a scalar if a parameter is several, then it’s array if parameter has [] then it’s an array, and [] is removed Support arrays in query string, e.g. PHP’s way is p[]=1&p=2
139 140 141 142 143 144 145 146 147 148 149 150 151 152 |
# File 'lib/aspera/rest.rb', line 139 def query_to_h(query) URI.decode_www_form(query).each_with_object({}) do |(key, value), h| if key.end_with?('[]') key = key[..-3] h[key] = [] unless h.key?(key) end if h.key?(key) h[key] = [h[key]] if !h[key].is_a?(Array) h[key].push(value) else h[key] = value end end end |
.remote_certificate_chain(url, as_string: true) ⇒ String
Returns PEM certificates of remote server.
215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 |
# File 'lib/aspera/rest.rb', line 215 def remote_certificate_chain(url, as_string: true) result = [] # initiate a session to retrieve remote certificate http_session = Rest.start_http_session(url) begin # retrieve underlying openssl socket result = Rest.io_http_session(http_session).io.peer_cert_chain rescue result = http_session.peer_cert ensure http_session.finish end result = result.map(&:to_pem).join("\n") if as_string return result end |
.start_http_session(base_url) ⇒ Net::HTTP
Start a HTTP/S session, also used for web sockets
188 189 190 191 192 193 194 195 196 197 198 199 |
# File 'lib/aspera/rest.rb', line 188 def start_http_session(base_url) uri = URI.parse(base_url) Aspera.assert_values(uri.scheme, %w[http https]){'URI scheme'} # This honors http_proxy env var http_session = Net::HTTP.new(uri.host, uri.port) http_session.use_ssl = uri.scheme.eql?('https') # Set http options in callback, such as timeout and cert. verification RestParameters.instance.session_cb&.call(http_session) # Manually start session for keep alive (if supported by server, else, session is closed every time) http_session.start return http_session end |
Instance Method Details
#call(operation:, subpath: nil, query: nil, content_type: nil, body: nil, headers: nil, save_to_file: nil, exception: true, ret: :data) ⇒ Array(Hash, Net::HTTPResponse), ...
HTTP/S REST call
358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 |
# File 'lib/aspera/rest.rb', line 358 def call( operation:, subpath: nil, query: nil, content_type: nil, body: nil, headers: nil, save_to_file: nil, exception: true, ret: :data ) subpath = subpath.to_s if subpath.is_a?(Symbol) subpath = '' if subpath.nil? Log.log.debug{"call #{operation} [#{subpath}]".red.bold.bg_green} Log.dump(:body, body, level: :trace1) Log.dump(:query, query, level: :trace1) Log.dump(:headers, headers, level: :trace1) Aspera.assert_type(subpath, String) # We must have a way to check return code Aspera.assert(exception || !ret.eql?(:data)) if headers.nil? headers = @headers.clone else h = headers headers = @headers.clone headers.merge!(h) end Aspera.assert_type(headers, Hash) case @auth_params[:type] when :none # no auth when :basic Log.log.debug('using Basic auth') # done in build_req when :oauth2 headers['Authorization'] = oauth. unless headers.key?('Authorization') when :url query ||= {} @auth_params[:url_query].each do |key, value| query[key] = value end else Aspera.error_unexpected_value(@auth_params[:type]) end result_http = nil result_data = nil # initialize with number of initial retries allowed, nil gives zero tries_remain_redirect = @redirect_max # start a block to be able to retry the actual HTTP request in case of OAuth token expiration begin Log.log.debug("send request (retries=#{tries_remain_redirect})") # TODO: shall we percent encode subpath (spaces) test with access key delete with space in id # URI.escape() separator = ['', '/'].include?(subpath) ? '' : '/' uri = self.class.build_uri("#{@base_url}#{separator}#{subpath}", query) Log.log.debug{"URI=#{uri}"} begin # instantiate request object based on string name req = Net::HTTP.const_get(operation.capitalize).new(uri) rescue NameError raise "unsupported operation : #{operation}" end case content_type when nil # ignore when Mime::JSON req.body = JSON.generate(body) # , ascii_only: true req['Content-Type'] = Mime::JSON when Mime::WWW req.body = URI.encode_www_form(body) req['Content-Type'] = Mime::WWW when Mime::TEXT req.body = body req['Content-Type'] = Mime::TEXT else Aspera.error_unexpected_value(content_type){'body type'} end # set headers headers.each do |key, value| req[key] = value end # :type = :basic req.basic_auth(@auth_params[:username], @auth_params[:password]) if @auth_params[:type].eql?(:basic) Log.dump(:req_body, req.body, level: :trace1) # we try the call, and will retry on some error types error_tries ||= 1 + RestParameters.instance.retry_max result_mime = nil file_saved = false # make http request (pipelined) http_session.request(req) do |response| result_http = response result_mime = self.class.parse_header(result_http['Content-Type'] || Mime::TEXT)[:type] Log.log.debug{"response: code=#{result_http.code}, mime=#{result_mime}, content-type=#{response['Content-Type']}"} # JSON data needs to be parsed, in case it contains an error code if !save_to_file.nil? && result_http.code.to_s.start_with?('2') && !Mime.json?(result_mime) total_size = result_http['Content-Length']&.to_i Log.log.debug('before write file') target_file = save_to_file # override user's path to path in header if !response['Content-Disposition'].nil? disposition = self.class.parse_header(response['Content-Disposition']) target_file = File.join(File.dirname(target_file), disposition[:parameters][:filename]) if disposition[:parameters].key?(:filename) && !disposition[:parameters][:filename].eql?('.') end # download with temp filename target_file_tmp = "#{target_file}#{RestParameters.instance.download_partial_suffix}" Log.log.debug{"saving to: #{target_file}"} written_size = 0 session_id = SecureRandom.uuid.freeze RestParameters.instance.&.event(:session_start, session_id: session_id) RestParameters.instance.&.event(:session_size, session_id: session_id, info: total_size) if total_size FileUtils.mkdir_p(File.dirname(target_file_tmp)) limiter = TimerLimiter.new(0.5) File.open(target_file_tmp, 'wb') do |file| result_http.read_body do |fragment| file.write(fragment) written_size += fragment.length RestParameters.instance.&.event(:transfer, session_id: session_id, info: written_size) if limiter.trigger? end end RestParameters.instance.&.event(:session_end, session_id: session_id) RestParameters.instance.&.event(:end) # rename at the end File.rename(target_file_tmp, target_file) file_saved = true end end Log.log.debug{"result: code=#{result_http.code} mime=#{result_mime}"} # sometimes there is a UTF8 char (e.g. © ) # TODO : related to mime type encoding ? # result_http.body.force_encoding('UTF-8') if result_http.body.is_a?(String) # Log.log.debug{"result: body=#{result_http.body}"} result_data = result_http.body Log.dump(:result_data_raw, result_data, level: :trace1) # TODO: Remove next 2 lines when bug in async node api is fixed. (Aspera/core/issues/4490) node_api_bug = result_data&.index('}HTTP/1.1 400 Bad Request') if result_data.is_a?(String) result_data = result_data[0..node_api_bug] if node_api_bug result_data = JSON.parse(result_data) if Mime.json?(result_mime) && !result_data.nil? && !result_data.empty? Log.dump(:result_data, result_data) RestErrorAnalyzer.instance.raise_on_error(req, result_data, result_http) unless file_saved || save_to_file.nil? FileUtils.mkdir_p(File.dirname(save_to_file)) File.write(save_to_file, result_http.body, binmode: true) end rescue RestCallError => e do_retry = false # AoC have some timeout , like Connect to platform.bss.asperasoft.com:443 ... do_retry ||= true if e.response.body.include?('failed: connect timed out') && RestParameters.instance.retry_on_timeout # AoC sometimes not available do_retry ||= true if RestParameters.instance.retry_on_unavailable && UNAVAILABLE_CODES.include?(result_http.code.to_s) # possibility to retry anything if it fails do_retry ||= true if RestParameters.instance.retry_on_error # not authorized: oauth token expired if @not_auth_codes.include?(result_http.code.to_s) && @auth_params[:type].eql?(:oauth2) begin # try to use refresh token req['Authorization'] = oauth.(refresh: true) rescue RestCallError => e_tok e = e_tok Log.log.error('refresh failed'.bg_red) # regenerate a brand new token req['Authorization'] = oauth.(cache: false) end Log.log.debug{"using new token=#{headers['Authorization']}"} do_retry ||= true end if do_retry && (error_tries -= 1).positive? sleep(RestParameters.instance.retry_sleep) unless RestParameters.instance.retry_sleep.eql?(0) retry end # redirect ? (any code beginning with 3) if e.response.is_a?(Net::HTTPRedirection) && tries_remain_redirect.positive? tries_remain_redirect -= 1 current_uri = URI.parse(@base_url) new_url = e.response['Location'] # special case: relative redirect if URI.parse(new_url).host.nil? # we don't manage relative redirects with non-absolute path Aspera.assert(new_url.start_with?('/')){"redirect location is relative: #{new_url}, but does not start with /."} new_url = "#{current_uri.scheme}://#{current_uri.host}#{new_url}" end # forwards the request to the new location return self.class.new( base_url: new_url, redirect_max: tries_remain_redirect ).call( operation: operation, subpath: new_url.end_with?('/') ? '/' : nil, query: query, body: body, content_type: content_type, save_to_file: save_to_file, exception: exception, headers: headers, ret: ret ) end # raise exception if could not retry and not return error in result raise e if exception end Log.log.debug{"result=http:#{result_http}, data:#{result_data.class}"} return case ret when :data then result_data when :resp then result_http when :both then [result_data, result_http] else Aspera.error_unexpected_value(ret){'Type of result for REST'} end end |
#cancel(subpath, **kwargs) ⇒ Object
Cancel: ‘CANCEL`
601 602 603 604 605 |
# File 'lib/aspera/rest.rb', line 601 def cancel(subpath, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') return call(operation: 'CANCEL', subpath: subpath, **kwargs) end |
#create(subpath, params, **kwargs) ⇒ Object
Create: ‘POST`
571 572 573 574 575 576 |
# File 'lib/aspera/rest.rb', line 571 def create(subpath, params, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') kwargs[:content_type] = Mime::JSON unless kwargs.key?(:content_type) return call(operation: 'POST', subpath: subpath, body: params, **kwargs) end |
#delete(subpath, params = nil, **kwargs) ⇒ Object
Delete: ‘DELETE`
594 595 596 597 598 |
# File 'lib/aspera/rest.rb', line 594 def delete(subpath, params = nil, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') return call(operation: 'DELETE', subpath: subpath, query: params, **kwargs) end |
#oauth ⇒ Object
Returns the OAuth object (create, or cached if already created).
334 335 336 337 338 339 340 341 342 |
# File 'lib/aspera/rest.rb', line 334 def oauth if @oauth.nil? Aspera.assert(@auth_params[:type].eql?(:oauth2)){'no OAuth defined'} oauth_parameters = @auth_params.reject{ |k, _v| k.eql?(:type)} Log.dump(:oauth_parameters, oauth_parameters) @oauth = OAuth::Factory.instance.create(**oauth_parameters) end return @oauth end |
#params ⇒ Object
Returns creation parameters.
279 280 281 282 283 284 285 286 287 |
# File 'lib/aspera/rest.rb', line 279 def params return { base_url: @base_url, # String auth: @auth_params, # Hash not_auth_codes: @not_auth_codes, # Array redirect_max: @redirect_max, # Integer headers: @headers # Hash } end |
#read(subpath, query = nil, **kwargs) ⇒ Object
Read: ‘GET`
579 580 581 582 583 |
# File 'lib/aspera/rest.rb', line 579 def read(subpath, query = nil, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') return call(operation: 'GET', subpath: subpath, query: query, **kwargs) end |
#update(subpath, params, **kwargs) ⇒ Object
Update: ‘PUT`
586 587 588 589 590 591 |
# File 'lib/aspera/rest.rb', line 586 def update(subpath, params, **kwargs) kwargs[:headers] ||= {} kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept') kwargs[:content_type] = Mime::JSON unless kwargs.key?(:content_type) return call(operation: 'PUT', subpath: subpath, body: params, **kwargs) end |