Class: VectorAmp::WebSource

Inherits:
Source
  • Object
show all
Defined in:
lib/vector_amp/source.rb

Overview

Web-crawl ingestion source.

Constant Summary

Constants inherited from Source

Source::SUPPORTED_SOURCE_TYPES

Instance Attribute Summary

Attributes inherited from Source

#config, #description, #id, #metadata, #name, #source_type

Instance Method Summary collapse

Methods inherited from Source

#[], from_api, #inspect, normalize_hash, #to_create_body, #to_h

Constructor Details

#initialize(start_urls:, name: nil, description: nil, metadata: nil, id: nil, **config) ⇒ WebSource

Parameters:

  • start_urls (String, Array<String>) —

    required seed URLs.

  • name (String, nil) (defaults to: nil) —

    defaults to web-<host> from the first URL, or web-source.

  • description (String, nil) (defaults to: nil) —

    optional description.

  • metadata (Hash, nil) (defaults to: nil) —

    optional metadata.

  • id (String, nil) (defaults to: nil) —

    optional API source id.

  • config (Hash) —

    additional web-source config forwarded to the API.

Raises:

  • (ArgumentError) 


187
188
189
190
191
192
193
194
195
196
197
198
199
 
# File 'lib/vector_amp/source.rb', line 187

def initialize(start_urls:, name: nil, description: nil, metadata: nil, id: nil, **config)
  urls = Array(start_urls)
  raise ArgumentError, "start_urls must not be empty" if urls.empty?

  super(
    id: id,
    source_type: "web",
    name: name || SourceNames.web(urls),
    description: description,
    metadata: metadata,
    config: Utils.compact_hash(config.merge(start_urls: urls))
  )
end