Surfliner Metadata Consumer

This service consumes a Surfliner metadata message queue, passing messages to a customizable handler.

Local development

The default rake task performs the following, with each step proceeding only if prior steps succeed.

1. runs unit tests, with code coverage
2. checks code style
3. builds the gem

Testing

• rake spec to run unit tests
  • test results are written to the console
  • test reports are written to artifacts/rspec in JUnit format
• rake coverage to run unit tests with coverage
  • summary coverage statistics are written to the console
  • detailed coverage reports are written to artifacts/coverage in HTML format; open artifacts/coverage/index.html to view

Enforcing code style

• rake standard to check code style
• rake standard:fix to make safe fixes automatically

Building the gem

• rake gem builds the gem
  • gem package is written to artifacts/surfliner_metadata_consumer-<VERSION>.gem
  • version is set in lib/surfliner_metadata_consumer/version.rb

Configuration

Note: Sample values taken from Surfliner's development daylight-listener.sh.)

An Mq::Connection object, suitable for listening or publishing, can be configured explicitly with a ConnectionConfig object, or implicitly, reading a default configuration with ConnectionConfig#from_env. ConnectionConfig#from_env expects the following environment variables:

Variable	Sample value	Description
RABBITMQ_HOST	rabbitmq	Hostname of RabbitMQ server
RABBITMQ_NODE_PORT_NUMBER	5672	Port name of RabbitMQ server
RABBITMQ_USERNAME	user	RabbitMQ username
RABBITMQ_PASSWORD	bitnami	RabbitMQ password
RABBITMQ_AWAIT_ON_CLOSE	false	Whether to wait on response when closing (default = true)

The Mq::Connection#with_topic and Mq::Connection#topic_from can either take an explicit TopicConfig object, or implicitly read a default configuration with TopicConfig#from_env. TopicConfig#from_env expects the following environment variables:

Variable	Sample value	Description
RABBITMQ_TOPIC	surfliner.metadata	RabbitMQ topic name

Note that TopicConfig#from_env also accepts keyword options, which are passed through to Bunny::Channel#topic. E.g.

topic_config = TopicConfig.from_env(durable: true, auto_delete: true)
connection.with_topic(topic_config) do |topic|
  # ...
end

TopicConfig#publish can either accept an explicit routing key, or read a default value from the environment:

Variable	Sample value	Description
RABBITMQ_PLATFORM_ROUTING_KEY	surfliner.metadata.daylight	RabbitMQ routing key

Similarly, Mq::Topic#bind_queue can either take an explicit QueueConfig, or implicitly read a default with QueConfig#from_env. QueueConfig#from_env expects the following environment variables:

Variable	Sample value	Description
RABBITMQ_QUEUE	surfliner.metadata	RabbitMQ queue name

And QueueConfig#from_env similarly accepts keyword options, which are forwarded to Bunny::Channel#queue:

connection.with_topic(topic_config) do |topic|
  queue_config = QueueConfig.from_env(exclusive: true, durable: false)
  queue = topic.bind_queue(queue_config)
  # ...
end

The Solr / Daylight handler implementation (see below) additionally expects a configured SOLR_URL, e.g.:

SOLR_URL=http://admin:admin@solr:8983/solr/daylight-dev

The bin/simulate-publish-event script expects an METADATA_API_URL_BASE, used to generate resource URLs for provided IDs:

METADATA_API_URL_BASE=http://superskunk:3000/resources

Solr / Daylight handler implementation

The Surfliner::MetadataConsumer::Solr package contains a handler that retrieves metadata for :published/:updated events, transforms results for indexing, and updates a target Solr index.

The bin/index-on-publish script starts a consumer using this handler, and accepts (but does not require) the following environment variables:

Variable	Sample value	Default	Description
OTEL_SDK_DISABLED	true	(not set)	Whether to disable OpenTelemetry
OTEL_SERVICE_NAME	surfliner-daylight-consumer	name of script	OpenTelemetry service name
OTEL_TRACER_NAME	DaylightConsumerTracer	name of script	OpenTelemetry tracer name

Running the service

Build the listener Dockerfile and tag it with the name solr-listener:

docker build . -t solr-listener

Run the listener container:

docker run --env-file <env-file> solr-listener

Utility scripts

• bin/index-on-publish script starts a consumer using MetadataConsumer::Solr::MessageHandler.
• bin/simulate-publish-event posts a publish event to the queue configured with Mq::Connection