GeoBlacklight Sidecar Images
Store local copies of remote imagery in GeoBlacklight.
Description
This GeoBlacklight plugin captures remote images from geographic web services and saves them locally. It borrows the concept of a SolrDocumentSidecar from Spotlight, to have an ActiveRecord-based "sidecar" to match each non-AR SolrDocument. This allows us to use ActiveStorage to attach images to our solr documents.
Example Screenshot
Requirements
- Ruby >= 3.3 (CI also runs on 3.4 and 4.0)
- Rails >= 7.2, < 9 (tested on 7.2 with GeoBlacklight 4 and 8.1 with GeoBlacklight 5/6)
- GeoBlacklight 4.x, 5.x, or 6.x
- libvips (Rails default) or ImageMagick
Suggested
- Background job processor — Solid Queue (Rails 8) or Sidekiq
Installation
Existing GeoBlacklight instance
gem "geoblacklight_sidecar_images", "~> 2.0"
GeoBlacklight v3 with GBL 1.0 metadata still uses the 0.9.x series:
gem "geoblacklight_sidecar_images", "~> 0.9.1", "< 1.0"
Run the generator.
$ bin/rails generate geoblacklight_sidecar_images:install
Use --skip-views on GeoBlacklight 5/6 apps that render results with ViewComponents rather than the GBL 4 split catalog partial. Use --skip-assets when the host is not using Sprockets.
Run the database migration.
$ bin/rails db:migrate
Complete any necessary Active Storage setup steps, for example:
- Add a config/storage.yml file
local:
service: Disk
root: <%= Rails.root.join("storage") %>
- Add config/environments declarations, development.rb for example:
# Store uploaded files on the local file system (see config/storage.yml for options)
config.active_storage.service = :local
The install generator appends Sidecar Images settings to config/settings.yml (GBLSI_THUMBNAIL_FIELD, optional GeoServer proxy keys, and optional GBLSI_OGM_API_URL). Leave the GeoServer URLs blank unless you need authenticated local WMS harvesting. Leave GBLSI_OGM_API_URL blank unless you want to skip harvest and render OpenGeoMetadata API thumbnails instead.
SolrDocument#sidecar is included by the engine. You do not need to copy a method into app/models/solr_document.rb. If you are upgrading from 1.x, you can remove the generator-injected sidecar method from that file.
New GeoBlacklight instance
$ rails new app-name -m https://raw.githubusercontent.com/geoblacklight/geoblacklight_sidecar_images/develop/template.rb
Ingest test documents
# Run your GBL instance
bundle exec rake geoblacklight:server
# Index the GBL test fixtures
bundle exec rake gblsci:sample_data:seed
Rake tasks
Harvest images
Harvest is only needed when you store thumbnails in Active Storage. If GBLSI_OGM_API_URL is set, skip these tasks and see OpenGeoMetadata API thumbnails.
Harvest all images
Spawns background jobs to harvest images for all documents in your Solr index (paginated with Solr cursorMarks).
bundle exec rake gblsci:images:harvest_all
Harvest an individual image
Allows you to add images one document id at a time. Pass a DOC_ID env var.
DOC_ID='stanford-cz128vq0535' bundle exec rake gblsci:images:harvest_doc_id
Harvest all incomplete states
Reattempt image harvesting for all non-successful state objects.
bundle exec rake gblsci:images:harvest_retry
Check image states
bundle exec rake gblsci:images:harvest_states
We use a state machine library to track success/failure of our harvest tasks. The states we track are:
- initialized - SolrDocumentSidecar created, no harvest attempt run
- queued - Harvest attempt queued as background job
- processing - Harvest attempt at work
- succeeded - Harvest was successful, image attached
- failed - Harvest failed, no image attached, error logged
- placeheld - Harvest was not successful, placeholder imagery will be used
SolrDocumentSidecar.in_state(:succeeded) => [#<SolrDocumentSidecar:0x0000000170697960 ... ]
SolrDocumentSidecar.image.attached? => false
SolrDocumentSidecar.image_state.current_state => "placeheld"
SolrDocumentSidecar.image_state.last_transition => #<SidecarImageTransition id: 207, to_state: "placeheld", metadata: {"solr_doc_id"=>"stanford-cg357zz0321", ...>
Destroy images
Destructive tasks require CONFIRM=1.
Remove everything
CONFIRM=1 bundle exec rake gblsci:images:harvest_purge_all
Remove orphaned AR objects
CONFIRM=1 bundle exec rake gblsci:images:harvest_purge_orphans
Remove a batch
Remove sidecar objects and attached images via a CSV file of document ids at tmp/destroy_batch.csv.
CONFIRM=1 bundle exec rake gblsci:images:harvest_destroy_batch
Troubleshooting
Harvest report
Generate a CSV file of sidecar objects and associated image state under tmp/.
bundle exec rake gblsci:images:harvest_report
Failed state inspect
Prints details for failed state harvest objects to stdout
bundle exec rake gblsci:images:harvest_failed_state_inspect
Prioritize Solr Thumbnail Field URIs
If you add a thumbnail uri to your geoblacklight solr documents...
Example Doc
{
...
"dct_format_s": "TIFF",
"dct_creator_sm": ["Minnesota. Department of Highways."],
"thumbnail_path_ss": "https://umedia.lib.umn.edu/sites/default/files/imagecache/square300/reference/562/image/jpeg/1089695.jpg",
"gbl_resourceClass_sm": ["Imagery"],
...
}
Then you can edit your GeoBlacklight settings.yml file to point at that solr field (Settings.GBLSI_THUMBNAIL_FIELD). Any docs in your index that have a value for that field will harvest the image at that URI instead of trying to retrieve an image via IIIF or the other web services.
OpenGeoMetadata API thumbnails
To skip harvest, Active Storage, and the sidecar state machine entirely, point views at the OpenGeoMetadata API. Thumbnails are loaded in the browser from:
https://ogm.geo4lib.app/api/v1/resources/{id}/thumbnail
Set the API root in config/settings.yml:
GBLSI_OGM_API_URL: 'https://ogm.geo4lib.app/api/v1'
sidecar_thumbnail_tag document then renders an img for that URL (for example stanford-mp692kw6192 → /resources/stanford-mp692kw6192/thumbnail). No sidecar row is created. Harvest rake tasks are unused in this mode.
The harvest/Active Storage path remains the default when GBLSI_OGM_API_URL is blank.
View customization
Use basic Active Storage patterns, or the engine helper, to display imagery in your application.
Example methods
# Helper (available in host views). Uses the OGM API when GBLSI_OGM_API_URL is
# set; otherwise uses a harvested Active Storage attachment.
<%= sidecar_thumbnail_tag document, size: [200, 200] %>
# Harvested Active Storage (only when OGM API thumbnails are not enabled)
document.sidecar.image.attached?
document.sidecar.image.variable?
<%= image_tag document.sidecar.image.variant(resize_to_fit: [100, 100]), {class: 'media-object'} %>
Search results
On GeoBlacklight 4, the install generator can copy a catalog _index_split_default.html.erb partial. GeoBlacklight 5/6 apps should call sidecar_thumbnail_tag (or document.sidecar.image) from their result component instead of replacing Blacklight helpers.
Show pages
Example for adding a thumbnail to the show page sidebar.
<%= sidecar_thumbnail_tag @document, size: [200, 200], class: "mr-3" %>
Development
# Run test suite
bundle exec rake ci
# Launch test app server
cd .internal_test_app/
bundle exec rake geoblacklight:server
# Load test fixtures
bundle exec rake gblsci:sample_data:seed
# Run harvest
bundle exec rake gblsci:images:harvest_all
# Tail image service log file
tail -f log/image_service_development.log
Test against a specific stack with environment variables:
RAILS_VERSION=8.1.3 GEOBLACKLIGHT_VERSION="~> 5.3" bundle exec rake ci