Yobi

Yobi is a Ruby interface for the Restic backup program.

Installation

Install the gem:

bundle add yobi

or, without Bundler:

gem install yobi

Installing Restic

Yobi shells out to a real restic binary; it doesn't install or bundle one. You need restic on PATH (or point Yobi at it explicitly, see below) before any of this works.

Install it however you'd install any other system tool:

# macOS
brew install restic

# Debian/Ubuntu
apt install restic

# Arch
pacman -S restic

# or download a binary directly
# https://github.com/restic/restic/releases

Confirm it's reachable:

restic version

Minimum supported version

Yobi requires Restic 0.17.1 or newer. This isn't arbitrary: Yobi::Restic's exit-code-based error dispatch (mapping Restic's own exit codes to Yobi::RepositoryNotFound/RepositoryLocked/AuthenticationFailed) depends on a convention that's only guaranteed since that version. An older Restic could reuse those same exit codes for different failures, so rather than risk silently misclassifying an error, Yobi checks the installed version before running any real command and raises Yobi::UnsupportedResticVersion if it's too old:

begin
  repo.snapshots
rescue Yobi::UnsupportedResticVersion => e
  puts "Restic #{e.installed_version} is too old, need >= #{e.minimum_version}"
end

See "Minimum-version enforcement, in detail" below for the mechanics. If you'd rather fail fast at application startup instead of on the first backup attempt, call it explicitly:

RESTIC = Yobi::Restic.new
RESTIC.ensure_minimum_version! # raises immediately if too old, otherwise a no-op

A non-default Restic binary

If restic isn't on PATH, or you want to pin a specific binary:

# via $RESTIC_PATH
ENV["RESTIC_PATH"] = "/opt/restic/bin/restic"
Yobi::Restic.new

# or explicitly
Yobi::Restic.new("/opt/restic/bin/restic")
Yobi::Repository.new(url: "...", password: "...", restic: "/opt/restic/bin/restic")

Core concepts

Two classes make up the whole API:

• Yobi::Restic: a specific restic binary plus its process-level settings that apply no matter which repository you're talking to (cache directory, bandwidth limits, compression, and so on).
• Yobi::Repository: one repository, covering its URL, password, and backend credentials, plus every Restic subcommand as a method. Holds a Restic internally, either one you hand it explicitly via restic: (useful when several repositories should share the same operational settings) or a default one it creates for you.

require "yobi"

repo = Yobi::Repository.new(
  url: "sftp:backup@nas.local:/repos/home",
  password: [:command, "security find-generic-password -a restic -s home -w"]
)

Quick start

A repository has to exist before anything else works: #init creates one at url. From there, #backup creates a snapshot, #snapshots lists what exists, and #restore pulls one back out:

repo.init

outcome = repo.backup(
  source: "/Users/zia/Documents",
  excludes: ["*.tmp", "node_modules"],
  tags: ["documents", "daily"]
)
outcome.success? # => true
outcome.report["snapshot_id"]
outcome.report["data_added"]

repo.snapshots.each do |snapshot|
  puts "#{snapshot.short_id}  #{snapshot.time}  #{snapshot.paths.join(", ")}"
end

repo.restore(snapshot_id: "latest", target: "/tmp/restore")

Every verb method returns something typed rather than a raw Hash. #backup returns a BackupOutcome (#success?, #report, #errors), #snapshots returns an Enumerable of Snapshot objects, and so on throughout the API.

Constructing a Repository

repo = Yobi::Repository.new(
  url: "sftp:backup@nas.local:/repos/home",
  password: [:command, "security find-generic-password -a restic -s home -w"]
)

url:

The repository location, in whatever form Restic itself accepts:

• a plain path for a local repository
• or a URL prefixed with backend type such as sftp:, s3:, azure:, b2:, gs:, rest:, rclone:, etc.

See Restic's documentation for the full list of supported backends and their URL-ish schemes.

password:

Identifies the repository's encryption password. Five shapes:

# A literal password
password = "correct horse battery staple"

# Restic resolves this itself, Yobi never sees the value
password = [:command, "security find-generic-password -a restic -w"]

# Same idea, from a file
password = [:file, "/etc/restic/password"]

# Resolved on the Ruby side, called fresh immediately before every Restic
# invocation, use this for a secret store (see Recipes below)
password = -> { Vault.logical.read("secret/restic").data[:password] }

# An explicit "there is no password" declaration (Restic's own
# --insecure-no-password flag)
password = :insecure_no_password

Yobi::Repository.new(url: "...", password: password)

If password: is left unspecified, Yobi adds nothing extra to the environment: Restic falls back to RESTIC_PASSWORD/RESTIC_PASSWORD_COMMAND if either happens to already be set in the calling process's own environment (a container, systemd unit, or parent shell outside Yobi's control). If nothing is set anywhere, Restic refuses a truly empty password (unless --insecure-no-password is given) and fails loudly and safely on its own.

backend_credentials:

Separate from password:: this is whatever the storage backend needs to authenticate, unrelated to whether the repository's contents can be decrypted. A Hash of env vars, or a callable returning one, resolved fresh on every call the same way a callable password: is:

# S3
Yobi::Repository.new(
  url: "s3:s3.amazonaws.com/my-bucket",
  password: "...",
  backend_credentials: {"AWS_ACCESS_KEY_ID" => "...", "AWS_SECRET_ACCESS_KEY" => "..."}
)

# Azure
Yobi::Repository.new(
  url: "azure:my-container:/",
  password: "...",
  backend_credentials: {"AZURE_ACCOUNT_NAME" => "...", "AZURE_ACCOUNT_KEY" => "..."}
)

# Google Cloud Storage
Yobi::Repository.new(
  url: "gs:my-bucket:/",
  password: "...",
  backend_credentials: {"GOOGLE_PROJECT_ID" => "...", "GOOGLE_APPLICATION_CREDENTIALS" => "/path/to/service-account.json"}
)

# REST server, credentials embedded directly in the URL instead of backend_credentials:
Yobi::Repository.new(url: "rest:https://user:pass@host/", password: "...")

For that last case, Yobi extracts the embedded username/password into RESTIC_REST_USERNAME/RESTIC_REST_PASSWORD and strips them from the stored url so it won't leak via repo#inspect, logs, or a stray puts. Explicit backend_credentials: values win over extracted ones if both are given.

See Restic's documentation for the full list of supported backends and their env vars.

restic:

A Yobi::Restic instance to share, a bare Restic binary path, or just leave it out to get a default one:

Yobi::Repository.new(url: "...", password: "...") # default Restic

Yobi::Repository.new(url: "...", password: "...", restic: "/opt/restic/bin/restic") # bare path

Yobi::Repository.new(url: "...", password: "...", restic: restic) # shared Yobi::Restic instance

See "Sharing a Restic across repositories" below for why you'd want to share one explicitly (global settings like bandwidth limits or cache directory).

Error handling

Every failure raises a specific, typed error rather than a generic one, so you can handle the cases that matter to you and let the rest propagate:

begin
  repo.snapshots
rescue Yobi::RepositoryLocked
  # another Restic process holds the lock, see Recipes below for a retry pattern
rescue Yobi::AuthenticationFailed
  # wrong password
rescue Yobi::RepositoryNotFound
  # url doesn't point to an initialized repository yet, maybe call #init first
end

The full hierarchy, all under Yobi::Error < StandardError:

• Yobi::ResticNotFound: the restic binary itself couldn't be found or executed.
• Yobi::UnsupportedResticVersion: the installed Restic is older than the minimum supported version.
• Yobi::RepositoryNotFound, Yobi::RepositoryLocked, Yobi::AuthenticationFailed: Restic's own typed exit codes (10/11/12).
• Yobi::ResticCommandFailed: any other non-zero exit, for a failure that doesn't fit one of the above.
• Yobi::MountTimeout: #mount didn't report itself ready within its timeout.

Every method below lives on Yobi::Repository unless noted otherwise.

Repository lifecycle

#init

Creates the repository at url.

repo.init
# => {"message_type"=>"initialized", "id"=>"...", "repository"=>"..."}

copy_chunker_params:/from_repo:/from_repository_file:/from_key_hint:/from_password_command:/from_password_file:/from_insecure_no_password: copy chunker parameters from a secondary repository. Useful when you plan to #copy between the two later, since chunker params have to match for deduplication to work across repositories. repository_version: picks a specific repository format version instead of Restic's current default.

#config

The repository's own config document (cat config), as a Hash. A cheap way to confirm a repository exists and the password is correct without doing anything else.

repo.config
# => {"version"=>2, "id"=>"...", "chunker_polynomial"=>"..."}

Raises Yobi::RepositoryNotFound if url doesn't point to an initialized repository, Yobi::AuthenticationFailed if the password is wrong.

Backup and restore

#backup

outcome = repo.backup(
  source: "/Users/zia/Documents",
  excludes: ["*.tmp", "node_modules"],
  tags: ["documents", "daily"]
)
outcome.success?          # => true
outcome.report["snapshot_id"]
outcome.report["files_new"]
outcome.report["data_added"]

source: is either a path String, or a [:stdin_from_command, command]/[:stdin_from_command, command, filename] tuple: Restic spawns and executes command itself, capturing its stdout as the backup content (see "Database dumps via stdin_from_command" below for a pg_dump example). command can be a String (tokenized with Shellwords.split, so quoted arguments survive) or an Array of already-discrete arguments (used as-is; needed when an argument itself contains a literal space, which Shellwords would otherwise split incorrectly).

Filtering:

• excludes:/exclude_files:/iexcludes:/iexclude_files: (case-insensitive): exclude by glob pattern or pattern file
• exclude_if_present:: skip a directory containing a named marker file
• exclude_larger_than:: skip by size
• exclude_caches:/exclude_cloud_files:: skip CACHEDIR.TAG-marked dirs and cloud-placeholder files respectively
• files_from:/files_from_raw:/files_from_verbatim:: read the backup list from a file instead of (or alongside) source:

Behavior:

• dry_run:: report what would happen without doing it
• force:: back up unchanged files anyway
• one_file_system:: don't cross filesystem boundaries
• ignore_ctime:/ignore_inode:: relax change detection
• no_scan:: skip the pre-backup scan, disabling percentage progress
• skip_if_unchanged:: don't create a snapshot if nothing changed
• parent:: pick a specific parent snapshot instead of the latest one
• group_by:: grouping used to find the parent snapshot instead, e.g. "host,paths", when parent: isn't given
• read_concurrency:: number of concurrent file reads
• time:: record a specific timestamp instead of now
• with_atime:: also store files' access times
• host:: hostname recorded on the new snapshot (a single value: this is metadata being written, not a filter, unlike every hosts: elsewhere in this API)

Returns a BackupOutcome: #success?/#partial? (exit code 3, some files were skipped, e.g. permission errors), #report (the summary Hash, with "backup_start"/"backup_end" parsed into Time), #errors (lazy Enumerable of BackupError), #command_output (the source: [:stdin_from_command, ...] subprocess's own stderr, de-prefixed, if any).

Also accepts a block, to receive typed progress objects as the backup runs instead of only getting the final BackupOutcome:

repo.backup(source: "/Users/zia/Documents") do |message|
  case message
  when Yobi::BackupStatus
    puts "#{((message.percent_done || 0) * 100).round}% (#{message.files_done}/#{message.total_files})"
  when Yobi::BackupError
    warn "#{message.item}: #{message.message}"
  when Yobi::BackupVerboseStatus
    puts "#{message.action} #{message.item}"
  end
end

Yobi::BackupVerboseStatus (one message per file) only streams when verbose: true is passed; it's not emitted by default, since a large backup can otherwise produce hundreds of MB of these. Behavior without a block is identical, just without the live callbacks; the returned BackupOutcome is the same either way.

#restore

repo.restore(snapshot_id: "latest", target: "/tmp/restore")

snapshot_id: accepts "latest" or a real ID; target: is the destination directory.

• excludes:/includes: (and their i-prefixed case-insensitive/_file file-based variants): filter what gets restored. includes: has no backup-time equivalent, since restoring is the one direction where "only these" makes sense as well as "all but these."
• exclude_xattrs:/include_xattrs:: filter extended attributes specifically
• dry_run:: report what would happen without doing it
• delete:: removes files in target: not present in the snapshot
• overwrite: ("always"/"if-changed"/"if-newer"): controls what happens to files that already exist there
• sparse:: writes sparse files
• ownership_by_name:: maps ownership by user/group name instead of numeric ID
• verify:: re-reads restored content against the repository to confirm it matches
• hosts:/paths:/tags:: only matter when snapshot_id: is "latest", same filters as everywhere else, used to resolve which snapshot "latest" means

Returns a RestoreOutcome (#report, no #success?/#partial?: restore has no exit code 3, any item-level failure is a hard error that raises before an outcome exists).

Accepts a block the same way #backup does, yielding Yobi::RestoreStatus/Yobi::RestoreVerboseStatus (the latter only when verbose: true) instead of Yobi::BackupStatus/Yobi::BackupVerboseStatus.

#dump

Extracts a single file (raw bytes) or a whole folder (as a tar/zip archive) from a snapshot, without a full restore:

# Straight to a file
repo.dump(snapshot_id: "latest", file: "/etc/hosts", target: "/tmp/hosts")

# Or stream it yourself
repo.dump(snapshot_id: "latest", file: "/etc/hosts") do |io|
  IO.copy_stream(io, "/tmp/hosts")
end

# A whole folder as an archive (default to tar):
repo.dump(snapshot_id: "latest", file: "/var/www", target: "/tmp/www.tar")
repo.dump(snapshot_id: "latest", file: "/var/www", target: "/tmp/www.zip", archive: "zip")

Give at most one of target: or a block. Without either, returns a Yobi::IOHandle instead:

handle = repo.dump(snapshot_id: "latest", file: "/var/www")
begin
  # ... hand handle.io to something else that reads from it later ...
ensure
  # put this in ensure block so the restic subprocess doesn't become a zombie:
  handle.close # may raise error based on restic's exit code.
end

#diff

outcome = repo.diff(from: older_snapshot.id, to: newer_snapshot.id)
outcome.report["changed_files"]
outcome.changes.each { |change| puts "#{change.modifier} #{change.path}" }

metadata: true also reports metadata-only changes (permissions, timestamps) alongside content changes. change.modifier is Restic's own single-character code: + added, - removed, U metadata updated, M content modified, T type changed, ? bitrot detected.

Snapshot management

#snapshots

repo.snapshots(tags: ["daily"], hosts: "web-1").each do |snapshot|
  puts "#{snapshot.short_id}  #{snapshot.time}  #{snapshot.paths.join(", ")}"
end

Returns an Enumerable of Snapshot (#id, #short_id, #time, #host, #tags, #paths, #parent_id). compact: compacts the underlying Restic output; group_by:/latest: group and limit results, e.g. latest: 1 per group for "the most recent backup of each host."

#tag

repo.tag(snapshot_ids: snapshot.id, add: "verified")

add:/remove:/set: (mutually exclusive with add:/remove: in Restic itself) modify tags on snapshots matched by snapshot_ids: (or by hosts:/paths:/tags: filters when no explicit IDs are given). Since tags are part of a snapshot's content-addressed identity, tagging produces a new snapshot ID for every snapshot touched. TagOutcome#changes gives you the old-ID/new-ID pairs.

#forget

repo.forget(keep_daily: 7, keep_weekly: 4, keep_monthly: 12, prune: true)

Applies a retention policy, removing snapshots that don't match any keep_* rule: keep_last:/keep_hourly:/keep_daily:/keep_weekly:/keep_monthly:/keep_yearly: (counts) and keep_within:/keep_within_hourly:/etc. (durations, e.g. "30d"). keep_tags: always keeps snapshots carrying any of the given tags regardless of the numeric rules. prune: true also reclaims the disk space the forgotten snapshots held (equivalent to a separate #prune call afterward); the max_unused:/max_repack_size:/repack_*: kwargs tune that implied prune step exactly like #prune's own. dry_run: reports what would be removed without doing it. unsafe_allow_remove_all: is required if a policy would remove every snapshot.

Returns an Enumerable of ForgetGroup (Restic evaluates the policy per group, by default grouped by host+paths). Each exposes #keep/#remove (Arrays of Snapshot) and #reasons (an Array of ForgetReason explaining which rule kept each surviving snapshot, e.g. "daily snapshot").

#find

repo.find(patterns: "*.pem").each do |matches|
  puts "in #{matches.snapshot["short_id"]}: #{matches.hits} hits"
  matches.matches.each { |m| puts "  #{m.path}" }
end

Searches for files/directories by name pattern across snapshots. blob:/pack:/tree: switch to matching object IDs instead, for low-level troubleshooting. Returns an Enumerable of MatchesPerSnapshot (Restic's own docs describe find's output as "organized by snapshot," not a flat list). Each has #hits and #matches (an Array of FindMatch, with the usual file metadata: #path, #size, #mtime, etc.).

#ls

outcome = repo.ls(snapshot_id: "latest", dirs: "/var/www")
outcome.snapshot.short_id
outcome.entries.each { |entry| puts entry.path }

Lists a snapshot's files/directories. recursive: descends into subdirectories; dirs: restricts to specific paths within the snapshot. Returns an LsOutcome: #snapshot (the resolved Snapshot, useful to see what "latest" actually resolved to) and lazy #entries (Enumerable of LsEntry).

Maintenance and health

#check

outcome = repo.check(read_data_subset: "5%")
outcome.report["num_errors"]
outcome.errors.each { |error| puts error.message }

Verifies repository integrity. read_data: also reads and verifies every pack file's actual contents, not just structure (slow, thorough); read_data_subset: does a partial version of the same (e.g. "5%", or "1/20" for a fifth each day in rotation); with_cache: uses the local cache instead of bypassing it. Returns a CheckOutcome (no #success?/#partial?: check has no exit code 3; check report["num_errors"] instead).

#prune

repo.prune(max_unused: "5%")

Removes data no longer referenced by any snapshot. dry_run: reports what would happen without doing it. max_unused: targets a maximum acceptable unused-space ratio after pruning; max_repack_size: bounds how much gets repacked in one run (for spreading a large prune across multiple scheduled invocations); repack_cacheable_only:/repack_uncompressed:/repack_smaller_than: tune which packs get rewritten. unsafe_recover_no_free_space: proceeds even without enough free space, at the given risk acknowledgement string. Returns true.

#repair_index, #repair_packs, #repair_snapshots

repo.repair_index
repo.repair_packs(ids: damaged_pack_ids)
repo.repair_snapshots(snapshot_ids: affected_ids, forget: true)

#repair_index rebuilds the index from the pack files present (the modern successor to Restic's now-deprecated rebuild-index). #repair_packs extracts intact blobs from damaged pack files and drops the rest. Restic also writes a backup copy of each given pack (named pack-<id>) into the calling process's own current working directory first, with no flag to disable this; be aware of where your process runs from before calling it. #repair_snapshots regenerates snapshots with damaged content removed. This is real data loss for whatever gets removed, so prefer a fresh #backup where the source data is still available, and run #repair_index first since this depends on a correct index.

#recover

repo.recover

Builds a new snapshot from any data present in the repository but not referenced by an existing snapshot (e.g. after an accidental #forget). Call #snapshots afterward to see whether anything was actually recovered. Restic reports "nothing to do" as plain text with no structured way to tell the two cases apart.

#rewrite

repo.rewrite(snapshot_ids: old.id, excludes: "*.log", new_time: Time.now.iso8601)

Creates new snapshots from existing ones with filters applied or metadata changed. With no snapshot_ids:/hosts:/tags:/paths: given, rewrites every snapshot in the repository. Restic's own default, not something Yobi adds. excludes:/exclude_files:/includes:/include_files: (and their i-prefixed case-insensitive variants) filter file content the same way as #backup/#restore. dry_run: reports what would happen without doing it. new_host:/new_time: change the recorded hostname/timestamp; snapshot_summary: regenerates the snapshot summary. forget: false (the default) tags the new snapshots "rewrite" and keeps the originals; forget: true removes the originals instead (their data isn't reclaimed until a later #prune).

#migrate

repo.migrate(names: "upgrade_repo_v2")

Applies the given named migrations; with names: left empty, only lists what's available (as Restic's own plain-text output; there's no programmatic way to enumerate migrations, since --json is ignored for this command).

#unlock

repo.unlock

Removes stale locks left by other Restic processes. remove_all: true removes every lock, not just stale ones.

#stats

repo.stats(mode: "raw-data")
# => {"total_size"=>..., "total_file_count"=>..., "snapshots_count"=>...}

Repository size/file-count statistics. mode: picks the counting strategy, as a String or Symbol: "restore-size"/:restore_size (default), "files-by-contents"/:files_by_contents, "blobs-per-file"/:blobs_per_file, "raw-data"/:raw_data (actual on-disk size, accounting for deduplication). Anything else raises ArgumentError.

Key management

repo.key_add(user: "ci-runner")
repo.key_list.each { |key| puts "#{key.id} #{key.user_name}@#{key.host_name} current=#{key.current?}" }
repo.key_passwd(new_password_file: "/etc/restic/new-password")
repo.key_remove(id: old_key.id)

#key_add/#key_list/#key_passwd/#key_remove (aliased #add_key/#keys/#change_password/#remove_key) manage passwords. Every key here is a different way to unlock the same underlying master key, not a separate encryption key of its own (see "Low-level (cat) and object listing"'s note on #cat_masterkey_and_game_over_if_this_leaks below). #key_passwd doesn't mutate the Repository you called it on; build a new one with the rotated password to keep talking to the repository afterward.

#key_add/#key_passwd share the same kwargs: host:/user: record a hostname/username on the new key (defaults to the OS's own, same as Restic itself); new_insecure_no_password: sets the new key's password to empty; new_password_file: reads the new password from a file instead of prompting interactively.

Restic refuses to #key_remove the key currently in use.

Low-level (cat) and object listing

repo.cat_snapshot(snapshot)      # => raw stored record, as a Hash
repo.cat_tree(snapshot, subfolder: "var/log")
repo.cat_pack(pack_id) { |io| ... }   # raw, still-encrypted bytes
repo.cat_blob(blob_id) { |io| ... }   # raw, decrypted bytes
repo.list(:packs)                     # => Array of pack IDs

#cat_snapshot/#cat_index/#cat_key/#cat_tree return a repository object's raw stored record as a Hash, lower-level than #snapshots/#key_list, which wrap entries in Snapshot/Key instead. #cat_snapshot/#cat_key/#cat_tree each accept either a bare ID String or the corresponding wrapper object (Snapshot/Key/Snapshot) directly; #cat_index only takes a bare ID String, since index entries have no wrapper class of their own. #cat_pack/#cat_blob are raw-bytes commands with the same block/IOHandle shape as #dump. #list(type) enumerates every object ID of a given type (:blobs/:packs/:index/:snapshots/:keys/:locks) as plain strings. Restic ignores --json for this command, so these come back exactly as Restic prints them.

#cat_masterkey_and_game_over_if_this_leaks returns the repository's actual encryption/MAC key material: not a redacted reference, the real thing. There is no operation that rotates this key; every password change made afterward protects nothing already backed up if this value ever leaks. Treat the return value with more care than you'd give password: itself: unlike a password, this can't be rotated away after a leak.

Across repositories

#copy

repo.copy(from_repo: source_repo, hosts: "web-1")

Replicates snapshots from another repository into this one. from_repo: accepts a plain URL String or a Repository instance. Given the latter, its own #url/#password are used automatically unless from_password: is given explicitly. from_key_hint:/from_repository_file: are passed straight through to Restic (a key ID hint for the source repository, and reading the source URL from a file, respectively). Already-copied snapshots are skipped automatically on a repeat run. Returns true.

Mounting

#mount

repo.mount(mountpoint: "/mnt/restic") do |mount|
  Dir.children("#{mount.mountpoint}/snapshots/latest")
end

Serves the repository as a read-only FUSE filesystem at mountpoint (which must already exist). With a block, yields a Yobi::Mount and stops it automatically afterward. Without one, returns the Mount for you to #stop yourself:

mount = repo.mount(mountpoint: "/mnt/restic")
begin
  # ... the mount stays live until something calls #stop ...
ensure
  mount.stop
end

#stop is safe to call more than once, and Mount#pid/Mount#stop are safe even if something else already triggered the unmount externally (kill -INT <pid>, or the OS's own umount/fusermount directly). hosts:/paths:/tags: restrict which snapshots appear under the mount's own snapshots/ directory; path_templates:/time_template: control the directory naming scheme Restic generates there. allow_other:/no_default_permissions:/owner_root: are FUSE-level access options (allow other users to access the mount, skip permission checks, mount files as owned by root, respectively). ready_timeout: (default 10 seconds) bounds how long #mount waits for Restic's own readiness message before raising Yobi::MountTimeout.

Restic-level (not scoped to a repository)

restic = Yobi::Restic.new
restic.version # => {"version"=>"0.19.1", "go_version"=>"...", ...}
restic.cache(cleanup: true)

#version and #cache are the two Restic subcommands that don't touch a repository at all, so they live on Yobi::Restic rather than Repository. #cache lists (or, with cleanup: true, cleans) local cache directories, returning true. max_age: limits cleanup to caches older than the given duration; no_size: skips computing directory sizes, for a faster listing.

Global Restic settings

Settings that apply regardless of which repository is being operated on live on Yobi::Restic, not Repository:

restic = Yobi::Restic.new(
  cache_dir: "/var/cache/restic",
  limit_upload: 5000,
  no_lock: true
)

Two genuinely different kinds, both exposed as plain attr_accessors:

• Env-var-backed: cache_dir:, compression:, pack_size:, read_concurrency:, host:, progress_fps:, cacert:, tls_client_cert:, key_hint:. Restic#env computes the corresponding RESTIC_* vars fresh from the current accessor values on every call. Mutating one after construction is picked up by the very next command.
• CLI-flag-only, no env var equivalent: limit_download:, limit_upload:, retry_lock:, no_lock:, no_cache:, cleanup_cache:, no_extra_verify:, stuck_request_timeout:, options: (Restic's own --option/-o, an escape hatch for backend-specific tuning like s3.connections=10), http_user_agent:, quiet:.

"Picked up by the very next command" assumes calls happen one at a time. If you mutate one of these accessors on a Restic shared across threads while another thread has a command already in flight on it, which value that in-flight command actually uses is a race with no ordering guarantee, though not a crash: each accessor read/write is still atomic, so you won't get a corrupted argv, just an unpredictable choice between the old and new value.

Sharing a Restic across repositories

restic = Yobi::Restic.new(limit_upload: 5000)

customer_a = Yobi::Repository.new(url: "s3:.../customer-a", password: "...", restic: restic)
customer_b = Yobi::Repository.new(url: "s3:.../customer-b", password: "...", restic: restic)

Both repositories now use the same limit_upload: value, and changing restic.limit_upload later affects every repository built from it. This isn't a shared, pooled cap: Restic applies --limit-upload per invocation, so two backups running concurrently through this same restic are each independently capped at 5000, not splitting one combined 5000 between them. Useful when one process backs up many repositories under one operational policy, without repeating the same settings on every Repository.new call.

Minimum-version enforcement, in detail

Yobi::Restic#run/#run_dump/#run_mount all call #ensure_minimum_version! before executing a real command. It's memoized per Restic instance. The check runs once, not on every call, since the binary on disk isn't expected to change mid-process. See "Minimum supported version" above for what triggers it and why 0.17.1 specifically.

Redaction

Both Repository#inspect and Restic#inspect redact sensitive values, so a stray pp/puts/log call never prints a credential in plaintext:

repo.inspect
# => #<Yobi::Repository url="s3:..." password="[FILTERED]" backend_credentials={"AWS_ACCESS_KEY_ID"=>"[FILTERED]", "AWS_SECRET_ACCESS_KEY"=>"[FILTERED]"}>

A literal or command/file-sourced password: shows as "[FILTERED]"; anything callable shows as "[RESOLVER]" without ever being called. Printing a Repository never triggers a real secret-store lookup as a side effect. :insecure_no_password shows plainly, since it's an explicit "there is no secret" declaration, not a value that could leak anything. backend_credentials: keys not in Restic's own documented list of non-identity operational env vars (Yobi::Restic::ALLOWED_ENV_VARS) are filtered the same way; the allowed ones (RESTIC_CACHE_DIR, AWS_DEFAULT_REGION, and similar non-secret operational vars) stay visible.

#env itself is never redacted. It's the method that actually builds what gets passed to Process.spawn, so it has to return the real values. Only #inspect (the thing a stray debug print might accidentally call) filters.

Recipes

Examples for common day-to-day scenarios.

Scheduled backup with retention

A nightly job (cron, whenever, a Sidekiq-cron/sidekiq-scheduler job, whatever runs on your schedule):

repo = Yobi::Repository.new(
  url: ENV.fetch("RESTIC_REPOSITORY"),
  password: [:file, "/etc/restic/password"]
)

outcome = repo.backup(
  source: "/var/myapp/uploads",
  tags: ["myapp", "nightly"],
  exclude_caches: true
)

unless outcome.success?
  outcome.errors.each { |e| log_error("backup: #{e.item}: #{e.message}") }
end

repo.forget(
  keep_daily: 7,
  keep_weekly: 4,
  keep_monthly: 12,
  tags: ["myapp"],
  prune: true
)

outcome.partial? (exit code 3) means some files were skipped but the snapshot was still created (often fine to continue past, e.g. permission-denied on a handful of files); a full failure raises before outcome even exists via the usual Yobi::ResticCommandFailed/Yobi::RepositoryLocked/etc. hierarchy, so wrap the whole thing in the usual rescue if you want to alert on that separately (see "Graceful lock-contention handling" below).

Restore workflows

Restoring the latest snapshot, the common case:

repo.restore(snapshot_id: "latest", target: "/var/myapp/uploads")

Restoring a specific snapshot, found by filtering:

snapshot = repo.snapshots(tags: "nightly").max_by(&:time)
repo.restore(snapshot_id: snapshot.id, target: "/tmp/restore-#{snapshot.short_id}")

Browsing before committing to a restore: mount the repository and look around, rather than restoring blind:

repo.mount(mountpoint: "/mnt/restic") do |mount|
  Dir.children("#{mount.mountpoint}/snapshots/latest/var/myapp/uploads")
end

or list without mounting anything:

repo.ls(snapshot_id: "latest", dirs: "/var/myapp/uploads").entries.each do |entry|
  puts "#{entry.type}  #{entry.size}  #{entry.path}"
end

Partial restore: only what you actually need back:

repo.restore(
  snapshot_id: "latest",
  target: "/tmp/restore",
  includes: "/var/myapp/uploads/2026-*"
)

Credentials from a secret store

Any callable works for password:/backend_credentials:. It's invoked fresh immediately before every Restic command, never cached by Yobi itself:

repo = Yobi::Repository.new(
  url: "s3:s3.amazonaws.com/my-bucket",
  password: -> { Rails.application.credentials.dig(:restic, :password) },
  backend_credentials: -> {
    {
      "AWS_ACCESS_KEY_ID" => Rails.application.credentials.dig(:aws, :access_key_id),
      "AWS_SECRET_ACCESS_KEY" => Rails.application.credentials.dig(:aws, :secret_access_key)
    }
  }
)

Or from Vault:

password: -> {
  Vault.logical.read("secret/data/restic")&.data&.dig(:data, :password)
}

If the resolver itself is slow or rate-limited (a network round-trip per Restic invocation adds up over many commands), add your own caching inside the lambda; how long the result stays valid is entirely your call.

Database dumps via stdin_from_command

Backing up a live database consistently means dumping it first, not copying its on-disk files while they might be mid-write. source: [:stdin_from_command, ...] has Restic spawn the dump command itself and back up its stdout directly, with no intermediate file ever touching disk:

repo.backup(
  source: [:stdin_from_command, "pg_dump --no-owner mydb", "mydb.sql"],
  tags: ["mydb", "nightly"]
)

The third element ("mydb.sql") names the resulting virtual file inside the snapshot. Omit it to fall back to Restic's own "stdin" default. If the command needs an argument containing a literal space (a password with a space in it, an unusual path), pass it as an Array of already-discrete arguments instead of a single String, since Shellwords-splitting a string would otherwise break it apart incorrectly. There's no shell involved either way (Restic execs the tokenized argv directly), but if any part of the String form is ever built from untrusted input rather than a hardcoded command like above, Shellwords.split will still tokenize whatever that input contains, letting it inject extra arguments into what gets executed. Use the Array form with untrusted values kept to their own discrete elements instead.

repo.backup(source: [:stdin_from_command, ["mysqldump", "-u", "backup", "my app db"]])

If the dump command itself fails partway (wrong credentials, database down), Restic detects the non-zero exit and does not persist a snapshot. It raises Yobi::ResticCommandFailed the same as any other command failure, so a broken dump never silently becomes a "successful" backup of garbage data.

Live progress reporting

Streaming #backup's block into something other than stdout: a logger, a progress bar, a broadcast to a browser:

# A logger
repo.backup(source: source_path) do |message|
  case message
  when Yobi::BackupStatus
    logger.info("backup: #{((message.percent_done || 0) * 100).round}%")
  when Yobi::BackupError
    logger.error("backup: #{message.item}: #{message.message}")
  end
end

# ActionCable, for a live-updating admin dashboard
repo.backup(source: source_path) do |message|
  next unless message.is_a?(Yobi::BackupStatus)

  BackupChannel.broadcast_to(job, {
    percent: message.percent_done,
    files_done: message.files_done,
    total_files: message.total_files
  })
end

Multi-repository operations

One Restic (holding shared operational settings) backing many Repository instances, e.g. one repository per customer or per managed server:

restic = Yobi::Restic.new(limit_upload: 10_000, cache_dir: "/var/cache/restic")

def repository_for(customer)
  Yobi::Repository.new(
    url: "s3:s3.amazonaws.com/backups/#{customer.id}",
    password: -> { customer.restic_password },
    restic: restic
  )
end

Customer.find_each do |customer|
  repository_for(customer).backup(source: customer.data_path, tags: [customer.id.to_s])
end

Running commands concurrently through a shared Repository/Restic is safe (see "Global Restic settings" above for the one caveat, mutating restic's tuning accessors from another thread while backups are in flight), so backing up many repositories doesn't have to run strictly one at a time. A Thread.new per customer would spawn an unbounded number of concurrent restic subprocesses, and Ruby silently drops an unhandled exception from a thread that's never joined, so a single customer's failure could vanish instead of getting logged. A small fixed-size worker pool avoids both:

queue = Queue.new
Customer.find_each { |customer| queue << customer }
queue.close

workers = 4.times.map do
  Thread.new do
    while (customer = queue.pop)
      begin
        repository_for(customer).backup(source: customer.data_path, tags: [customer.id.to_s])
      rescue Yobi::Error => e
        log_error("backup failed for customer #{customer.id}: #{e.message}")
      end
    end
  end
end
workers.each(&:join)

Copying between repositories (e.g. replicating to a second, offsite backend for a 3-2-1 strategy):

offsite = Yobi::Repository.new(url: "b2:my-bucket:/", password: "...")
offsite.copy(from_repo: primary_repo, tags: "nightly")

Repository maintenance and health checks

A separate, less-frequent job (weekly, say) than the nightly backup itself:

check = repo.check(read_data_subset: "10%")
if check.report["num_errors"].to_i.positive?
  check.errors.each { |e| AlertService.notify("Restic check: #{e.message}") }
end

repo.prune(max_unused: "5%")

read_data_subset: rotates through the repository over time (e.g. "1/20" today, "2/20" tomorrow) rather than reading everything in one run, so a full verification eventually happens without one job spending hours reading the entire dataset.

Key rotation

repo.key_passwd(new_password_file: "/etc/restic/new-password")

# repo itself still holds the OLD password - rebuild before continuing
rotated_repo = Yobi::Repository.new(
  url: repo.url,
  password: [:file, "/etc/restic/new-password"]
)

This rotates the password, not the underlying encryption key itself (see "Key management" above for why repo still needs rebuilding afterward). See "Low-level (cat) and object listing"'s note on #cat_masterkey_and_game_over_if_this_leaks if that distinction matters for your threat model.

Graceful lock-contention handling

Two processes touching the same repository at once (a backup job and a prune job overlapping, say): retry instead of failing the whole run:

def with_retry(max_attempts: 3, delay: 30)
  attempts = 0
  begin
    yield
  rescue Yobi::RepositoryLocked
    attempts += 1
    raise if attempts >= max_attempts

    sleep(delay + rand(delay)) # jittered, so lockstep retries against the same lock spread out
    retry
  end
end

with_retry { repo.backup(source: source_path) }

For a one-off manual fix instead of retrying, repo.unlock removes stale locks left by a process that crashed without cleaning up after itself, but don't call it reflexively inside a retry loop that might be racing a legitimately still-running second process; that would double-run the backup.

Non-goals

• Windows. Relies on Process.spawn/POSIX unlink semantics; only verified on Linux and macOS so far.
• Bundled secret-vault integration. Credentials are your own responsibility. Pass a callable for password:/backend_credentials: if you need one resolved from Vault, 1Password, or similar (see "Credentials from a secret store" above).
• A built-in scheduler/orchestrator. Yobi runs a command when you call it; deciding when that happens (cron, whenever, a Sidekiq-cron job) is your own responsibility, see "Scheduled backup with retention" above.

Contributing

Bug reports and pull requests are welcome at https://codeberg.org/ukazap/yobi.

After checking out the repo, run bin/setup to install dependencies.

• bin/test-unit: the unit suite, no I/O, no Restic/minio needed.
• bin/test-integration: the integration suite, shelling out to a real Restic binary; skipped automatically if Restic isn't installed.
• bin/test: both, in parallel.
• bin/standardrb: lint.
• bin/console: an interactive prompt with the gem loaded.
• bin/docs: serves YARD docs at http://localhost:8808, reparsing on every request.

See MAINTAINERS.md for what to keep in sync as the API changes, and the release checklist.

License

The gem is available as open source under the terms of the MIT License.