Class: RSpecTracer::RemoteCache::LocalFsBackend Private

Inherits:
Object
  • Object
show all
Defined in:
lib/rspec_tracer/remote_cache/local_fs_backend.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Filesystem implementation of ‘RemoteCache::Backend`. Target is a shared directory: an NFS mount, a per-host dev cache, or a CI workspace volume. Two-tier layout mirrors `S3Backend` bit-for-bit; a LocalFs root directory can be rsync’d to/from S3 without any transform (same ‘cache.tar.gz` per ref, same `branch_refs.json` path, same tier prefixes).

<root>/main/<sha>/[<test_suite_id>/]cache.tar.gz
<root>/pr/<branch>/<sha>/[<test_suite_id>/]cache.tar.gz
<root>/pr/<branch>/branch_refs.json

Uploads are atomic: the archive is staged at a sibling tmp path on the same filesystem, then ‘File.rename`d into place. POSIX rename is atomic on same-filesystem moves, which covers every shared-mount topology LocalFs targets.

Concurrent writes to the same ref: last-write-wins is correct because the archive content is a deterministic function of the local cache (two workers on the same SHA produce identical bytes). No file locking - flock is unreliable over NFS (lockd sharp edges) and buys nothing when contents match.

NFS caveat: on a network filesystem, cross-node consistency is eventual. A download issued by node B immediately after an upload on node A may miss; retries converge. Document as user concern, not a backend correctness issue.

Defined Under Namespace

Classes: LocalFsBackendError

Constant Summary collapse

MAIN_TIER =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Internal constant.

'main'
PR_TIER =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Internal constant.

'pr'
BRANCH_REFS_FILENAME =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Internal constant.

'branch_refs.json'
LAST_RUN_FILENAME =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Internal constant.

'last_run.json'
CACHE_ARCHIVE_FILENAME =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Internal constant.

Archive::CACHE_FILENAME
ENCODING =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Internal constant.

'UTF-8'

Instance Method Summary collapse

Constructor Details

#initialize(root:, branch:, default_branch:, cache_path:, test_suite_id: nil, logger: nil) ⇒ LocalFsBackend

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

rubocop:disable Metrics/ParameterLists



66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 66

def initialize(root:, branch:, default_branch:, cache_path:,
               test_suite_id: nil, logger: nil)
  validate_required!(root: root, branch: branch,
                     default_branch: default_branch, cache_path: cache_path)

  @root = File.expand_path(root.to_s)
  @branch = branch.to_s.chomp
  @default_branch = default_branch.to_s.chomp
  @test_suite_id = normalize_test_suite_id(test_suite_id)
  @cache_path = cache_path.to_s
  @logger = logger
end

Instance Method Details

#branch_refs(branch_name) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Read branch_refs for the given branch. Returns ‘=> ts_epoch` or `{}` on missing / malformed. PR tier only.



128
129
130
131
132
133
134
135
136
137
138
139
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 128

def branch_refs(branch_name)
  return {} if blank?(branch_name)

  path = branch_refs_path(branch_name)
  return {} unless File.file?(path)

  parsed = JSON.parse(File.read(path, encoding: ENCODING))
  parsed.is_a?(Hash) ? parsed.transform_values(&:to_i) : {}
rescue StandardError => e
  log_debug("branch_refs read failed (#{e.class}: #{e.message}); treating as empty")
  {}
end

#download(ref, tree_sha: nil) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Download the cache for ‘ref` into `cache_path`. Tries the backend’s own tier first; on miss, falls back to the main tier for the same ref. Validates via ‘schema_version` before declaring success. Returns true on validated success, false otherwise. Cleans up partially-extracted state on failure.

‘tree_sha:` is accepted for protocol uniformity with S3Backend but is currently a no-op: the tree-SHA secondary index is an S3-only feature. Future enhancement may extend it here; the orchestrator already forwards the kwarg.



90
91
92
93
94
95
96
97
98
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 90

def download(ref, tree_sha: nil)
  _ = tree_sha
  return false if blank?(ref)

  tiers_to_try = [own_tier_prefix]
  tiers_to_try << main_tier_prefix if pr_tier?

  tiers_to_try.any? { |tier| try_download_from(tier, ref) }
end

#prune!(count: nil, duration_seconds: nil, pr_branch_ttl_seconds: nil) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Apply retention to the backend’s own tier. Returns count removed. Semantics match S3Backend: count keeps newest N, duration prunes by mtime, pr_branch_ttl deletes the whole branch prefix when idle. Two or more params may be set simultaneously; all nil/0 is a no-op. Never raises on partial I/O failure.



166
167
168
169
170
171
172
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 166

def prune!(count: nil, duration_seconds: nil, pr_branch_ttl_seconds: nil)
  removed = 0
  removed += prune_by_count!(count) if count&.positive?
  removed += prune_by_duration!(duration_seconds) if duration_seconds&.positive?
  removed += prune_dead_pr_branch!(pr_branch_ttl_seconds) if pr_tier? && pr_branch_ttl_seconds&.positive?
  removed
end

#prune_all!(pr_branch_ttl_seconds: nil) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Cross-tier PR-branch cleanup. Enumerates every branch dir under ‘pr/`, applies the ttl to each, deletes branches with no ref newer than the cutoff. Returns total refs removed. No-op on nil / non-positive ttl.



178
179
180
181
182
183
184
185
186
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 178

def prune_all!(pr_branch_ttl_seconds: nil)
  return 0 unless pr_branch_ttl_seconds&.positive?

  cutoff = Time.now.to_i - pr_branch_ttl_seconds.to_i
  pr_root = File.join(@root, PR_TIER)
  return 0 unless File.directory?(pr_root)

  branch_dirs(pr_root).sum { |branch_dir| maybe_prune_branch(branch_dir, cutoff) }
end

#unbounded_warning(warn_threshold: 500) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Warn when the main tier has grown beyond a soft threshold and no retention is configured. Called from the orchestrator.



190
191
192
193
194
195
196
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 190

def unbounded_warning(warn_threshold: 500)
  refs = list_refs_in_tier(MAIN_TIER)
  return nil unless refs.length > warn_threshold

  "rspec-tracer remote cache has #{refs.length} refs in #{@root}/#{MAIN_TIER}; " \
    'configure cache_retention_count or cache_retention_duration to cap growth'
end

#upload(ref, tree_sha: nil) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Upload the local cache to this backend’s own tier under ‘ref`. Packs the 15-file local layout into a `cache.tar.gz` via `Archive.pack`, renames into place atomically. Raises on a malformed local cache or an I/O failure.

‘tree_sha:` is accepted for protocol uniformity with S3Backend (no-op here; see `download`).

Raises:



107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 107

def upload(ref, tree_sha: nil)
  _ = tree_sha
  raise LocalFsBackendError, 'ref is required' if blank?(ref)

  run_id = read_local_run_id
  raise LocalFsBackendError, "no local cache to upload (missing #{LAST_RUN_FILENAME})" if run_id.nil?

  dest = archive_path(own_tier_prefix, ref)
  FileUtils.mkdir_p(File.dirname(dest))
  staging = "#{dest}.tmp.#{Process.pid}.#{SecureRandom.hex(4)}"
  begin
    Archive.pack(cache_path: @cache_path, run_id: run_id, dest_path: staging)
    File.rename(staging, dest)
    log_debug("uploaded cache for #{ref} to #{own_tier_prefix} (#{File.size(dest)} bytes)")
  ensure
    FileUtils.rm_f(staging)
  end
end

#write_branch_refs(branch_name, refs) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Persist branch_refs for the given branch. No-op for main-branch writes. Atomic via tmp+rename. Raises on I/O failure for PR tier.



144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/rspec_tracer/remote_cache/local_fs_backend.rb', line 144

def write_branch_refs(branch_name, refs)
  return if blank?(branch_name)
  return if branch_name.to_s.chomp == @default_branch
  return if refs.nil? || refs.empty?

  path = branch_refs_path(branch_name)
  FileUtils.mkdir_p(File.dirname(path))
  staging = "#{path}.tmp.#{Process.pid}.#{SecureRandom.hex(4)}"
  begin
    File.write(staging, JSON.pretty_generate(refs), encoding: ENCODING)
    File.rename(staging, path)
    log_debug("wrote branch_refs for #{branch_name}")
  ensure
    FileUtils.rm_f(staging)
  end
end