Class: Gigatoken::Hub

Inherits:
Object
  • Object
show all
Defined in:
lib/gigatoken/hub.rb

Overview

HuggingFace Hub file fetch, mirroring huggingface_hub.hf_hub_download: same endpoint and URL layout, same token discovery (HF_TOKEN env var, then the token file written by hf auth login), same cache directory resolution — without requiring huggingface_hub, tokenizers, or transformers. Files already present in the standard HF cache are served with a pure-filesystem lookup (no network); on a miss the file is downloaded straight into the shared cache, so later loads (ours and huggingface_hub's) are served from it.

Network I/O runs on async-http, on the reactor: #hub_file wraps its fetch in Sync, so it composes whether the caller is already inside a reactor or is plain sync code.

Constant Summary collapse

TOKENIZER_FILE_SUFFIXES =

Filename suffixes of local tokenizer files (tokenizer.json contents and raw sentencepiece models). A name ending in one of these is never treated as a Hub repo id, so a mistyped local path fails fast instead of hitting the network. Keep in sync with src/load_tokenizer/hub.rs's TOKENIZER_FILE_SUFFIXES.

[".json", ".model"].freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(endpoint: DEFAULT_ENDPOINT) ⇒ Hub

Returns a new instance of Hub.



129
130
131
132
 
# File 'lib/gigatoken/hub.rb', line 129

def initialize(endpoint: DEFAULT_ENDPOINT)
  @endpoint = endpoint.chomp("/")
  @internet = Async::HTTP::Internet.new
end

Class Method Details

.cached_file(repo_id, filename, revision) ⇒ Object

Path of filename in the local HF cache, or nil when not cached. A pure-filesystem lookup — no request is made. revision may be a commit hash (used directly as the snapshot name) or a branch/tag name (followed through the cached ref).



82
83
84
85
86
87
88
89
 
# File 'lib/gigatoken/hub.rb', line 82

def cached_file(repo_id, filename, revision)
  repo_dir = repo_cache_dir(repo_id)
  commit = commit_hash?(revision) ? revision : cached_ref(repo_dir, revision)
  return nil unless commit

  path = repo_dir.join("snapshots", commit, filename)
  path if path.file?
end

.commit_hash?(revision) ⇒ Boolean

A full git commit hash: cache snapshot directories are named by these.

Returns:

  • (Boolean) 


98
99
100
 
# File 'lib/gigatoken/hub.rb', line 98

def commit_hash?(revision)
  revision.match?(/\A[0-9a-f]{40}\z/)
end

.hf_home ⇒ Object

$HF_HOME, defaulting to $XDG_CACHE_HOME/huggingface then ~/.cache/huggingface — the root for both the hub cache and the token file.



54
55
56
 
# File 'lib/gigatoken/hub.rb', line 54

def hf_home
  Pathname.new(env("HF_HOME") || File.join(env("XDG_CACHE_HOME") || File.join(Dir.home, ".cache"), "huggingface"))
end

.hf_hub_cache_dir ⇒ Object

The standard HuggingFace hub cache directory, resolved like huggingface_hub does it: HF_HUB_CACHE, then $HF_HOME/hub.



60
61
62
 
# File 'lib/gigatoken/hub.rb', line 60

def hf_hub_cache_dir
  Pathname.new(env("HF_HUB_CACHE") || hf_home.join("hub").to_s)
end

.hf_token ⇒ Object

The HuggingFace access token, discovered like huggingface_hub does it: the HF_TOKEN (or legacy HUGGING_FACE_HUB_TOKEN) environment variable, then the token file (HF_TOKEN_PATH, default $HF_HOME/token).



67
68
69
70
71
72
73
74
75
76
 
# File 'lib/gigatoken/hub.rb', line 67

def hf_token
  token = env("HF_TOKEN") || env("HUGGING_FACE_HUB_TOKEN")
  return token.strip if token

  token_path = env("HF_TOKEN_PATH") || hf_home.join("token").to_s
  return nil unless File.file?(token_path)

  token = File.read(token_path).strip
  token unless token.empty?
end

.looks_like_repo_id?(name) ⇒ Boolean

Whether name is shaped like a HuggingFace Hub repo id: org/name, or a bare legacy repo name like gpt2. At most one slash, and not something that is obviously a filesystem path to a local tokenizer file.

Returns:

  • (Boolean) 


40
41
42
43
44
45
46
47
48
49
 
# File 'lib/gigatoken/hub.rb', line 40

def looks_like_repo_id?(name)
  parts = name.split("/", -1)
  return false if parts.empty? || parts.size > 2

  org, rest = parts
  return false unless word_part?(org, first_alnum: true)
  return false if rest && !word_part?(rest, first_alnum: false)

  TOKENIZER_FILE_SUFFIXES.none? { |suffix| name.end_with?(suffix) }
end

.repo_cache_dir(repo_id) ⇒ Object

The cache directory of a repo (models--org--name).



92
93
94
 
# File 'lib/gigatoken/hub.rb', line 92

def repo_cache_dir(repo_id)
  hf_hub_cache_dir.join("models--#{repo_id.gsub("/", "--")}")
end

Instance Method Details

#hub_file(repo_id, filename = "tokenizer.json", revision: "main") ⇒ Object

Path of filename from Hub repo repo_id at revision, served from the standard HF cache, downloading into it first when absent.



136
137
138
139
 
# File 'lib/gigatoken/hub.rb', line 136

def hub_file(repo_id, filename = "tokenizer.json", revision: "main")
  self.class.cached_file(repo_id, filename, revision) ||
    Sync { fetch(repo_id, filename, revision) }
end