Class: Bundler::Spinel::Site

Inherits:
Object
  • Object
show all
Defined in:
lib/bundler/spinel/site.rb

Overview

Builds the static spinelgems.org deploy tree — the *apex double-duty* layout, where one directory served by one static host is both a human website and a machine RubyGems source:

out/index.html             presentation (copied from the repo's site/ dir)
out/catalog.html           landing: verdict-mix chips + per-verdict links
out/catalog-<verdict>.html one page per verdict (split for browser perf —
                           one 90MB HTML was painful, even on a fast box)
out/assets/…               shared CSS
out/versions  out/names  out/info/<gem>  out/gems/<file>.gem
                           the Compact Index (only when a --store is given)

The reserved Compact Index paths and the human paths don’t collide, so ‘source “spinelgems.org”` in a Gemfile resolves against the curated source while a browser at the same origin gets the site. The catalog is the human-readable view of the same ledger the gate and the curated source are built on, joined with rubygems.org metadata (a meta.jsonl sidecar). It renders offline from committed data — no Spinel, no network — so it builds on the deploy host; the engine rev is read from the ledger, not probed.

Defined Under Namespace

Classes: Row

Constant Summary collapse

SRC =

the repo’s site/ source dir

File.expand_path("../../../site", __dir__)
VERDICT_ORDER = 
%w[verified loaded clean risky rejected].freeze
GLYPH = 
{ "verified" => "★", "loaded" => "○", "clean" => "✓", "risky" => "~", "rejected" => "✗" }.freeze
VERDICT_RANK =

Verdict ladder by strength. Used to pick the strongest current-rev signal for a gem when multiple probes wrote different verdicts (the survey writes clean, an earlier harness pass at the same rev wrote loaded; the harness ran something the survey didn’t, so it wins). Rejected is handled separately by ‘pick_current` — a caught failure beats any success-shaped signal regardless of where it ranks here.

{ "rejected" => 0, "risky" => 1, "clean" => 2, "loaded" => 3, "verified" => 4 }.freeze
MIN_DOWNLOADS =

Default downloads floor for the catalog’s “hide low-signal gems” toggle —weeds out test / security-researcher / throwaway gems (the exfil PoC has ~580 downloads; rake has ~1.3B). Tunable via SPINEL_CATALOG_MIN_DOWNLOADS.

Integer(ENV.fetch("SPINEL_CATALOG_MIN_DOWNLOADS", "1000"))
REJECTED_CAP =

Cap on the rejected page (~113k full would be ~50MB of HTML — still slow to load even split out). Top-N by downloads keeps the signal — popular gems we want and can’t yet have — and drops the long tail of obscure rejects, which is just noise for browsing. The complete machine-readable list stays in compat.jsonl / candidates.tsv for anyone who needs it.

Integer(ENV.fetch("SPINEL_CATALOG_REJECTED_CAP", "2000"))
BLURB =

One-line semantics per verdict — used as the lede on each per-verdict page.

{
  "verified" => "<strong>Full surface</strong> compiles and a behaviour smoke matches CRuby under a Spinel-compiled harness — every <code>lib/</code> file force-required (no <code>autoload</code> masking, no missing-dependency rescue), not just the entrypoint. The only verdict to trust where it matters. A constant/VERSION-only smoke that loads the entrypoint but leaves the gem's real code behind <code>autoload</code> is <em>not</em> enough — that overstated usability, so the bar was tightened to whole-surface. Sticky across engine revisions until a re-run catches a regression.",
  "loaded"   => "Compiles and loads identically under CRuby and Spinel via a require-only differential. Logic untested — a gem can load fine and still silently miscompile in the code paths the require-only smoke doesn't exercise. Weaker than <strong>verified</strong>; not a trust signal.",
  "clean"    => "Compiles clean (cheap static lower bound). No behaviour was exercised — the survey doesn't run the gem. Massively overstates compatibility; the harness is the trustworthy check.",
  "risky"    => "Compiles, but the source uses constructs Spinel degrades silently (<code>eval</code>, <code>define_method</code>, …). Allowed by default; fails under <code>spinel-compat check --strict</code>.",
  "rejected" => "Doesn't compile, or compiles to silent no-ops we detected — including <code>rejected:miscompile</code> caught by the harness. Each reason names the missing feature; the histogram is the prioritized roadmap of Spinel asks."
}.freeze
BADGE =

Composable signal badges layered on top of the verdict (spinelgems#4/#6). The verdict is the rank (how far the gem compiled/ran); badges are *orthogonal signals* a gem can additionally carry. A gem can be ★ verified AND 👤 human-attested AND ✪ own-tests-passing all at once.

human — a person attested it works in real use (attestations.jsonl).
        The strongest signal there is: a human's "I built on this."
tests — the gem's OWN test suite compiles, runs, and matches CRuby under
        Spinel (verify --tests). Zero per-gem human effort, strictly
        stronger than a hand smoke. Rare today (whole-program inference,
        #1043/#1062 — every suite tried still surfaces a miscompile).

{
  "human" => { glyph: "👤", label: "human-attested",
    blurb: "A person has used this gem in a real Spinel/Tep program and attests it works — the highest-trust signal we carry. Version-pinned in <code>attestations.jsonl</code>." },
  "tests" => { glyph: "✪", label: "own-tests pass",
    blurb: "The gem's own test suite compiles, runs, and matches CRuby under a Spinel-compiled harness (<code>verify --tests</code>) — a stronger, zero-human-effort behaviour signal than a hand smoke." },
}.freeze
<<~'FOOT'
  <footer><div class="foot-wrap">
    <div class="foot-built">
      <span class="by"><svg width="16" height="16" viewBox="0 0 24 24" aria-hidden="true"><path d="M6 3h12l4 6-10 13L2 9z" fill="#b31217"/><path d="M6 3 2 9l10 13z" fill="#7a0c0f"/><path d="M18 3l4 6-10 13z" fill="#d42b2b"/><path d="M6 3h12l-6 6z" fill="#e86a6a"/></svg> <a href="https://github.com/matz/spinel">Spinel</a>-compiled Ruby</span>
      <span class="by">Built with <a href="https://github.com/OriPekelman/tep">Tep</a></span>
      <span class="by"><svg width="16" height="16" viewBox="0 0 24 24" aria-hidden="true"><circle cx="12" cy="12" r="4.4" fill="#ff6b57"/><g stroke="#ff6b57" stroke-width="1.7" stroke-linecap="round"><path d="M12 2.2v2.6"/><path d="M12 19.2v2.6"/><path d="M2.2 12h2.6"/><path d="M19.2 12h2.6"/><path d="M5.1 5.1l1.8 1.8"/><path d="M17.1 17.1l1.8 1.8"/><path d="M18.9 5.1l-1.8 1.8"/><path d="M6.9 17.1l-1.8 1.8"/></g></svg> Hosted on <a href="https://upsun.com">Upsun</a></span>
    </div>
    <p class="foot-note">Pre-release · verdicts keyed on the Spinel engine revision ·
      <a href="https://github.com/OriPekelman/spinelgems">source &amp; RFC on GitHub</a></p>
  </div></footer>
FOOT
ATTESTATIONS =

Sidecar signal sources (repo-root, committed, version-pinned). Absent →the badge is simply never granted.

File.expand_path("../../../attestations.jsonl", __dir__)
TEST_RESULTS = 
File.expand_path("../../../test-results.jsonl", __dir__)

Instance Method Summary collapse

Constructor Details

#initialize(ledger: Ledger.new, engine: nil, src: SRC, meta_path: nil, attestations_path: ATTESTATIONS, test_results_path: TEST_RESULTS) ⇒ Site

Returns a new instance of Site.



101
102
103
104
105
106
107
108
109
 
# File 'lib/bundler/spinel/site.rb', line 101

def initialize(ledger: Ledger.new, engine: nil, src: SRC, meta_path: nil,
               attestations_path: ATTESTATIONS, test_results_path: TEST_RESULTS)
  @ledger = ledger
  @engine = engine
  @src = src
  @meta_path = meta_path
  @attestations_path = attestations_path
  @test_results_path = test_results_path
end

Instance Method Details

#build(out, store: nil, min_verdict: :verified) ⇒ Object

out: deploy dir. store: optional dir of vetted .gem files → Compact Index.



112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
 
# File 'lib/bundler/spinel/site.rb', line 112

def build(out, store: nil, min_verdict: :verified)
  FileUtils.mkdir_p(out)
  copy_presentation(out)

  rs = rows
  counts = Hash.new(0)
  rs.each { |r| counts[r.verdict] += 1 }

  File.write(File.join(out, "catalog.html"), catalog_landing_html(rs, counts))
  VERDICT_ORDER.each do |v|
    File.write(File.join(out, "catalog-#{v}.html"), verdict_page_html(v, rs, counts))
  end

  compact_index(out, store, min_verdict) if store
  out
end

#build_db(db_path) ⇒ Object

Materialize the catalog into a SQLite DB for the dynamic (Tep) server: one row per gem, with ‘rows`’ stickiness already applied, so the runtime only does indexed SELECTs (no 209k-line ledger replay per request, no 90MB-HTML split, no REJECTED_CAP). Built at deploy and served read-only.

We shell out to the ‘sqlite3` CLI (TSV `.import`) rather than the Ruby sqlite3 gem: no native-gem build at deploy, and the file is read by Tep::SQLite © at runtime — same on-disk format. `rows` stays the single source of verdict truth.



138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
 
# File 'lib/bundler/spinel/site.rb', line 138

def build_db(db_path)
  require "open3"
  require "tempfile"
  require "time"

  rs = rows
  # Strip control chars incl. the ASCII unit/record separators (\x1c-\x1f)
  # we use as the .import delimiters, so no field value can break a row.
  clean = ->(s) { s.to_s.gsub(/[\t\r\n\x1c-\x1f]+/, " ").strip }
  us = "\x1f" # unit (field) separator
  rs_ = "\x1e" # record separator
  tsv = Tempfile.new(["catalog", ".asv"])
  begin
    rs.each do |r|
      tsv.write([r.gem, r.gem.downcase, clean.(r.version), r.verdict,
                 r.downloads.to_i, clean.(r.info), clean.(r.updated),
                 clean.(r.homepage), clean.(r.notes),
                 r.human ? 1 : 0, r.tests ? 1 : 0, clean.(r.rubric),
                 fmt_n(r.downloads)].join(us) + rs_)
    end
    tsv.flush
    FileUtils.rm_f(db_path)
    # .mode ascii uses \x1f/\x1e separators with NO quote processing —
    # robust for arbitrary description text (.mode tabs/csv quote-swallows
    # rows whose info contains a `"`, silently dropping ~10% of gems).
    sql = <<~SQL
      PRAGMA journal_mode=OFF;
      CREATE TABLE catalog (
        gem TEXT PRIMARY KEY, gem_lower TEXT NOT NULL, version TEXT,
        verdict TEXT NOT NULL, downloads INTEGER NOT NULL DEFAULT 0,
        info TEXT, updated TEXT, homepage TEXT, notes TEXT,
        human INTEGER NOT NULL DEFAULT 0, tests INTEGER NOT NULL DEFAULT 0,
        rubric TEXT,
        -- pre-formatted "3.4B" for display: the Tep server reads downloads
        -- via sqlite3_column_int (C 32-bit), which wraps values >2^31
        -- (bundler's 3.4B showed as -867M). ORDER BY / filtering still use
        -- the 64-bit `downloads` column SQL-side; only the display string
        -- comes from here, so the truncation never reaches the page.
        downloads_fmt TEXT
      );
      .mode ascii
      .import #{tsv.path} catalog
      CREATE INDEX idx_verdict_dl ON catalog(verdict, downloads DESC);
      CREATE INDEX idx_downloads  ON catalog(downloads DESC);
      CREATE INDEX idx_gem_lower  ON catalog(gem_lower);
      CREATE INDEX idx_human ON catalog(human, downloads DESC);
      CREATE INDEX idx_tests ON catalog(tests, downloads DESC);
      CREATE TABLE catalog_meta (key TEXT PRIMARY KEY, value TEXT);
      INSERT INTO catalog_meta VALUES
        ('rev', #{sql_str(rev)}),
        ('total', '#{rs.size}'),
        ('built_at', '#{Time.now.utc.strftime("%Y-%m-%dT%H:%M:%SZ")}');
    SQL
    out, st = Open3.capture2e("sqlite3", db_path, stdin_data: sql)
    raise Error, "sqlite3 build failed: #{out}" unless st.success?
  ensure
    tsv.close!
  end
  db_path
end