Unmagic::Browser

A real browser you can drive from Ruby — for one-off page grabs and multi-step automation. It runs locally in development and against Cloudflare Browser Rendering (or any remote Puppeteer endpoint) in production, behind one API. Built so an LLM agent — or plain Ruby code — can browse the web the way a person does.

browser = Unmagic::Browser.new

# one-off: grab a JS-rendered page as Markdown
browser.markdown("https://example.com")

# multi-step: log in, search, read the result
browser.open("https://news.ycombinator.com/login") do |session|
  session.fill_in "acct", with: "pg"
  session.fill_in "pw",   with: ENV["HN_PASSWORD"]
  session.click_button "login"
  session.wait_for text: "logout"
  session.screenshot
end

Installation

gem "unmagic-browser"

Bundler requires it for you. Outside Bundler, the gem's name maps to its path in the usual way:

require "unmagic/browser"

Drivers

The same API runs against three drivers. Pick one when you construct the browser; nothing else in your code changes. A symbol builds the driver with its defaults — when a driver needs options, construct it yourself and pass the instance.

# A symbol builds the driver with no arguments…
Unmagic::Browser.new(driver: :local)      # same as driver: Unmagic::Browser::Driver::Local.new
Unmagic::Browser.new(driver: :remote)
Unmagic::Browser.new(driver: :cloudflare)

# …pass an instance to hand a driver options.

# Local Chrome — great for development. headless: false opens a window you can watch.
Unmagic::Browser.new(driver: Unmagic::Browser::Driver::Local.new(headless: false))

# Any remote Puppeteer / CDP websocket — browserless, your own pool, etc.
Unmagic::Browser.new(driver: Unmagic::Browser::Driver::Remote.new(ws: "ws://browserless.internal:3000"))

# Cloudflare Browser Rendering — no browser runs on your servers.
Unmagic::Browser.new(driver: Unmagic::Browser::Driver::Cloudflare.new(
  account_id: ENV["CF_ACCOUNT_ID"],
  token:      ENV["CF_BROWSER_TOKEN"],
))

With no arguments, Unmagic::Browser.new uses :local in development and the configured default_driver everywhere else.

One-off page grabs

For "just get me this page", call straight on the browser. Each call takes the cheapest path its driver offers: the REST rendering API on Cloudflare (no browser to hold open), a throwaway page on local and remote.

browser.markdown("https://example.com")              # => String  (clean text)
browser.html("https://example.com")                  # => String  (fully JS-rendered HTML)
browser.screenshot("https://example.com")            # => PNG bytes
browser.screenshot("https://example.com",
                   full_page: true,                  # the whole scrollable page
                   selector:  "#main")               # or a single element
browser.pdf("https://example.com")                   # => PDF bytes

One-offs always render with default settings. To capture a page in dark mode, on a phone viewport, or in another locale, open a session and use emulation.

On local and remote the throwaway pages share one browser, started on first use and shut down when the process exits. browser.close lets it go sooner; sessions are unaffected, because each holds its own.

Sessions

A session is a live browser page you drive step by step. browser.open works like File.open: give it a URL and a block, and you get a session — already on that page — that's cleaned up when the block ends.

browser.open("https://store.example.com") do |session|
  session.fill_in "Search", with: "espresso"
  session.click_button "Go"
  session.wait_for text: "results"

  session.all(".product").map { |el| el.text }
end

Unmagic::Browser.open(url) is shorthand for Unmagic::Browser.new.open(url).

Call open without a block and it returns the session for you to keep — see sessions that outlive one call.

Verbs

The API reads like Capybara, so it's instantly familiar.

# navigate
session.visit(url)
session.reload
session.go_back
session.go_forward
session.current_url
session.title

# act
session.fill_in("Email", with: "me@example.com")
session.click_button("Sign in")
session.click_link("Forgot password?")
session.click("Anything at all")
session.select("Australia", from: "Country")
session.check("Remember me")
session.uncheck("Send me email")
session.choose("Express shipping")
session.attach_file("Résumé", "/path/to/cv.pdf")
session.press(:enter)
session.hover("Profile")

# read
session.text
session.html
session.markdown
session.value("Email")
session.screenshot
session.pdf
session.evaluate("document.title")

# wait & ask
session.wait_for(text: "Welcome")
session.wait_for(selector: ".loaded")
session.has_text?("Welcome")
session.has_button?("Sign in")
session.has_link?("Log out")
session.has_field?("Email")
session.has_selector?(".row")

# scope
session.find(".cart")
session.all(".product")
session.within(".checkout") { session.click_button("Pay") }

Verbs auto-wait: click_button "Sign in" keeps looking until the button appears and is actionable, up to a timeout — so you rarely need an explicit wait_for.

Questions don't. has_button? answers about the page as it stands and returns straight away, because a caller asking whether a challenge appeared wants "no" now, not in ten seconds. When you mean to wait, say so with wait_for.

Smart locators

You don't have to hand-write CSS. A plain string is resolved by label, then visible text, then ARIA role, falling back to CSS — whichever matches. That's what keeps the API working when the caller (an LLM, say) doesn't know the page's markup.

session.fill_in "Email", with: "me@example.com"   # finds <label>Email</label> → its input
session.click_button "Sign in"                    # finds the button reading "Sign in"

Be explicit when you want to be:

session.fill_in css:   "#user_email", with: "me@example.com"
session.click_button text: "Sign in"
session.find label: "Date of birth"

Emulation (dark mode, viewport, devices, locale…)

Everything about how the page renders — colour scheme, viewport, device, locale, timezone — lives in one emulate: config, set in exactly two places: passed to open, or changed live with session.emulate.

browser.open("https://example.com",
             emulate: { color_scheme: :dark,
                        viewport: { width: 1440, height: 900 } }) do |session|
  session.screenshot                   # dark mode, at 1440×900
end

The full set of knobs — valid as keys in any emulate: hash, or as a reusable object:

emulation = Unmagic::Browser::Emulation.new(
  color_scheme:   :dark,                           # :light | :dark | :no_preference
  reduced_motion: :reduce,                         # :reduce | :no_preference
  viewport:       { width: 390, height: 844,
                    scale: 2, mobile: true, touch: true },   # scale = device pixel ratio
  device:         "iPhone 13",                     # shortcut: sets viewport + UA + touch
  user_agent:     "MyBot/1.0",
  locale:         "en-AU",                         # Accept-Language
  timezone:       "Australia/Sydney",
  geolocation:    { latitude: -33.87, longitude: 151.21 },
  media:          :print,                          # :screen | :print (handy before #pdf)
)

# build it once, reuse it across sessions
browser.open(url, emulate: emulation) { |session| ... }

# change it live — applies to everything after
session.emulate(color_scheme: :light)
session.emulate(viewport: { width: 1280, height: 800 })

# or scope a change to a block — reverts when the block ends
session.emulate(color_scheme: :dark) do
  session.screenshot                   # dark…
end
session.screenshot                     # …light again

Whatever you leave out renders at a fixed default — desktop viewport, light, screen media — not at whatever the machine underneath prefers. Headless Chrome inherits the host's appearance if you let it, which would mean the same page rendering dark on a laptop after sunset and light on CI. Pass color_scheme: :no_preference if you actually want the host's.

Niche emulation (CPU/network throttling, offline, vision-deficiency simulation) isn't wrapped — reach it through the escape hatch.

Sessions that outlive one call

A block session is perfect inside a single method. An LLM agent, though, works turn by turn — open a page in one call, act on it in the next, let a human step in, then continue. So open without a block hands you the session to keep. It's a plain object; hold it wherever your app keeps state.

session = browser.open("https://example.com/login")
session.fill_in "Email", with: "me@example.com"
session.click_button "Continue"

# ...later, same object, same live browser...
session.fill_in "Password", with: secret
session.click_button "Sign in"
session.text

session.close

A kept session holds a real browser — and on Cloudflare, one of a limited number of concurrent slots — until you close it.

Operations on a session are serialized, so two concurrent callers can't interleave a keystroke. When a sequence must be atomic against other callers, wrap it:

session.lock do
  session.fill_in "Amount", with: "100"
  session.click_button "Transfer"
end

Human in the loop (captcha / 2FA)

Because a kept session stays live between calls, a human can step in mid-flow: submit the login, detect a challenge, show the person a screenshot, and resume once they've answered — all against the same session.

session = browser.open("https://example.com/login")
session.fill_in "Email", with: "me@example.com"
session.fill_in "Password", with: secret
session.click_button "Sign in"

if session.has_text?("Enter the code we texted you")
  # ...show session.screenshot to a person, collect the code...
end

# later, once the human has answered — same session, still live:
session.fill_in "Code", with: human_supplied_code
session.click_button "Verify"

For visual captchas, use a local driver with headless: false and let the person click the real window, then keep driving.

Errors

Failures are typed, so you classify them by rescuing — never by parsing a message.

begin
  browser.markdown(url)
rescue Unmagic::Browser::Error::TimedOut      # the page took too long
rescue Unmagic::Browser::Error::Unreachable   # couldn't reach the page or the rendering service
rescue Unmagic::Browser::Error::RenderFailed  # the service answered but couldn't render
rescue Unmagic::Browser::Error::RateLimited   # hit a concurrency / rate limit
rescue Unmagic::Browser::Error::Misconfigured # missing credentials / config
rescue Unmagic::Browser::Error::NotFound      # nothing on the page matched a locator
rescue Unmagic::Browser::Error::SessionClosed # the session has been closed
rescue Unmagic::Browser::Error                # base class for all of the above
end

NotFound is a TimedOut, because locators auto-wait: "not found" always means "still not there when we ran out of patience".

Configuration

Unmagic::Browser.configure do |config|
  # used when Unmagic::Browser.new is called without a driver — a symbol or an instance
  config.default_driver = Unmagic::Browser::Driver::Cloudflare.new(
    account_id: ENV["CF_ACCOUNT_ID"],
    token:      ENV["CF_BROWSER_TOKEN"],
  )

  # how long a locator keeps looking, and a `wait_for` waits, before giving up
  config.timeout = 10

  # how long a single navigation is given — longer, because a cold page on a
  # slow network legitimately takes a while and there's nothing to retry against
  config.navigation_timeout = 30
end

Either timeout can be overridden per browser (Unmagic::Browser.new(timeout: 5)), per session (browser.open(url, timeout: 5)), or per call (session.find(".row", wait: 2)).

The Cloudflare driver falls back to CLOUDFLARE_ACCOUNT_ID and, for the token, CLOUDFLARE_BROWSER_RENDERING_TOKEN then CLOUDFLARE_API_TOKEN — so a machine already set up for Cloudflare needs nothing, and a token scoped to just Browser Rendering still wins over the account-wide one. The remote driver falls back to BROWSER_WS_ENDPOINT. In a container you can usually skip the block entirely.

Escape hatch

The curated API stays small on purpose. When you need something it doesn't wrap (network interception, tracing, the full Puppeteer surface), drop down to the object doing the real work. It's the mirror of the driver: option: you pick a driver when you construct the browser, and driver hands the underlying instance back.

browser.driver        # => the driver instance (e.g. Unmagic::Browser::Driver::Cloudflare)

browser.open("https://example.com") do |session|
  puppeteer_page = session.driver     # => the live Puppeteer::Page this session is driving
  puppeteer_page.request_interception = true
  # ...full puppeteer-ruby API...
end

Every driver drives the browser over CDP through puppeteer-ruby, so session.driver is always a Puppeteer::Page. Session operations stay serialized as usual; wrap multi-step native work in session.lock when it must be atomic against other callers.