🪙 Token::Resolver

if ci_badges.map(&:color).detect { it != "green"} ☝️ let me know, as I may have missed the discord notification.

if ci_badges.map(&:color).all? { it == "green"} 👇️ send money so I can do more of this. FLOSS maintenance is now my full-time job.

👣 How will this project approach the September 2025 hostile takeover of RubyGems? 🚑️

I've summarized my thoughts in [this blog post](https://dev.to/galtzo/hostile-takeover-of-rubygems-my-thoughts-5hlo).

🌻 Synopsis

Token::Resolver is a configurable PEG-based token parser and resolver for structured token detection and replacement in arbitrary text.

Detects structured tokens like {KJ|GEM_NAME} in any file format and resolves them against a replacement map. The token structure (delimiters, separators, segment count) is fully configurable.

# One-liner: parse and resolve
result = Token::Resolver.resolve(
  "Hello {KJ|NAME}, welcome to {KJ|PROJECT}!",
  {"KJ|NAME" => "World", "KJ|PROJECT" => "my-app"},
)
# => "Hello World, welcome to my-app!"

💡 Info you can shake a stick at

Tokens to Remember
Works with JRuby
Works with Truffle Ruby
Works with MRI Ruby 4
Works with MRI Ruby 3
Support & Community
Source
Documentation
Compliance
Style
Maintainer 🎖️
`...` 💖	🧊 🐙 🛖 🧪

Compatibility

Compatible with MRI Ruby 3.2.0+, and concordant releases of JRuby, and TruffleRuby. CI workflows and Appraisals are generated for MRI Ruby 3.2.0+. This test floor is configured by ruby.test_minimum in .kettle-jem.yml and may be higher than the gem's runtime compatibility floor when legacy Rubies are not practical for the current toolchain.

🚚 Amazing test matrix was brought to you by	🔎 appraisal2 🔎 and the color 💚 green 💚
👟 Check it out!	✨ github.com/appraisal-rb/appraisal2 ✨

Federated DVCS

Find this repo on federated forges (Coming soon!)

| Federated [DVCS][💎d-in-dvcs] Repository | Status | Issues | PRs | Wiki | CI | Discussions | |-------------------------------------------------|-----------------------------------------------------------------------|---------------------------|--------------------------|---------------------------|--------------------------|------------------------------| | 🧪 [kettle-rb/token-resolver on GitLab][📜src-gl] | The Truth | [💚][🤝gl-issues] | [💚][🤝gl-pulls] | [💚][📜gl-wiki] | 🐭 Tiny Matrix | ➖ | | 🧊 [kettle-rb/token-resolver on CodeBerg][📜src-cb] | An Ethical Mirror ([Donate][🤝cb-donate]) | [💚][🤝cb-issues] | [💚][🤝cb-pulls] | ➖ | ⭕️ No Matrix | ➖ | | 🐙 [kettle-rb/token-resolver on GitHub][📜src-gh] | Another Mirror | [💚][🤝gh-issues] | [💚][🤝gh-pulls] | [💚][📜gh-wiki] | 💯 Full Matrix | [💚][gh-discussions] | | 🎮️ [Discord Server][✉️discord-invite] | [![Live Chat on Discord][✉️discord-invite-img-ftb]][✉️discord-invite] | [Let's][✉️discord-invite] | [talk][✉️discord-invite] | [about][✉️discord-invite] | [this][✉️discord-invite] | [library!][✉️discord-invite] |

Enterprise Support

Available as part of the Tidelift Subscription.

Need enterprise-level guarantees?

The maintainers of this and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source packages you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact packages you use. [![Get help from me on Tidelift][🏙️entsup-tidelift-img]][🏙️entsup-tidelift] - 💡Subscribe for support guarantees covering _all_ your FLOSS dependencies - 💡Tidelift is part of [Sonar][🏙️entsup-tidelift-sonar] - 💡Tidelift pays maintainers to maintain the software you depend on!
📊`@`Pointy Haired Boss: An [enterprise support][🏙️entsup-tidelift] subscription is "[never gonna let you down][🧮kloc]", and *supports* open source maintainers Alternatively: - [![Live Chat on Discord][✉️discord-invite-img-ftb]][✉️discord-invite] - [![Get help from me on Upwork][👨🏼‍🏫expsup-upwork-img]][👨🏼‍🏫expsup-upwork] - [![Get help from me on Codementor][👨🏼‍🏫expsup-codementor-img]][👨🏼‍🏫expsup-codementor]

✨ Installation

Install the gem and add to the application's Gemfile by executing:

bundle add token-resolver

If bundler is not being used to manage dependencies, install the gem by executing:

gem install token-resolver

⚙️ Configuration

Token Config Options

Option	Default	Description
`pre`	`"{"`	Opening delimiter
`post`	`"}"`	Closing delimiter
`separators`	`["\|"]` (pipe)	Segment separators (sequential; last repeats)
`min_segments`	`2`	Minimum segments for a valid token
`max_segments`	`nil`	Maximum segments (`nil` = unlimited)
`segment_pattern`	`"[A-Za-z0-9_]"`	Parslet character class for valid segment content

Segment Character Constraints

Token segments (the parts between delimiters and separators) only match characters that conform to the segment_pattern. By default, this is word characters: uppercase and lowercase letters, digits, and underscores.

This prevents false positives with syntax that structurally resembles tokens but isn't:

# These are NOT parsed as tokens (spaces, punctuation disqualify them):
"items.map { |x| x.to_s }"                 # Ruby block parameters
"${CLASSPATH:+:$CLASSPATH}"                 # Shell variable expansion
"cert_chain.select! { |fp| File.exist? }"  # Ruby block with expressions

If you need different characters in your token segments, provide a custom pattern:

# Allow hyphens in segments: {NS|my-key}
config = Token::Resolver::Config.new(segment_pattern: "[A-Za-z0-9_-]")

🔧 Basic Usage

Basic Token Resolution

require "token/resolver"

# Parse a document to inspect tokens
doc = Token::Resolver.parse("Deploy {KJ|GEM_NAME} to {KJ|GH_ORG}")
doc.token_keys  # => ["KJ|GEM_NAME", "KJ|GH_ORG"]
doc.text_only?  # => false

# Resolve tokens
result = Token::Resolver.resolve(
  "Deploy {KJ|GEM_NAME} to {KJ|GH_ORG}",
  {"KJ|GEM_NAME" => "my-gem", "KJ|GH_ORG" => "my-org"},
)
# => "Deploy my-gem to my-org"

Handling Missing Tokens

# Default: raise on unresolved tokens
Token::Resolver.resolve("{KJ|MISSING}", {})
# => raises Token::Resolver::UnresolvedTokenError

# Keep unresolved tokens as-is
Token::Resolver.resolve("{KJ|MISSING}", {}, on_missing: :keep)
# => "{KJ|MISSING}"

# Remove unresolved tokens
Token::Resolver.resolve("{KJ|MISSING}", {}, on_missing: :remove)
# => ""

Custom Token Structure

# Tokens like <<SECTION:NAME>>
config = Token::Resolver::Config.new(
  pre: "<<",
  post: ">>",
  separators: [":"],
)
Token::Resolver.resolve("Hello <<NS:NAME>>!", {"NS:NAME" => "World"}, config: config)
# => "Hello World!"

Multi-Segment Tokens with Sequential Separators

# Tokens like {KJ|SECTION:SUBSECTION}
config = Token::Resolver::Config.new(
  separators: ["|", ":"],  # First boundary uses |, second uses :, rest repeat :
)
doc = Token::Resolver.parse("{KJ|META:AUTHOR}", config: config)
doc.tokens.first.key       # => "KJ|META:AUTHOR"
doc.tokens.first.prefix    # => "KJ"
doc.tokens.first.segments  # => ["KJ", "META", "AUTHOR"]

Step-by-Step API

# Parse
doc = Token::Resolver::Document.new("Hello {KJ|NAME}!")

# Inspect
doc.nodes       # => [Text("Hello "), Token(["KJ", "NAME"]), Text("!")]
doc.tokens      # => [Token(["KJ", "NAME"])]
doc.token_keys  # => ["KJ|NAME"]
doc.to_s        # => "Hello {KJ|NAME}!"  (roundtrip fidelity)

# Resolve
resolver = Token::Resolver::Resolve.new(on_missing: :raise)
result = resolver.resolve(doc, {"KJ|NAME" => "World"})
# => "Hello World!"

🔐 Security

See SECURITY.md.

🤝 Contributing

If you need some ideas of where to help, you could work on adding more code coverage, or if it is already 💯 (see below) check issues or PRs, or use the gem and think about how it could be better.

We so if you make changes, remember to update it.

See CONTRIBUTING.md for more detailed instructions.

🚀 Release Instructions

See CONTRIBUTING.md.

Code Coverage

Coverage service badges

[![Coverage Graph][🏀codecov-g]][🏀codecov] [![Coveralls Test Coverage][🏀coveralls-img]][🏀coveralls] [![QLTY Test Coverage][🏀qlty-covi]][🏀qlty-cov]

🪇 Code of Conduct

Everyone interacting with this project's codebases, issue trackers, chat rooms and mailing lists agrees to follow the .

🌈 Contributors

Made with contributors-img.

Also see GitLab Contributors: https://gitlab.com/kettle-rb/token-resolver/-/graphs/main

⭐️ Star History

📌 Versioning

This library follows for its public API where practical. For most applications, prefer the Pessimistic Version Constraint with two digits of precision.

For example:

spec.add_dependency("token-resolver", "~> 2.0")

📌 Is "Platform Support" part of the public API? More details inside.

Dropping support for a platform can be a breaking change for affected users. If a release changes supported platforms, it should be called out clearly in the changelog and versioned with that impact in mind. To get a better understanding of how SemVer is intended to work over a project's lifetime, read this article from the creator of SemVer: - ["Major Version Numbers are Not Sacred"][📌major-versions-not-sacred]

See CHANGELOG.md for a list of releases.

📄 License

The gem is available under the following licenses: AGPL-3.0-only, PolyForm-Small-Business-1.0.0. See LICENSE.md for details.

If none of the available licenses suit your use case, please contact us to discuss a custom commercial license.

© Copyright

See LICENSE.md for the official copyright notice.

🤑 A request for help

Maintainers have teeth and need to pay their dentists. After getting laid off in an RIF in March, and encountering difficulty finding a new one, I began spending most of my time building open source tools. I'm hoping to be able to pay for my kids' health insurance this month, so if you value the work I am doing, I need your support. Please consider sponsoring me or the project.

To join the community or get help 👇️ Join the Discord.

To say "thanks!" ☝️ Join the Discord or 👇️ send money.

💌 💌 💌

Please give the project a star ⭐ ♥.

Thanks for RTFM. ☺️

Field	Value
Package	token-resolver
Description	🪙 Token::Resolver provides configurable PEG-based (parslet) parsing and resolution of structured tokens (e.g., {KJ\
Homepage	https://github.com/kettle-rb/token-resolver
Source	https://github.com/kettle-rb/token-resolver/tree/v2.0.0
License	`AGPL-3.0-only` OR `PolyForm-Small-Business-1.0.0`
Funding	https://github.com/sponsors/pboling, https://issuehunt.io/u/pboling, https://ko-fi.com/pboling, https://liberapay.com/pboling/donate, https://opencollective.com/kettle-rb, https://patreon.com/galtzo, https://polar.sh/pboling, https://thanks.dev/u/gh/pboling, https://tidelift.com/funding/github/rubygems/token-resolver, https://www.buymeacoffee.com/pboling