pgoutput-parser

A high-performance, Ractor-safe PostgreSQL pgoutput logical replication protocol parser written in pure Ruby.

pgoutput-parser parses PostgreSQL logical replication CopyData payloads into immutable protocol message objects. It focuses on the pgoutput wire format: transaction boundaries, relation metadata, DML message structure, tuple payload markers, and raw tuple bytes.

It intentionally does not convert PostgreSQL values into application-specific Ruby objects. That belongs to a higher-level decoder layer, such as a future pgoutput-decoder gem.

Requirements

  • Ruby 3.4+
  • PostgreSQL 10+

Features

  • Pure Ruby implementation
  • Ruby 3.4+
  • Ractor-safe parsed messages
  • Immutable protocol message objects
  • PostgreSQL logical replication protocol support
  • Relation metadata tracking
  • Binary-safe tuple parsing
  • RBS type signatures included
  • YARD documentation included
  • No runtime dependencies

Why Another pgoutput Library?

This gem focuses exclusively on protocol parsing.

It intentionally separates:

  • Protocol parsing (pgoutput-parser)
  • Type decoding (pgoutput-decoder)
  • Replication transport/client management

This keeps the parser small, predictable, dependency-free, and faithful to PostgreSQL's wire format.

Supported MVP Scope

Supports the core pgoutput row-change replication messages:

  • Begin (B)
  • Relation (R)
  • Insert (I)
  • Update (U)
  • Delete (D)
  • Commit (C)

The currently supported message formats are stable across PostgreSQL 10 through PostgreSQL 18.

TupleData supports all base column markers:

Tuple Value Tag Meaning
n NULL
u Unchanged TOAST value
t Text-formatted raw value
b Binary-formatted raw value

Planned Support

Future releases may add support for:

  • Message (M)
  • Truncate (T)
  • Origin (O)
  • Type (Y)
  • Stream Start (S)
  • Stream Stop (E)
  • Stream Commit (c)
  • Stream Abort (A)
  • Two-Phase Commit messages

What This Gem Does

PostgreSQL CopyData payload
           │
           ▼
    pgoutput-parser
           │
           ▼
Immutable protocol messages

The parser understands:

  • Message tags and binary field sizes
  • Transaction begin metadata
  • Transaction commit metadata
  • Relation metadata
  • Column flags
  • Column names
  • PostgreSQL type OIDs
  • PostgreSQL type modifiers
  • Insert tuples
  • Update old-key tuples
  • Update old full tuples
  • Update new tuples
  • Delete old-key tuples
  • Delete old full tuples
  • Tuple value markers (n, u, t, b)

What This Gem Does Not Do

The parser does not perform application-level type decoding.

It does not convert:

  • UUID
  • JSONB
  • Timestamp
  • Numeric
  • Array
  • Range
  • PostGIS
  • Custom PostgreSQL types

Example:

value.raw
# => "2026-05-31 12:34:56+00"

The raw value is preserved exactly as received.

A higher-level decoder may later interpret it.

Non-goals

This project intentionally does not:

  • Manage replication slots
  • Open replication connections
  • Maintain WAL positions
  • Reconnect to PostgreSQL
  • Decode PostgreSQL types
  • Integrate with ActiveRecord
  • Publish events
  • Build CDC pipelines

Its sole responsibility is parsing pgoutput protocol messages.

Installation

Add this line to your Gemfile:

gem "pgoutput-parser"

Then run:

bundle install

Require the library:

require "pgoutput"

Quick Start

require "pgoutput"

stream = Pgoutput::RelationTracker.new

stream.process(relation_payload)

insert = stream.process(insert_payload)

insert.relation_id
# => 42

insert.tuple.first.raw
# => "7"

insert.tuple.first.oid
# => 23

Binary Tuple Values

When PostgreSQL publishes tuple values using binary format (b), the parser preserves the raw bytes exactly as received.

value.raw
# => "\x00\x00\x00\x07".b

The parser does not interpret binary values.

Update Messages

PostgreSQL Update messages may contain:

  • No old tuple
  • An old key tuple (K)
  • An old full tuple (O)

They always contain a new tuple (N).

update = stream.process(update_payload)

update.old_key_tuple
# => [Pgoutput::Messages::TupleValue, ...] or nil

update.old_tuple
# => [Pgoutput::Messages::TupleValue, ...] or nil

update.new_tuple
# => [Pgoutput::Messages::TupleValue, ...]

Update Tuple Example

update = stream.process(update_payload)

update.old_key_tuple
update.old_tuple
update.new_tuple

Delete Messages

PostgreSQL Delete messages contain either:

  • An old key tuple (K)
  • An old full tuple (O)
delete = stream.process(delete_payload)

delete.old_key_tuple
# => [Pgoutput::Messages::TupleValue, ...] or nil

delete.old_tuple
# => [Pgoutput::Messages::TupleValue, ...] or nil

Relation Metadata Tracking

RelationTracker keeps a local relation cache so tuple values can be associated with PostgreSQL column OIDs defined by preceding Relation (R) messages.

No type conversion is performed.

Only protocol metadata is attached.

stream.process(relation_payload)

message = stream.process(update_payload)

message.new_tuple.map(&:oid)
# => [23, 25, 16]

The relation tracker itself is stateful and maintains relation metadata encountered in the replication stream.

Ractor Safety

message = stream.process(update_payload)

Ractor.shareable?(message)
# => true

Passing parsed messages to a Ractor:

message = stream.process(update_payload)

result = Ractor.new(message) do |update|
  update.new_tuple.map(&:raw)
end.take

Architecture

PostgreSQL
      │
      ▼
CopyData payload
      │
      ▼
Pgoutput::BinaryParser
      │
      ▼
Parsed protocol message
      │
      ▼
Pgoutput::RelationTracker
      │
      ▼
Protocol message with relation metadata
      │
      ▼
Ractor-safe protocol message

Public API

Pgoutput::BinaryParser

Parses a single pgoutput payload without stream state.

message = Pgoutput::BinaryParser.new(payload).parse

Pgoutput::RelationTracker

Parses messages in stream order and remembers relation metadata.

stream = Pgoutput::RelationTracker.new

stream.process(relation_payload)

message = stream.process(insert_payload)

Optional Usage

RelationTracker is optional.

If relation metadata tracking is not required, payloads can be parsed directly:

message =
  Pgoutput::BinaryParser
    .new(payload)
    .parse

RelationTracker Lifecycle

A RelationTracker should be created per logical replication stream.

stream = Pgoutput::RelationTracker.new

The tracker maintains relation metadata discovered during the stream and therefore should not be reused across unrelated replication sessions.

Message Classes

Pgoutput::Messages::Begin
Pgoutput::Messages::Relation
Pgoutput::Messages::Column
Pgoutput::Messages::TupleValue
Pgoutput::Messages::Insert
Pgoutput::Messages::Update
Pgoutput::Messages::Delete
Pgoutput::Messages::Commit

Type Signatures

RBS signatures are included:

sig/pgoutput.rbs

Run Steep:

bundle exec steep check

Testing

Run all tests:

bundle exec rake test

Run with coverage:

COVERAGE=true bundle exec rake test

Development

Generate YARD documentation:

bundle exec yard doc

Ecosystem Direction

This gem is the protocol layer.

pgoutput-parser
      │
      ▼
Protocol messages
      │
      ▼
pgoutput-decoder
      │
      ▼
Application objects

pgoutput-parser should remain small, dependency-free, binary-safe, and faithful to PostgreSQL's wire format.

License

MIT.