fluentd-json-size-limit

Fluentd filter that limits the byte size of JSON.generate(record) by truncating nested string values. Keys, containers, non-string values, and string encodings are preserved.

Requirements

  • Ruby >= 3.3, < 5
  • Fluentd >= 1.16.11, < 2

Installation

fluent-gem install fluentd-json-size-limit

Configuration

All options with their default values:

<filter **>
  @type jsonsizelimit
  max_size 250k
  enabled true
  overflow_action pass
  error_action pass
  warning_interval 60s
  processing_timeout 5s
  max_record_nodes 1000000
  max_input_size 64m
  max_nesting 100
</filter>
Option Description
max_size Maximum generated JSON size. Must be greater than zero.
enabled Enables measurement and reduction.
overflow_action pass, drop, or raise when immutable record data exceeds the limit.
error_action pass, drop, or raise on invalid input or processing failure.
warning_interval Per-worker duplicate log suppression interval. 0 disables suppression.
processing_timeout Per-record processing deadline.
max_record_nodes Maximum nodes processed during one measurement or reduction phase.
max_input_size Maximum input or generated JSON size. Effective value is at least max_size.
max_nesting Maximum Hash/Array nesting depth.

Behavior

  • The byte limit is exact for Ruby JSON.generate output.
  • Strings are truncated only on valid character and JSON-escape boundaries.
  • Nested arrays, hashes, shared containers, and non-UTF-8 Ruby strings are supported.
  • Invalid byte sequences, cyclic structures, unsupported objects, excessive nesting, oversized inputs, and processing timeouts follow error_action.
  • Records that cannot fit after all strings are emptied follow overflow_action.
  • Record contents are not written to operational logs.
  • No concurrency option is required. The filter is stateless and runs in every Fluentd worker/thread assigned to it.

Metrics

Fluentd metrics are exported under fluentd_json_size_limit:

  • oversized_records
  • reduced_records
  • unreducible_records
  • error_records
  • dropped_records
  • processing_limit_records
  • unsupported_records
  • removed_bytes

Development

bundle install
bundle exec rake
bundle exec rake benchmark
COVERAGE=true bundle exec rake test
bundle exec rake build

Release tags must exactly match the gem version, for example 0.2.0.