Class: WaterDrop::ConnectionPool

Inherits:
Object
  • Object
show all
Extended by:
Forwardable
Defined in:
lib/waterdrop/connection_pool.rb

Overview

Connection pool wrapper for WaterDrop producers using the proven connection_pool gem.

This provides a clean WaterDrop-specific API while leveraging the battle-tested, connection_pool gem underneath. The wrapper hides the direct usage of the connection_pool gem and provides WaterDrop-specific configuration.

Examples:

Basic usage

pool = WaterDrop::ConnectionPool.new(size: 10) do |config|
  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
  config.deliver = true
end

pool.with do |producer|
  producer.produce_sync(topic: 'events', payload: 'data')
end

Transactional producers with unique IDs

pool = WaterDrop::ConnectionPool.new(size: 5) do |config, index|
  config.kafka = {
    'bootstrap.servers': 'localhost:9092',
    'transactional.id': "my-app-#{index}"
  }
end

Global connection pool

WaterDrop::ConnectionPool.setup(size: 20) do |config|
  config.kafka = { 'bootstrap.servers': ENV['KAFKA_BROKERS'] }
end

WaterDrop::ConnectionPool.with do |producer|
  producer.produce_async(topic: 'events', payload: 'data')
end

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(size: 5, timeout: 5000, &producer_config) {|config, index| ... } ⇒ ConnectionPool

Creates a new WaterDrop connection pool

Parameters:

  • size (Integer) (defaults to: 5) —

    Pool size (default: 5)

  • timeout (Numeric) (defaults to: 5000) —

    Connection timeout in milliseconds (default: 5000)

  • producer_config (Proc) —

    Block to configure each producer in the pool

Yields:

  • (config, index) —

    Block to configure each producer in the pool, receives config and pool index



199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
 
# File 'lib/waterdrop/connection_pool.rb', line 199

def initialize(size: 5, timeout: 5000, &producer_config)
  self.class.send(:ensure_connection_pool_gem!)

  @producer_config = producer_config
  @pool_index = 0
  @pool_mutex = Mutex.new

  @pool = ::ConnectionPool.new(size: size, timeout: timeout / 1000.0) do
    producer_index = @pool_mutex.synchronize { @pool_index += 1 }

    WaterDrop::Producer.new do |config|
      if @producer_config.arity == 2
        @producer_config.call(config, producer_index)
      else
        @producer_config.call(config)
      end
    end
  end

  # Emit event when a connection pool is created
  WaterDrop.instrumentation.instrument(
    "connection_pool.created",
    pool: self,
    size: size,
    timeout: timeout
  )
end

Class Attribute Details

.default_pool ⇒ ConnectionPool?

Returns the global connection pool instance.

Returns:



46
47
48
 
# File 'lib/waterdrop/connection_pool.rb', line 46

def default_pool
  @default_pool
end

Instance Attribute Details

#pool ⇒ ::ConnectionPool (readonly)

Returns the underlying connection_pool instance This allows access to advanced connection_pool features if needed

Returns:

  • (::ConnectionPool) —

    The underlying connection pool



300
301
302
 
# File 'lib/waterdrop/connection_pool.rb', line 300

def pool
  @pool
end

Class Method Details

.active? ⇒ Boolean

Check if the global connection pool is active (configured)

Returns:

  • (Boolean) —

    true if global pool is configured, false otherwise



149
150
151
 
# File 'lib/waterdrop/connection_pool.rb', line 149

def active?
  !@default_pool.nil?
end

.close ⇒ Object

Shutdown the global connection pool Alias for shutdown to align with producer API WaterDrop producers use #close, so we alias connection pool #shutdown to #close for API consistency across both individual producers and connection pools

Parameters:

  • force (Boolean) —

    when true, force-close each producer, purging unflushed messages. Defaults to false (graceful close) so in-flight messages are not silently dropped.



131
132
133
134
135
136
137
138
139
 
# File 'lib/waterdrop/connection_pool.rb', line 131

def shutdown(force: false)
  return unless @default_pool

  # The instrumentation event is emitted by the instance-level #shutdown we delegate to
  # (with `pool: self`, i.e. this same global pool). We deliberately do not emit it again
  # here - doing so fired `connection_pool.shutdown` twice for a single global shutdown.
  @default_pool.shutdown(force: force)
  @default_pool = nil
end

.reload ⇒ Object

Reload the global connection pool



134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/waterdrop/connection_pool.rb', line 134

def reload
  return unless @default_pool

  @default_pool.reload

  # Emit global event for pool reload
  WaterDrop.instrumentation.instrument(
    "connection_pool.reload",
    pool: @default_pool
  )
end

.setup(size: 5, timeout: 5000, &producer_config) {|config, index| ... } ⇒ ConnectionPool

Sets up a global connection pool

Examples:

Basic setup

WaterDrop::ConnectionPool.setup(size: 15) do |config|
  config.kafka = { 'bootstrap.servers': ENV['KAFKA_BROKERS'] }
  config.deliver = true
end

Transactional setup with unique IDs

WaterDrop::ConnectionPool.setup(size: 5) do |config, index|
  config.kafka = {
    'bootstrap.servers': ENV['KAFKA_BROKERS'],
    'transactional.id': "my-app-#{index}"
  }
end

Parameters:

  • size (Integer) (defaults to: 5) —

    Pool size (default: 5)

  • timeout (Numeric) (defaults to: 5000) —

    Connection timeout in milliseconds (default: 5000)

  • producer_config (Proc) —

    Block to configure each producer in the pool

Yields:

  • (config, index) —

    Block to configure each producer in the pool, receives config and pool index

Returns:



70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/waterdrop/connection_pool.rb', line 70

def setup(size: 5, timeout: 5000, &producer_config)
  ensure_connection_pool_gem!

  @default_pool = new(size: size, timeout: timeout, &producer_config)

  # Emit global event for pool setup
  WaterDrop.instrumentation.instrument(
    "connection_pool.setup",
    pool: @default_pool,
    size: size,
    timeout: timeout
  )

  @default_pool
end

.shutdown(force: false) ⇒ Object

Shutdown the global connection pool

Parameters:

  • force (Boolean) (defaults to: false) —

    when true, force-close each producer, purging unflushed messages. Defaults to false (graceful close) so in-flight messages are not silently dropped.



118
119
120
121
122
123
124
125
126
 
# File 'lib/waterdrop/connection_pool.rb', line 118

def shutdown(force: false)
  return unless @default_pool

  # The instrumentation event is emitted by the instance-level #shutdown we delegate to
  # (with `pool: self`, i.e. this same global pool). We deliberately do not emit it again
  # here - doing so fired `connection_pool.shutdown` twice for a single global shutdown.
  @default_pool.shutdown(force: force)
  @default_pool = nil
end

.stats ⇒ Hash?

Get statistics about the global pool

Returns:

  • (Hash, nil) —

    Pool statistics or nil if no global pool



105
106
107
108
109
110
111
112
 
# File 'lib/waterdrop/connection_pool.rb', line 105

def stats
  return nil unless @default_pool

  {
    size: @default_pool.size,
    available: @default_pool.available
  }
end

.transaction {|producer| ... } ⇒ Object

Execute a transaction with a producer from the global connection pool Only available when connection pool is configured

Examples:

WaterDrop::ConnectionPool.transaction do |producer|
  producer.produce(topic: 'events', payload: 'data1')
  producer.produce(topic: 'events', payload: 'data2')
end

Yields:

  • (producer) —

    Producer from the global pool with an active transaction

Returns:

  • (Object) —

    Result of the block

Raises:

  • (RuntimeError) —

    If no global pool is configured



165
166
167
168
169
 
# File 'lib/waterdrop/connection_pool.rb', line 165

def transaction(...)
  raise "No global connection pool configured. Call setup first." unless @default_pool

  @default_pool.transaction(...)
end

.with {|producer| ... } ⇒ Object

Executes a block with a producer from the global pool

Examples:

WaterDrop::ConnectionPool.with do |producer|
  producer.produce_sync(topic: 'events', payload: 'data')
end

Yields:

  • (producer) —

    Producer from the global pool

Returns:

  • (Object) —

    Result of the block

Raises:

  • (RuntimeError) —

    If no global pool is configured



96
97
98
99
100
 
# File 'lib/waterdrop/connection_pool.rb', line 96

def with(...)
  raise "No global connection pool configured. Call setup first." unless @default_pool

  @default_pool.with(...)
end

Instance Method Details

#reload ⇒ Object

Note:

Producers are always closed gracefully (never force-closed): a reload must not drop in-flight messages, so it waits for them to flush rather than purging the queue.

Reload all connections in the pool. Useful for configuration changes or error recovery



266
267
268
269
270
271
272
273
274
275
276
 
# File 'lib/waterdrop/connection_pool.rb', line 266

def reload
  @pool.reload do |producer|
    producer.close if producer&.status&.active?
  end

  # Emit event after pool is reloaded
  WaterDrop.instrumentation.instrument(
    "connection_pool.reloaded",
    pool: self
  )
end

#shutdown(force: false) ⇒ Object Also known as: close

Shutdown the connection pool

Parameters:

  • force (Boolean) (defaults to: false) —

    when true, force-close each producer, purging any messages that do not flush within the producer's max wait timeout. Defaults to false: producers are closed gracefully so in-flight messages are flushed instead of being silently dropped when the broker is slow or unreachable.



243
244
245
246
247
248
249
250
251
252
253
254
255
 
# File 'lib/waterdrop/connection_pool.rb', line 243

def shutdown(force: false)
  @pool.shutdown do |producer|
    next unless producer&.status&.active?

    force ? producer.close! : producer.close
  end

  # Emit event after pool is shut down
  WaterDrop.instrumentation.instrument(
    "connection_pool.shutdown",
    pool: self
  )
end

#stats ⇒ Hash

Get pool statistics

Returns:

  • (Hash) —

    Pool statistics



230
231
232
233
234
235
 
# File 'lib/waterdrop/connection_pool.rb', line 230

def stats
  {
    size: @pool.size,
    available: @pool.available
  }
end

#transaction {|producer| ... } ⇒ Object

Execute a transaction with a producer from this connection pool

Examples:

pool.transaction do |producer|
  producer.produce(topic: 'events', payload: 'data1')
  producer.produce(topic: 'events', payload: 'data2')
end

Yields:

  • (producer) —

    Producer from the pool with an active transaction

Returns:

  • (Object) —

    Result of the block



288
289
290
291
292
293
294
 
# File 'lib/waterdrop/connection_pool.rb', line 288

def transaction
  with do |producer|
    producer.transaction do
      yield(producer)
    end
  end
end