Class: Arachni::Support::Database::Queue

Inherits:
Base show all
Defined in:
lib/arachni/support/database/queue.rb

Overview

Flat-file Queue implementation

Behaves pretty much like a Ruby Queue however it transparently serializes and saves its entries to the file-system under the OS's temp directory *after* a specified #max_buffer_size (for in-memory entries) has been exceeded.

It's pretty useful when you want to reduce memory footprint without having to refactor any code since it behaves just like a Ruby Queue implementation.

Author:

  • Tasos “Zapotek” Laskos <tasos.laskos@arachni-scanner.com>

Constant Summary collapse

DEFAULT_MAX_BUFFER_SIZE =

Default #max_buffer_size.

100

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#serialize, #unserialize

Constructor Details

#initialize(*args) ⇒ Queue

Returns a new instance of Queue.

See Also:

  • Database::Base#initialize


41
42
43
44
45
46
47
 
# File 'lib/arachni/support/database/queue.rb', line 41

def initialize( *args )
    super( *args )
    @disk    = []
    @buffer  = []
    @waiting = []
    @mutex   = Mutex.new
end

Instance Attribute Details

#buffer ⇒ Array<Object> (readonly)

Returns Objects stored in the memory buffer.

Returns:

  • (Array<Object>) —

    Objects stored in the memory buffer.



34
35
36
 
# File 'lib/arachni/support/database/queue.rb', line 34

def buffer
  @buffer
end

#disk ⇒ Array<String> (readonly)

Returns Paths to files stored to disk.

Returns:



38
39
40
 
# File 'lib/arachni/support/database/queue.rb', line 38

def disk
  @disk
end

#max_buffer_size ⇒ Integer

Note:

Defaults to DEFAULT_MAX_BUFFER_SIZE.

Returns How many entries to keep in memory before starting to off-load to disk.

Returns:

  • (Integer) —

    How many entries to keep in memory before starting to off-load to disk.



30
31
32
 
# File 'lib/arachni/support/database/queue.rb', line 30

def max_buffer_size
  @max_buffer_size
end

Instance Method Details

#<<(obj) ⇒ Object Also known as: push, enq

Parameters:

  • obj (Object) —

    Object to add to the queue.



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/arachni/support/database/queue.rb', line 59

def <<( obj )
    synchronize do
        if @buffer.size < max_buffer_size
            @buffer << obj
        else
            @disk << dump( obj )
        end

        begin
            t = @waiting.shift
            t.wakeup if t
        rescue ThreadError
            retry
        end
    end
end

#buffer_size ⇒ Object 



108
109
110
 
# File 'lib/arachni/support/database/queue.rb', line 108

def buffer_size
    @buffer.size
end

#clear ⇒ Object

Removes all objects from the queue.



125
126
127
128
129
130
131
132
133
134
135
 
# File 'lib/arachni/support/database/queue.rb', line 125

def clear
    synchronize do
        @buffer.clear

        while !@disk.empty?
            path = @disk.pop
            next if !path
            delete_file path
        end
    end
end

#disk_size ⇒ Object 



112
113
114
 
# File 'lib/arachni/support/database/queue.rb', line 112

def disk_size
    @disk.size
end

#empty? ⇒ Bool

Returns `true` if the queue if empty, `false` otherwise.

Returns:

  • (Bool) —

    `true` if the queue if empty, `false` otherwise.



118
119
120
121
122
 
# File 'lib/arachni/support/database/queue.rb', line 118

def empty?
    synchronize do
        internal_empty?
    end
end

#free_buffer_size ⇒ Object 



104
105
106
 
# File 'lib/arachni/support/database/queue.rb', line 104

def free_buffer_size
    max_buffer_size - buffer_size
end

#num_waiting ⇒ Object 



137
138
139
 
# File 'lib/arachni/support/database/queue.rb', line 137

def num_waiting
    @waiting.size
end

#pop(non_block = false) ⇒ Object Also known as: deq, shift

Returns Removes an object from the queue and returns it.

Returns:

  • (Object) —

    Removes an object from the queue and returns it.



80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/arachni/support/database/queue.rb', line 80

def pop( non_block = false )
    synchronize do
        loop do
            if internal_empty?
                raise ThreadError, 'queue empty' if non_block
                @waiting.push Thread.current
                @mutex.sleep
            else
                return @buffer.shift if !@buffer.empty?
                return load_and_delete_file( @disk.shift )
            end
        end
    end
end

#size ⇒ Integer Also known as: length

Returns Size of the queue, the number of objects it currently holds.

Returns:

  • (Integer) —

    Size of the queue, the number of objects it currently holds.



99
100
101
 
# File 'lib/arachni/support/database/queue.rb', line 99

def size
    buffer_size + disk_size
end