Class: Traject::ExperimentalNokogiriStreamingReader::PathTracker

Inherits:
Object
  • Object
show all
Defined in:
lib/traject/experimental_nokogiri_streaming_reader.rb

Overview

initialized with the specification (a very small subset of xpath) for what records to yield-on-each. Tests to see if a Nokogiri::XML::Reader node matches spec.

'//record'

or anchored to root:

'/body/head/meta' same thing as './body/head/meta' or 'head/meta'

Elements can (and must, to match) have XML namespaces, if and only if they are registered with settings nokogiri.namespaces

sadly JRuby Nokogiri has an incompatibility with true nokogiri, and doesn't preserve our namespaces on outer_xml, so in JRuby we have to track them ourselves, and then also do yet ANOTHER parse in nokogiri. This may make this in Java even LESS performant, I'm afraid.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(str_spec, clipboard:, namespaces: {}, extra_xpath_hooks: {}) ⇒ PathTracker

Returns a new instance of PathTracker.



135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 135

def initialize(str_spec, clipboard:, namespaces: {}, extra_xpath_hooks: {})
  @inverted_namespaces  = namespaces.invert
  @clipboard = clipboard
  # We're guessing using a string will be more efficient than an array
  @current_path         = +""
  @floating             = false

  @path_spec, @floating = parse_path(str_spec)

  @namespaces_stack = []


  @extra_xpath_hooks = extra_xpath_hooks.collect do |path, callable|
    bare_path, floating = parse_path(path)
    {
      path: bare_path,
      floating: floating,
      callable: callable
    }
  end
end

Instance Attribute Details

#clipboard ⇒ Object (readonly)

Returns the value of attribute clipboard.



134
135
136
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 134

def clipboard
  @clipboard
end

#current_path ⇒ Object (readonly)

Returns the value of attribute current_path.



134
135
136
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 134

def current_path
  @current_path
end

#extra_xpath_hooks ⇒ Object (readonly)

Returns the value of attribute extra_xpath_hooks.



134
135
136
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 134

def extra_xpath_hooks
  @extra_xpath_hooks
end

#inverted_namespaces ⇒ Object (readonly)

Returns the value of attribute inverted_namespaces.



134
135
136
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 134

def inverted_namespaces
  @inverted_namespaces
end

#namespaces_stack ⇒ Object (readonly)

Returns the value of attribute namespaces_stack.



134
135
136
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 134

def namespaces_stack
  @namespaces_stack
end

#path_spec ⇒ Object (readonly)

Returns the value of attribute path_spec.



134
135
136
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 134

def path_spec
  @path_spec
end

Instance Method Details

#current_node_doc ⇒ Object 



197
198
199
200
201
202
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 197

def current_node_doc
  return nil unless @current_node

  # yeah, sadly we got to have nokogiri parse it again
  fix_namespaces(Nokogiri::XML.parse(@current_node.outer_xml))
end

#fix_namespaces(doc) ⇒ Object

no-op unless it's jruby, and then we use our namespace stack to correctly add namespaces to the Nokogiri::XML::Document, cause in Jruby outer_xml on the Reader doesn't do it for us. :(



243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 243

def fix_namespaces(doc)
  if is_jruby?
    # Only needed in jruby, nokogiri's jruby implementation isn't weird
    # around namespaces in exactly the same way as MRI. We need to keep
    # track of the namespaces in outer contexts ourselves, and then see
    # if they are needed ourselves. :(
    namespaces = namespaces_stack.compact.reduce({}, :merge)
    default_ns = namespaces.delete("xmlns")

    namespaces.each_pair do |attrib, uri|
      ns_prefix = attrib.sub(/\Axmlns:/, '')

      # gotta make sure it's actually used in the doc to not add it
      # unecessarily. GAH.
      if    doc.xpath("//*[starts-with(name(), '#{ns_prefix}:')][1]").empty? &&
            doc.xpath("//@*[starts-with(name(), '#{ns_prefix}:')][1]").empty?
        next
      end
      doc.root.add_namespace_definition(ns_prefix, uri)
    end

    if default_ns
      doc.root.default_namespace = default_ns
      # OMG nokogiri, really?
      default_ns = doc.root.namespace
      doc.xpath("//*[namespace-uri()='']").each do |node|
        node.namespace = default_ns
      end
    end

  end
  return doc
end

#floating? ⇒ Boolean

Returns:

  • (Boolean) 


214
215
216
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 214

def floating?
  !!@floating
end

#is_jruby? ⇒ Boolean

Returns:

  • (Boolean) 


172
173
174
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 172

def is_jruby?
  Traject::Util.is_jruby?
end

#match? ⇒ Boolean

Returns:

  • (Boolean) 


218
219
220
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 218

def match?
  match_path?(path_spec, floating: floating?)
end

#match_path?(path_to_match, floating:) ⇒ Boolean

Returns:

  • (Boolean) 


222
223
224
225
226
227
228
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 222

def match_path?(path_to_match, floating:)
  if floating?
    current_path.end_with?(path_to_match)
  else
    current_path == path_to_match
  end
end

#pop ⇒ Object

removes the last slash-separated component from current_path



205
206
207
208
209
210
211
212
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 205

def pop
  current_path.slice!( current_path.rindex('/')..-1 )
  @current_node = nil

  if is_jruby?
    namespaces_stack.pop
  end
end

#push(reader_node) ⇒ Object

adds a component to slash-separated current_path, with namespace prefix.



177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 177

def push(reader_node)
  namespace_prefix = reader_node.namespace_uri && inverted_namespaces[reader_node.namespace_uri]

  # gah, reader_node.name has the namespace prefix in there
  node_name = reader_node.name.gsub(/[^:]+:/, '')

  node_str = if namespace_prefix
    namespace_prefix + ":" + node_name
  else
    reader_node.name
  end

  current_path << ("/" + node_str)

  if is_jruby?
    namespaces_stack << reader_node.namespaces
  end
  @current_node = reader_node
end

#run_extra_xpath_hooks ⇒ Object 



230
231
232
233
234
235
236
237
238
 
# File 'lib/traject/experimental_nokogiri_streaming_reader.rb', line 230

def run_extra_xpath_hooks
  return unless @current_node

  extra_xpath_hooks.each do |hook_spec|
    if match_path?(hook_spec[:path], floating: hook_spec[:floating])
      hook_spec[:callable].call(current_node_doc, clipboard)
    end
  end
end