Class: Spacy::Language

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby-spacy.rb

Overview

See also spaCy Python API document for Language.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(model = "en_core_web_sm", max_retrial: MAX_RETRIAL, timeout: 60) ⇒ Language

Creates a language model instance, which is conventionally referred to by a variable named nlp.

Parameters:

  • model (String) (defaults to: "en_core_web_sm") —

    A language model installed in the system

  • timeout (Numeric, nil) (defaults to: 60) —

    Seconds to wait for the model to load before raising a RuntimeError. nil waits indefinitely. The timeout is enforced on the Python side (a loading thread with join(timeout)) because Ruby's Timeout cannot fire while PyCall holds the GVL. When it fires, the loading thread is left running as a daemon until the process exits (accepted: timeouts are an abnormal path).



625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
 
# File 'lib/ruby-spacy.rb', line 625

def initialize(model = "en_core_web_sm", max_retrial: MAX_RETRIAL, timeout: 60)
  unless model.to_s.match?(/\A[a-zA-Z0-9_\-\.\/]+\z/)
    raise ArgumentError, "Invalid model name: #{model.inspect}"
  end

  retrial = 0
  begin
    @py_nlp = PyHelpers.load_with_timeout(model, timeout)
  rescue StandardError => e
    retrial += 1
    if retrial <= max_retrial
      sleep 0.5
      retry
    else
      raise "Failed to initialize Spacy after #{max_retrial} attempts: #{e.message}"
    end
  end
  # A timeout is not retried; it almost certainly means a hung load
  raise "PyCall execution timed out after #{timeout} seconds" if @py_nlp.nil?
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args) ⇒ Object

Methods defined in Python but not wrapped in ruby-spacy can be called by this dynamic method handling mechanism.



818
819
820
 
# File 'lib/ruby-spacy.rb', line 818

def method_missing(name, *args)
  Spacy.safe_py_send(@py_nlp, name, args)
end

Instance Attribute Details

#py_nlp ⇒ Object (readonly)

Returns a Python Language instance accessible via PyCall.

Returns:

  • (Object) —

    a Python Language instance accessible via PyCall



600
601
602
 
# File 'lib/ruby-spacy.rb', line 600

def py_nlp
  @py_nlp
end

Instance Method Details

#get_lexeme(text) ⇒ Object

A utility method to get a Python Lexeme object.

Parameters:

  • text (String) —

    A text string representing a lexeme

Returns:



686
687
688
 
# File 'lib/ruby-spacy.rb', line 686

def get_lexeme(text)
  @py_nlp.vocab[text]
end

#instance_variables_to_inspect ⇒ Object 



826
827
828
 
# File 'lib/ruby-spacy.rb', line 826

def instance_variables_to_inspect
  [:@spacy_nlp_id]
end

#matcher ⇒ Matcher

Generates a matcher for the current language model.

Returns:



654
655
656
 
# File 'lib/ruby-spacy.rb', line 654

def matcher
  Matcher.new(@py_nlp)
end

#memory_zone { ... } ⇒ Object

Executes a block within spaCy's memory zone for efficient memory management. Requires spaCy >= 3.8.

Yields:

  • the block to execute within the memory zone

Raises:

  • (NotImplementedError) —

    if spaCy version does not support memory zones



808
809
810
811
812
813
814
815
 
# File 'lib/ruby-spacy.rb', line 808

def memory_zone(&block)
  major, minor = SpacyVersion.split(".").map(&:to_i)
  unless major > 3 || (major == 3 && minor >= 8)
    raise NotImplementedError, "memory_zone requires spaCy >= 3.8 (current: #{SpacyVersion})"
  end

  PyCall.with(@py_nlp.memory_zone, &block)
end

#most_similar(vector, num) ⇒ Array<Hash{:key => Integer, :text => String, :best_rows => Array<Float>, :score => Float}>

Returns n lexemes having the vector representations that are the most similar to a given vector representation of a word.

Parameters:

  • vector (Object) —

    A vector representation of a word (whether existing or non-existing)

Returns:

  • (Array<Hash{:key => Integer, :text => String, :best_rows => Array<Float>, :score => Float}>) —

    An array of hash objects each contains the key, text, best_row and similarity score of a lexeme



700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
 
# File 'lib/ruby-spacy.rb', line 700

def most_similar(vector, num)
  vec_array = PyNp.asarray([vector])
  py_result = @py_nlp.vocab.vectors.most_similar(vec_array, n: num)
  key_texts = PyCall::List.call(PyHelpers.key_texts(@py_nlp, py_result[0][0].tolist))
  keys = key_texts.map { |kt| kt[0] }
  texts = key_texts.map { |kt| kt[1] }
  best_rows = PyCall::List.call(py_result[1])[0]
  scores = PyCall::List.call(py_result[2])[0]

  results = []
  num.times do |i|
    result = { key: keys[i].to_i,
               text: texts[i],
               best_row: best_rows[i],
               score: scores[i] }
    result.each_key do |key|
      result.define_singleton_method(key) { result[key] }
    end
    results << result
  end
  results
end

#phrase_matcher(attr: "ORTH") ⇒ PhraseMatcher

Generates a phrase matcher for the current language model. PhraseMatcher is more efficient than Matcher for matching large terminology lists.

Examples:

matcher = nlp.phrase_matcher(attr: "LOWER")
matcher.add("PRODUCT", ["iPhone", "MacBook Pro"])

Parameters:

  • attr (String) (defaults to: "ORTH") —

    the token attribute to match on (default: "ORTH"). Use "LOWER" for case-insensitive matching.

Returns:



666
667
668
 
# File 'lib/ruby-spacy.rb', line 666

def phrase_matcher(attr: "ORTH")
  PhraseMatcher.new(self, attr: attr)
end

#pipe(texts, disable: [], batch_size: 50) ⇒ Array<Doc>

Utility function to batch process many texts

Parameters:

  • texts (String)
  • disable (Array<String>) (defaults to: [])
  • batch_size (Integer) (defaults to: 50)

Returns:



728
729
730
731
732
 
# File 'lib/ruby-spacy.rb', line 728

def pipe(texts, disable: [], batch_size: 50)
  PyCall::List.call(@py_nlp.pipe(texts, disable: disable, batch_size: batch_size)).map do |py_doc|
    Doc.new(@py_nlp, py_doc: py_doc)
  end
end

#pipe_names ⇒ Array<String>

A utility method to list pipeline components.

Returns:

  • (Array<String>) —

    An array of text strings representing pipeline components



679
680
681
 
# File 'lib/ruby-spacy.rb', line 679

def pipe_names
  PyCall::List.call(@py_nlp.pipe_names).to_a
end

#read(text) ⇒ Object

Reads and analyze the given text.

Parameters:

  • text (String) —

    a text to be read and analyzed



648
649
650
 
# File 'lib/ruby-spacy.rb', line 648

def read(text)
  Doc.new(py_nlp, text: text)
end

#respond_to_missing?(sym, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


822
823
824
 
# File 'lib/ruby-spacy.rb', line 822

def respond_to_missing?(sym, include_private = false)
  Spacy.py_hasattr?(@py_nlp, sym) || super
end

#spacy_nlp_id ⇒ String

Deprecated.

The Python object is no longer stored in a global variable at initialization time. Referencing this method creates a global variable in Python's __main__ on demand (which then stays alive until the process exits). Use #py_nlp instead.

Returns an identifier string that can be used to refer to the Python Language object inside PyCall::exec or PyCall::eval.

Returns:

  • (String) —

    an identifier string that can be used to refer to the Python Language object inside PyCall::exec or PyCall::eval



607
608
609
610
611
612
613
614
615
 
# File 'lib/ruby-spacy.rb', line 607

def spacy_nlp_id
  @spacy_nlp_id ||= begin
    warn "[DEPRECATION] `Spacy::Language#spacy_nlp_id` is deprecated. " \
         "It creates a Python global variable that is never released; use `py_nlp` instead."
    id = "nlp_#{@py_nlp.object_id}"
    Builtins.setattr(PyMain, id, @py_nlp)
    id
  end
end

#vocab(text) ⇒ Lexeme

Returns a ruby lexeme object

Parameters:

  • text (String) —

    a text string representing the vocabulary item

Returns:



693
694
695
 
# File 'lib/ruby-spacy.rb', line 693

def vocab(text)
  Lexeme.new(@py_nlp.vocab[text])
end

#vocab_string_lookup(id) ⇒ String

A utility method to lookup the string of the given vocabulary id.

Parameters:

  • id (Integer) —

    a vocabulary id (unsigned 64-bit values are supported)

Returns:

  • (String) —

    the string corresponding to the given vocabulary id



673
674
675
 
# File 'lib/ruby-spacy.rb', line 673

def vocab_string_lookup(id)
  PyHelpers.string_lookup(@py_nlp, Integer(id).to_s)
end

#with_llm(provider: :openai, **opts) {|OpenAIHelper, AnthropicHelper| ... } ⇒ Object

Yields a provider-specific LLM helper for making API calls within a block. The helper is configured once and reused for all calls within the block, making it efficient for batch processing with #pipe.

Providers:

Examples:

Claude

nlp.with_llm(provider: :anthropic) do |ai|
  ai.chat(system: "Analyze.", user: doc.linguistic_summary)
end

Local model via Ollama

nlp.with_llm(provider: :ollama, model: "llama3.2") do |ai|
  ai.chat(user: "Say hello.")
end

Parameters:

  • provider (Symbol) (defaults to: :openai) —

    :openai (default), :anthropic, or :ollama

  • opts (Hash) —

    helper options (access_token:, model:, max_tokens:, temperature:, base_url:, ...) — see OpenAIHelper#initialize and AnthropicHelper#initialize

Yields:

Returns:

  • (Object) —

    the block's return value



760
761
762
763
764
765
766
767
768
769
770
771
772
773
 
# File 'lib/ruby-spacy.rb', line 760

def with_llm(provider: :openai, **opts)
  helper = case provider.to_sym
           when :openai
             OpenAIHelper.new(**opts)
           when :anthropic
             AnthropicHelper.new(**opts)
           when :ollama
             OpenAIHelper.new(**{ base_url: "http://localhost:11434/v1",
                                  access_token: "ollama" }.merge(opts))
           else
             raise ArgumentError, "Unknown LLM provider: #{provider} (use :openai, :anthropic, or :ollama)"
           end
  yield helper
end

#with_openai(access_token: nil, model: OpenAIClient::DEFAULT_MODEL, max_completion_tokens: 1000, temperature: nil, base_url: nil) {|OpenAIHelper| ... } ⇒ Object

Yields an OpenAIHelper instance for making OpenAI API calls within a block. Equivalent to #with_llm with provider: :openai.

Examples:

Batch processing with pipe

nlp.with_openai(model: "gpt-5-mini") do |ai|
  nlp.pipe(texts).map do |doc|
    ai.chat(system: "Analyze.", user: doc.linguistic_summary)
  end
end

Parameters:

  • access_token (String, nil) (defaults to: nil) —

    OpenAI API key (defaults to OPENAI_API_KEY env var)

  • model (String) (defaults to: OpenAIClient::DEFAULT_MODEL) —

    the default model for chat requests

  • max_completion_tokens (Integer) (defaults to: 1000) —

    default maximum tokens in responses

  • temperature (Float, nil) (defaults to: nil) —

    default sampling temperature (omitted from requests when nil)

  • base_url (String, nil) (defaults to: nil) —

    OpenAI-compatible API endpoint override

Yields:

  • (OpenAIHelper) —

    the helper instance for making API calls

Returns:

  • (Object) —

    the block's return value



792
793
794
795
796
797
798
799
800
801
802
 
# File 'lib/ruby-spacy.rb', line 792

def with_openai(access_token: nil, model: OpenAIClient::DEFAULT_MODEL,
                max_completion_tokens: 1000, temperature: nil, base_url: nil)
  helper = OpenAIHelper.new(
    access_token: access_token,
    model: model,
    max_completion_tokens: max_completion_tokens,
    temperature: temperature,
    base_url: base_url
  )
  yield helper
end