Class: Spacy::Language

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby-spacy.rb

Overview

See also spaCy Python API document for Language.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(model = "en_core_web_sm", max_retrial: MAX_RETRIAL, timeout: 60) ⇒ Language

Creates a language model instance, which is conventionally referred to by a variable named nlp.

Parameters:

  • model (String) (defaults to: "en_core_web_sm") —

    A language model installed in the system



513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
 
# File 'lib/ruby-spacy.rb', line 513

def initialize(model = "en_core_web_sm", max_retrial: MAX_RETRIAL, timeout: 60)
  unless model.to_s.match?(/\A[a-zA-Z0-9_\-\.\/]+\z/)
    raise ArgumentError, "Invalid model name: #{model.inspect}"
  end

  @spacy_nlp_id = "nlp_#{model.object_id}"
  retrial = 0
  begin
    Timeout.timeout(timeout) do
      PyCall.exec("import spacy; #{@spacy_nlp_id} = spacy.load('#{model}')")
    end
    @py_nlp = PyCall.eval(@spacy_nlp_id)
  rescue Timeout::Error
    raise "PyCall execution timed out after #{timeout} seconds"
  rescue StandardError => e
    retrial += 1
    if retrial <= max_retrial
      sleep 0.5
      retry
    else
      raise "Failed to initialize Spacy after #{max_retrial} attempts: #{e.message}"
    end
  end
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args) ⇒ Object

Methods defined in Python but not wrapped in ruby-spacy can be called by this dynamic method handling mechanism.



710
711
712
 
# File 'lib/ruby-spacy.rb', line 710

def method_missing(name, *args)
  @py_nlp.send(name, *args)
end

Instance Attribute Details

#py_nlp ⇒ Object (readonly)

Returns a Python Language instance accessible via PyCall.

Returns:

  • (Object) —

    a Python Language instance accessible via PyCall



509
510
511
 
# File 'lib/ruby-spacy.rb', line 509

def py_nlp
  @py_nlp
end

#spacy_nlp_id ⇒ String (readonly)

Returns an identifier string that can be used to refer to the Python Language object inside PyCall::exec or PyCall::eval.

Returns:

  • (String) —

    an identifier string that can be used to refer to the Python Language object inside PyCall::exec or PyCall::eval



506
507
508
 
# File 'lib/ruby-spacy.rb', line 506

def spacy_nlp_id
  @spacy_nlp_id
end

Instance Method Details

#get_lexeme(text) ⇒ Object

A utility method to get a Python Lexeme object.

Parameters:

  • text (String) —

    A text string representing a lexeme

Returns:



578
579
580
 
# File 'lib/ruby-spacy.rb', line 578

def get_lexeme(text)
  @py_nlp.vocab[text]
end

#instance_variables_to_inspect ⇒ Object 



718
719
720
 
# File 'lib/ruby-spacy.rb', line 718

def instance_variables_to_inspect
  [:@spacy_nlp_id]
end

#matcher ⇒ Matcher

Generates a matcher for the current language model.

Returns:



546
547
548
 
# File 'lib/ruby-spacy.rb', line 546

def matcher
  Matcher.new(@py_nlp)
end

#memory_zone { ... } ⇒ Object

Executes a block within spaCy's memory zone for efficient memory management. Requires spaCy >= 3.8.

Yields:

  • the block to execute within the memory zone

Raises:

  • (NotImplementedError) —

    if spaCy version does not support memory zones



700
701
702
703
704
705
706
707
 
# File 'lib/ruby-spacy.rb', line 700

def memory_zone(&block)
  major, minor = SpacyVersion.split(".").map(&:to_i)
  unless major > 3 || (major == 3 && minor >= 8)
    raise NotImplementedError, "memory_zone requires spaCy >= 3.8 (current: #{SpacyVersion})"
  end

  PyCall.with(@py_nlp.memory_zone, &block)
end

#most_similar(vector, num) ⇒ Array<Hash{:key => Integer, :text => String, :best_rows => Array<Float>, :score => Float}>

Returns n lexemes having the vector representations that are the most similar to a given vector representation of a word.

Parameters:

  • vector (Object) —

    A vector representation of a word (whether existing or non-existing)

Returns:

  • (Array<Hash{:key => Integer, :text => String, :best_rows => Array<Float>, :score => Float}>) —

    An array of hash objects each contains the key, text, best_row and similarity score of a lexeme



592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
 
# File 'lib/ruby-spacy.rb', line 592

def most_similar(vector, num)
  vec_array = Numpy.asarray([vector])
  py_result = @py_nlp.vocab.vectors.most_similar(vec_array, n: num)
  key_texts = PyCall.eval("[[str(num), #{@spacy_nlp_id}.vocab[num].text] for num in #{py_result[0][0].tolist}]")
  keys = key_texts.map { |kt| kt[0] }
  texts = key_texts.map { |kt| kt[1] }
  best_rows = PyCall::List.call(py_result[1])[0]
  scores = PyCall::List.call(py_result[2])[0]

  results = []
  num.times do |i|
    result = { key: keys[i].to_i,
               text: texts[i],
               best_row: best_rows[i],
               score: scores[i] }
    result.each_key do |key|
      result.define_singleton_method(key) { result[key] }
    end
    results << result
  end
  results
end

#phrase_matcher(attr: "ORTH") ⇒ PhraseMatcher

Generates a phrase matcher for the current language model. PhraseMatcher is more efficient than Matcher for matching large terminology lists.

Examples:

matcher = nlp.phrase_matcher(attr: "LOWER")
matcher.add("PRODUCT", ["iPhone", "MacBook Pro"])

Parameters:

  • attr (String) (defaults to: "ORTH") —

    the token attribute to match on (default: "ORTH"). Use "LOWER" for case-insensitive matching.

Returns:



558
559
560
 
# File 'lib/ruby-spacy.rb', line 558

def phrase_matcher(attr: "ORTH")
  PhraseMatcher.new(self, attr: attr)
end

#pipe(texts, disable: [], batch_size: 50) ⇒ Array<Doc>

Utility function to batch process many texts

Parameters:

  • texts (String)
  • disable (Array<String>) (defaults to: [])
  • batch_size (Integer) (defaults to: 50)

Returns:



620
621
622
623
624
 
# File 'lib/ruby-spacy.rb', line 620

def pipe(texts, disable: [], batch_size: 50)
  PyCall::List.call(@py_nlp.pipe(texts, disable: disable, batch_size: batch_size)).map do |py_doc|
    Doc.new(@py_nlp, py_doc: py_doc)
  end
end

#pipe_names ⇒ Array<String>

A utility method to list pipeline components.

Returns:

  • (Array<String>) —

    An array of text strings representing pipeline components



571
572
573
 
# File 'lib/ruby-spacy.rb', line 571

def pipe_names
  PyCall::List.call(@py_nlp.pipe_names).to_a
end

#read(text) ⇒ Object

Reads and analyze the given text.

Parameters:

  • text (String) —

    a text to be read and analyzed



540
541
542
 
# File 'lib/ruby-spacy.rb', line 540

def read(text)
  Doc.new(py_nlp, text: text)
end

#respond_to_missing?(sym, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


714
715
716
 
# File 'lib/ruby-spacy.rb', line 714

def respond_to_missing?(sym, include_private = false)
  Spacy.py_hasattr?(@py_nlp, sym) || super
end

#vocab(text) ⇒ Lexeme

Returns a ruby lexeme object

Parameters:

  • text (String) —

    a text string representing the vocabulary item

Returns:



585
586
587
 
# File 'lib/ruby-spacy.rb', line 585

def vocab(text)
  Lexeme.new(@py_nlp.vocab[text])
end

#vocab_string_lookup(id) ⇒ Object

A utility method to lookup a vocabulary item of the given id.

Parameters:

  • id (Integer) —

    a vocabulary id

Returns:



565
566
567
 
# File 'lib/ruby-spacy.rb', line 565

def vocab_string_lookup(id)
  PyCall.eval("#{@spacy_nlp_id}.vocab.strings[#{Integer(id)}]")
end

#with_llm(provider: :openai, **opts) {|OpenAIHelper, AnthropicHelper| ... } ⇒ Object

Yields a provider-specific LLM helper for making API calls within a block. The helper is configured once and reused for all calls within the block, making it efficient for batch processing with #pipe.

Providers:

Examples:

Claude

nlp.with_llm(provider: :anthropic) do |ai|
  ai.chat(system: "Analyze.", user: doc.linguistic_summary)
end

Local model via Ollama

nlp.with_llm(provider: :ollama, model: "llama3.2") do |ai|
  ai.chat(user: "Say hello.")
end

Parameters:

  • provider (Symbol) (defaults to: :openai) —

    :openai (default), :anthropic, or :ollama

  • opts (Hash) —

    helper options (access_token:, model:, max_tokens:, temperature:, base_url:, ...) — see OpenAIHelper#initialize and AnthropicHelper#initialize

Yields:

Returns:

  • (Object) —

    the block's return value



652
653
654
655
656
657
658
659
660
661
662
663
664
665
 
# File 'lib/ruby-spacy.rb', line 652

def with_llm(provider: :openai, **opts)
  helper = case provider.to_sym
           when :openai
             OpenAIHelper.new(**opts)
           when :anthropic
             AnthropicHelper.new(**opts)
           when :ollama
             OpenAIHelper.new(**{ base_url: "http://localhost:11434/v1",
                                  access_token: "ollama" }.merge(opts))
           else
             raise ArgumentError, "Unknown LLM provider: #{provider} (use :openai, :anthropic, or :ollama)"
           end
  yield helper
end

#with_openai(access_token: nil, model: OpenAIClient::DEFAULT_MODEL, max_completion_tokens: 1000, temperature: nil, base_url: nil) {|OpenAIHelper| ... } ⇒ Object

Yields an OpenAIHelper instance for making OpenAI API calls within a block. Equivalent to #with_llm with provider: :openai.

Examples:

Batch processing with pipe

nlp.with_openai(model: "gpt-5-mini") do |ai|
  nlp.pipe(texts).map do |doc|
    ai.chat(system: "Analyze.", user: doc.linguistic_summary)
  end
end

Parameters:

  • access_token (String, nil) (defaults to: nil) —

    OpenAI API key (defaults to OPENAI_API_KEY env var)

  • model (String) (defaults to: OpenAIClient::DEFAULT_MODEL) —

    the default model for chat requests

  • max_completion_tokens (Integer) (defaults to: 1000) —

    default maximum tokens in responses

  • temperature (Float, nil) (defaults to: nil) —

    default sampling temperature (omitted from requests when nil)

  • base_url (String, nil) (defaults to: nil) —

    OpenAI-compatible API endpoint override

Yields:

  • (OpenAIHelper) —

    the helper instance for making API calls

Returns:

  • (Object) —

    the block's return value



684
685
686
687
688
689
690
691
692
693
694
 
# File 'lib/ruby-spacy.rb', line 684

def with_openai(access_token: nil, model: OpenAIClient::DEFAULT_MODEL,
                max_completion_tokens: 1000, temperature: nil, base_url: nil)
  helper = OpenAIHelper.new(
    access_token: access_token,
    model: model,
    max_completion_tokens: max_completion_tokens,
    temperature: temperature,
    base_url: base_url
  )
  yield helper
end