Module: Spacy

Defined in:: lib/ruby-spacy.rb,
lib/ruby-spacy/version.rb,
lib/ruby-spacy/openai_client.rb,
lib/ruby-spacy/openai_helper.rb,
lib/ruby-spacy/llm_client_base.rb,
lib/ruby-spacy/anthropic_client.rb,
lib/ruby-spacy/anthropic_helper.rb

Overview

This module covers the areas of spaCy functionality for using many varieties of its language models, not for building ones.

Defined Under Namespace

Classes: AnthropicClient, AnthropicHelper, Doc, LLMClientBase, Language, Lexeme, Matcher, OpenAIClient, OpenAIHelper, PhraseMatcher, Span, Token

Constant Summary collapse

MAX_RETRIAL =

Builtins =

PyCall.import_module("builtins")

PySpacy = Python spacy module

spacy

PyMain = Python __main__ module (used for the deprecated Language#spacy_nlp_id)

PyCall.import_module("__main__")

PyBase64 = Python base64 module (used for Doc.from_bytes)

PyCall.import_module("base64")

SpacyVersion =

spacy.__version__

PyLanguage = Python Language class

spacy.language.Language

PyDoc = Python Doc class object

spacy.tokens.Doc

PySpan = Python Span class object

spacy.tokens.Span

PyToken = Python Token class object

spacy.tokens.Token

PyMatcher = Python Matcher class object

spacy.matcher.Matcher

PyPhraseMatcher = Python PhraseMatcher class object

spacy.matcher.PhraseMatcher

PyDisplacy = Python displacy object

PyCall.import_module('spacy.displacy')

PyHelpers =

Python-side helpers, kept in a dedicated module so that nothing is added to the user's __main__ namespace. They exist because spaCy's vocabulary keys, matcher ids, and integer attributes (e.g. Token#orth) are unsigned 64-bit integers that do not survive the PyCall boundary in either direction: on the Python -> Ruby side, values from 262 up to 263-1 are silently corrupted into negative numbers and values from 2**63 up come back as nil; on the Ruby -> Python side, passing a large Ruby Integer raises TypeError ("an integer is required"). Moving ids as decimal strings and resolving them in Python sidesteps both directions.

key_texts(nlp, keys): used by Language#most_similar; returns [[key_string, text], ...]

string_lookup(nlp, key_str): used by Language#vocab_string_lookup; returns the string for the given id

matcher_matches(matcher, doc): used by Matcher#match; returns [(match_id_string, start, end, label_string), ...]

attr_or_call(obj, name, args): used by Spacy.safe_py_send; returns the attribute (called with args if given), with Python ints stringified with INT_PREFIX so they survive the PyCall boundary

load_with_timeout(model, seconds): used by Language#initialize; loads a model on a Python thread and returns nil if it does not finish in time (Ruby's Timeout cannot fire while PyCall holds the GVL)

PyCall.import_module("types").ModuleType.new("ruby_spacy_helpers")

INT_PREFIX = Marks integer values that PyHelpers.attr_or_call stringified so they can survive the PyCall boundary (see the comment above PyHelpers)

"__ruby_spacy_int__:"

PyNp = Python numpy module (always present as a spaCy dependency)

PyCall.import_module("numpy")

VERSION = The version number of the module

"0.6.0"

Class Method Summary collapse

.generator_to_array(py_generator) ⇒ Object
A utility module method to convert Python's generator object to a Ruby array, mainly used on the items inside the array returned from dependency-related methods such as Span#rights, Span#lefts and Span#subtree.
.py_hasattr?(py_obj, attr) ⇒ Boolean
Checks if a Python object has a given attribute using builtins.hasattr.
.safe_py_send(py_obj, name, args) ⇒ Object
Calls an attribute or method on a Python object.

Class Method Details

.generator_to_array(py_generator) ⇒ `Object`

A utility module method to convert Python's generator object to a Ruby array, mainly used on the items inside the array returned from dependency-related methods such as Spacy::Span#rights, Spacy::Span#lefts and Spacy::Span#subtree.



150
151
152

# File 'lib/ruby-spacy.rb', line 150

def self.generator_to_array(py_generator)
  PyCall::List.call(py_generator)
end

.py_hasattr?(py_obj, attr) ⇒ `Boolean`

Checks if a Python object has a given attribute using builtins.hasattr. Falls back to true if the check itself fails (e.g. due to PyCall issues).

Parameters:

py_obj (Object) —
a Python object
attr (String, Symbol) —
the attribute name to check

Returns:

(Boolean)

# File 'lib/ruby-spacy.rb', line 159

def self.py_hasattr?(py_obj, attr)
  Builtins.hasattr(py_obj, attr.to_s)
rescue StandardError
  true
end

.safe_py_send(py_obj, name, args) ⇒ `Object`

Calls an attribute or method on a Python object. spaCy exposes several unsigned 64-bit integer attributes (e.g. Token#orth) that are silently corrupted when they cross the PyCall boundary (see the comment above PyHelpers), so when the result looks corrupted (nil, or a negative Integer, which cannot occur legitimately for these ids) the value is fetched again via PyHelpers.attr_or_call, which stringifies ints on the Python side. Ordinary values (strings, booleans, floats, Python objects, small non-negative integers) are returned as-is.

# File 'lib/ruby-spacy.rb', line 136

def self.safe_py_send(py_obj, name, args)
  v = py_obj.send(name, *args)
  return v unless v.nil? || (v.is_a?(Integer) && v.negative?)

  r = PyHelpers.attr_or_call(py_obj, name.to_s, args)
  r.is_a?(String) && r.start_with?(INT_PREFIX) ? r.delete_prefix(INT_PREFIX).to_i : r
end