Class: OpenC3::PythonPackageModel

Inherits:
Object
  • Object
show all
Extended by:
Api
Defined in:
lib/openc3/models/python_package_model.rb

Overview

This class acts like a Model but doesn't inherit from Model because it doesn't actual interact with the Store (Redis). Instead we implement names, get, put and destroy to allow interaction with python package files from the PluginModel and the PackagesController.

Constant Summary collapse

DIST_INFO = 
'.dist-info'
PLUGIN_VENVS_DIR =

Per-plugin isolated venvs created by uvinstall

'/gems/plugin_venvs'
SYSTEM_VENV_DIR =

Core openc3 Python library venv (read-only)

'/openc3/python/.venv'
DEFAULT_UV_CACHE_DIR =

UV wheel cache, seeded from the Docker image at init

'/gems/uv'
UPLOADS_DIR_NAME =

Subdirectory under UV cache for uploaded .whl files

'uploads'

Constants included from Api

Api::DELAY_METRICS, Api::DURATION_METRICS, Api::SUBSCRIPTION_DELIMITER, Api::SUM_METRICS

Constants included from ApiShared

ApiShared::DEFAULT_TLM_POLLING_RATE

Constants included from Extract

Extract::SCANNING_REGULAR_EXPRESSION

Class Method Summary collapse

Methods included from Api

_cal_to_epoch, _cmd_implementation, _extract_target_command_names, _extract_target_command_parameter_names, _extract_target_packet_item_names, _extract_target_packet_names, _get_and_set_cmd, _get_item, _limits_group, _set_tlm_process_args, _tlm_process_args, _validate_tlm_type, build_cmd, cmd, cmd_no_checks, cmd_no_hazardous_check, cmd_no_range_check, cmd_raw, cmd_raw_no_checks, cmd_raw_no_hazardous_check, cmd_raw_no_range_check, commit_timeline_activity, config_tool_names, connect_interface, connect_router, count_timeline_activities, create_timeline, create_timeline_activity, delete_config, delete_limits_set, delete_timeline, delete_timeline_activity, disable_cmd, disable_limits, disable_limits_group, disconnect_interface, disconnect_router, enable_cmd, enable_limits, enable_limits_group, get_all_cmd_names, get_all_cmds, get_all_interface_info, get_all_router_info, get_all_settings, get_all_tlm, get_all_tlm_item_names, get_all_tlm_names, get_cmd, get_cmd_buffer, get_cmd_cnt, get_cmd_cnts, get_cmd_hazardous, get_cmd_time, get_cmd_value, get_interface, get_interface_names, get_item, get_limits, get_limits_events, get_limits_groups, get_limits_set, get_limits_sets, get_metrics, get_out_of_limits, get_overall_limits_state, get_overrides, get_packet_derived_items, get_packets, get_param, get_router, get_router_names, get_setting, get_settings, get_target, get_target_interfaces, get_target_names, get_timeline, get_timeline_activities, get_timeline_activity, get_tlm, get_tlm_available, get_tlm_buffer, get_tlm_cnt, get_tlm_cnts, get_tlm_packet, get_tlm_values, inject_tlm, interface_cmd, interface_details, interface_protocol_cmd, interface_target_disable, interface_target_enable, limits_enabled?, list_configs, list_settings, list_timelines, load_config, map_target_to_interface, map_target_to_router, normalize_tlm, offline_access_needed, override_tlm, router_cmd, router_details, router_protocol_cmd, router_target_disable, router_target_enable, save_config, send_raw, set_limits, set_limits_set, set_offline_access, set_setting, set_state_color, set_timeline_color, set_timeline_execute, set_tlm, start_raw_logging_interface, start_raw_logging_router, stash_all, stash_delete, stash_get, stash_keys, stash_set, stop_raw_logging_interface, stop_raw_logging_router, subscribe_packets, tlm, tlm_formatted, tlm_raw, tlm_with_units, unmap_target_from_interface, unmap_target_from_router, update_news, update_plugin_store, update_timeline_activity

Methods included from CmdLog

#_build_cmd_output_string

Class Method Details

.cached_packages ⇒ Object

List unique packages in the UV download cache. UV cache structure: wheels-v/// e.g. wheels-v6/pypi/numpy/2.4.6-cp312-cp312-musllinux_1_2_aarch64 Also scans the uploads/ subdirectory for user-uploaded .whl files.



137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
 
# File 'lib/openc3/models/python_package_model.rb', line 137

def self.cached_packages
  cache_dir = ENV.fetch('UV_CACHE_DIR', DEFAULT_UV_CACHE_DIR)
  return [] unless File.directory?(cache_dir)

  packages = Set.new
  # Glob 4 levels deep: wheels-v<N>/<registry>/<package-name>/<version-entry>
  Dir.glob("#{cache_dir}/wheels-v*/*/*/*").each do |entry|
    basename = File.basename(entry)
    # Skip UV metadata sidecar files (.http, .msgpack)
    next if basename.end_with?('.http', '.msgpack')

    # Version entries start with a digit
    match = basename.match(/\A(\d[^-]*)/)
    next unless match

    # Parent directory name is the package name
    pkg_name = File.basename(File.dirname(entry)).tr('_', '-').downcase
    packages.add("#{pkg_name}-#{match[1]}")
  end

  # Scan uploaded wheels stored at <cache_dir>/uploads/*.whl
  uploads_dir = File.join(cache_dir, UPLOADS_DIR_NAME)
  if File.directory?(uploads_dir)
    Dir.glob("#{uploads_dir}/*.whl").each do |whl_path|
      parsed = parse_wheel_filename(File.basename(whl_path))
      next unless parsed

      packages.add("#{parsed[0]}-#{parsed[1]}")
    end
  end

  packages.to_a
end

.destroy(name, scope:, plugin: nil) ⇒ Object

Uninstall a Python package. When plugin is provided, the package is removed from that plugin's per-plugin venv; otherwise from the shared venv.



272
273
274
275
276
277
278
279
280
281
282
283
 
# File 'lib/openc3/models/python_package_model.rb', line 272

def self.destroy(name, scope:, plugin: nil)
  package_name, version = self.extract_name_and_version(name)
  Logger.info "Uninstalling package: #{name}"
  pip_args = [package_name]
  spawn_env = {}
  if plugin
    venv_path = "#{PLUGIN_VENVS_DIR}/#{plugin}/.venv"
    spawn_env['PIPINSTALL_VENV'] = venv_path
  end
  result = OpenC3::ProcessManager.instance.spawn(["/openc3/bin/pipuninstall"] + pip_args, "package_uninstall", name, Time.now + 3600.0, scope: scope, env: spawn_env)
  return result.name
end

.extract_name_and_version(name) ⇒ Object 



305
306
307
308
309
310
311
312
313
314
315
316
 
# File 'lib/openc3/models/python_package_model.rb', line 305

def self.extract_name_and_version(name)
  split_name = name.split('-')
  if split_name.length > 1
    package_name = split_name[0..-2].join('-')
    version = File.basename(split_name[-1], DIST_INFO)
  else
    package_name = name
    version = "Unknown"
  end

  return package_name, version
end

.get(name) ⇒ Object 



190
191
192
193
194
195
196
197
198
 
# File 'lib/openc3/models/python_package_model.rb', line 190

def self.get(name)
  path = "#{ENV['PYTHONUSERBASE']}/cache"
  FileUtils.mkdir_p(path) unless Dir.exist?(path)
  result = Pathname.new(path).children.select { |c| c.file? and File.basename(c, File.extname(c)) == name }
  if result.length > 0
    return result[0] if File.exist?(result[0])
  end
  raise "Package '#{name}' not found"
end

.install(name_or_path, scope:, plugin: nil) ⇒ Object

Install a Python package via pipinstall. When plugin is provided, the package is installed into that plugin's per-plugin venv instead of the shared PYTHONUSERBASE. This is used by the Admin Packages tab when a user selects a specific plugin venv as the install target.



230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
 
# File 'lib/openc3/models/python_package_model.rb', line 230

def self.install(name_or_path, scope:, plugin: nil)
  if File.exist?(name_or_path)
    package_file_path = name_or_path
  else
    package_file_path = get(name_or_path)
  end
  package_filename = File.basename(package_file_path)
  begin
    pypi_url = get_setting('pypi_url', scope: scope)
    if pypi_url
      pypi_url += '/simple'
    end
  rescue => e
    Logger.error("Failed to retrieve pypi_url: #{e.formatted}")
  ensure
    if pypi_url.nil?
      # If Redis isn't running try the ENV, then simply pypi.org/simple
      pypi_url = ENV['PYPI_URL']
      if pypi_url
        pypi_url += '/simple'
      end
      pypi_url ||= PypiUrl::DEFAULT
    end
  end
  pypi_url = PypiUrl.validate(pypi_url)
  Logger.info "Installing python package: #{name_or_path}"
  if ENV['PIP_ENABLE_TRUSTED_HOST'].nil?
    pip_args = ["-i", pypi_url, package_file_path]
  else
    pip_args = ["-i", pypi_url, "--trusted-host", URI.parse(pypi_url).host, package_file_path]
  end
  spawn_env = {}
  if plugin
    venv_path = "#{PLUGIN_VENVS_DIR}/#{plugin}/.venv"
    spawn_env['PIPINSTALL_VENV'] = venv_path
  end
  result = OpenC3::ProcessManager.instance.spawn(["/openc3/bin/pipinstall"] + pip_args, "package_install", package_filename, Time.now + 3600.0, scope: scope, env: spawn_env)
  return result.name
end

.names ⇒ Object 



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
 
# File 'lib/openc3/models/python_package_model.rb', line 75

def self.names
  result = {}

  # Collect all packages available in the UV download cache
  # This includes system packages (seeded from the Docker image) plus
  # any additional packages downloaded during plugin installs
  cached = cached_packages
  result['cached'] = cached.sort unless cached.empty?

  # Collect packages from per-plugin venvs
  if File.directory?(PLUGIN_VENVS_DIR)
    Dir.glob("#{PLUGIN_VENVS_DIR}/*/").each do |plugin_dir|
      plugin_name = File.basename(plugin_dir)
      venv_dir = File.join(plugin_dir, '.venv')
      next unless File.directory?(venv_dir)

      # Always include plugin venvs even if empty so they remain visible
      # in the Admin UI and selectable as install targets
      packages = packages_in_venv(venv_dir)
      result[plugin_name] = packages.sort
    end
  end

  # Also collect packages from the shared venv for backwards compatibility
  shared_packages = shared_venv_packages
  result['shared'] = shared_packages.sort unless shared_packages.empty?

  return result
end

.normalize_pkg_name(name) ⇒ Object

Normalize a package name to lowercase with hyphens (PEP 503 canonical form)



38
39
40
 
# File 'lib/openc3/models/python_package_model.rb', line 38

def self.normalize_pkg_name(name)
  name.tr('_', '-').downcase
end

.packages_in_venv(venv_dir) ⇒ Object

List packages in a specific venv by scanning dist-info directories. Returns normalized names (lowercase, hyphens) for consistent display.



107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
 
# File 'lib/openc3/models/python_package_model.rb', line 107

def self.packages_in_venv(venv_dir)
  packages = []
  # Look for site-packages in the venv's lib directory
  Dir.glob("#{venv_dir}/lib/*/site-packages").each do |site_packages|
    next unless File.directory?(site_packages)
    Pathname.new(site_packages).children.each do |child|
      if child.directory? && File.extname(child) == DIST_INFO
        raw_name = File.basename(child, DIST_INFO)
        # Normalize: split name from version, normalize name, rejoin
        match = raw_name.match(/\A(.+?)-(\d.*)/)
        if match
          packages << "#{normalize_pkg_name(match[1])}-#{match[2]}"
        else
          packages << normalize_pkg_name(raw_name)
        end
      end
    end
  end
  packages
end

.parse_wheel_filename(filename) ⇒ Object

Parse a PEP 427 wheel filename into [normalized_name, version]. Handles browser-appended duplicate suffixes like "(1)" on the stem. Returns nil for non-wheel files or malformed names. Example: "numpy-2.4.6-cp312-cp312-musllinux_1_2_aarch64.whl" => ["numpy", "2.4.6"]



46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/openc3/models/python_package_model.rb', line 46

def self.parse_wheel_filename(filename)
  # Strip browser duplicate suffixes like " (1)" before the .whl extension
  # Two steps to avoid any space quantifier in the regex:
  # 1. Remove "(N).whl" at end, replacing with just ".whl"
  # 2. Clean up leftover spaces before .whl with rstrip
  clean = filename.sub(/\(\d+\)\.whl\z/i, '.whl')
  clean = "#{clean.chomp('.whl').rstrip}.whl" if clean.end_with?('.whl')
  return nil unless clean.end_with?('.whl')

  # PEP 427: {name}-{version}(-{build})?-{python}-{abi}-{platform}.whl
  # We need at least name-version-python-abi-platform (5 segments)
  stem = clean.chomp('.whl')
  parts = stem.split('-')
  return nil if parts.length < 5

  # The version is always the second segment; everything before it is the name
  # (some packages have hyphens in their name that become underscores in the wheel)
  # PEP 427 guarantees: last 3 segments are python-abi-platform,
  # optional build tag before those, then version, then name (may have multiple segments)
  # Simplest reliable approach: version is always at index 1
  name = normalize_pkg_name(parts[0])
  version = parts[1]

  # Sanity check: version should start with a digit
  return nil unless version.match?(/\A\d/)

  [name, version]
end

.put(package_file_path, package_install: true, scope:, plugin: nil) ⇒ Object 



200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
 
# File 'lib/openc3/models/python_package_model.rb', line 200

def self.put(package_file_path, package_install: true, scope:, plugin: nil)
  if File.file?(package_file_path)
    package_filename = File.basename(package_file_path)
    FileUtils.mkdir_p("#{ENV['PYTHONUSERBASE']}/cache") unless Dir.exist?("#{ENV['PYTHONUSERBASE']}/cache")
    cache_path = "#{ENV['PYTHONUSERBASE']}/cache/#{File.basename(package_file_path)}"
    FileUtils.cp(package_file_path, cache_path)

    # Copy uploaded .whl files to the UV uploads directory so they appear
    # in the "Cached" section of the Admin Packages UI
    if package_filename.end_with?('.whl')
      uploads_dir = File.join(ENV.fetch('UV_CACHE_DIR', DEFAULT_UV_CACHE_DIR), UPLOADS_DIR_NAME)
      FileUtils.mkdir_p(uploads_dir)
      FileUtils.cp(package_file_path, File.join(uploads_dir, package_filename))
    end

    if package_install
      return self.install(cache_path, scope: scope, plugin: plugin)
    end
  else
    message = "Package file #{package_file_path} does not exist!"
    Logger.error message
    raise message
  end
  return nil
end

.shared_venv_packages ⇒ Object

List packages in the shared venv (backwards compatibility)



172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
 
# File 'lib/openc3/models/python_package_model.rb', line 172

def self.shared_venv_packages
  packages = []
  pythonuserbase = ENV.fetch('PYTHONUSERBASE', nil)
  return packages unless pythonuserbase

  paths = Dir.glob("#{pythonuserbase}/lib/*")
  paths.each do |path|
    site_packages = File.join(path, 'site-packages')
    next unless File.directory?(site_packages)
    Pathname.new(site_packages).children.each do |child|
      if child.directory? && File.extname(child) == DIST_INFO
        packages << File.basename(child, DIST_INFO)
      end
    end
  end
  packages
end

.system_venv_packages ⇒ Object

List packages in the system venv (/openc3/python/.venv)



129
130
131
 
# File 'lib/openc3/models/python_package_model.rb', line 129

def self.system_venv_packages
  packages_in_venv(SYSTEM_VENV_DIR)
end

.trees ⇒ Object

Returns a hash of plugin_name => "uv pip list" text output for each plugin venv.



286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
 
# File 'lib/openc3/models/python_package_model.rb', line 286

def self.trees
  result = {}

  if File.directory?(PLUGIN_VENVS_DIR)
    Dir.glob("#{PLUGIN_VENVS_DIR}/*/").each do |plugin_dir|
      plugin_name = File.basename(plugin_dir)
      venv_dir = File.join(plugin_dir, '.venv')
      next unless File.directory?(venv_dir)

      stdout, status = Open3.capture2('uv', 'pip', 'list', '--python', venv_dir)
      if status.success? && stdout.lines.length > 2
        result[plugin_name] = stdout.rstrip
      end
    end
  end

  result
end