Class: Labimotion::AiTemplate

Inherits:
Object
  • Object
show all
Defined in:
lib/labimotion/libs/ai_template.rb

Overview

AiTemplate generates a generic dataset template (DatasetKlass properties template) from a natural-language description using an LLM.

The client targets the KIT KI-Toolbox (https://ki-toolbox.scc.kit.edu), an OpenAI-compatible chat-completions gateway (Open WebUI), but is fully config-driven (config/labimotion_ai.yml, then ENV) so it can point at any compatible service. Settings resolve yml -> ENV -> default:

:api_key  / KI_TOOLBOX_API_KEY    (required) bearer token (created in the UI)
:base_url / KI_TOOLBOX_BASE_URL   default https://ki-toolbox.scc.kit.edu
:api_path / KI_TOOLBOX_API_PATH   default /api/v1/chat/completions
:model    / KI_TOOLBOX_MODEL      default kit.mistral-small-4-119b-a8b
:max_tokens, :timeout, :system_prompt

The LLM is asked to return a JSON document describing the template layers and fields in the LabIMotion format. Only the data part is returned here; wrapping it into a persistable DatasetKlass (uuid, pkg, klass, ...) is the caller's job (see Labimotion::DatasetHelpers#create_ai_dataset_klass).

Constant Summary collapse

DEFAULT_BASE_URL = 
'https://ki-toolbox.scc.kit.edu'
DEFAULT_API_PATH = 
'/api/v1/chat/completions'
DEFAULT_MODEL = 
'kit.mistral-small-4-119b-a8b'
DEFAULT_MAX_TOKENS = 
8000
PING_MAX_TOKENS =

A connection test only needs a valid round-trip, not a useful completion, so cap the reply hard to keep the ping cheap and fast.

8
REQUEST_TIMEOUT = 
120
MAX_FILES = 
10
MAX_FILE_CHARS =

Slightly above FileExtractor::MAX_OUT_CHARS (8000) so its "[truncated]" marker is not chopped off by the slice below.

9000
MAX_INSTRUCTION_CHARS =

Cap on a template's own auto-fill guidance. Generous for the paragraph or two this is meant to hold, small enough that it cannot crowd out the document.

4000
MAX_ONTOLOGY_TERMS =

Ontology vocabularies are sent verbatim; past this many terms in ONE request the remaining ontology fields go unconstrained rather than starve the document of context. Sized well above a realistic template (Viability: ~470 terms).

Labimotion::OntologyTerms::MAX_TERMS_TOTAL
FIELD_TYPES =

Field "type" values allowed in generated templates — a practical subset of the LabIMotion field-base schema enum.

%w[
  text textarea integer number checkbox date datetime
  select select-multi ontology-select system-defined label upload table
].freeze
SELECT_TYPES =

Field types whose choices live in the shared select_options map.

%w[select select-multi].freeze
UNIT_CANDIDATE_TYPES =

Field types that may carry a physical unit (mapped to a system-defined group).

%w[number integer system-defined].freeze
TABLE_COL_TYPES =

Column types allowed inside a "table" field's sub_fields; others -> text.

%w[text number integer checkbox date datetime].freeze
DURATION_UNITS =

The duration units a "datetime-range" field can carry, as chem-generic-ui's DateTimeRange renders them. The model picks one alongside its number, so "three days" cannot land in a field that then reads it as three hours.

%w[d h min s].freeze
MAX_TABLE_ROWS =

Rows past this are dropped: a table is a summary of a document, and a model that starts emitting hundreds of rows has lost the thread.

50
MAX_FIELD_DESC_CHARS =

Per-field cap on the designer's description. Generous enough to carry a full ontology definition intact (the longest in the Viability template is 771), while still bounding a template that puts an essay on every field.

1000
PLAN_MAX_TOKENS =

A structural plan is a handful of operations, never a template. Capping the reply this hard is the whole point of the path: it bounds the cheap route to a rounding error, and a model that starts echoing a template instead of planning hits the ceiling and is escalated rather than billed for 16k tokens.

1200
PLAN_BUILDERS =

The closed vocabulary of STRUCTURAL operations the designer already applies itself (chem-generic-ui: action-handler / group-handler / sorting-handler), each mapped to the builder that validates it. Anything outside this table is design work and goes back through refine. A table rather than a case: the vocabulary is data, and the client dispatches on exactly these strings.

{
  'delete_layer' => :delete_layer_op,
  'ungroup_layer' => :ungroup_layer_op,
  'update_layer' => :update_layer_op,
  'delete_field' => :delete_field_op,
  'set_field' => :field_update_op,
  'reorder_layers' => :reorder_layers_op,
  'group_layers' => :group_layers_op
}.freeze
PLAN_LAYER_ATTRS =

Layer attributes update_layer may change.

%w[label cols color].freeze
PLAN_FIELD_ATTRS =

Field attributes set_field may change. Deliberately EXCLUDES "type": a type change cascades into units, numeric config, restrictions and display-name references, and the designer's own handler owns that logic — routing it through here would reimplement it worse.

%w[label description placeholder required readonly cols hasOwnRow].freeze
PLAN_FIELD_FLAGS =

Field attributes above that are flags, not text.

%w[required readonly hasOwnRow].freeze
MAX_PLAN_OPERATIONS =

A plan longer than this is not a structural edit any more.

25
MAX_PROVIDER_ERROR_CHARS =

How much of a provider's error message is shown to the user.

300

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(kind: 'dataset', ols_term_id: nil, subject: nil, desc: nil, cols: 1, references: [], files: [], history: [], model: nil, api_key: nil, base_url: nil, api_path: nil) ⇒ AiTemplate

Returns a new instance of AiTemplate.



271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
 
# File 'lib/labimotion/libs/ai_template.rb', line 271

def initialize(kind: 'dataset', ols_term_id: nil, subject: nil, desc: nil, cols: 1, references: [], files: [], history: [], model: nil, api_key: nil, base_url: nil, api_path: nil)
  @kind = kind.to_s
  @subject = subject.to_s.strip
  @ols_term_id = ols_term_id.to_s.strip
  @desc = desc.to_s.strip
  @cols = cols.to_i.clamp(1, 6)
  @references = Array(references).map(&:to_s).reject(&:blank?)
  @files = Array(files)
  @history = Array(history)
  # Per-user overrides (from the caller); blank -> fall back to yml/ENV/default.
  @model_override = model.to_s.strip
  @api_key_override = api_key.to_s.strip
  # A user may only redirect to their OWN provider when they also bring their
  # own key — the shared server key must never be sent to a user-supplied URL.
  @base_url_override = base_url.to_s.strip
  @api_path_override = api_path.to_s.strip
end

Class Method Details

.build_unit_index ⇒ Object 



229
230
231
232
233
234
235
236
 
# File 'lib/labimotion/libs/ai_template.rb', line 229

def self.build_unit_index
  collect_unit_tokens.each_with_object({}) do |(token, groups), index|
    next unless groups.size == 1

    gkey, ukey = groups.first
    index[token] = { group: gkey, value_system: ukey }
  end
end

.collect_unit_tokens ⇒ Object

token -> { group => first_unit_key }. A token seen under >1 group signals it is ambiguous (kept here, filtered out in build_unit_index).



240
241
242
243
244
245
246
247
 
# File 'lib/labimotion/libs/ai_template.rb', line 240

def self.collect_unit_tokens
  tokens = {}
  Labimotion::Units::FIELDS.each do |group|
    gkey = group[:field].to_s
    Array(group[:units]).each { |unit| record_unit_tokens(tokens, gkey, unit) }
  end
  tokens
end

.fill(properties:, context_text:, instructions: nil, model: nil, api_key: nil, base_url: nil, api_path: nil) ⇒ Hash

Extract DATA VALUES for an existing generic element/segment/dataset instance from a document's text, to pre-fill the working copy for human review. This does NOT design or modify a template — it only reads values for the fields the given properties template already defines. Nothing is persisted here.

Parameters:

  • properties (Hash) —

    the element instance properties ({ 'layers' => ..., 'select_options' => ... })

  • context_text (String) —

    plain text extracted from the source document(s)

  • instructions (String, nil) (defaults to: nil) —

    the TEMPLATE's own extraction guidance, set by the designer in Template settings (klass.settings). Appended to the system prompt; it steers WHAT to look for, never the response format.

  • model (String, nil) (defaults to: nil) —

    per-user model override (falls back to yml/ENV/default)

  • api_key (String, nil) (defaults to: nil) —

    per-user API key override (falls back to yml/ENV)

  • base_url (String, nil) (defaults to: nil) —

    per-user provider base URL (honored only with api_key)

  • api_path (String, nil) (defaults to: nil) —

    per-user provider chat-completions path (honored only with api_key)

Returns:

  • (Hash) —

    { 'values' => { layer_key => { field_key => value } }, 'summary' => String }



199
200
201
202
 
# File 'lib/labimotion/libs/ai_template.rb', line 199

def self.fill(properties:, context_text:, instructions: nil, model: nil, api_key: nil, base_url: nil, api_path: nil)
  new(ols_term_id: '', model: model, api_key: api_key, base_url: base_url, api_path: api_path)
    .fill(properties: properties, context_text: context_text, instructions: instructions)
end

.generate(kind: 'dataset', ols_term_id: nil, subject: nil, desc: nil, cols: 1, references: [], files: [], model: nil, api_key: nil, base_url: nil, api_path: nil) ⇒ Hash

Generate a metadata template for a generic dataset, element or segment.

The JSON template schema (layers / fields / select_options) is IDENTICAL across all three kinds — only the prompt wording changes. The dataset path is driven by a CHMO ontology term; the element/segment paths are driven by a subject string (the element/segment name/label).

Parameters:

  • kind (String) (defaults to: 'dataset') —

    'dataset' (default), 'element' or 'segment'

  • ols_term_id (String) (defaults to: nil) —

    CHMO term id, e.g. "CHMO:0000470 | mass spectrometry (MS)" (dataset only)

  • subject (String) (defaults to: nil) —

    the element/segment name/label (element/segment only)

  • desc (String) (defaults to: nil) —

    free-text description of the template

  • cols (Integer) (defaults to: 1) —

    default columns per row for generated layers (1-6)

  • references (Array<String>) (defaults to: []) —

    reference links (publications / standards)

  • files (Array<Hash>) (defaults to: []) —

    [{ 'filename' => ..., 'content' => ... }, ...]

Returns:

  • (Hash) —

    { 'label' => String, 'layers' => Hash, 'select_options' => Hash }



129
130
131
132
 
# File 'lib/labimotion/libs/ai_template.rb', line 129

def self.generate(kind: 'dataset', ols_term_id: nil, subject: nil, desc: nil, cols: 1, references: [], files: [], model: nil, api_key: nil, base_url: nil, api_path: nil)
  new(kind: kind, ols_term_id: ols_term_id, subject: subject, desc: desc, cols: cols, references: references,
      files: files, model: model, api_key: api_key, base_url: base_url, api_path: api_path).generate
end

.normalize_unit_token(str) ⇒ Object

Fold a unit string to a comparable token: strip HTML (2 -> 2), map unicode superscripts to digits, micro sign to "u", downcase, drop spaces. So "°C", "µmol/L", "g/cm3" match the model's "°c"/"umol/l"/"g/cm3".



264
265
266
267
268
269
 
# File 'lib/labimotion/libs/ai_template.rb', line 264

def self.normalize_unit_token(str)
  s = str.to_s.gsub(/<[^>]+>/, '')
  s = s.tr('²³¹⁰⁴⁵⁶⁷⁸⁹', '2310456789')
  s = s.gsub(/[µμ]/, 'u').downcase
  s.gsub(/\s+/, '')
end

.ping(model: nil, api_key: nil, base_url: nil, api_path: nil) ⇒ Hash

Verify the (per-user or server) AI settings by sending one tiny chat request and reporting whether it round-trips. Nothing is generated or persisted. Same resolution as the real calls: base_url/api_path are honored only ALONGSIDE a personal key (custom_endpoint?), and a custom base_url is SSRF-validated.

Returns:

  • (Hash) —

    { 'ok' => true, 'model' => .., 'endpoint' => .., 'ms' => .. }

Raises:

  • (RuntimeError) —

    with a user-facing message on any failure



211
212
213
 
# File 'lib/labimotion/libs/ai_template.rb', line 211

def self.ping(model: nil, api_key: nil, base_url: nil, api_path: nil)
  new(ols_term_id: '', model: model, api_key: api_key, base_url: base_url, api_path: api_path).ping
end

.plan(index:, instruction:, ols_term_id: nil, history: [], model: nil, api_key: nil, base_url: nil, api_path: nil) ⇒ Hash

Turn a natural-language instruction into a list of STRUCTURAL operations, without sending the template or asking the model to re-emit it.

Refine's cost is set by the template, not the request: the model is told to reproduce every layer and field verbatim, so "delete the processing layer" bills the same thousands of output tokens as a redesign — and on a long template the model stops mid-copy, which surfaces as an unparseable response. Neither is inherent to the request. Deleting a layer, grouping two, reordering, flipping required are all things chem-generic-ui already does deterministically, cascades and workflow guard included; the model is only needed to read the sentence and name the layer.

So this path sends an INDEX (keys, labels, types) and takes back a short op list the designer applies itself. When the instruction needs judgement the ops cannot express — new fields, wording, units, ontology terms — the model says so and the caller falls back to refine.

Parameters:

  • index (Hash) —

    compact template index (see chem-generic-ui buildTemplateIndex)

  • instruction (String) —

    the change to apply

  • history (Array<Hash>) (defaults to: []) —

    prior turns [{ 'role' => .., 'content' => .. }]

Returns:

  • (Hash) —

    { 'operations' => Array, 'summary' => String, 'needs_design' => Boolean, 'reason' => String, 'usage' => Hash }



179
180
181
182
 
# File 'lib/labimotion/libs/ai_template.rb', line 179

def self.plan(index:, instruction:, ols_term_id: nil, history: [], model: nil, api_key: nil, base_url: nil, api_path: nil)
  new(ols_term_id: ols_term_id, history: history, model: model, api_key: api_key, base_url: base_url, api_path: api_path)
    .plan(index: index, instruction: instruction)
end

.record_unit_tokens(tokens, gkey, unit) ⇒ Object 



249
250
251
252
253
254
255
256
257
258
259
 
# File 'lib/labimotion/libs/ai_template.rb', line 249

def self.record_unit_tokens(tokens, gkey, unit)
  ukey = unit[:key].to_s
  return if ukey.empty?

  [unit[:label], ukey].each do |raw|
    token = normalize_unit_token(raw)
    next if token.empty?

    (tokens[token] ||= {})[gkey] ||= ukey
  end
end

.refine(current:, instruction:, ols_term_id: nil, history: [], cols: 1, model: nil, api_key: nil, base_url: nil, api_path: nil) ⇒ Hash

Refine an existing dataset template through a natural-language instruction.

The caller passes the CURRENT template (label, layers, select_options) — which already reflects any previously-applied refinements — plus the latest instruction and an optional short history of prior turns for continuity. The LLM returns the FULL revised template in the same shape, together with a one-line summary of what it changed. Nothing is persisted here; the admin reviews the result in the designer and saves it (human-in-the-loop).

Parameters:

  • current (Hash) —

    { 'label' => String, 'layers' => Hash, 'select_options' => Hash }

  • instruction (String) —

    the change to apply

  • history (Array<Hash>) (defaults to: []) —

    prior turns [{ 'role' => 'user'|'assistant', 'content' => String }]

  • cols (Integer) (defaults to: 1) —

    fallback columns per row when a layer omits its own (1-6)

  • model (String, nil) (defaults to: nil) —

    per-user model override (falls back to yml/ENV/default)

  • api_key (String, nil) (defaults to: nil) —

    per-user API key override (falls back to yml/ENV)

  • base_url (String, nil) (defaults to: nil) —

    per-user provider base URL (honored only with api_key)

  • api_path (String, nil) (defaults to: nil) —

    per-user provider chat-completions path (honored only with api_key)

Returns:

  • (Hash) —

    { 'label' => String, 'layers' => Hash, 'select_options' => Hash, 'summary' => String }



152
153
154
155
 
# File 'lib/labimotion/libs/ai_template.rb', line 152

def self.refine(current:, instruction:, ols_term_id: nil, history: [], cols: 1, model: nil, api_key: nil, base_url: nil, api_path: nil)
  new(ols_term_id: ols_term_id, cols: cols, history: history, model: model, api_key: api_key, base_url: base_url, api_path: api_path)
    .refine(current: current, instruction: instruction)
end

.unit_groups ⇒ Object

All valid unit-group keys (the option_layers value of a system-defined field).



225
226
227
 
# File 'lib/labimotion/libs/ai_template.rb', line 225

def self.unit_groups
  @unit_groups ||= Labimotion::Units::FIELDS.map { |g| g[:field].to_s }.freeze
end

.unit_index ⇒ Object

Reverse index of physical units -> the LabIMotion system-defined unit group, built once from Labimotion::Units::FIELDS. Maps a normalized unit token (from a unit's label OR its key) to { group:, value_system: }. Tokens used by more than one group are AMBIGUOUS and dropped, so a lookup never guesses the wrong group (e.g. "Pa" -> pressure vs elastic modulus falls back to a plain number).



220
221
222
 
# File 'lib/labimotion/libs/ai_template.rb', line 220

def self.unit_index
  @unit_index ||= build_unit_index
end

Instance Method Details

#fill(properties:, context_text:, instructions: nil) ⇒ Object 



365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
 
# File 'lib/labimotion/libs/ai_template.rb', line 365

def fill(properties:, context_text:, instructions: nil)
  raise 'AI API key is not configured (set config/labimotion_ai.yml :api_key or KI_TOOLBOX_API_KEY)' if api_key.blank?
  raise 'No readable text could be extracted from the selected source' if context_text.to_s.strip.blank?

  props = properties.is_a?(Hash) ? properties : {}
  @fill_instructions = instructions.to_s.strip
  # Resolved ONCE and reused by both the prompt and the validation below, so the
  # model is never offered a term the answer is then rejected against.
  @ontology_vocabulary = build_ontology_vocabulary(props)
  schema = fill_field_schema(props)
  raise 'This element template has no fields to fill' if schema.empty?

  response = post_chat(fill_messages(schema, context_text))
  request_failed!('fill', response) unless response.code == 200

  body = JSON.parse(response.body)
  guard_finish_reason!(body)

  text = extract_text(body)
  raise 'AI returned an empty response' if text.blank?

  data = parse_json(text)
  raw_values = data.is_a?(Hash) ? data['values'] : nil
  {
    'values' => normalize_fill_values(props, raw_values),
    'summary' => (data.is_a?(Hash) ? data['summary'].to_s.strip : ''),
    'usage' => usage_from(body),
    'model' => model
  }
rescue JSON::ParserError => e
  log_ai_response('fill could not parse response', response&.body)
  Labimotion.log_exception(e)
  raise 'AI returned a response that could not be parsed'
end

#generate ⇒ Object 



289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
 
# File 'lib/labimotion/libs/ai_template.rb', line 289

def generate
  raise 'AI API key is not configured (set config/labimotion_ai.yml :api_key or KI_TOOLBOX_API_KEY)' if api_key.blank?
  if @kind == 'dataset'
    raise 'An ontology term (CHMO) is required' if @ols_term_id.blank?
  else
    raise 'A name/label is required' if @subject.blank?
  end

  response = post_messages
  request_failed!('generate', response) unless response.code == 200

  body = JSON.parse(response.body)
  guard_finish_reason!(body)

  text = extract_text(body)
  raise 'AI returned an empty response' if text.blank?

  normalize(parse_json(text)).merge('usage' => usage_from(body))
rescue JSON::ParserError => e
  log_ai_response('generate could not parse response', response&.body)
  Labimotion.log_exception(e)
  raise 'AI returned a response that could not be parsed as a template'
end

#ping ⇒ Object 



400
401
402
403
404
405
406
407
408
409
 
# File 'lib/labimotion/libs/ai_template.rb', line 400

def ping
  raise 'AI API key is not configured' if api_key.blank?

  started  = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  response = post_chat([{ role: 'user', content: 'ping' }], PING_MAX_TOKENS)
  ms       = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round
  raise ping_error(response.code) unless response.code == 200

  { 'ok' => true, 'model' => model, 'endpoint' => "#{base_url}#{api_path}", 'ms' => ms }
end

#plan(index:, instruction:) ⇒ Object 



346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
 
# File 'lib/labimotion/libs/ai_template.rb', line 346

def plan(index:, instruction:)
  raise 'AI API key is not configured (set config/labimotion_ai.yml :api_key or KI_TOOLBOX_API_KEY)' if api_key.blank?
  raise 'An instruction is required' if instruction.to_s.strip.blank?

  response = post_chat(plan_messages(index, instruction), PLAN_MAX_TOKENS)
  request_failed!('plan', response) unless response.code == 200

  body = JSON.parse(response.body)
  raise 'AI request was declined by the content filter' if finish_reason(body) == 'content_filter'

  plan_from(body)
rescue JSON::ParserError => e
  log_ai_response('plan could not parse response', response&.body)
  Labimotion.log_exception(e)
  # An unparseable PLAN is not a failed request — it is a signal to take the
  # slower path, which is exactly what escalation asks the caller to do.
  escalate('The model did not return a usable plan.')
end

#refine(current:, instruction:) ⇒ Object 



313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
 
# File 'lib/labimotion/libs/ai_template.rb', line 313

def refine(current:, instruction:)
  raise 'AI API key is not configured (set config/labimotion_ai.yml :api_key or KI_TOOLBOX_API_KEY)' if api_key.blank?
  raise 'An instruction is required' if instruction.to_s.strip.blank?

  response = post_chat(refine_messages(current, instruction))
  request_failed!('refine', response) unless response.code == 200

  body = JSON.parse(response.body)
  guard_finish_reason!(body)

  text = extract_text(body)
  raise 'AI returned an empty response' if text.blank?

  data = parse_json(text)
  result = normalize_preserving(data)
  # Guard against a malformed response silently wiping the template — a
  # dataset template always has at least one layer.
  if result['layers'].empty?
    log_ai_response('refine produced no layers', text)
    raise 'AI returned an empty template (the model response contained no layers). ' \
          'Try a more capable model.'
  end

  result['summary'] = (data.is_a?(Hash) ? data['summary'].to_s.strip : '')
  result['usage'] = usage_from(body)
  result['model'] = model
  result
rescue JSON::ParserError => e
  log_ai_response('refine could not parse response', response&.body)
  Labimotion.log_exception(e)
  raise 'AI returned a response that could not be parsed as a template'
end