Class: Odin::Transform::TransformEngine

Inherits:
Object
  • Object
show all
Defined in:
lib/odin/transform/transform_engine.rb

Defined Under Namespace

Modules: ErrorCodes Classes: CodedTransformError, CompiledValidation, MappingMods, TransformAbortError, TransformError, TransformWarning

Constant Summary collapse

CORE_VERBS =

Verb registry — populated in Phase 9-10. For now, core verbs only.

{}.freeze
SORT_VERBS =

Verbs whose cost grows with an array argument. Charged proportional to the array width so large-array work cannot escape the fuel budget.

%w[sort].freeze
WIDTH_VERBS = 
%w[
  distinct groupBy keyBy countBy reduce
  sum avg min max count sumIf avgIf countIf
  union intersection difference symmetricDifference
  map filter window explode flatten reverse
].freeze
CLOCK_CHECK_INTERVAL =

Read the wall clock once per this many charged units.

1024
SOURCE_MISSING =

The required-source-missing code: a present-but-null :required field.

"SOURCE_MISSING"
NUMERIC_ARG_VERBS =

Verbs whose leading arguments must be numeric; checked under strictTypes.

%w[
  sqrt abs round floor ceil negate sign trunc ln log log10 log2 exp pow
  add subtract multiply divide mod between formatNumber formatInteger
  formatCurrency toRadians toDegrees
].freeze
NUMERIC_DYN_TYPES = 
%i[integer float float_raw currency currency_raw percent null].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ TransformEngine

Returns a new instance of TransformEngine.



184
185
186
187
 
# File 'lib/odin/transform/transform_engine.rb', line 184

def initialize
  @verb_registry = build_verb_registry
  reset_guard!
end

Instance Attribute Details

#verb_registry ⇒ Object (readonly)

Returns the value of attribute verb_registry.



182
183
184
 
# File 'lib/odin/transform/transform_engine.rb', line 182

def verb_registry
  @verb_registry
end

Class Method Details

.accumulator_overflow_error(name, value) ⇒ Object

Create a T008 Accumulator Overflow error.



170
171
172
 
# File 'lib/odin/transform/transform_engine.rb', line 170

def self.accumulator_overflow_error(name, value)
  TransformError.new("Accumulator '#{name}' overflow with value #{value}", code: ErrorCodes::T008_ACCUMULATOR_OVERFLOW)
end

.budget_exceeded_error(limit) ⇒ Object

Create a T016 Transform Budget Exceeded error.



93
94
95
96
97
98
 
# File 'lib/odin/transform/transform_engine.rb', line 93

def self.budget_exceeded_error(limit)
  TransformError.new(
    "Transform fuel budget exceeded (limit #{limit})",
    code: ErrorCodes::T016_TRANSFORM_BUDGET_EXCEEDED
  )
end

.dangling_branch_error(directive) ⇒ Object

Create a T012 Dangling Branch error (elif/else with no preceding if).



127
128
129
130
131
132
 
# File 'lib/odin/transform/transform_engine.rb', line 127

def self.dangling_branch_error(directive)
  TransformError.new(
    "'#{directive}' segment has no preceding 'if'",
    code: ErrorCodes::T012_DANGLING_BRANCH
  )
end

.expression_depth_exceeded_error(limit) ⇒ Object

Create a T018 Expression Depth Exceeded error.



109
110
111
112
113
114
 
# File 'lib/odin/transform/transform_engine.rb', line 109

def self.expression_depth_exceeded_error(limit)
  TransformError.new(
    "Expression evaluation depth exceeded (limit #{limit})",
    code: ErrorCodes::T018_EXPRESSION_DEPTH_EXCEEDED
  )
end

.incompatible_conversion_error(verb_name, detail) ⇒ Object

Create a T011 Incompatible Conversion error. Used when a verb receives an unknown or incompatible conversion target (e.g., unknown unit in dateDiff or distance).



137
138
139
140
141
142
 
# File 'lib/odin/transform/transform_engine.rb', line 137

def self.incompatible_conversion_error(verb_name, detail)
  TransformError.new(
    "#{verb_name}: incompatible conversion — #{detail}",
    code: ErrorCodes::T011_INCOMPATIBLE_CONVERSION
  )
end

.invalid_output_format_error(format) ⇒ Object

Create a T006 Invalid Output Format error.



165
166
167
 
# File 'lib/odin/transform/transform_engine.rb', line 165

def self.invalid_output_format_error(format)
  TransformError.new("Invalid or unsupported output format: #{format}", code: ErrorCodes::T006_INVALID_OUTPUT_FORMAT)
end

.lookup_key_not_found_error(table_name, key) ⇒ Object

Create a T004 Lookup Key Not Found error.



155
156
157
 
# File 'lib/odin/transform/transform_engine.rb', line 155

def self.lookup_key_not_found_error(table_name, key)
  TransformError.new("Lookup key '#{key}' not found in table '#{table_name}'", code: ErrorCodes::T004_LOOKUP_KEY_NOT_FOUND)
end

.lookup_table_not_found_error(table_name) ⇒ Object

Create a T003 Lookup Table Not Found error.



150
151
152
 
# File 'lib/odin/transform/transform_engine.rb', line 150

def self.lookup_table_not_found_error(table_name)
  TransformError.new("Lookup table not found: #{table_name}", code: ErrorCodes::T003_LOOKUP_TABLE_NOT_FOUND)
end

.loop_source_not_array_error(path) ⇒ Object

Create a T009 Loop Source Not Array error.



175
176
177
 
# File 'lib/odin/transform/transform_engine.rb', line 175

def self.loop_source_not_array_error(path)
  TransformError.new("Loop source path '#{path}' does not resolve to an array", code: ErrorCodes::T009_LOOP_SOURCE_NOT_ARRAY)
end

.nested_interpolation_error(expr, segment = nil) ⇒ Object

Create a T014 Nested Interpolation error.



117
118
119
120
121
122
123
124
 
# File 'lib/odin/transform/transform_engine.rb', line 117

def self.nested_interpolation_error(expr, segment = nil)
  err = TransformError.new(
    "Nested interpolation is not allowed: ${#{expr}}",
    code: ErrorCodes::T014_NESTED_INTERPOLATION
  )
  err.segment = segment if segment
  err
end

.source_path_not_found_error(path) ⇒ Object

Create a T005 Source Path Not Found error.



160
161
162
 
# File 'lib/odin/transform/transform_engine.rb', line 160

def self.source_path_not_found_error(path)
  TransformError.new("Source path not found: #{path}", code: ErrorCodes::T005_SOURCE_PATH_NOT_FOUND)
end

.timeout_exceeded_error(limit_ms) ⇒ Object

Create a T017 Transform Timeout Exceeded error.



101
102
103
104
105
106
 
# File 'lib/odin/transform/transform_engine.rb', line 101

def self.timeout_exceeded_error(limit_ms)
  TransformError.new(
    "Transform timeout exceeded (limit #{limit_ms}ms)",
    code: ErrorCodes::T017_TRANSFORM_TIMEOUT_EXCEEDED
  )
end

.unknown_verb_error(verb_name) ⇒ Object

Create a T001 Unknown Verb error.



145
146
147
 
# File 'lib/odin/transform/transform_engine.rb', line 145

def self.unknown_verb_error(verb_name)
  TransformError.new("Unknown verb: #{verb_name}", code: ErrorCodes::T001_UNKNOWN_VERB)
end

Instance Method Details

#abort_result(err, output, context) ⇒ Object

Surface a guard abort as a failed result; the abort is never raised past the execute boundary. The guard error joins context.errors so the result reports failure.



477
478
479
480
481
 
# File 'lib/odin/transform/transform_engine.rb', line 477

def abort_result(err, output, context)
  context.errors << err.transform_error
  plain_output = deep_to_ruby(output)
  TransformResult.new(output: plain_output, formatted: "", errors: context.errors, warnings: context.warnings)
end

#charge(units) ⇒ Object

Charge fuel and, at a coarse interval, the wall clock. Both are no-ops unless their cap is set (> 0), so unbounded transforms pay nothing.

Raises:



490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
 
# File 'lib/odin/transform/transform_engine.rb', line 490

def charge(units)
  if @fuel_cap.positive?
    @fuel_used += units
    raise TransformAbortError.new(self.class.budget_exceeded_error(@fuel_cap)) if @fuel_used > @fuel_cap
  end
  return unless @timeout_ms.positive?

  @ops_since_clock += units
  return if @ops_since_clock < CLOCK_CHECK_INTERVAL

  @ops_since_clock = 0
  return unless monotonic_ms - @start_time > @timeout_ms

  raise TransformAbortError.new(self.class.timeout_exceeded_error(@timeout_ms))
end

#charge_verb_width(verb, args) ⇒ Object

Charge width for a verb doing O(n)/O(n log n) work over an array argument, so large-array work cannot escape the budget at a flat unit. Runs when fuel OR timeout is set, so a timeout-only host can interrupt wide verbs.



509
510
511
512
513
514
515
516
517
518
519
520
 
# File 'lib/odin/transform/transform_engine.rb', line 509

def charge_verb_width(verb, args)
  return if @fuel_cap <= 0 && @timeout_ms <= 0

  n = first_array_width(args)
  return if n <= 0

  if SORT_VERBS.include?(verb)
    charge(n * Math.log2([n, 2].max).ceil)
  elsif WIDTH_VERBS.include?(verb)
    charge(n)
  end
end

#check_verb_arg_types!(verb_name, args) ⇒ Object

T002: under strictTypes, a numeric verb argument that is not a numeric (or null) value fails the field.



457
458
459
460
461
462
463
464
465
466
467
468
469
470
 
# File 'lib/odin/transform/transform_engine.rb', line 457

def check_verb_arg_types!(verb_name, args)
  return unless NUMERIC_ARG_VERBS.include?(verb_name)

  args.each_with_index do |arg, i|
    next if arg.nil?
    next if NUMERIC_DYN_TYPES.include?(arg.type)

    err = TransformError.new(
      "Verb '#{verb_name}' arg #{i + 1}: expected number, got #{arg.type}",
      code: ErrorCodes::T002_INVALID_VERB_ARGS
    )
    raise CodedTransformError.new(err)
  end
end

#evaluate(expr, context) ⇒ Object

Guard boundary: enforce depth, charge one base unit of fuel, and batch the wall-clock check before delegating to the evaluator. All evaluation paths funnel through here, so charging stays concentrated at this single point.



535
536
537
538
539
540
541
542
543
544
545
546
547
 
# File 'lib/odin/transform/transform_engine.rb', line 535

def evaluate(expr, context)
  @expr_depth += 1
  if @expr_depth > @max_expr_depth
    @expr_depth -= 1
    raise TransformAbortError.new(self.class.expression_depth_exceeded_error(@max_expr_depth))
  end
  charge(1)
  begin
    evaluate_inner(expr, context)
  ensure
    @expr_depth -= 1
  end
end

#evaluate_inner(expr, context) ⇒ Object 



549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
 
# File 'lib/odin/transform/transform_engine.rb', line 549

def evaluate_inner(expr, context)
  case expr
  when LiteralExpr
    val = expr.value
    if val.is_a?(Types::DynValue) && val.string? && val.value.include?("${")
      return interpolate_string(val.value, context)
    end
    val
  when CopyExpr
    val = resolve_path(expr.source_path, context)
    # Apply CopyExpr-level extraction directives only for compatible source formats
    # (fixed-width, csv, delimited, flat — NOT odin, json, xml)
    if expr.directives && !expr.directives.empty?
      src_fmt = context.source_format
      if src_fmt == "fixed-width" || src_fmt == "csv" || src_fmt == "delimited" || src_fmt == "flat"
        val = apply_extraction_directives(val, expr.directives)
      end
    end
    val
  when VerbExpr
    evaluate_verb(expr, context)
  when ObjectExpr
    evaluate_object(expr, context)
  else
    Types::DynValue.of_null
  end
end

#execute(transform_def, source_data, import_resolver: nil, max_transform_fuel: nil, transform_timeout_ms: nil, max_expression_depth: nil) ⇒ Object 



202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
 
# File 'lib/odin/transform/transform_engine.rb', line 202

def execute(transform_def, source_data, import_resolver: nil,
            max_transform_fuel: nil, transform_timeout_ms: nil, max_expression_depth: nil)
  reset_guard!(max_transform_fuel: max_transform_fuel,
               transform_timeout_ms: transform_timeout_ms,
               max_expression_depth: max_expression_depth)

  # Merge imported tables, constants, accumulators, and segments.
  if import_resolver && transform_def.imports && !transform_def.imports.empty?
    resolve_imports(transform_def, import_resolver)
  end

  # Check for multi-record mode (discriminator dispatch)
  disc_config = transform_def.discriminator_config
  if disc_config
    raw_str = case source_data
              when String then source_data
              when Types::DynValue
                source_data.string? ? source_data.value : nil
              end
    return execute_multi_record(transform_def, raw_str, disc_config) if raw_str
  end

  # 1. Normalize source data to DynValue
  source = normalize_source(source_data, transform_def.source_format)

  # 2. Build context
  context = build_context(transform_def, source)

  # 3. Process segments (multi-pass support)
  output = {}
  begin
    passes = transform_def.passes
    if passes.empty?
      # Single implicit pass
      process_segment_list(transform_def.segments, source, context, output)
    else
      # Multi-pass: explicit passes first, then pass-0 (implicit)
      all_passes = passes.include?(0) ? passes : passes + [0]
      first_pass = true
      all_passes.each do |pass_num|
        unless first_pass
          reset_non_persist_accumulators(context, transform_def.accumulators)
        end
        first_pass = false

        pass_segments = transform_def.segments.select { |s| (s.pass || 0) == pass_num }
        process_segment_list(pass_segments, source, context, output)
      end
    end
  rescue TransformAbortError => e
    return abort_result(e, output, context)
  end

  # 4. Apply confidential enforcement
  if transform_def.header.enforce_confidential != ConfidentialMode::NONE
    apply_confidential(output, transform_def.header.enforce_confidential, context.field_modifiers)
  end

  # 5. Convert output to DynValue (preserves types like date, timestamp)
  output_dv = Types::DynValue.from_ruby(output)

  # 6. Format output
  formatted = format_output(output_dv, transform_def, context)

  # 7. Convert output to plain Ruby for result (DynValues -> native Ruby)
  plain_output = deep_to_ruby(output)

  TransformResult.new(output: plain_output, formatted: formatted, output_dv: output_dv, errors: context.errors, warnings: context.warnings)
end

#execute_multi_record(transform_def, raw_input, disc_config) ⇒ Object

── Multi-Record Execution (discriminator-based routing) ──



274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
 
# File 'lib/odin/transform/transform_engine.rb', line 274

def execute_multi_record(transform_def, raw_input, disc_config)
  # Parse discriminator config
  disc = parse_discriminator_config(disc_config)
  return TransformResult.new(output: {}, formatted: "", errors: []) unless disc

  source_format = transform_def.source_format
  delimiter = transform_def.header.source_options["delimiter"] || ","

  # Build segment routing map: _type literal value -> segment
  segment_map = {}
  transform_def.segments.each do |seg|
    next unless seg.discriminator_value

    seg.discriminator_value.split(",").each do |type_val|
      segment_map[type_val.strip] = seg
    end
  end

  context = build_context(transform_def, Types::DynValue.of_null)
  context.source_format = source_format

  output = {}
  array_accumulators = {}

  # Initialize array accumulators
  transform_def.segments.each do |seg|
    if seg.is_array
      array_accumulators[seg.name] = []
    end
  end

  # Process each record/line
  begin
    lines = raw_input.split(/[\r\n]+/)
    lines.each do |line|
      next if line.strip.empty?

      disc_value = extract_discriminator_value(line, disc, delimiter)
      segment = segment_map[disc_value]
      next unless segment

      record_source = parse_record(line, source_format, delimiter)
      record_output = {}

      # Set the record as the current source for path resolution
      context.source = record_source

      # Process field mappings
      segment.field_mappings.each do |mapping|
        process_mapping(mapping, record_source, context, record_output)
      end

      # Process children
      segment.children.each do |child|
        process_segment(child, record_source, context, record_output)
      end

      # Merge into output
      seg_name = segment.name

      if segment.is_array
        array_accumulators[seg_name] ||= []
        array_accumulators[seg_name] << record_output
      else
        # Merge fields into existing segment object
        if output[seg_name].is_a?(Hash)
          record_output.each { |k, v| output[seg_name][k] = v }
        else
          output[seg_name] = record_output
        end
      end
    end
  rescue TransformAbortError => e
    return abort_result(e, output, context)
  end

  # Merge array accumulators into output in segment order
  transform_def.segments.each do |seg|
    next unless seg.is_array

    items = array_accumulators[seg.name]
    next unless items

    output[seg.name] = items
  end

  # Convert output to DynValue
  output_dv = Types::DynValue.from_ruby(output)

  # Format output
  formatted = format_output(output_dv, transform_def, context)

  # Convert output to plain Ruby
  plain_output = deep_to_ruby(output)

  TransformResult.new(output: plain_output, formatted: formatted, output_dv: output_dv, errors: context.errors, warnings: context.warnings)
end

#first_array_width(args) ⇒ Object

Width of the first array-typed argument, or 0 if none.



523
524
525
526
527
528
 
# File 'lib/odin/transform/transform_engine.rb', line 523

def first_array_width(args)
  args.each do |arg|
    return arg.value.length if arg.is_a?(Types::DynValue) && arg.array?
  end
  0
end

#invoke_verb(name, args, context) ⇒ Object

Public for unit testing verbs directly

Raises:



439
440
441
442
443
444
 
# File 'lib/odin/transform/transform_engine.rb', line 439

def invoke_verb(name, args, context)
  verb_fn = @verb_registry[name]
  raise CodedTransformError.new(self.class.unknown_verb_error(name)) unless verb_fn

  verb_fn.call(args, context)
end

#monotonic_ms ⇒ Object

Monotonic clock in milliseconds for the wall-clock guard.



484
485
486
 
# File 'lib/odin/transform/transform_engine.rb', line 484

def monotonic_ms
  Process.clock_gettime(Process::CLOCK_MONOTONIC, :millisecond)
end

#reset_guard!(max_transform_fuel: nil, transform_timeout_ms: nil, max_expression_depth: nil) ⇒ Object

Reset execution-guard state for a single execute call. Fuel and timeout charge only when their cap is set (> 0); depth uses its standing default. A per-call value overrides the global limit; nil falls back to it.



192
193
194
195
196
197
198
199
200
 
# File 'lib/odin/transform/transform_engine.rb', line 192

def reset_guard!(max_transform_fuel: nil, transform_timeout_ms: nil, max_expression_depth: nil)
  @fuel_cap = max_transform_fuel.nil? ? Utils::SecurityLimits.max_transform_fuel : max_transform_fuel
  @timeout_ms = transform_timeout_ms.nil? ? Utils::SecurityLimits.transform_timeout_ms : transform_timeout_ms
  @max_expr_depth = max_expression_depth.nil? ? Utils::SecurityLimits.max_expression_depth : max_expression_depth
  @fuel_used = 0
  @expr_depth = 0
  @ops_since_clock = 0
  @start_time = @timeout_ms.positive? ? monotonic_ms : 0
end