Class: Toy::RMSNorm

Inherits:
Object
  • Object
show all
Defined in:
lib/toy.rb

Overview

Toy::RMSNorm — root-mean-square LayerNorm. No mean subtraction, no beta. Standard in llama-family models.

y = x / sqrt(mean(x^2) + eps) * gamma   (row-wise)

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(d) ⇒ RMSNorm

eps defaults to 1.0e-5 (matches Llama / SmolLM2). Override via ‘rms.eps = …` after construction — seeding the ivar with a Float literal pins Spinel’s type inference.



362
363
364
365
366
 
# File 'lib/toy.rb', line 362

def initialize(d)
  @d     = d
  @eps   = RMS_EPS_DEFAULT
  @gamma = Array.new(d, 1.0)
end

Instance Attribute Details

#d ⇒ Object

Returns the value of attribute d.



357
358
359
 
# File 'lib/toy.rb', line 357

def d
  @d
end

#eps ⇒ Object

Returns the value of attribute eps.



357
358
359
 
# File 'lib/toy.rb', line 357

def eps
  @eps
end

#gamma ⇒ Object

Returns the value of attribute gamma.



357
358
359
 
# File 'lib/toy.rb', line 357

def gamma
  @gamma
end

Instance Method Details

#algorithm_card ⇒ Object 



396
397
398
 
# File 'lib/toy.rb', line 396

def algorithm_card
  "RMSNorm(x; γ, ε) := x / √(mean(x²) + ε) ⊙ γ"
end

#forward(x) ⇒ Object

x: [T, D] → [T, D]



369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
 
# File 'lib/toy.rb', line 369

def forward(x)
  t   = x.nrows
  d   = @d
  out = Mat.new(t, d)
  i = 0
  while i < t
    sumsq = 0.0
    j = 0
    while j < d
      v = x.flat[i * d + j]
      sumsq = sumsq + v * v
      j += 1
    end
    inv = 1.0 / Math.sqrt(sumsq / d + @eps)
    j = 0
    while j < d
      out.flat[i * d + j] = x.flat[i * d + j] * inv * @gamma[j]
      j += 1
    end
    i += 1
  end
  out
end

#param_count ⇒ Object

gamma only



394
 
# File 'lib/toy.rb', line 394

def param_count; @d; end

#summary ⇒ Object 



393
 
# File 'lib/toy.rb', line 393

def summary;     "RMSNorm(d=" + @d.to_s + ", eps=" + @eps.to_s + ")"; end