Module: Toy::LLM::Primitives::SwiGLU

Defined in:
lib/toy/llm/primitives/swiglu.rb,
lib/toy/llm/primitives/swiglu_cuda.rb,
lib/toy/llm/primitives/swiglu_metal.rb

Constant Summary collapse

NAME = 
:swiglu

Class Method Summary collapse

Class Method Details

.gate(sess, t_gate, t_up) ⇒ Object

SiLU-gate composition: silu(gate) * up. t_gate and t_up are the already-projected gate/up tensors (block-owned mp_matmul outputs). Argument order matters: tnn_mul’s first operand (silu(gate)) is the shape-driver under ggml broadcast semantics — preserve silu(gate) first, up second. Returns the gated tensor handle (the input to the down projection).



34
35
36
37
 
# File 'lib/toy/llm/primitives/swiglu.rb', line 34

def self.gate(sess, t_gate, t_up)
  t_silug = TinyNN.tnn_silu(sess, t_gate)
  TinyNN.tnn_mul(sess, t_silug, t_up)
end