Class: GeminiAI::Utils::Moderation

Inherits:
Object
  • Object
show all
Defined in:
lib/utils/moderation.rb

Overview

Content moderation utility for filtering potentially harmful or inappropriate content

Constant Summary collapse

HARMFUL_PATTERNS =

Patterns for detecting potentially harmful content

[
  /\b(hack|exploit|malware|virus|trojan|ransomware)/i,
  /\b(illegal|unlawful|criminal)/i,
  /\b(violence|kill|harm|attack)/i,
  /\b(drug|weapon|nuclear)/i
].freeze

Class Method Summary collapse

Class Method Details

.moderate_text(text) ⇒ Object

Moderate text by checking for harmful patterns and redacting them Returns [moderated_text, warnings_array]



17
18
19
20
21
22
23
24
25
26
27
28
29
30
 
# File 'lib/utils/moderation.rb', line 17

def self.moderate_text(text)
  return [text, []] unless text.is_a?(String)

  moderated = text.dup
  warnings = []

  HARMFUL_PATTERNS.each do |pattern|
    if moderated.gsub!(pattern, '[REDACTED]')
      warnings << "Detected potentially harmful pattern: #{pattern.inspect}"
    end
  end

  [moderated, warnings]
end