Module: Insika::Knowledge

Defined in:
lib/insika/knowledge.rb

Overview

the one place knowledge extraction asks a model for anything, and the concept format itself (markdown + YAML frontmatter, the same shape a SKILL.md uses).

The engine's generic prompt (what a concept worth keeping is, the answer shape, the "never invent, never state as policy" rules). A pack knowledge.prompt REPLACES this wholesale (the forge's half), like distill.prompt / harvest.prompt.

Defined Under Namespace

Modules: Concept, ConsolidatorFactory, ExtractorFactory, GraphmlExport, Index Classes: Consolidator, Extractor

Constant Summary collapse

DEFAULT_TYPES = 
%w[fact entity procedure policy objection].freeze
DEFAULT_PROMPT = 
<<~PROMPT.freeze
  You are extracting durable KNOWLEDGE from one finished conversation, for
  a store agent's shared memory. A concept is something worth remembering
  across DIFFERENT conversations: a policy, a recurring question, an
  objection customers raise, a fact about how the business operates. It
  is NOT a fact about one customer (that belongs to per-customer memory)
  and NOT a summary of this conversation.

  Answer with a single JSON array and NOTHING else. No prose, no fences.

  Each element is an object with:
  - "name" — a short, lowercase, hyphen-separated slug (max 80 chars);
  - "description" — one line (max 300 chars): what the concept says;
  - "type" — one of: fact, entity, procedure, policy, objection;
  - "body" — the concept's content (max 2000 chars): the durable claim,
    in your own words, plus `[[other-concept-name]]` links to any related
    concept you are also proposing in this same answer.

  Rules:
  - Never invent: only concepts the conversation actually supports.
  - Never state something the agent merely PROMISED as if it were
    official policy — describe it as what was said, not as a guarantee.
  - Never include a customer id, a session id or anything that identifies
    one person — the engine stamps provenance itself.
  - A concept every good agent already assumes is not worth proposing.
  - Fewer, better concepts beat filling a quota.
PROMPT
CONCEPT_SCHEMA =

The safe-subset JSON Schema (Workflow::Schema, the house zero-dep validator). Anything outside this set — a model-authored provenance, confidence, sources, occurrences — is a provenance lie: the schema refuses it by not having the key, and the extractor drops+counts it rather than trusting the model's self-assessment (RFC's "provenance only as ids" rule, enforced here, not just documented).

Insika::Workflow::Schema.coerce({
  "type" => "array",
  "items" => {
    "type" => "object",
    "properties" => {
      "name" => { "type" => "string" },
      "description" => { "type" => "string" },
      "type" => { "type" => "string" },
      "body" => { "type" => "string" }
    },
    "required" => %w[name description type body]
  }
})
ITEM_SCHEMA = 
Insika::Workflow::Schema.coerce(CONCEPT_SCHEMA.json_schema["items"])
CONFIDENCE_BASE =

The layer-2 confidence formula: more independent sightings, more confidence, never certainty. confidence_for(1) is the first-sighting value PR 1 hardcoded (0.6) — spelled out here as the formula's own degenerate case, not a separate constant to keep in sync.

0.5
CONFIDENCE_STEP = 
0.1
CONFIDENCE_CAP = 
0.95
CONTRADICTION_CONFIDENCE =

A contradiction is never silently resolved into a confidence climb — it is a flat drop, regardless of how confirmed the concept was before.

0.4
CONTRADICTION_HEADING = 
"## Contradiction"

Class Method Summary collapse

Class Method Details

.bump(existing, session_id) ⇒ Object

Same claim, reworded or reconfirmed: the body stays (no operator edit is silently discarded by a repeat sighting), only the evidence grows.



136
137
138
139
140
141
142
143
 
# File 'lib/insika/knowledge.rb', line 136

def bump(existing, session_id)
  sources = (existing[:sources] + [session_id.to_s]).uniq
  Concept.render(
    name: existing[:name], description: existing[:description], type: existing[:type], body: existing[:body],
    provenance: "observed", confidence: confidence_for(sources.size), sources: sources,
    occurrences: existing[:occurrences] + 1, created_at: existing[:created_at], updated_at: Time.now.utc.iso8601
  )
end

.confidence_for(distinct_sources) ⇒ Object 



77
 
# File 'lib/insika/knowledge.rb', line 77

def confidence_for(distinct_sources) = [CONFIDENCE_CAP, CONFIDENCE_BASE + (CONFIDENCE_STEP * distinct_sources)].min

.contradict(existing, new_body, session_id) ⇒ Object

Contradicting claim: never overwritten. The new claim joins the body under a heading a human resolves in the Studio; occurrences do NOT bump (a conflict is not a confirmation), but the sighting still joins sources for the audit trail.



160
161
162
163
164
165
166
167
168
 
# File 'lib/insika/knowledge.rb', line 160

def contradict(existing, new_body, session_id)
  sources = (existing[:sources] + [session_id.to_s]).uniq
  body = "#{existing[:body]}\n\n#{CONTRADICTION_HEADING}\n\n#{new_body}"
  Concept.render(
    name: existing[:name], description: existing[:description], type: existing[:type], body: body,
    provenance: "observed", confidence: CONTRADICTION_CONFIDENCE, sources: sources,
    occurrences: existing[:occurrences], created_at: existing[:created_at], updated_at: Time.now.utc.iso8601
  )
end

.first_sighting(concept, redacted_body, session_id) ⇒ Object 



125
126
127
128
129
130
131
132
 
# File 'lib/insika/knowledge.rb', line 125

def first_sighting(concept, redacted_body, session_id)
  now = Time.now.utc.iso8601
  Concept.render(
    name: concept["name"], description: concept["description"], type: concept["type"], body: redacted_body,
    provenance: "observed", confidence: confidence_for(1), sources: [session_id.to_s], occurrences: 1,
    created_at: now, updated_at: now
  )
end

.merge(existing, merged_body, session_id) ⇒ Object

Related claim: the consolidator's merged text replaces the body — the only branch where a second model call decided the wording.



147
148
149
150
151
152
153
154
 
# File 'lib/insika/knowledge.rb', line 147

def merge(existing, merged_body, session_id)
  sources = (existing[:sources] + [session_id.to_s]).uniq
  Concept.render(
    name: existing[:name], description: existing[:description], type: existing[:type], body: merged_body,
    provenance: "observed", confidence: confidence_for(sources.size), sources: sources,
    occurrences: existing[:occurrences] + 1, created_at: existing[:created_at], updated_at: Time.now.utc.iso8601
  )
end

.normalize_claim(text) ⇒ Object 



173
 
# File 'lib/insika/knowledge.rb', line 173

def normalize_claim(text) = text.to_s.strip.downcase.gsub(/\s+/, " ")

.same_claim?(a, b) ⇒ Boolean

Normalized equality (strip/downcase/collapse whitespace) — cheap and deterministic, no model call spent confirming a reworded repeat.

Returns:

  • (Boolean) 


172
 
# File 'lib/insika/knowledge.rb', line 172

def same_claim?(a, b) = normalize_claim(a) == normalize_claim(b)

.write_concept(store:, agent_id:, concept:, session_id:, tenant: nil, consolidator: nil) ⇒ Object

The ONE entry point every write path uses (the Executor's terminal hook, the backfill CLI, and a Studio-authored concept). Decides new/same/related/contradicting when concept:<name> already exists, and stamps the result — the model never gets to (RFC's "provenance only as ids" rule). The ONE place a concept's body is redacted (every write goes through this, so a caller cannot forget).

store: KnowledgeStore. concept: "name","description","type","body" — the extractor's candidate; body not yet redacted. consolidator: Consolidator | nil. nil = the conservative default: a differing body is always :contradicting (never silently overwritten) rather than spending a model call to guess. -> { verdict: :new | :same | :related | :contradicting, name:, type: }



99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
 
# File 'lib/insika/knowledge.rb', line 99

def write_concept(store:, agent_id:, concept:, session_id:, tenant: nil, consolidator: nil)
  name = concept["name"].to_s
  redacted_body, = Insika::Safety::Detectors.redact(concept["body"].to_s)
  current = store.get(agent_id, name, tenant: tenant)

  if current.nil?
    store.write(agent_id, name, first_sighting(concept, redacted_body, session_id), tenant: tenant)
    return { verdict: :new, name: name, type: concept["type"].to_s }
  end

  existing = Concept.parse(current)
  if same_claim?(existing[:body], redacted_body)
    store.write(agent_id, name, bump(existing, session_id), tenant: tenant)
    return { verdict: :same, name: existing[:name], type: existing[:type] }
  end

  resolution = consolidator && consolidator.resolve(existing_body: existing[:body], new_body: redacted_body)
  if resolution && resolution[:verdict] == :related
    store.write(agent_id, name, merge(existing, resolution[:merged_body], session_id), tenant: tenant)
    { verdict: :related, name: existing[:name], type: existing[:type] }
  else
    store.write(agent_id, name, contradict(existing, redacted_body, session_id), tenant: tenant)
    { verdict: :contradicting, name: existing[:name], type: existing[:type] }
  end
end