Class: Suma::StagedCollectionBuilder

Inherits:
Object
  • Object
show all
Defined in:
lib/suma/staged_collection_builder.rb

Overview

Memory-bounded, resumable staged collection build (metanorma/suma#94).

A normal collection build renders every member document in one OS process, so peak memory accumulates across the whole collection and a large collection (e.g. the STEP SRL) OOMs. This builder instead:

1. compiles each member in its OWN fresh OS process, preserving that
 member's unresolved cross-document references as stubs and writing the
 stub-bearing Semantic XML + anchors to a durable content-addressed store
 (the metanorma engine's +preserve_unresolved:+ / +artifact_store_dir:+
 render options, metanorma/metanorma#576);
2. runs those per-member processes SEQUENTIALLY (never in parallel --
 parallelism reintroduces the cross-linking / sectionsplit races that are
 deliberately avoided); each process exits and the OS reclaims its memory
 before the next, so peak memory is bounded by a single member; then
3. REINFLATES: a final pass assembles a manifest of the stored stubs and
 resolves them into real cross-document links (+reinflate:+), producing
 the same output an all-present build would.

The empirical case (metanorma/metanorma#576): on the real SRL a single-process build peaks ~4.1 GB and OOMs; each member compiled in isolation peaks ~1 GB, so process isolation is a ~4x peak reduction that converts the OOM into a bounded footprint.

This is opt-in; Suma::Processor uses the normal single-process build unless staging is requested.

Constant Summary collapse

STORE_DIRNAME = 
".metanorma-collection-cache"

Instance Method Summary collapse

Constructor Details

#initialize(collection_config_path:, output_directory:, coverpage: nil, store_dir: nil, formats: %i[xml html])) ⇒ StagedCollectionBuilder

collection_config_path is the emitted collection manifest (the same collection-output.yaml the normal build renders). Member filerefs are relative to its directory, so all per-member manifests are written there.



40
41
42
43
44
45
46
47
48
 
# File 'lib/suma/staged_collection_builder.rb', line 40

def initialize(collection_config_path:, output_directory:, coverpage: nil,
               store_dir: nil, formats: %i[xml html])
  @config_path = collection_config_path
  @config_dir = File.dirname(File.expand_path(collection_config_path))
  @output_directory = output_directory
  @coverpage = coverpage
  @store_dir = store_dir || File.join(output_directory, STORE_DIRNAME)
  @formats = formats
end

Instance Method Details

#build ⇒ Object 



50
51
52
53
54
55
56
57
58
 
# File 'lib/suma/staged_collection_builder.rb', line 50

def build
  FileUtils.mkdir_p(@output_directory)
  FileUtils.mkdir_p(@store_dir)
  members = collection_members
  Utils.log "[staged] #{members.size} members; one process per member, " \
            "sequential; store: #{@store_dir}"
  members.each_with_index { |member, i| stage_member(member, i) }
  reinflate(members)
end