Module: TWPipeline

Defined in:
lib/twpipeline.rb,
lib/twpipeline/cli.rb,
lib/twpipeline/bench.rb,
lib/twpipeline/jsonl.rb,
lib/twpipeline/stage.rb,
lib/twpipeline/rusage.rb,
lib/twpipeline/sorting.rb,
lib/twpipeline/version.rb,
lib/twpipeline/parallel.rb,
lib/twpipeline/frequency.rb,
lib/twpipeline/resources.rb,
lib/twpipeline/segmenter.rb,
lib/twpipeline/sources/hplt.rb,
lib/twpipeline/stages/count.rb,
lib/twpipeline/stages/dedup.rb,
lib/twpipeline/stages/ingest.rb,
lib/twpipeline/stages/script.rb,
lib/twpipeline/stages/lexicon.rb,
lib/twpipeline/stages/segment.rb,
lib/twpipeline/stages/evidence.rb,
lib/twpipeline/stages/tokenize.rb,
lib/twpipeline/stages/normalize.rb,
sig/twpipeline.rbs

Defined Under Namespace

Modules: Bench, Frequency, Jsonl, Parallel, Rusage, Sorting, Sources, Stages Classes: CLI, Error, MapStage, Resources, Segmenter, Stage

Constant Summary collapse

ROOT =

Returns:

  • (Pathname) 
Pathname(__dir__).parent
VERSION =

Returns:

  • (String) 
"0.1.0"

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.resources ⇒ Resources

Returns:



38
 
# File 'lib/twpipeline.rb', line 38

def resources = @resources ||= Resources.detect

Class Method Details

.data(*parts) ⇒ Pathname

Parameters:

  • parts (::String)

Returns:

  • (Pathname) 


42
 
# File 'lib/twpipeline.rb', line 42

def data(*parts) = ROOT.join("data", *parts)

.warm_up ⇒ singleton(TWPipeline)

Returns:



44
45
46
47
48
49
50
51
52
53
54
55
56
57
 
# File 'lib/twpipeline.rb', line 44

def warm_up
  TWFilter::Checks::Script.tiers
  TWFilter::Checks::Lexicon.hard_terms
  TWFilter::Checks::Lexicon.soft_terms
  TWFilter::Checks::Lexicon.exceptions
  TWFilter::Checks::Lexicon.regional_terms
  TWFilter::Checks::Lexicon.foreign_topics
  TWFilter::Checks::Lexicon.cantonese_chars
  TWFilter::Checks::Erhua.headed
  TWFilter::Checks::Erhua.tailed
  TWFilter::Checks::Wenyan.markers
  TWFilter::Evidence.terms
  self
end

.work ⇒ Pathname

Returns:

  • (Pathname) 


40
 
# File 'lib/twpipeline.rb', line 40

def work = Pathname(ENV.fetch("TWP_WORK", ROOT.parent.join("work").to_s))