twpipeline

Corpus construction for Taiwan Mandarin. Nine stages, one JSONL record contract, stdin → stdout. Every stage is independently runnable and independently verifiable. No record is ever discarded silently: rejection sets ok: false and appends a typed finding; --drop is what removes it.

Linguistic decisions live in twfilter; this repository is the execution layer — sources, parallelism, external sorting, measurement.

For corpus linguists and dataset engineers who need Taiwan Mandarin (cmn-Hant-TW) material with verifiable provenance. In: a web crawl or any JSONL of documents. Out: filtered sentence types, word segmentation, n-gram and frequency tables with per-host counting and a publication floor — the process behind twngrams. Single machine, no cluster: the full HPLT Taiwan slice (2.0 M documents) builds in under two hours on a laptop.

Pipeline

flowchart TD
  subgraph acquire["acquisition"]
    S1["<b>01_ingest</b> — source adapter<br/>HPLT gate: <code>.tw</code> ∪ <code>html_lang ∈ zh-TW</code> ∪ allowlist<br/>hard exclusion: <code>.cn .hk .mo .sg .my</code><br/><code>--host-cap N</code> bounds per-host concentration"]
  end

  subgraph shape["orthographic normalization — mandatory for publishable text"]
    S2["<b>02_normalize</b> — MOE 《重訂標點符號手冊》<br/>“ ” → 「 」 · → ‧ … → …… — → ──<br/>fullwidth Latin → ASCII, invisibles stripped"]
    S3["<b>03_segment</b> — sentence boundaries<br/>terminators <code>[。！？…；]+[」』）》〉]*</code>, <code>、</code> is not a boundary<br/>shape: han ∈ [6, 60], han ratio ≥ 0.65"]
  end

  subgraph origin["origin filter — mandatory"]
    S4["<b>04_script</b> — character inventory<br/>MOE 常用 4 808 / 次常用 6 343 / 罕用 18 356 + 12 exceptions<br/>OpenCC STCharacters ∖ TSCharacters round-trip<br/>converted orthography 裏着衞爲説麽…"]
    S5["<b>05_lexicon</b> — lexical origin<br/>149 hard + 38 soft mainland pairs, corpus-verified<br/>written Cantonese, Hong Kong and Singapore forms, foreign-topic tag, erhua two-window rule<br/>literary-Chinese density > 0.05 with ≥ 2 hits and ≥ 8 han"]
  end

  subgraph corpus["corpus hygiene — optional, source-dependent"]
    S6["<b>06_evidence</b> — block judgment<br/>window 300 sentences; reject block if any member fails<br/>require ≥ 1.0 Taiwan evidence per 100 sentences"]
    S7["<b>07_dedup</b> — exact type dedup<br/>SHA-256 → 62-bit fingerprint<br/>flags types occurring on ≥ 5 distinct hosts as boilerplate"]
  end

  subgraph measure["measurement"]
    S8["<b>08_tokenize</b> — Viterbi over a unigram cost model<br/>EM re-estimation, max word length 8"]
    S9["<b>09_count</b> — n-gram counting by external sort<br/><code>sort -S … --parallel …</code> ∘ <code>uniq -c</code><br/>frequency floor for publication (Google Ngram rule)"]
  end

  S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 --> S9
  S9 --> OUT["frequency, dispersion (DP),<br/>collocations (G², logDice)"]

  classDef must fill:#0b3d2e,stroke:#0b3d2e,color:#fff
  classDef may fill:#3d3a0b,stroke:#3d3a0b,color:#fff
  class S2,S3,S4,S5 must
  class S6,S7 may

Green stages are mandatory for any Taiwan Mandarin corpus. Amber stages depend on the source: block judgment assumes documents are topically coherent, type deduplication assumes the source is web-scale.

Record contract

One JSON object per line. Stages add fields; they never remove them.

field	stage	meaning
`id`, `source`	01	stable identity
`url`, `host`, `gate`	01	provenance and the gate that admitted it
`register`, `doc_score`	01	HPLT web-register argmax, mean segment score
`text`	01–02	document text, then punctuation-normalized
`doc`, `index`, `han`	03	parent document, ordinal, han count
`tier`	04	0 常用, 1 次常用, 2 罕用
`evidence`	06	Taiwan evidence count over the block
`tokens`	08	word segmentation
`ok`, `findings[]`	all	verdict and every reason for it

A finding is {check, code, severity, detail} with severity ∈ {reject, mark}.

Install

Ruby ≥ 3.4. The only runtime dependency is twfilter, pulled from RubyGems:

bundle install
bin/twpipeline stages

External sorting uses gsort (GNU coreutils) when installed and the system sort otherwise; both understand -S and --parallel. Override with TWP_SORT.

Usage

twpipeline run --source hplt --input cmn_Hant/7_1.jsonl.zst --cores 16 --memory 8G
cat raw.jsonl | twpipeline 04_script | twpipeline 05_lexicon > checked.jsonl
echo '這個政策有沒有經過評估？' | twpipeline check
twpipeline run --from 03_segment --to 07_dedup --drop --policy publishable
twpipeline resources
twpipeline stages

--policy corpus (default) keeps everything Taiwan-attested. --policy publishable restricts to MOE 常用 4 808, rejects soft mainland markers, and enforces the strict MOE punctuation inventory — the boundary of what may reach a commercial learner product.

Resources

Cores and memory are read from --cores / --memory, then TWP_CORES / TWP_MEMORY, then autodetection (all cores, half of physical memory). --ask prompts interactively.

Streaming stages fork cores workers per chunk; chunk size is bounded by bytes (memory / 32, capped at 1 GiB, overridable with TWP_CHUNK_BYTES), not by line count, so a source of 12 KiB documents and a source of 40-byte sentences both stay within the same envelope. Counting never holds a hash table: keys go through sort -S <memory> --parallel <cores>, so peak resident size is set by the flag, not by the corpus.

Every stage writes one line to work/benchmarks.jsonl — wall time, CPU seconds, mean cores used, peak RSS, and the stage's own counters. The work directory defaults to ../work beside the checkout; override with TWP_WORK. See BENCHMARKS.md.

Findings

Measured on HPLT 3.0 cmn_Hant, quality bins 7–10 (4 116 754 documents). The admission gate kept 2 001 161 documents: 541 467 by .tw host, 1 459 694 by html_lang. All of them were carried through stages 02–07 in 28 minutes (twfilter 0.1.0, the published tables):

stage	in	out	lost
03_segment	2 001 161 docs	37 772 352 sentences	—
04_script	37 772 352	37 122 017	1.72 %
05_lexicon	37 122 017	36 455 533	1.80 %
06_evidence	36 455 533	29 089 952	20.20 %
07_dedup	29 089 952	12 339 761 types	57.58 %

Three results worth stating plainly.

Deduplication dominates every linguistic filter. More than half of all segmented sentences are exact repeats of another sentence — an order of magnitude more than the three origin filters remove between them. HPLT deduplicates documents, not sentences, and the Taiwanese long tail is largely search-optimized loan and hotel templates. Without stage 07 an n-gram table over this material measures template frequency, not language.

A stratified 279 198-document sample drawn from the head of each bin file put duplication at 71.0 % and evidence loss at 5.6 %; the true figures are 57.6 % and 20.2 %. Documents cluster by host inside a bin file, so a head sample overstates local repetition and understates topical variety. Sample to find the shape of a problem; measure on everything before publishing a number.

The two gates differ, but less than a head sample suggests. Unique Taiwan-attested types produced per admitted document:

gate	documents	unique types	types per document
`.tw` host	541 467	4 230 061	7.81
`html_lang` on a generic TLD	1 459 694	8 109 700	5.56

A 1.41× gradient, not the 2.4× the sample showed. The html_lang gate supplies 65.7 % of the final corpus at about two thirds of the per-document yield — worth keeping, not worth trusting blindly. The gate field is preserved on every record, so restricting to .tw afterwards costs nothing and remains available as a higher-precision subset.

Boilerplate is sub-sentential, so no amount of sentence deduplication removes it. The phrase 提供相關細節的諮詢服務 occurs 263 210 times in the deduplicated corpus, inside 263 210 distinct sentences, across 222 900 documents — but only 1 948 hosts. Sentence dedup cannot see it (every string differs), per-document counting cannot see it (every document differs), and --host-cap cannot see it (the network spans 11 691 throwaway domains, none of them large). 貸款 comes out as the 7th most frequent word in Taiwan Mandarin, which is false.

The fix is to change what a count means. --once-per host counts each n-gram once per host instead of once per occurrence, so a frequency list measures how many independent sites use a phrase rather than how many times a template was published. On the phrase above that is a 135-fold correction. For web-derived frequency this is the correct semantics regardless of spam: a generated page is not an independent act of authorship.

Occurrence counting remains the default, because for an edited corpus — statutes, news, dictionaries — repetition is evidence rather than noise.

Measured effect on the full corpus. 貸款 falls from rank 7 to rank 1 299; 借錢, 包養, 二胎 and 信貸 drop below the 40-host floor and disappear entirely. The 1-gram list becomes 的是有一在了我會也要不個, the 4-gram list 越來越多, 這是一個, 最重要的是, 更重要的是. In the run where both semantics were built, published rows fell by 7 % at order 1 and 28 % at order 4.

Known residual. Eleven of the top hundred 4-grams are fragments of one string: 讓人們盡情分享，將這個世界變得更開闊，聯繫更緊密 — the Chinese Facebook tagline, carried by the social-plugin embed on roughly 3 900 hosts, about once per host, so per-host counting cannot suppress it: those hosts really are independent. This is third-party embedded interface text, a different problem from spam, and it needs either HTML-level boilerplate stripping upstream or an explicit stoplist. It contaminates 0.03 % of sentences but a visible share of the 4-gram head.

The surviving corpus spans 149 022 hosts: top 10 hold 8.8 % of sentences, top 100 20.8 %, top 1000 46.3 %, top 10 000 82.8 %. Diffuse enough to be a corpus rather than a scrape of a few sites, but the single largest host is chinatimes.com at 411 005 sentences — 3.3 % of everything, one newspaper. Concentration of one rightsholder's expression, not total size, is what has drawn takedowns against comparable datasets; cap per host before publishing anything that carries text.

Layout

lib/twpipeline/
  resources.rb   cores and memory resolution
  parallel.rb    fork-based map, byte-bounded streaming
  sorting.rb     external sort, tally with a frequency floor
  jsonl.rb       record contract
  bench.rb       measurement, appended to work/benchmarks.jsonl
  rusage.rb      getrusage(2) via Fiddle
  segmenter.rb   Viterbi, EM re-estimation
  frequency.rb   deviation of proportions
  stage.rb       Stage, MapStage, registry
  stages/        the nine stages
  sources/       source adapters
tools/
  regression.rb  twfilter against every reference corpus

Tests

bundle exec rake

RSpec suite plus RBS signature validation. Fixtures carry no corpus material; the suite needs no data and no credentials.