twfilter

Provenance analysis for Chinese text: is this span Taiwan Mandarin (臺灣華語), and if not, on what evidence. Pure Ruby, no runtime dependencies, no native extensions, no network, deterministic output.

Traditional Chinese is written in Taiwan, Hong Kong and Macau, and by mainland publishers converting from simplified. Script alone does not identify origin. Every string below is written in traditional characters, and only three of them are Taiwan Mandarin.

TWFilter.keep?("這個政策有沒有經過完整評估？")     # => true
TWFilter.keep?("请把这个信息转发给同事")           # => false  simplified script
TWFilter.keep?("請把這個信息轉發給同事")           # => false  lexicon: 信息, Taiwan writes 資訊
TWFilter.keep?("他站在門裏面等了很久")             # => false  orthography: 裏, Taiwan writes 裡
TWFilter.keep?("佢哋唔知道發生咗咩事")             # => false  Cantonese particles
TWFilter.keep?("這事兒辦得挺漂亮的")               # => false  erhua suffix on 事
TWFilter.keep?("他的女兒今年考上大學")             # => true   兒 here is a root morpheme

Depth matters as much as breadth. Conversion tables are not provenance signals, and subject matter is not provenance either:

# 程序 is ordinary Taiwanese legal vocabulary, not a mainland form
TWFilter.keep?("行政程序法所定之程序應由主管機關進行")   # => true

# 了嗎 is attested in every native Taiwanese source; it is not a marker
TWFilter.keep?("你吃飯了嗎？我們等一下再出發。")         # => true

# Ambiguous vocabulary is flagged for review, not discarded
TWFilter.examine("這個視頻的畫質很不錯").marks.map(&:code)     # => [:mainland_soft]

# Taiwanese reporting about China is Taiwan Mandarin; the topic is tagged, not rejected
TWFilter.examine("行政院昨天回應了北京的最新聲明").marks.map(&:code)   # => [:prc_topic]

# 南京 is also a Taipei street; an address suffix suppresses the topic tag
TWFilter.examine("臺北市南京東路五段的捷運站出口").findings   # => []

Provenance is decided on orthography, character inventory, lexical choice, morphology and a positive evidence signal, and every criterion that fired is reported.

What it does and does not do

Does — decide provenance of a span of text; normalize punctuation to the Taiwan standard; segment into sentences; report a character-difficulty tier; report every reason for every decision.

Does not — segment words, tag parts of speech, detect language (assume Han script input), translate, convert between simplified and traditional, or judge grammaticality. A span rejected here is not ungrammatical; it is not Taiwanese.

Applications

  1. Learner-facing corpora. Extracted from TaiwanCards, where it gates every sentence, gloss and collocation before it can reach a card. The publishable policy encodes the boundary of the pedagogical inventory: MOE 常用 4 808 only, no ambiguous vocabulary, MOE punctuation only.
  2. Web corpus construction. Filtering crawl output where the language identifier reports zho_Hant but the provenance is unknown. Traditional-script crawls conflate four written standards; this separates them at 10⁴ sentences per second per core.
  3. Training-set curation. Building or auditing a monolingual set for a model intended to produce Taiwan Mandarin rather than converted mainland text. Report#findings is machine-readable, so exclusions stay attributable.
  4. Editorial and localization QA. Detecting mainland vocabulary, orthography and punctuation in copy that is supposed to be Taiwanese — the mainland_soft band is designed for review queues rather than automatic rejection.

Decision procedure

flowchart TD
  T["input span"] --> N["Punctuation.normalize<br/>教育部《重訂標點符號手冊》"]
  N --> SH

  subgraph SH["Checks::Shape"]
    A1["han count ∈ policy.han_range"]
    A2["han ratio ≥ policy.min_han_ratio"]
    A3["inventory ⊆ Han ∪ Latin ∪ Bopomofo ∪ punctuation"]
  end

  SH --> SC
  subgraph SC["Checks::Script — character inventory"]
    B1["simplified-only set<br/>OpenCC STCharacters ∖ TSCharacters ∖ Taiwan variants"]
    B2["converted orthography 裏着衞爲説麽綫峯牀墻眞啓衆産囯覈麪"]
    B3["MOE tier ≤ policy.max_tier"]
  end

  SC --> LX
  subgraph LX["Checks::Lexicon, Erhua, Wenyan"]
    C1["mainland hard ×168 → reject"]
    C2["mainland soft ×36 → mark"]
    C3["Cantonese particles ×19 → reject"]
    C4["mainland topics ×60 → mark"]
    C5["erhua: 兒 outside a headed or tailed window → reject"]
    C6["literary density > 0.05 → reject"]
  end

  LX --> EV["Evidence — Taiwan lexicon, particles, institutions,<br/>有 + V perfective, A-not-A"]
  EV --> R["Report{ok?, findings[], tier, han, evidence}"]

Positive evidence never rejects. 了嗎 is not a defect: it is attested in every native source measured. 有 as a perfective marker counts in favour, never against.

Input and output

Input — anything responding to to_s. Empty, whitespace-only, Latin-only and supplementary-plane input are reported, not raised on. Invalid byte sequences must be scrubbed by the caller. There is no length limit; segmentation and normalization are linear in input length.

Output — TWFilter::Report:

member meaning
ok? true iff no finding has severity :reject
findings every criterion that fired, in check order
rejects, marks the two severity partitions; their union is findings
tier highest MOE chart index reached: 0 常用, 1 次常用, 2 罕用, nil if unlisted
han Han character count
evidence count of positive Taiwan signals
reasons rejects rendered as "code: detail" strings
to_h plain data, JSON-serializable

A Finding is {check, code, severity, detail}. Codes are stable across releases; severity depends on the policy. Nothing is ever discarded silently — keep? is a convenience over examine, not a separate path.

Policies

Immutable value objects. Derive with #with.

corpus publishable permissive
han count 6–60 6–40 1–∞
han ratio ≥ 0.65 ≥ 0.70 0
character tier ≤ 罕用 ≤ 常用 4 808 ≤ 罕用
soft lexical markers mark reject mark
mainland topics mark reject mark
punctuation inventory wide MOE strict wide
converted orthography reject reject reject 
TWFilter::Policy.corpus.with(max_tier: :secondary, han_range: (4..80))

Usage

require "twfilter"

# Provenance, with reasons
report = TWFilter.examine("请把这个信息转发给同事")
report.ok?                    # => false
report.reasons                # => ["simplified: 请这个转", "unlisted: 请这个转", "mainland: 信息"]
report.findings.map(&:code)   # => [:simplified, :unlisted, :mainland]

# Editorial review rather than rejection
report = TWFilter.examine("這個程式的運行狀況非常穩定")
report.ok?                    # => true
report.marks.map(&:detail)    # => ["運行"]
TWFilter::Checks::Lexicon.taiwan_form("運行")   # => "執行"

# Pedagogical gate
TWFilter.keep?("他每天都會去公園散步運動。", policy: TWFilter::Policy.publishable)  # => true
TWFilter.keep?("饕餮紋飾在青銅器上十分常見。", policy: TWFilter::Policy.publishable)  # => false

# Punctuation to the Taiwan standard
TWFilter.normalize("他說“好”...")        # => "他說「好」……"
TWFilter.normalize("馬丁·路德ＰＤＦ")     # => "馬丁‧路德PDF"

# Segmentation: 。！？…； terminate, closers stay attached, 、 does not break
TWFilter.sentences("他說：「好。」我走了。")   # => ["他說：「好。」", "我走了。"]

# Character difficulty
TWFilter.tier("散步")   # => 0
TWFilter.tier("饕餮")   # => 1

# Block judgement — a paragraph is accepted as a unit or not at all
TWFilter::Block.judge(paragraph_sentences)

# Table fingerprint, to record beside derived measurements
TWFilter.provenance
# => {version: "1.0.0", dir: "…/lib/twfilter/data", tables: 19, digest: "14cd9fd7fc97dc9a"}

Placement in a pipeline

Normalize before segmenting: the ellipsis and dash rules change sentence boundaries. Segment before examining: the shape thresholds are defined on sentences. Examine before deduplicating and counting, so rejected material never reaches the statistics.

acquire → normalize → segment → examine → deduplicate → tokenize → count
             │           │         │
       Punctuation   Sentences   Report

Block.judge is an optional stage between segmentation and examination for sources whose documents are topically coherent: it rejects a whole window when any member fails, and requires a minimum density of positive evidence across the window. Use it on crawls, not on dictionaries.

Performance

Single core, Apple M5 Max, on 199 936 sentences of Taiwanese web text (6.2 M characters):

operation throughput per 10⁷ sentences
Sentences.split 741 000 /s · 68 MB/s 13 s
Punctuation.normalize 67 000 /s · 6.1 MB/s 149 s
tier 134 000 /s · 12 MB/s 75 s
examine (all five checks) 10 700 /s · 1.0 MB/s 934 s

Suitable for corpora of arbitrary size. Cost is linear in input length and dominated by Checks::Lexicon, which scans 264 union-compiled patterns per span; the other four checks together account for under a third of the total. Memory is constant — tables load once (≈ 3 MB resident) and every entry point is a pure function over a string, so the work parallelizes by fork or by thread without coordination.

A 37.7 M-sentence corpus takes ≈ 3.5 minutes of wall time across 18 cores. In practice I/O, JSON parsing and deduplication dominate; this library is not usually the bottleneck.

Reference tables

lib/twfilter/data/ — plain text, one record per line, tab-separated where a record has fields. Diffable, reviewable and versioned with the gem.

table rows content
moe_common/secondary/rare.txt 4 808 / 6 343 / 18 356 教育部標準字體表
moe_exception.txt 12 dictionary headword characters outside all three charts
simplified_only.txt 3 783 derived from the OpenCC round-trip
variants_used_in_taiwan.txt 18 simplified-looking forms standard in Taiwan
converted_orthography.txt 17 mainland traditional variants
mainland_hard.tsv 168 rejecting pairs, with attestation counts
mainland_soft.tsv 36 marking pairs, ambiguous or dominated
mainland_exceptions.tsv 12 superstrings in which a marker is legitimate
cantonese.txt 19 Cantonese-exclusive particles and pronouns
prc_topics.txt 60 mainland toponyms and institutions — subject matter, not provenance
taiwan_markers/lexicon/particles/grammar.txt 67 / 65 / 12 / 35 positive evidence
erhua_headed/tailed.txt 14 / 36 兒 as a root morpheme
wenyan.txt 4 literary particles for density estimation
MANIFEST.json — row count and SHA-256 per table

Grading. Candidate pairs are graded by attestation ratio against a reference corpus of natively Taiwanese sources: hard when the mainland form is unattested and the Taiwan form occurs at least three times; soft when the Taiwan form dominates by 20 : 1 or the pair is hand-graded ambiguous; excluded otherwise. Most candidates are excluded — conversion tables built for orthographic mapping contain many pairs where both forms are ordinary Taiwanese.

Topics are not provenance. prc_topics.txt is a subject-matter tag. Taiwanese media covers the mainland constantly, several entries are ordinary ROC administrative vocabulary, and Taipei streets are named after mainland cities. It marks under corpus and rejects only under publishable, where excluding foreign subject matter is an editorial decision.

Influencing the criteria

Preferred: contribute to the published dataset. The tables are maintained at huggingface.co/datasets/taiwan-corpora/twfilter-tables, where each release carries its grading evidence, and the gem vendors a copy of a specific version. A correction accepted there reaches every consumer at the next release, with a record of what changed and why — which a local edit does not.

Three table sources are supported. All three are reported by provenance.

Bundled — the default, versioned with the gem, the only configuration whose results are comparable with anyone else's.

Replacement — a directory holding the full set, e.g. a checkout of the dataset newer than the gem:

TWFilter::Tables.dir = "path/to/twfilter-tables"   # or TWFILTER_DATA_DIR

Overlay — the bundled set plus your additions. Filename suffixes select the operation:

file in the overlay directory effect on mainland_hard.tsv
mainland_hard.add.tsv rows appended
mainland_hard.remove.tsv rows removed, matched on the full row or its first field
mainland_hard.tsv table replaced 
TWFilter::Tables.overlay = "path/to/house-style"    # or TWFILTER_OVERLAY_DIR

Row format is validated on load and a violation raises TWFilter::InvalidTableError:

shape tables row
:chars the MOE charts, simplified_only, variants_used_in_taiwan, converted_orthography, cantonese, wenyan one Han character
:pairs mainland_hard, mainland_soft, mainland_exceptions two or more tab-separated fields, first non-empty
:terms everything else, including tables you invent one term, no tabs

Files named *.remove.* carry keys only and are not shape-checked. UTF-8, one record per line, no header, no comments; blank lines ignored.

Record provenance alongside any measurement derived this way. Two corpora filtered with different tables are not comparable, and the digest is what makes that detectable.

Policy thresholds are the supported knob and need no table changes:

lenient = TWFilter::Policy.corpus.with(soft_lexicon_rejects: false, prc_topics_reject: false)

Development

bundle install
bundle exec rake          # specs and RBS validation
bundle exec rake tables   # regenerate the tables and their manifest

Specs require no data, no network and no credentials.

License

Code: MIT. Tables: mixed, all permitting commercial use. See NOTICES.md for per-table provenance and terms.

Two attribution strings must be reproduced by anything shipping this gem:

數位發展部，CNS11643中文標準交換碼全字庫網站
OpenCC — https://github.com/BYVoid/OpenCC, Apache License 2.0

No table carries a NonCommercial, ShareAlike or copyleft term. Verdicts and counts produced by the gem are the caller's, not derivatives of the tables.