Class: Opencdd::Parcel::ScrapeVerifier

Inherits:
Object
  • Object
show all
Defined in:
lib/opencdd/parcel/scrape_verifier.rb

Overview

Verifies that scrape output produced populated .xls files, not just schema stubs. Catches the silent failure mode discovered on 2026-07-08: iec61360/iec63213/iec61360-7 scrapes appeared complete (10,338 PROPERTY .xls files for iec61360) but every file had 0 data rows. The build pipeline exported 574 classes

  • 0 properties, and the browser showed "No declared properties" for every class. Nothing failed loudly.

This verifier reads every export_*.xls under a directory and reports which ones are empty. Use via rake verify_scrape[dict] or directly:

results = Opencdd::Parcel::ScrapeVerifier.verify("downloads/iec61360")
empty = results.select(&:empty?)
abort "#{empty.size} empty files" unless empty.empty?

Defined Under Namespace

Classes: Result

Constant Summary collapse

FILE_PATTERN = 
/\Aexport_(CLASS|PROPERTY|RELATION|UNIT|VALUELIST|VALUETERMS)_[^\.]+\.(xls|xlsx)\z/i

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(directory) ⇒ ScrapeVerifier

Returns a new instance of ScrapeVerifier.



37
38
39
 
# File 'lib/opencdd/parcel/scrape_verifier.rb', line 37

def initialize(directory)
  @directory = directory
end

Instance Attribute Details

#directory ⇒ Object (readonly)

Returns the value of attribute directory.



35
36
37
 
# File 'lib/opencdd/parcel/scrape_verifier.rb', line 35

def directory
  @directory
end

Class Method Details

.verify(directory) ⇒ Object 



30
31
32
 
# File 'lib/opencdd/parcel/scrape_verifier.rb', line 30

def verify(directory)
  new(directory).verify
end

Instance Method Details

#data_row?(row) ⇒ Boolean

A "data row" in the Parcel wire format is a row with content in column 1+ whose first cell isn't a # directive or comment. Column 0 may be empty (the IEC CDD .xls layout leaves it blank; data starts in column 1). Public so tests can verify the rule without send.

Returns:

  • (Boolean) 


62
63
64
65
66
67
 
# File 'lib/opencdd/parcel/scrape_verifier.rb', line 62

def data_row?(row)
  return false unless row.is_a?(Array) && row.size > 1
  first = row.first.to_s.strip
  return false if first.start_with?("#")
  row[1..].any? { |c| !c.nil? && !c.to_s.strip.empty? }
end

#summary ⇒ Object

Summary counts — useful for CLI output.



46
47
48
49
50
51
52
53
54
55
 
# File 'lib/opencdd/parcel/scrape_verifier.rb', line 46

def summary
  results = verify
  {
    total:    results.size,
    ok:       results.count(&:ok?),
    empty:    results.count(&:empty?),
    by_type:  results.group_by(&:entity_type).transform_values(&:size),
    empty_by_type: results.select(&:empty?).group_by(&:entity_type).transform_values(&:size),
  }
end

#verify ⇒ Object 



41
42
43
 
# File 'lib/opencdd/parcel/scrape_verifier.rb', line 41

def verify
  xls_files.map { |f| check_file(f) }
end