Gem Version Build Status Pull Requests Commits since latest

Purpose

Commenter is a Ruby gem for working with ISO comment sheets in DOCX and XLSX format.

It provides utilities for parsing, manipulating, and serializing ISO comment data, converting between DOCX/XLSX and structured YAML with schema validation.

The supported input formats are:

  • "ISO/IEC/CEN/CENELEC electronic balloting commenting template/version 2012-03" (DOCX)

  • ISO Online Standards Development (OSD) XLSX exports (resolved and unresolved comment sheets)

  • Redlined Word documents containing tracked changes (e.g. ISO/CS editor redlines), imported with --format redline

This gem only supports plain text comment extraction and filling. Only use this to handle plain text comments and resolutions.

Note
 Mathematical formulas, images, and complex formatting are not supported due to limitations in the underlying docx gem. Comments containing such elements will have their text content extracted, but formatting and embedded objects will be lost.

Installation

Add this line to your application’s Gemfile:

gem 'commenter'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install commenter

Usage

Importing comments

Convert an ISO comment sheet (DOCX or XLSX) to structured YAML. The format is auto-detected from the file extension; redline DOCX must be selected explicitly with --format redline because the extension cannot distinguish a redline from a comment sheet.

# From DOCX (ISO 2012-03 template or OSD DOCX export)
$ commenter import "ISO 80000-2 review comments.docx" -o comments.yaml

# From XLSX (ISO OSD resolved comment export)
$ commenter import "91855-Comments-resolved.xlsx" -o comments.yaml

# From XLSX (ISO OSD comments-only export)
$ commenter import "ISO 5843-6-Comments.xlsx" -o comments.yaml

# From a redlined DOCX with tracked changes (ISO/CS editor redline)
$ commenter import "ISO_DIS 2533 Enquiry Trackchange Word.docx" --format redline \
    --body CS --document "ISO 2533:2026" --stage DIS \
    --observations "Accepted. ISO/CS tracked change accepted." \
    -o comments-cs.yaml

Each tracked change (w:ins, w:del, w:moveFrom, w:moveTo) becomes a comment entry whose proposed_change renders the change itself (Insert: "…​" / Delete: "…​" / Move …​: "…​") and whose locality.clause is resolved from the nearest preceding heading (including sub-clauses such as 4.2.1 and Annex A), with locality.element resolved from the nearest Table/Figure/Formula/NOTE reference in the same clause. Self-closing paragraph-mark markers are skipped because they carry no content.

Reviewer comment threads (w:comment) are emitted after the track changes with -CNNN ids: the remark verbatim in comments, its instruction reworded as the proposed_change (a leading "Please " is dropped), and empty observations for the document owner to draft.

This will create two files:

comments.yaml

The structured comment data

schema/iso_comment_2012-03.yaml or schema/iso_comment_osd.yaml

The YAML schema for validation (selected automatically based on input format)

Import options

$ commenter import input.docx -o comments.yaml --exclude-observations --schema-dir schemas/

Options:

-o, --output FILE

Output YAML file (default: comments.yaml)

-e, --exclude-observations

Skip the observations column

--schema-dir DIR

Directory for schema file (default: schema)

--format FORMAT

Force input format (docx, xlsx or redline; auto-detected by default)

--sheet NAME

XLSX: sheet name to parse (default: first sheet)

--resolved-only

XLSX: use resolved comments sheet only

--unresolved-only

XLSX: use unresolved comments sheet only

--body CODE

Redline: member body code for comment IDs (default: CS)

--observations TEXT

Redline: observations stamped on all track changes

--accept-all

Redline: stamp Accepted. Tracked change accepted. on all track changes (an explicit --observations wins)

--document ID

Redline: document identifier (e.g. ISO 2533:2026)

--stage STAGE

Redline: approval stage (e.g. DIS)

Metadata (extraction limitation)

Warning
 Due to a limitation in the underlying docx gem, metadata fields (Date, Document, Project) from the DOCX header cannot be automatically extracted. This is because the required functionality is not yet merged (see docx gem PR #73).

The generated YAML will have empty metadata fields that you can manually populate:

# yaml-language-server: $schema=schema/iso_comment_2012-03.yaml

version: "2012-03"
date: ""           # Manually add: e.g., "2023-04-25"
document: ""       # Manually add: e.g., "ISO 80000-2:2019"
project: ""        # Manually add: e.g., "Project name"
comments:

Alternatively, you can provide metadata during import using CLI options (planned for future release).

Example YAML output

# yaml-language-server: $schema=schema/iso_comment_2012-03.yaml

version: "2012-03"
date: "2023-04-25"
document: "ISO 80000-2:2019"
project: "Mathematics review"
comments:
  - id: DE-001
    body: DE
    locality:
      line_number:
      clause: "_whole document"
      element:
    type: general
    comments: |
      The document should include more examples
      to clarify the implementation requirements.
    proposed_change: |
      Add section 4.5 with practical examples
      showing typical use cases.
    observations: |
      Accepted. Examples will be added in the
      next revision.
  - id: US-002
    body: US
    locality:
      line_number: "45"
      clause: "5.2.1"
      element: "Table 3"
    type: technical
    comments: |
      The values in Table 3 appear to be
      inconsistent with the formula in 5.1.
    proposed_change: |
      Correct the values in column 2 of Table 3
      to match the calculation method.
    observations:

Filling DOCX templates from YAML

This gem contains a command-line utility to fill a DOCX template with comments from a YAML file. It generates a filled comment sheet that can be used for review and resolution tracking.

The base template is the ISO comment sheet template located at data/iso_comment_template_2012-03.docx. You can also provide a custom template file using the --template option.

Syntax:

$ commenter fill comments.yaml -o filled_comments.docx

Fill options

Options:

-o, --output FILE

Output DOCX file (default: filled_comments.docx)

-t, --template FILE

Custom template file

-s, --shading

Apply status-based cell shading

GitHub Integration

Overview

The commenter gem provides comprehensive GitHub integration for collaborative comment review and tracking. The workflow consists of two main commands:

  • github-create - Creates GitHub issues from comments and tracks them in YAML

  • github-retrieve - Retrieves final observations from closed GitHub issues

Creating GitHub issues from comments

Basic usage
$ commenter github-create --config github_config.yaml comments.yaml

This command creates GitHub issues for each comment and automatically updates the YAML file with GitHub issue information (issue numbers, URLs, status, timestamps).

Configuration file

Create a GitHub configuration file to specify repository, authentication, and issue settings:

github:
  repository: "owner/repo-name"
  token: "ghp_xxxxxxxxxxxx"
  milestone:
    name: "ISO 80000-2 DIS Review"
  default_labels: ["comment-review", "iso-standard"]
  stage_labels:
    WD: ["working-draft"]
    DIS: ["draft-international-standard"]

A sample configuration file is provided at data/github_config_sample.yaml.

Issue templates

General

The gem uses Liquid templates to format GitHub issue titles and bodies. Default templates are provided, but you can customize them.

Title template variables

The title template (data/github_issue_title_template.liquid) supports these variables:

  • stage - Approval stage (WD/CD/DIS/FDIS/PRF/PUB)

  • document - Document identifier

  • comment_id - Comment identifier

  • brief_summary - Generated summary combining locality and description

  • body - Member body abbreviation

  • type - Comment type code

  • clause, element, line_number - Location information

  • unique_id - Rendered unique identifier (see Duplicate detection)

Body template variables

The body template (data/github_issue_body_template.liquid) supports all title variables plus:

  • comments - Full comment text

  • proposed_change - Proposed change text

  • observations - Secretariat observations

  • has_observations - Boolean for conditional rendering

  • has_proposed_change - Boolean for conditional rendering

  • project - Project name

  • date - Comment sheet date

  • version - Template version

  • type_full_name - Full comment type name (General/Technical/Editorial)

  • locality_summary - Formatted locality string

Custom templates

Create custom templates and reference them in your configuration:

github:
  templates:
    title: "custom_title.liquid"
    body: "custom_body.liquid"
    unique_id: "[{{ stage | upcase }}] {{ comment_id }}"  # Optional: customize unique ID

Command options

  • -c, --config FILE - GitHub configuration YAML file (required)

  • --stage STAGE - Override approval stage

  • --milestone NAME - Override milestone name

  • --assignee HANDLE - Override assignee GitHub handle

  • --title-template FILE - Custom title template

  • --body-template FILE - Custom body template

  • --dry-run - Preview issues without creating them

Examples

Create issues with custom stage:

$ commenter github-create --config github_config.yaml --stage DIS comments.yaml

Preview issues before creation:

$ commenter github-create --config github_config.yaml --dry-run comments.yaml

Security considerations

For security reasons, it is recommended to use the GITHUB_TOKEN environment variable instead of storing the token in the configuration file.

$ export GITHUB_TOKEN=ghp_xxxxxxxxxxxx
$ commenter github comments.yaml --config github_config.yaml

Duplicate detection

The gem automatically checks for existing issues to avoid duplicates. The duplicate detection uses a configurable unique identifier that is:

  1. Rendered from a Liquid template

  2. Embedded in the issue title for human identification

  3. Used as the search pattern to find existing issues

This ensures that the search pattern always matches what appears in the title, preventing synchronization issues.

Unique ID template

The unique identifier is defined by the unique_id template in your configuration:

github:
  templates:
    # Default: "[STAGE] COMMENT_ID"
    unique_id: "[{{ stage | upcase }}] {{ comment_id }}"

The default renders to: [DIS] GB-001

Why stage-aware unique IDs?

ISO standards go through multiple ballot stages (WD, CD, DIS, FDIS). The same comment ID (e.g., GB-001) appears at each stage, but represents different comments:

| Stage | Comment ID | Unique ID | Same Issue? | |-------|------------|-----------|-------------| | CD | GB-001 | [CD] GB-001 | No | | DIS | GB-001 | [DIS] GB-001 | No |

Without stage-aware unique IDs, CD GB-001 and DIS GB-001 would be treated as duplicates, causing the DIS comment to be skipped.

Custom unique ID patterns

You can customize the unique ID pattern to match your title template format:

github:
  templates:
    # Different order
    unique_id: "{{ comment_id }} [{{ stage | upcase }}]"

    # With document prefix
    unique_id: "{{ document }} - {{ comment_id }}"

    # With body prefix (for multi-body projects)
    unique_id: "{{ body }}/{{ comment_id }}"

    # Single stage project (no stage needed)
    unique_id: "{{ comment_id }}"
Using unique_id in templates

The unique_id is available as a variable in title and body templates:

{{ unique_id }}: {{ brief_summary }} {% if document %}({{ document }}){% endif %}

This renders to: [DIS] GB-001: Clause 5.1 summary (ISO/DIS 2533)

Retrieving observations from GitHub issues

Basic usage

After GitHub issues have been created and reviewed, use the github-retrieve command to extract final observations from closed issues:

$ commenter github-retrieve --config github_config.yaml comments.yaml

This command:

  • Reads GitHub issue information from the YAML file (no searching required)

  • Fetches observations from closed GitHub issues only

  • Updates the same YAML file with extracted observations

  • Preserves all existing comment data

Magic comment syntax

To provide official observations in GitHub issues, use markdown blockquotes with special markers:

> **OBSERVATION:**
> Accepted. The proposed change will be included in clause 5.2.1 of the next revision.
> Technical details have been reviewed and approved by the working group.

Alternative shorter syntax:

> **COMMENTER OBSERVATION:**
> Noted. This will be considered for future revisions.
Retrieval options

  • -c, --config FILE - GitHub configuration YAML file (required)

  • -o, --output FILE - Output YAML file (default: update original)

  • --include-open - Include observations from open issues (not recommended)

  • --dry-run - Preview observations without updating YAML

Examples

Preview observations before updating:

$ commenter github-retrieve --config github_config.yaml --dry-run comments.yaml

Save to new file instead of updating original:

$ commenter github-retrieve --config github_config.yaml -o final_comments.yaml comments.yaml

Include observations from open issues:

$ commenter github-retrieve --config github_config.yaml --include-open comments.yaml
Enhanced YAML structure

After using github-create, your YAML file will include GitHub integration information:

comments:
  - id: US-001
    body: US
    locality:
      clause: "5.2.1"
      element: "Table 3"
    type: technical
    comments: "The values in Table 3 appear inconsistent..."
    proposed_change: "Correct the values in column 2..."
    observations: ""
    github:
      issue_number: 123
      issue_url: "https://github.com/owner/repo/issues/123"
      status: "open"
      created_at: "2024-01-15T10:30:00Z"

After using github-retrieve (when the issue is closed):

comments:
  - id: US-001
    # ... other fields unchanged ...
    observations: "Accepted. The proposed change will be included in clause 5.2.1."
    github:
      issue_number: 123
      issue_url: "https://github.com/owner/repo/issues/123"
      status: "closed"
      created_at: "2024-01-15T10:30:00Z"
      updated_at: "2024-01-20T14:45:00Z"
Configuration for retrieval

The GitHub configuration file supports retrieval-specific settings:

github:
  repository: "owner/repo-name"
  retrieval:
    # Magic comment markers to look for
    observation_markers:
      - "**OBSERVATION:**"
      - "**COMMENTER OBSERVATION:**"

    # Fallback to last comment if no magic comment found
    fallback_to_last_comment: true

    # Only retrieve from closed issues (recommended)
    closed_issues_only: true

Comment ID format

Typical comment IDs follow the pattern: {MB/NC}-{number} or {MB/NC}-{org_id}-{seq_id}

  • US-001 - First comment from ANSI (US)

  • DE-01-002 - Second comment from organization 01 within DIN (DE)

  • **-001 - First comment from ISO secretariat

Where:

  • US = ANSI (American National Standards Institute)

  • DE = DIN (Deutsches Institut für Normung)

  • ** = ISO Secretariat

  • CC = CalConnect

Comment types

The comment types are defined as follows:

ge

General comment

te

Technical comment

ed

Editorial comment

Note
 In generated YAML, types are stored in expanded form (general, technical, editorial). Short codes are accepted as input anywhere and are expanded when the comment is loaded.

Workflow integration

flowchart LR
    A1[ISO Comment Sheet DOCX] --> B[commenter import]
    A2[ISO OSD XLSX export] --> B
    B --> C[YAML + Schema]
    C --> D[commenter github-create]
    D --> E[YAML + GitHub Info]
    E --> F[GitHub Issues + Review Process]
    F --> G[commenter github-retrieve]
    G --> H[YAML + Observations]
    H --> I[commenter fill]
    I --> J[Final DOCX]
    J --> K[ISO Secretariat]

Shading rules

When the --shading option is used, the following status patterns are recognized and applied to the observations column:

Status Pattern Intended Color Hex Code Example

accept(ed)?

Green

#92D050

"Accepted"

awm|accept with modifications

Olive Green

#C4D79B

"Accept with modifications"

noted

Blue

#8DB4E2

"Noted"

reject(ed)?

Pink

#FF99CC

"Rejected"

todo

Diagonal stripes

#D9D9D9

"TODO: Review"

Data model

ISO 2012-03 template (DOCX)

The comment structure follows this schema:

version: "2012-03"  # Template version
date: string | null # Comment sheet date (manually populated)
document: string | null # Document being reviewed (manually populated)
project: string | null  # Project name (manually populated)
comments:           # Array of comment objects
  - id: string      # Comment identifier
    body: string    # Member body abbreviation
    locality:       # Location information
      line_number: string | null
      clause: string
      element: string | null
    type: "ge" | "te" | "ed"  # Comment type
    comments: string          # Comment text
    proposed_change: string   # Proposed solution
    observations: string | null  # Secretariat observations (optional)
    github:         # GitHub integration information (optional)
      issue_number: integer   # GitHub issue number
      issue_url: string       # GitHub issue URL
      status: "open" | "closed"  # GitHub issue status
      created_at: string      # ISO 8601 timestamp
      updated_at: string      # ISO 8601 timestamp (optional)

ISO OSD format (XLSX)

The OSD format includes additional metadata and resolution fields extracted from the ISO Online Standards Development platform:

version: "osd"           # OSD format identifier
date: string             # Date from XLSX header (auto-extracted)
document: string         # Document reference, e.g. "ISO/DIS 5843-6(en)"
project: string          # Project name (auto-extracted)
stage: string            # Stage: WD, CD, DIS, FDIS, PRF, PUB (auto-extracted)
title_en: string         # Document title in English (auto-extracted)
title_fr: string         # Document title in French (auto-extracted)
comments:
  - id: string           # Comment ID (numeric from OSD)
    body: string         # User name / member body
    locality:
      clause: string     # Clause number
      element: string    # Clause title
    type: "ge" | "te" | "ed"
    comments: string     # Comment text
    proposed_change: string
    observations: string # Built from resolution_status + motivation
    user_name: string         # OSD user name
    comment_type: string      # Subtype: Editorial, General, Technical
    resolution_status: string # Accepted, Partially accepted, Rejected, etc.
    resolution_date: string   # Date resolved
    feedbacks: string         # Discussion replies
    motivation: string        # Resolution justification
    created_date: string      # Date comment was created
    stage_code: string        # ISO stage code (e.g. "40.20")

The OSD XLSX format comes in two variants, both auto-detected:

Resolved: 17 columns starting with Comment ID, includes resolution data (Resolution status, Motivation, Resolution Date, Stage code)

Unresolved/comments-only: 15 columns starting with User name, includes Comment type, Comment/Motivation, Replies, Comment number

Schema validation

Each exported YAML file includes a schema reference for IDE support. The schema is selected automatically based on the input format:

# For ISO 2012-03 DOCX imports:
# yaml-language-server: $schema=schema/iso_comment_2012-03.yaml

# For ISO OSD XLSX imports:
# yaml-language-server: $schema=schema/iso_comment_osd.yaml

This enables:

  • Auto-completion in VS Code and other editors

  • Real-time validation

  • Inline documentation

Development

Getting started

After checking out the repo, run bin/setup to install dependencies:

$ git clone https://github.com/metanorma/commenter.git
$ cd commenter
$ bin/setup

Then, run rake spec to run the tests:

$ bundle exec rake spec

You can also run bin/console for an interactive prompt that will allow you to experiment:

$ bin/console

Running tests

The gem includes comprehensive test coverage for all major components:

# Run all tests
$ bundle exec rspec

# Run specific test files
$ bundle exec rspec spec/commenter/comment_spec.rb
$ bundle exec rspec spec/commenter/comment_sheet_spec.rb
$ bundle exec rspec spec/commenter/github_integration_spec.rb

# Run tests with coverage
$ bundle exec rspec --format documentation

Testing GitHub integration

To test the GitHub integration features:

  1. Create a test repository on GitHub

  2. Generate a personal access token with appropriate permissions

  3. Create a test configuration file:

github:
  repository: "your-username/test-repo"
  default_labels: ["test-comment"]
  default_assignee: "your-username"

  1. Test with dry-run mode first:

$ GITHUB_TOKEN=your_token bundle exec exe/commenter github test_comments.yaml --config test_config.yaml --dry-run
Note
 For testing template rendering and dry-run functionality without a real GitHub token, you can use a dummy token: 
$ GITHUB_TOKEN=dummy_token bundle exec exe/commenter github test_comments.yaml --config test_config.yaml --dry-run

This allows you to test the issue preview functionality, template rendering, and configuration parsing without making actual GitHub API calls.

Code structure

The gem is organized into several key components:

Core classes

Commenter::Comment

Represents individual comments with locality, type, and content

Commenter::CommentSheet

Container for multiple comments with metadata

Commenter::Parser

Handles DOCX and XLSX parsing with auto-detection

Commenter::Parser::OsdXlsxParser

Parses ISO OSD XLSX exports (resolved and unresolved variants)

Commenter::Filler

Fills DOCX templates with comment data

Commenter::GitHubIssueCreator

Creates GitHub issues from comments

CLI interface

Commenter::Cli

Thor-based command-line interface with subcommands:

  • import - Convert DOCX or XLSX to YAML

  • fill - Fill DOCX template from YAML

  • github-create - Create GitHub issues from comments

  • github-retrieve - Retrieve observations from GitHub issues

Templates and configuration

  • data/iso_comment_template_2012-03.docx - Base DOCX template

  • data/github_issue_title_template.liquid - GitHub issue title template

  • data/github_issue_body_template.liquid - GitHub issue body template

  • data/github_config_sample.yaml - Sample GitHub configuration

  • schema/iso_comment_2012-03.yaml - YAML schema for ISO 2012-03 validation

  • schema/iso_comment_osd.yaml - YAML schema for ISO OSD format validation

Debugging

Enable debug output for troubleshooting:

# Enable verbose output
$ bundle exec exe/commenter import input.docx --verbose

# Debug GitHub API calls
$ OCTOKIT_DEBUG=true bundle exec exe/commenter github comments.yaml --config config.yaml --dry-run

Troubleshooting

Common issues

DOCX parsing errors

  • Ensure the DOCX file follows the ISO comment template format

  • Check for corrupted or password-protected files

  • Verify table structure matches expected format

GitHub API errors

  • Verify your GitHub token has appropriate permissions

  • Check rate limiting if making many requests

  • Ensure repository exists and is accessible

Template rendering errors

  • Validate Liquid template syntax

  • Check that all referenced variables are available

  • Test templates with sample data first

Schema validation errors

  • Ensure YAML follows the required structure

  • Check for missing required fields

  • Validate comment ID format

Getting help

  • Check the issue tracker on GitHub

  • Review existing test cases for usage examples

  • Run commands with --help for detailed options

  • Use --dry-run mode to preview operations safely

Copyright

This gem is developed, maintained and funded by Ribose

License

The gem is available as open source under the terms of the 2-Clause BSD License.