Class: GitlabQuality::TestTooling::CodeCoverage::ClickHouse::PerTestCoverageTable

Inherits:
Table
  • Object
show all
Defined in:
lib/gitlab_quality/test_tooling/code_coverage/click_house/per_test_coverage_table.rb

Overview

Inserts per-test, per-source-file line-set coverage rows into ‘code_coverage.test_coverage_per_file`. The `covered_lines` column is `AggregateFunction(groupBitmap, UInt32)` so JSONEachRow can’t carry it; rows go in via raw ‘INSERT … VALUES` statements wrapping `bitmapBuild(CAST([line, …] AS Array(UInt32)))` per row.

Dedup across runs is handled by the table’s ‘SharedReplacingMergeTree(version)` engine on `(ci_project_path, test_file, source_file)` ORDER BY. Within a single run, callers must pre-aggregate at the (test_file, source_file) grain before pushing: multiple examples within the same test_file should be unioned into one row by the loader, not handed in as duplicates.

Constant Summary collapse

TABLE_NAME = 
"test_coverage_per_file"
BATCH_SIZE = 
500
MAX_LINE_NUMBER =

Intentionally generous ceiling on line numbers. Real source files are thousands of lines; generated artifacts (large GraphQL schemas, bundled JS, JSON manifests) can run past 100k. The cap is set to flag clearly bogus values (negative, garbage casts, anything past ~1M) without rejecting realistic generated files. ClickHouse’s UInt32 ceiling is ~4.3B, so we still have orders of magnitude of headroom above this. Tighten only with evidence.

1_000_000

Instance Method Summary collapse

Methods inherited from Table

#initialize

Constructor Details

This class inherits a constructor from GitlabQuality::TestTooling::CodeCoverage::ClickHouse::Table

Instance Method Details

#push(data) ⇒ void

This method returns an undefined value.

Parameters:

  • data (Array<Hash>) —

    one entry per (test_file, source_file). Each entry needs: :test_file [String] :source_file [String] :covered_lines [Array<Integer>] line numbers covered by this test on this file :total_lines [Integer] executable lines in the source file :feature_category, :group, :stage, :section [String, optional]



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
 
# File 'lib/gitlab_quality/test_tooling/code_coverage/click_house/per_test_coverage_table.rb', line 40

def push(data) # rubocop:disable Metrics/AbcSize
  return logger.warn("#{LOG_PREFIX} No data found, skipping ClickHouse export!") if data.empty?

  logger.debug("#{LOG_PREFIX} Starting per-test coverage export to ClickHouse")
  sanitized_data = sanitize(data)

  return logger.warn("#{LOG_PREFIX} No valid data found after sanitization, skipping ClickHouse export!") if sanitized_data.empty?

  total_batches = (sanitized_data.size.to_f / BATCH_SIZE).ceil
  sanitized_data.each_slice(BATCH_SIZE).with_index do |batch, index|
    logger.debug("#{LOG_PREFIX} Pushing batch #{index + 1} of #{total_batches} (#{batch.size} rows)")
    client.query(build_insert_sql(batch), format: "TabSeparated")
  end
  logger.info("#{LOG_PREFIX} Successfully pushed #{sanitized_data.size} records to #{full_table_name}")
rescue StandardError => e
  logger.error("#{LOG_PREFIX} Error occurred while pushing data to #{full_table_name}: #{e.message}")
  raise
end