Class: GitlabQuality::TestTooling::CodeCoverage::ClickHouse::PerTestCoverageTable

Inherits:
Table
  • Object
show all
Defined in:
lib/gitlab_quality/test_tooling/code_coverage/click_house/per_test_coverage_table.rb

Overview

Inserts per-test, per-source-file line-set coverage rows into code_coverage.test_coverage_per_file. The covered_lines column is AggregateFunction(groupBitmap, UInt32) so JSONEachRow can't carry it; rows go in via raw INSERT ... VALUES statements wrapping bitmapBuild(CAST([line, ...] AS Array(UInt32))) per row.

Dedup across runs is handled by the table's SharedReplacingMergeTree(version) engine on (ci_project_path, test_file, source_file) ORDER BY. Within a single run, callers must pre-aggregate at the (test_file, source_file) grain before pushing: multiple examples within the same test_file should be unioned into one row by the loader, not handed in as duplicates.

Constant Summary collapse

TABLE_NAME = 
"test_coverage_per_file"
MAX_BATCH_BYTES =

Cap each INSERT by rendered byte size rather than a fixed row count. test_coverage_per_file is a SharedReplacingMergeTree partitioned by month, so every INSERT lands a part in one partition; hundreds of tiny inserts pile up parts faster than background merges collapse them, and ClickHouse throttles inserts once the per-partition part count climbs. Batching by bytes keeps the part count low while bounding each statement well within ClickHouse's limits, regardless of how wide a row's covered-line bitmap is.

4 * 1024 * 1024
MAX_LINE_NUMBER =

Intentionally generous ceiling on line numbers. Real source files are thousands of lines; generated artifacts (large GraphQL schemas, bundled JS, JSON manifests) can run past 100k. The cap is set to flag clearly bogus values (negative, garbage casts, anything past ~1M) without rejecting realistic generated files. ClickHouse's UInt32 ceiling is ~4.3B, so we still have orders of magnitude of headroom above this. Tighten only with evidence.

1_000_000

Instance Method Summary collapse

Methods inherited from Table

#initialize

Constructor Details

This class inherits a constructor from GitlabQuality::TestTooling::CodeCoverage::ClickHouse::Table

Instance Method Details

#push(data) ⇒ void

This method returns an undefined value.

Parameters:

  • data (Array<Hash>) —

    one entry per (test_file, source_file). Each entry needs: :test_file [String] :source_file [String] :covered_lines [Array] line numbers covered by this test on this file :total_lines [Integer] executable lines in the source file :feature_category, :group, :stage, :section [String, optional]



48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/gitlab_quality/test_tooling/code_coverage/click_house/per_test_coverage_table.rb', line 48

def push(data) # rubocop:disable Metrics/AbcSize
  return logger.warn("#{LOG_PREFIX} No data found, skipping ClickHouse export!") if data.empty?

  logger.debug("#{LOG_PREFIX} Starting per-test coverage export to ClickHouse")
  sanitized_data = sanitize(data)

  return logger.warn("#{LOG_PREFIX} No valid data found after sanitization, skipping ClickHouse export!") if sanitized_data.empty?

  batch_index = 0
  each_byte_limited_batch(sanitized_data) do |rows_sql|
    batch_index += 1
    logger.debug("#{LOG_PREFIX} Pushing batch #{batch_index} (#{rows_sql.size} rows)")
    client.query(build_insert_sql(rows_sql), format: "TabSeparated")
  end
  logger.info("#{LOG_PREFIX} Successfully pushed #{sanitized_data.size} records to #{full_table_name}")
rescue StandardError => e
  logger.error("#{LOG_PREFIX} Error occurred while pushing data to #{full_table_name}: #{e.message}")
  raise
end