CemAcpt

CemAcpt is an acceptance testing library / command line application for running acceptance tests for the CEM modules.

CemAcpt uses Terraform for provisioning test nodes, and Goss to execute acceptance tests.

Installation

gem install cem_acpt

cem_acpt was developed using Ruby 3.2.1, but other Ruby versions may work.

Usage

Quickstart

Make sure Terraform is installed and in your PATH. Instructions for installing Terraform can be found here.Then, navigate to the root of the cem_linux module and run the following command:

cem_acpt --config ./cem_acpt_config.yaml

Command-line

cem_acpt -h | --help

If you do not delete Gemfile.lock before running bundle install, you may encounter dependency version errors. Just delete Gemfile.lock and run bundle install again to get past these.

Test directory structure

CemAcpt expects a specific directory structure for your acceptance tests. The directory structure is as follows:

spec
└── acceptance
    └── <framework>_<os>-<version>_firewalld_<profile>_<level>
        ├── goss.yaml # Goss test file
        └── manifest.pp # Puppet manifest to apply

For example, the following directory structure would be valid:

spec
└── acceptance
    └── cis_rhel-8_firewalld_server_2
        ├── goss.yaml
        └── manifest.pp

The directory name is used to generate test data. See Test data for more information. The file goss.yaml is used as the Goss test file. The file manifest.pp is used as the Puppet manifest to apply.

Configuration

There are several ways to configure CemAcpt which are outlined below in order of precendence. Options specified later in the list will be merged with or override options specified earlier in the list. If an option is specified in multiple places, the last place it is specified will be used.

  1. Environment variables
  2. User-specific config file
  3. Config file specified by --config option
  4. Command-line options

You can view your working config by adding the -Y flag to any command. This will print the merged config to STDOUT as a YAML document. You can also use the -X flag to print an explanation of how the config was merged.

Environment variables

Environment variables are the most basic way to configure CemAcpt. They are useful for setting sensitive information like API keys or passwords. Environment variables are prefixed with CEM_ACPT_ and are converted to the nested structure of the config file, if applicable. Double underscores (__) are used to separate key levels. For example, the environment variable CEM_ACPT_NODE_DATA__DISK_SIZE=40 would be converted to the following config:

node_data:
  disk_size: 40

All environment variables are optional. If an environment variable is not set, the value will be taken from the config file or command-line option. If an environment variable is set, it will be overridden by the same value from the config file or command-line option.

Config file

The most common way is to create a config file and pass it to CemAcpt using the --config option. The config file should be a YAML file. See sample_config.yaml for an example.

You can also create a user-specific config file at $HOME/.cem_acpt/config.yaml. This file will be merged with the config file specified by the --config option. This is useful for storing sensitive information or making config changes that you don't want to commit to a repo.

Options set in the user-specific config file will be overridden by options set in the config file specified by the --config option. Both config files will override options specified by environment variables.

Command-line options

CemAcpt can be configured using command-line options. These options are specified using the --<option> syntax. For a full list of options, run cem_acpt -h. Options specified on the command-line will override options specified in the config file and environment variables.

Goss

Goss is the core infrastructure testing tool used by CemAcpt. The goss.yaml file is used to specify the tests to run. Please see the Goss documentation for more information on how to write Goss tests.

Terraform

CemAcpt uses Terraform for managing the lifecycle of test nodes. Users don't interact with Terraform directly, but you will need it installed to use CemAcpt.

Platforms

Platforms are the underlying infrastructure that nodes are provisioned on. Currently, only GCP is supported. Each platform has two parts to it: the platform and the node data.

Platform

A platform represents where nodes are provisioned. Currently, only GCP is supported. Platforms are configured using the top-level key platform. For example, the following config would configure a GCP platform:

platform:
  name: gcp
  project: my-project
  region: us-west1
  zone: us-west1-b
  subnetwork: my-subnetwork

Node data

Node data is a more generic complement to the platform that specifies node-level details for test nodes. Node data is configured using the top-level key node_data. For example, the following config would configure node data:

node_data:
  disk_size: 40
  machine_type: e2-small

Developing new platforms

Platforms are defined as Ruby files in the lib/cem_acpt/platform directory. Each platform file should define a module called Platform that implements the following methods:

module Platform
  # @return [Hash] A hash of platform data
  def platform_data
    # Return a hash of platform data
  end

  # @return [Hash] A hash of node data
  def node_data
    # Destroy a node
  end
end

Both the #platform_data and #node_data methods should return a hash. Each method should also also ensure the config is queried for values for each key in each hash. This is enabled by the exposed instance variables @config and @test_data in the Platform module. For an example, see the GCP platform.

Tests

Each acceptance test should be specified in the config under the top-level key tests. These tests SHOULD NOT be paths, just the test directory name. For example, if you have an acceptance test directory spec/acceptance/cis_rhel_acceptance_test/*, the tests config would look like this:

tests:
  - our_acceptance_test

Aside from ways of manipulating test data outlined below, tests are typically matched one-to-one with a test node.

Test data

Test data is a collection of data about a specific test that persists through the entire acceptance test suite lifecycle. Specifically, test data is implemented as an array of hashes with each hash representing a single run through the acceptance test suite lifecycle. What this means is that each item in the test data array is couple with a single test node that is provisioned and a single test run against that node. Test data is used to store values that are test-specific, as opposed to node data which is generic.

Test data can be configured using the top-level key test_data. There are several supported options for manipulating test data:

for_each

When specified, should be a hash of key -> Array pairs. For each of these pairs, a copy of the test data is made for each item in the array, with that item becoming the value of a variable key.

Example:

test_data:
  for_each:
    collection:
      - puppet6
      - puppet7

tests:
  - our_acceptance_test

In the above config, instead of test data consisting of a single hash, it will have two hashes that have the collection variable set to puppet6 and puppet7 respectively. This means that there will be two test nodes provisioned and two runs of the test our_acceptance_test, one run against each node.

vars

Aribitrary key-value pairs that are injected into the test data hashes. Think of these as constants.

name_pattern_vars

A Ruby regex pattern that is matched against the test name with the intent of creating variables from named capture groups. See sample_config.yaml for an example.

vars_post_processing

Rules that allow for processing variables after all other test data rules are ran. See sample_config.yaml for an example.

Image name builder

Much like name_pattern_vars, specifying the image_name_builder top-level key in the config allows you to manipulate acceptance test names to create a special test data variable called image_name. This is helpful for when you have multiple platform base images and want to use the correct image with the correct test. See sample_config.yaml for an example.

The acceptance test lifecycle

  • Load and merge the config
  • Create the local test directory under $HOME/.cem_acpt
  • Build the Puppet module. Uses current dir if no module_dir is specified in the config
  • Copy all relevant files into the local test directory
    • This includes the Terraform files provided by CemAcpt, as well as the files under the specified acceptance test directory, and the built Puppet module
  • Provision test nodes using Terraform
    • After the node is created, the contents of the local test directory are copied to the node
    • Additionally, the Puppet module is installed on the node and Puppet is ran once to apply manifest.pp
    • After, the Goss server endpoints are started and exposed on the node
  • Once node is provisioned, make HTTP get requests to the Goss server endpoints to run the tests
  • Destroy the test nodes
  • Report the results of the tests