CemAcpt

CemAcpt is an acceptance testing library / command line application for running acceptance tests against the CEM modules. It is heavily inspired by Puppet Litmus. CemAcpt is fully compatible with Puppet Litmus acceptance tests.

CemAcpt was written to facilitate running a single acceptance test file against one or more test nodes concurrently. This is useful when your module supports a lot of different operating systems and modifies low-level components of those operating systems. For CEM, we manage things such as firewall and bootloader configurations which requires us to test against multiple base images of a single operating system (i.e. RHEL 8 with iptables and RHEL 8 with firewalld). Additionally, we test our module using both Puppet 6 and Puppet 7. As you can see, we now need to test against four test nodes that are all RHEL 8 with slightly different baseline configs. CemAcpt allows us to do so quickly in parallel.

Major differences between Puppet Litmus and CemAcpt are:

  • CemAcpt revolves around mapping a single acceptance test file to one or more test hosts.
  • Test node image names can be dynamically defined by configurable parsing of the acceptance test name.
    • This allows your acceptance test names to define the node it will run against.
  • CemAcpt has it's own CLI for running provisioning of hosts and the test suite and does not use rake.
  • CemAcpt runs everything, including provisioning the test node, creating a temporary manifest on the test node, and running the acceptance test on the test node, in parallel. No matter how many test / host combinations there are, running the test suite will only take as long as the longest running single test / host combo.
  • CemAcpt is configurable via a YAML file.

Concepts

CemAcpt is underpinned with a few core concepts. These concepts shape the architecture and functionality of CemAcpt.

Platforms

Platforms are backends that handle provisioning, interacting with, and destroying test nodes. Platforms typically represent something like a cloud provider or hypervisor, but as long as a platform implements the required methods it could do anything.

Platforms are created by creating a file in lib/cem_acpt/platform. The name of the platform file is significant as it will be capitalized and used as a classname. PLATFORMS MUST implement the methods stipulated by lib/cem_acpt/platform/base.rb in a single module called Platform because the platform system will dynamically create the platform class from this module. How the methods are implemented is irrelevant as long as they take the correct input and give the correct output.

Platforms will be passed all configuration options under the top-level key node_data. Node configuration should be generic and should apply to all test nodes regardless of the test being ran against them. For test-specific configurations, use test data.

See lib/cem_acpt/platform/gcp.rb for an example of a platform file.

Tests

Each acceptance test should be specified in the config under the top-level key tests. These tests SHOULD NOT be paths and should not contain the suffix _spec.rb. For example, if you have an acceptance test spec/acceptance/our_acceptance_test_spec.rb, the tests config would look like this:

tests:
  - our_acceptance_test

Aside from ways of manipulating test data outlined below, tests are typically matched one-to-one with a test node.

Test data

Test data is a collection of data about a specific test that persists through the entire acceptance test suite lifecycle. Specifically, test data is implemented as an array of hashes with each hash representing a single run through the acceptance test suite lifecycle. What this means is that each item in the test data array is couple with a single test node that is provisioned and a single test run against that node. Test data is used to store values that are test-specific, as opposed to node data which is generic.

Test data can be configured using the top-level key test_data. There are several supported options for manipulating test data:

for_each

When specified, should be a hash of key -> Array pairs. For each of these pairs, a copy of the test data is made for each item in the array, with that item becoming the value of a variable key.

Example:

test_data:
  for_each:
    collection:
      - puppet6
      - puppet7

tests:
  - our_acceptance_test

In the above config, instead of test data consisting of a single hash, it will have two hashes that have the collection variable set to puppet6 and puppet7 respectively. This means that there will be two test nodes provisioned and two runs of the test our_acceptance_test, one run against each node.

vars

Aribitrary key-value pairs that are injected into the test data hashes. Think of these as constants.

name_pattern_vars

A Ruby regex pattern that is matched against the test name with the intent of creating variables from named capture groups. See sample_config.yaml for an example.

vars_post_processing

Rules that allow for processing variables after all other test data rules are ran. See sample_config.yaml for an example.

Image name builder

Much like name_pattern_vars, specifying the image_name_builder top-level key in the config allows you to manipulate acceptance test names to create a special test data variable called image_name. This is helpful for when you have multiple platform base images and want to use the correct image with the correct test. See sample_config.yaml for an example.

Installation

Add this line to your module's Gemfile (usually under group :development):

gem 'cem_acpt', require: false

Then, add the following to your spec_helper_acceptance.rb (replacing the Puppet Litmus configuration lines):

require 'cem_acpt'

CemAcpt.configure_spec_helper!

To use CemAcpt in your acceptance tests, you must call the method initialize_test_environment! in your acceptance test's describe block before doing anything else:

describe 'cem_linux CIS Level 1' do
  initialize_test_environment!
  ...

Usage

RSpec methods

CemAcpt enables you to use all ServerSpec methods in your acceptance tests, and adds three new methods that can be used (these methods are used the same as in Litmus):

  • apply_manifest(manifest, opts = {}): Applies a Puppet manifest given as a string.
  • idempotent_apply(manifest, opts = {}): Applies a Puppet manifest given as a string twice and fails if the second apply reports changes.
  • run_shell(command, opts = {}): Runs a shell command against the test node.

Config file

CemAcpt can use a config file (default path is ./cem_acpt_config.yaml) to configure it's settings. Below are some of the options available in the config file:

  • tests: An array of test names you want to run. These are assumed to exist in the path spec/acceptance. You should leave off the suffix _spec.rb from the test file names.
  • platform: The backend platform that acceptance tests will run on. Currently, only gcp is supported.
  • test_data: Configurations for test data. Allows assigning variables to test data either dynamically or statically.
  • node_data: Configurations for nodes created by the platform. The available options are platform-dependent.
  • image_name_builder: If this key is specified, platforms will be passed a dynamically generated image name based on the configurations under this key.

Semantic acceptance test names

With CemAcpt, acceptance test names can have semantic meaning and influence how your acceptance test nodes are provisioned and how test data is created. The semantic meaning of the test names can be configured in the config file.

Test data

Test data can

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/cem_acpt.

License

The gem is available as open source under the terms of the MIT License.