Abide-Data-Processor

This gem provides the functionality to process data parsed from a Hiera file for Puppetlabs CEM modules.

• Abide-Data-Processor
  • About the Gem
  • Installation
  • Usage
  • Development
  • CEM Resource Data Specification
  • 1: Meta Information
  • 2: Common.yaml
  • 3: Top-level key
  • 4: Resources
    • 4.1: Tile Key
    • Example 4.1.1 - Basic plain-text title key
    • Example 4.1.2 - File path title key
    • 4.2: Type
    • Example 4.2.1 - Basic type property
    • Example 4.2.2 - Complex type property
  • 5: Controls
    • Example 5.1 - Basic controls property
    • Example 5.2 - Multiple controls map to the same parameter and value
    • Example 5.3 - Multiple controls map to the same parameter with different values
    • Example 5.4 - One control maps to multiple different parameters
    • Example 5.5 - A control does not supply parameters, but maps to the whole resource
  • 6: Metaparameters
    • Example 6.0.1 - Declaring metaparameters
    • Example 6.0.1.1 - Declaring a metaparameter at the resource level
    • Example 6.0.1.2 - Declaring a metaparameter at the control level
    • 6.1: Dependent

About the Gem

Installation

Add this line to your application's Gemfile:

gem 'abide-data-processor'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install abide-data-processor

Usage

Since this gem is more of a library, there is no executable to run it. This gem is designed to be use by Puppetlabs CEM modules to process the data that those modules generate.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

CEM Resource Data Specification

CEM resource data is the basis for how the CEM modules know what code to enforce, and how, for each of our supported compliance frameworks. Resource data is implemented as module-level Hiera data in each of the CEM modules. The goal of resource data is to provide detailed information about how specific Puppet resources and their parameters enforce compliance standards, which is then used in the modules to include the correct Puppet resources and values in the catalog. This is included in this repo because the data processor is the parser for CEM resource data.

1: Meta Information

• File Type: YAML version 1.2
• File Extension: .yaml
• Base Directory: data/
• Hiera hierarchy levels:
  • %{facts.os.family}/%{facts.os.name}/%{facts.os.release.major}.yaml
  • %{facts.os.family}/%{facts.os.name}.yaml
  • %{facts.os.family}/%{facts.os.release.major}.yaml
  • %{facts.os.family}.yaml
  • common.yaml

2: Common.yaml

common.yaml must declare lookup options for the key <module name>::resources: as follows:

lookup_options:
  <module name>::resources:
    merge:
      strategy: deep
      merge_hash_arrays: true

3: Top-level key

There must only be one top-level key in each resource data file. This key serves as a universal lookup key for the entire data structure. The top-level key in each resource data file must follow the convention: <module name>::resources:.

4: Resources

Under the top-level key, all sub-hashes constitute a singular Puppet resource, whether that resource is a class, a defined type, etc. Each resource consists of a title-key, a type, various controls, and optional metaparameters.

4.1: Tile Key

Title keys exist beneath the top-level key. There may be any amount of title keys. Title keys must consist of resource titles. A resource title is the unique string assigned to a Puppet resource after it's type declaration:

exec { 'This is the resource title':
    ...
}

Title keys must be unique once the entire data structure is resolved via Hiera lookup. This is partially because Puppet resource names must be unique in Puppet manifests, but also because keys at the same level in Ruby hashes must be unique. This requirement is less flexible than resource naming in Puppet itself, because types with multiple name variables that are combined to make the actual resource name are not considered.

Example 4.1.1 - Basic plain-text title key

<module name>::resources:
  'This is the resource title':
    ...

Example 4.1.2 - File path title key

<module name>::resources:
  '/opt/puppetlabs/cem/test_file.txt':

In this example, we specify a file path that would be used by a Puppet resource such as file. However, we cannot use the file path /opt/puppetlabs/cem/test_file.txt again as a title key.

4.2: Type

Type is a property of a resource defined by a title key. Type must be a Puppet resource type. When Puppet resources are created during catalog compilation, this is the type of resource that will be created.

this_is_the_resource_type { 'This is the resource title':
  ...
}

Type is defined as a single key-value pair under the title key.

Example 4.2.1 - Basic type property

<module name>::resources:
  'This is the resource title':
    type: exec

Example 4.2.2 - Complex type property

<module name>::resources:
  'This is the resource title':
    type: 'cem_linux::utils::bootloader::grub2::password'

5: Controls

Controls is a property of a resource that maps compliance framework controls to the various Puppet resource parameters that those controls are concerned with. Each mapping in controls consists of one or more keys that use an compliance control identifier.

Example 5.1 - Basic controls property

We have a compliance framework that defines a control 'Ensure you run the ls command'. The Puppet code to enforce this control looks like this:

# Enforces 'Ensure you run the ls command
exec { 'This is the resource title':
    command => 'ls',
}

The above Puppet code translates to the following resource data:

<module name>::resources:
  'This is the resource title':
    type: exec
    controls:
      'Ensure you run the ls command':
        command: 'ls'

Example 5.2 - Multiple controls map to the same parameter and value

We have two compliance frameworks. Both frameworks define a control that is equal in all but name. Framework 1 defines the control 'Ensure you run the ls command' and Framework 2 defines the control 'The ls command must be ran'. The Puppet code to enforce this control looks like this:

# Framework 1 - Enforces 'Ensure you run the ls command
# Framework 2 - The ls command must be ran
exec { 'This is the resource title':
    command => 'ls',
}

The above Puppet code translates to the following resource data:

<module name>::resources:
  'This is the resource title':
    type: exec
    controls:
      ? - 'Ensure you run the ls command'
        - 'The ls command must be ran'
      :  command: 'ls'

When two or more controls map to the same parameter and the same value, YAML complex mapping keys must be used.

Example 5.3 - Multiple controls map to the same parameter with different values

<module name>::resources:
  'This is the resource title':
    type: exec
    controls:
      'Ensure you run the ls command':
        command: 'ls'
      'The ls -l command must be ran':
        command: 'ls -l'

Example 5.4 - One control maps to multiple different parameters

<module name>::resources:
  'This is the resource title':
    type: exec
    controls:
      'Ensure you run the ls command only if test.txt exists':
        command: 'ls'
        onlyif: 'test -f test.txt'

When the Puppet resources are created from resource data during catalog compilation, the resources cem_linux::utils::packages::absenter { 'avahi': } and cem_linux::utils::packages::absenter { 'avahi-autoipd': } will be created before the resource cem_linux::utils::disable_service { 'avahi-daemon': }. You do not need any other resource data to specify cem_linux::utils::packages::absenter { 'avahi': } and cem_linux::utils::packages::absenter { 'avahi-autoipd': }.

Example 5.5 - A control does not supply parameters, but maps to the whole resource

Sometimes, controls will not map to specific resource parameters but to the whole resource itself. This is often the case with resources that represent defined types with very specific functionality. In this case, the controls key must be an Array, not a Hash. Each item of that Array must be a control name that maps to the declared resource.

Below is an example of Puppet code where the control would not manage a parameter:

cem_linux::utils::disable_service { 'avahi-daemon': }

With this particular defined type, the resource title can also serve as the one and only parameter. In resource data, we declares a resource with control as an Array.

cem_linux::resources:
  'avahi-daemon':
    type: 'cem_linux::utils::disable_service'
    controls:
      - 'Ensure Avahi server is not installed'

6: Metaparameters

Both resources and controls support five metaparameters: before, require, notify, subscribe, and dependent. Metaparameters are used for ordering resources, and are used just as they are in Puppet code. Dependent is a special metaparameter with no Puppet code equivalent. Metaparameters are applied to a resource either directly, by declaring a metaparameter key at the same level as the type key or the controls key, or at the control level just as you would declare control parameters. Regardless of if a metaparameter is declared at the resource or control level, the metaparameter applies to the whole resource.

Example 6.0.1 - Declaring metaparameters

Below is an example of Puppet code that uses the Puppet metaparameter before:

cem_linux::utils::disable_service { 'avahi-daemon':
  before => [
      Cem_linux::Utils::Packages::Absenter['avahi'],
      Cem_linux::Utils::Packages::Absenter['avahi-autoipd'],
  ],
}

This Puppet code infers, through the use of Puppet resource references, that the Puppet resources cem_linux::utils::packages::absenter { 'avahi': } and cem_linux::utils::packages::absenter { 'avahi-autoipd': } are declared somewhere in the manifest. In resource data, we declare the dependent resources in the metaparameter itself.

There is no need for resources declared in metaparameters to exist already in resource data. Additionally, if a resource declared in resource data is also declared in a metaparameter, only one copy of that resource will be created.

Example 6.0.1.1 - Declaring a metaparameter at the resource level

cem_linux::resources:
  'avahi-daemon':
    type: 'cem_linux::utils::disable_service'
    before:
      'avahi-autoipd':
        type: 'cem_linux::utils::packages::absenter'
      'avahi':
        type: 'cem_linux::utils::packages::absenter'
    controls:
      - 'Ensure Avahi server is not installed'

Example 6.0.1.2 - Declaring a metaparameter at the control level

cem_linux::resources:
  'avahi-daemon':
    type: 'cem_linux::utils::disable_service'
    controls:
      'Ensure Avahi server is not installed':
        before:
          'avahi-autoipd':
            type: 'cem_linux::utils::packages::absenter'
          'avahi':
            type: 'cem_linux::utils::packages::absenter'

When the Puppet resources are created from resource data during catalog compilation, the resources cem_linux::utils::packages::absenter { 'avahi': } and cem_linux::utils::packages::absenter { 'avahi-autoipd': } will be created before the resource cem_linux::utils::disable_service { 'avahi-daemon': }.

6.1: Dependent

Dependent is a special metaparameter used in resource data used to ensure mutual inclusion / exclusion of resources that depend on each other to function. When Resource A declares Resource B in the dependent metaparameter, Resource A will only be enforced if Resource B is included in the catalog as well. Dependent ensures that controls residing in the ignore list will not disrupt the entire catalog.