Blackstart

Blackstart is a small, subdued library for automated testing in Ruby. It doesn’t depend on anything beyond the Ruby platform and doesn’t modify the environment beyond its conventional namespace. The program that tests it is written in primitive Ruby, with no other testing library, so it can be verified straightforwardly.

The blackstart is a small, subdued bird that eats bugs. A black start is when an inactive power plant restarts by means of an independent power source rather than the electrical grid.

Here’s an example of a test program that uses Blackstart:

require "blackstart"

exit Blackstart.run [
  lambda {
    unless "Hello, World!" == ["He", "", "o, Wor", "d!"].join("l")
      raise "join did not work as expected"
    end
  },

  lambda {
    unless "sample" == "simple".gsub(/i/, "a")
      raise "gsub did not work as expected"
    end
  }
]

Blackstart.run runs a sequence of tests and reports information about any that fail. It returns false if there were failures, true otherwise.

An array of procs is an easy way to represent a sequence of tests. In general, the sequence can be any object that responds to an each message in the conventional way and each test object in the sequence can be any object that (1) is a proc or converts to one via to_proc and (2) converts to a string via to_s.

Each test is run by converting the test object to a proc if necessary and then calling it in the context of a new instance of Blackstart::Scratchpad; if and only if this raises an error (any exception that’s a kind of StandardError), Blackstart.run interprets it as a failed test.

After each test failure, Blackstart.run sends a puts message with failure information to its second parameter, which is $stdout by default. The failure information includes a description of the test (the test object converted to a string) and a description of the error raised (its class, string representation, and backtrace).

If a test raises a non-StandardError exception, Blackstart.run interprets it as a system-level event or fundamental problem: it immediately raises the exception and doesn’t run any more tests.

The library provides little beyond this, but it supports more sophisticated uses. Some examples follow.

• Exiting with the appropriate status

You may want your test program to exit with a successful status only if there were no failures, perhaps so it can work as part of a testing script. You can do this by using the object returned by Blackstart.run as the argument to Kernel#exit. Blackstart.run returns false if there were failures, true otherwise.

• Defining helpers

You may want to enable all your tests to assert something, generate test data, or perform some other task by sending a message. You could implement this statelessly in a singleton object or all instances of Object, for example, but there’s another option that may be more convenient. When Blackstart.run calls a test proc, it sets the self object to a new instance of Blackstart::Scratchpad. You can define instance methods in that class. For example:

class Blackstart::Scratchpad
  def assert boolean
    raise "assertion failed" unless boolean
  end

  def make_products
    @cabbage = { :description => "head of cabbage", :price => 125 }
    @orange = { :description => "Cara Cara navel orange", :price => 100 }
    nil
  end
end

All your test procs can use them by sending messages to self. These methods, like the test proc itself, can see and modify instance variables and other elements of the scratchpad’s state. The scratchpad is disposable: Blackstart.run discards it after the test is complete.

• Running code before and after each test

You may want to run code before and after each test – for example, to create objects needed in tests or clean up resources without requiring every test to do these explicitly. You can do this by defining a custom Blackstart::Scratchpad#instance_exec. For example:

class Blackstart::Scratchpad
  def instance_exec(*)
    puts "before test"
    @variable = "example"
    super
  ensure
    puts "after test"
  end
end

Use this technique with care because the test proc could send instance_exec messages to self.

• Building a test collection in stages

You may want to build a test collection in stages rather than all at once. This can be done straightforwardly:

require "blackstart"

TESTS = []

TESTS.concat [
  lambda {
    unless "Hello, World!" == ["He", "", "o, Wor", "d!"].join("l")
      raise "join did not work as expected"
    end
  }
]

TESTS.concat [
  lambda {
    unless "sample" == "simple".gsub(/i/, "a")
      raise "gsub did not work as expected"
    end
  }
]

exit Blackstart.run TESTS

This pattern is useful if you want to define your tests in multiple files: you create a collection with an agreed-upon name, load multiple files, each of which adds test objects to that collection, and then run all the tests.

• Running tests in random order

To check if you have any tests that depend on other tests having run, or not having run, earlier, you may want to run your tests in random order. Blackstart.run runs a sequence of tests in order, but you can pass it a randomly-ordered sequence. Array#shuffle may be helpful for this. If you use Array#shuffle or something similar, you may also want to print the random seed just before the tests are shuffled so you can rerun your tests in the same order by setting the random seed.

• Reporting detailed test descriptions

After a test fails, Blackstart.run writes a string describing the test to the output stream. It gets this string by sending a to_s message to the test object. When the test object is an instance of Proc, the returned string typically includes the file path and line number where it was defined: helpful, but it won’t be immediately clear what was being tested.

You can improve this by designing your own test objects. Blackstart.run does not strictly need a sequence of procs; it also works with a sequence of objects that convert to procs in response to to_proc messages. Your custom test objects can respond to to_s with detailed descriptions instead of mere file paths and line numbers.

• Handling failures differently

Blackstart.run handles each failure by sending a puts message to the output stream, one of its parameters, with failure information. By default, the output stream is $stdout, so the default behavior is to print unadorned failure information to standard output. But you can specify any object as the output stream – even if it’s not really a stream – allowing you to handle failure information however you want.

• Running multiple tests at the same time

Although Blackstart.run is written in a sequential style, it can run multiple tests at the same time. Blackstart.run sends an each message with a block to your test collection, expecting the block to be executed once for each test; your test collection can execute the block in multiple threads concurrently. For your program to work well in this mode, your tests and the output stream must work correctly when used from multiple threads.