Search documentation of common_test

# Using Common Test for Large-Scale Testing

View Source

## General

Large-scale automated testing requires running multiple independent test sessions in parallel. This is accomplished by running some `Common Test` nodes on one or more hosts, testing different target systems. Configuring, starting, and controlling the test nodes independently can be a cumbersome operation. To aid this kind of automated large-scale testing, `Common Test` offers a master test node component, `Common Test` Master, which handles central configuration and control in a system of distributed `Common Test` nodes.

The `Common Test` Master server runs on one dedicated Erlang node and uses distributed Erlang to communicate with any number of `Common Test` test nodes, each hosting a regular `Common Test` server. Test specifications are used as input to specify what to test on which test nodes, using what configuration.

The `Common Test` Master server writes progress information to HTML log files similarly to the regular `Common Test` server. The logs contain test statistics and links to the log files written by each independent `Common Test` server.

The `Common Test` Master API is exported by module `ct_master`.

## Use

`Common Test` Master requires all test nodes to be on the same network and share a common file system. `Common Test` Master cannot start test nodes automatically. The nodes must be started in advance for `Common Test` Master to be able to start test sessions on them.

Tests are started by calling `ct_master:run(TestSpecs)` or `ct_master:run(TestSpecs, InclNodes, ExclNodes)`

`TestSpecs` is either the name of a test specification file (string) or a list of test specifications. If it is a list, the specifications are handled (and the corresponding tests executed) in sequence. An element in a `TestSpecs` list can also be list of test specifications. The specifications in such a list are merged into one combined specification before test execution.

Example:

    ct_master:run(["ts1","ts2",["ts3","ts4"]])

Here, the tests specified by "ts1" run first, then the tests specified by "ts2", and finally the tests specified by both "ts3" and "ts4".

The `InclNodes` argument to `run/3` is a list of node names. Function `run/3` runs the tests in `TestSpecs` just like `run/1`, but also takes any test in `TestSpecs`, which is not explicitly tagged with a particular node name, and execute it on the nodes listed in `InclNodes`. By using `run/3` this way, any test specification can be used, with or without node information, in a large-scale test environment.

`ExclNodes` is a list of nodes to be excluded from the test. That is, tests that are specified in the test specification to run on a particular node are not performed if that node is listed in `ExclNodes` at runtime.

If `Common Test` Master fails initially to connect to any of the test nodes specified in a test specification or in the `InclNodes` list, the operator is prompted with the option to either start over again (after manually checking the status of the nodes in question), to run without the missing nodes, or to abort the operation.

When tests start, `Common Test` Master displays information to console about the involved nodes. `Common Test` Master also reports when tests finish, successfully or unsuccessfully. If connection is lost to a node, the test on that node is considered finished. `Common Test` Master does not attempt to re-establish contact with the failing node.

At any time, to get the current status of the test nodes, call function `ct_master:progress()`.

To stop one or more tests, use function `ct_master:abort()` (to stop all) or `ct_master:abort(Nodes)`.

For details about the `Common Test` Master API, see module `ct_master`.

## Test Specifications

The test specifications used as input to `Common Test` Master are fully compatible with the specifications used as input to the regular `Common Test` server. The syntax is described in section Test Specifications in section Running Tests and Analyzing Results.

All test specification terms can have a `NodeRefs` element. This element specifies which node or nodes a configuration operation or a test is to be executed on. `NodeRefs` is defined as follows:

`NodeRefs = all_nodes | [NodeRef] | NodeRef`

`NodeRef = NodeAlias | node() | master`

A `NodeAlias` (`atom/0`) is used in a test specification as a reference to a node name (so the node name only needs to be declared once, which also can be achieved using constants). The alias is declared with a `node` term as follows:

`{node, NodeAlias, NodeName}`

If `NodeRefs` has the value `all_nodes`, the operation or test is performed on all specified test nodes. (Declaring a term without a `NodeRefs` element has the same effect). If `NodeRefs` has the value `master`, the operation is only performed on the `Common Test` Master node (namely set the log directory or install an event handler).

Consider the example in section Test Specifications in section Running Tests and Analysing Results, now extended with node information and intended to be executed by `Common Test` Master:

    {define, 'Top', "/home/test"}.
    {define, 'T1', "'Top'/t1"}.
    {define, 'T2', "'Top'/t2"}.
    {define, 'T3', "'Top'/t3"}.
    {define, 'CfgFile', "config.cfg"}.
    {define, 'Node', ct_node}.

    {node, node1, 'Node@host_x'}.
    {node, node2, 'Node@host_y'}.

    {logdir, master, "'Top'/master_logs"}.
    {logdir, "'Top'/logs"}.

    {config, node1, "'T1'/'CfgFile'"}.
    {config, node2, "'T2'/'CfgFile'"}.
    {config, "'T3'/'CfgFile'"}.

    {suites, node1, 'T1', all}.
    {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
    {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
    {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}.

    {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}.
    {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}.

    {skip_suites, 'T3', all, "Not implemented"}.

This example specifies the same tests as the original example. But now if started with a call to `ct_master:run(TestSpecName)`, test `t1` is executed on node `ct_node@host_x` (`node1`), test `t2` on `ct_node@host_y` (`node2`) and test `t3` on both `node1` and `node2`. Configuration file `t1` is only read on `node1` and configuration file `t2` only on `node2`, while the configuration file `t3` is read on both `node1` and `node2`. Both test nodes write log files to the same directory. (However, the `Common Test` Master node uses a different log directory than the test nodes.)

If the test session is instead started with a call to `ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x])`, the result is that test `t1` does not run on `ct_node@host_x` (or any other node) while test `t3` runs on both `ct_node@host_y` and `ct_node@host_z`.

A nice feature is that a test specification that includes node information can still be used as input to the regular `Common Test` server (as described in section Test Specifications). The result is that any test specified to run on a node with the same name as the `Common Test` node in question (typically `ct@somehost` if started with the `ct_run` program), is performed. Tests without explicit node association are always performed too, of course.

## Automatic Startup of Test Target Nodes

Initial actions can be started and performed automatically on test target nodes using test specification term `init`.

Two subterms are supported, `node_start` and `eval`.

Example:

    {node, node1, node1@host1}.
    {node, node2, node1@host2}.
    {node, node3, node2@host2}.
    {node, node4, node1@host3}.
    {init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}.
    {init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}.
    {init, node4, {eval, {module, function, []}}}.

This test specification declares that `node1@host1` is to be started using the user callback function `callback_module:my_slave_callback/0`, and nodes `node1@host2` and `node2@host2` are to be started with the default callback module `ct_slave`. The specified username and password are used to log on to remote host `host2`. Also, function `module:function/0` is evaluated on `node1@host3`, and the result of this call is printed to the log.

The default callback module `ct_slave`, has the following features:

- Starting Erlang target nodes on local or remote hosts (application `SSH` is used for communication).
- Ability to start an Erlang emulator with more flags (any flags supported by `erl` are supported).
- Supervision of a node being started using internal callback functions. Used to prevent hanging nodes. (Configurable.)
- Monitoring of the master node by the slaves. A slave node can be stopped if the master node terminates. (Configurable.)
- Execution of user functions after a slave node is started. Functions can be specified as a list of `{Module, Function, Arguments}` tuples.

#### Note

An `eval` term for the node and `startup_functions` in the `node_start` options list can be specified. In this case, the node is started first, then the `startup_functions` are executed, and finally functions specified with `eval` are called.

← Previous Page Code Coverage Analysis
Next Page → Event Handling