Copyright (C) 1992, 1993 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

The Appendix "Tool Command Language Overview" was written by John Ousterhout, of the University of California at Berkeley. Special copyright terms apply to that Appendix; see section Tcl Overview.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided also that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions.

@raggedbottom

What is DejaGnu?

DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Beyond this, DejaGnu offers several advantages for testing:

1. The flexibility and consistency of the DejaGnu framework make it easy to write tests for any program.
2. DejaGnu provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test for GDB can run (from any Unix based host) on any target architecture that DejaGnu supports. Currently DejaGnu runs tests on several single board computers, whose operating software ranges from just a boot monitor to a full-fledged, Unix-like realtime OS.
3. All tests have the same output format. This makes it easy to integrate testing into other software development processes.

DejaGnu is written in expect, which in turn uses Tcl---Tool command language.

Running tests requires two things: the testing framework, and the test suites themselves. Tests are usually written in expect using Tcl, but you can also use a Tcl script to run a test suite that is not based on expect. (expect script filenames conventionally use `.exp' as a suffix; for example, the main implementation of the DejaGnu test driver is in the file `runtest.exp'.)

• Running Tests: A first look at running DejaGnu tests
• Sample Test: What does a DejaGnu test case look like?
• Design Goals: Goals behind DejaGnu
• Posix: DejaGnu conforms to POSIX 1003.3
• Future Directions: Where is DejaGnu going?
• Tcl and Expect: Reading more about Tcl and Expect

Running existing tests

To run tests from an existing collection, first use configure as usual to set up the source directory containing the tests. Then try running

make check

If the check target exists, it usually saves you some trouble--for instance, it can set up any auxiliary programs or other files needed by the tests.

Once you have run `make check' to build any auxiliary files, you might want to call the test driver runtest directly to repeat the tests. You may also have to call runtest directly for test collections with no check target in the `Makefile'.

Typically, you must use two command-line options: `--tool', to specify which set of tests to run(1), and `--srcdir', to specify where to find test directories.

For example, if the directory `gdb/testsuite' contains a collection of DejaGnu tests for GDB, you can run them like this:

eg$ cd gdb/testsuite
eg$ runtest --tool gdb
Test output follows, ending with:

		=== gdb Summary ===

# of expected passes 508
# of expected failures 103
/usr/latest/bin/gdb version 4.11.2 -nx

You can use the option `--srcdir' to point to some other directory containing a collection of tests:

eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite

These examples assume a native configuration, where the same computer runs both runtest and the tests themselves. When you have a cross configuration, the tests run on a different computer, controlled by the host running runtest. In this situation, you need the option `--name' to specify the network address for the other computer:

eg$ runtest --tool gdb --name vx9.munist.com

If you always use the same option values, you can record them in a file called `site.exp', rather than typing them each time. See section Defaults for runtest options.

By default, runtest prints only the names of the tests it runs, output from any tests that have unexpected results, and a summary showing how many tests passed and how many failed. To display output from all tests (whether or not they behave as expected), use the `--all' option. For more verbose output about processes being run, communication, and so on, use `--verbose'. To see even more output, use multiple `--verbose' options. See section Using runtest, for a more detailed explanation of each runtest option.

Test output goes into two files in your current directory: summary output in `tool.sum', and detailed output in `tool.log'. (tool refers to the collection of tests; for example, after a run with `--tool gdb', look for output files `gdb.sum' and `gdb.log'.) See section The files DejaGnu writes.

What does a DejaGnu test look like?

Each DejaGnu test is an expect script; the tests vary widely in complexity, depending on the nature of the tool and the feature tested.

Here is a very simple GDB test--one of the simplest tests shipped with DejaGnu (extracted from `gdb.t00/echo.exp'):(2)

# send a string to the GDB stdin:
send "echo Hello world!\n"

# inspect the GDB stdout for the correct reply,
# and determine whether the test passes or fails:
expect {
  -re "Hello world.*$prompt $"    { pass "Echo test" }
  -re "$prompt $"                 { fail "Echo test" }
  timeout                         { fail "(timeout) Echo test" }
  }

Though brief, this example is a complete test. It illustrates some of the main features of DejaGnu test scripts:

• The test case does not start the tested program (GDB in this case); all test scripts for interactive tools can assume the corresponding tool is running.
• Comments start with `#'.
• The main commands you use to control a tested program are send (to give it commands) and expect (to analyze its responses).
• The expect command uses a list of pairs; a pattern (regular expression if `-re' specified), followed by an action to run if the pattern matches output from the program. Only the action for the first matching pattern will execute.
• Test cases use the commands pass and fail to record the test outcome.

Design goals

DejaGnu grew out of the internal needs of Cygnus Support. Cygnus maintains and enhances a variety of free programs in many different environments, and we needed a testing tool that:

• is useful to developers while fixing bugs;
• automates running many tests during a software release process;
• is portable among a variety of host computers;
• supports cross-development testing;
• permits testing interactive programs, like GDB; and
• permits testing batch oriented programs, like GCC.

Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important: for instance, it is imperative to make sure that GDB works as well when cross-debugging as it does in a native configuration.

Probably the greatest challenge was testing in a cross-development environment (which can be a real nightmare). Most cross-development environments are customized by each developer. Even when buying packaged boards from vendors there are many differences. The communication interfaces vary from a serial line to ethernet. DejaGnu was designed with a modular communication setup, so that each kind of communication can be added as required, and supported thereafter. Once a communication procedure is coded, any test can use it. Currently DejaGnu can use rsh, rlogin, telnet, tip, kermit, and mondfe for remote communications.

Julia Menapace first coined the term "Deja Gnu" to describe an earlier testing framework at Cygnus Support. When we replaced it with the Expect-based framework, it was like DejaGnu all over again...

A POSIX conforming test framework

DejaGnu conforms to the POSIX standard for test frameworks.

POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET.(3)

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertion was written.

As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. DejaGnu supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:

PASS

A test has succeeded. POSIX 1003.3 says that a known error can also be considered "success". Rather than use `PASS' in a confusing manner, DejaGnu recommends that tests restrict the meaning of `PASS' to a test case that has not reproduced the bug it was intended to capture. For known failures, DejaGnu tests can use the message `XFAIL' (or `XPASS', should a "known failure" start passing unexpectedly). (See section Using runtest.) However, a test suite that may produce `XFAIL' will not conform to POSIX 1003.3. If you use DejaGnu to write POSIX test suites, you must use PASS ambiguously to mean either "no failure", or "known failure".

FAIL

A test has produced the bug it was intended to capture. The `FAIL' message is based on the test case only. Other messages are used to indicate a failure of the framework.

UNRESOLVED

A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to `PASS' or `FAIL' before a test run can be considered finished. Here are some of the ways a test may wind up `UNRESOLVED':

• A test's execution is interrupted.
• A test does not produce a clear result. This is usually because there was an `ERROR' from DejaGnu while processing the test, or because there were three or more `WARNING' messages. Any `WARNING' or `ERROR' messages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened--and to improve the test case.
• A test depends on a previous test, which fails.
• The test was set up incorrectly.

UNTESTED

A test was not run. This is a placeholder, used when there is no real test case yet.

The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:

UNSUPPORTED

There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. DejaGnu also uses this message when a testing environment (often a "bare board" target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine gethostname would never work on a target board running only a boot monitor.

DejaGnu uses the same output procedures to produce these messages for all test suites, and these procedures are already known to conform to POSIX 1003.3. Therefore, any DejaGnu test suite (so long as it avoids the setup_xfail procedure, and hence the `XFAIL' and `XPASS' messages) can be considered to conform to POSIX 1003.3.

Future directions

In the near future, there are two parallel directions for DejaGnu development. The first is to add support for more hosts and targets.

The second would permit testing programs with a more complex interface, whether text based or GUI based. Two components already exist: a Tcl based X window toolkit, and a terminal package for expect. Both of these could be merged into DejaGnu in a way that permits testing programs that run in each environment.

Meanwhile, we hope DejaGnu enables the creation of test suites for conformance to ANSI C and C++, to POSIX, and to other standards. We encourage you to make any test suites you create freely available, under the same terms as DejaGnu itself.

Tcl and Expect

Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter Usenix conference, Tcl: An Embeddable Command Language. That paper is included in PostScript form in the `doc' subdirectory of the Tcl distribution. See section Tcl Overview, for details on Tcl. The version of Tcl included in DejaGnu at this time is Tcl 6.7.

Don Libes introduced expect in his paper expect: Curing Those Uncontrollable Fits of Interaction at the 1990 Summer Usenix conference. The paper is included in PostScript form in the expect distribution (as are several other papers about expect). See section Expect: Programmed Dialogue, for details on expect. The version of expect included in DejaGnu at this time is expect 4.7.6.1.