• Home |
• Products |
• Support |
• KB

There are five steps to setting up the CDRouter-based automated regression test system presented here:

1. Identify configurations and test combinations that you want to perform
2. Create the necessary configuration files
3. Write a bash script that executes multiple tests using buddy or BuddyWeb commands
4. Set up a cron entry so that the script automatically runs every day
5. Set up dedicated hardware with multiple interfaces (the "pod") to execute multiple configurations/tests across multiple devices

This article focuses primarily on steps 3 through 5 and assumes that steps 1 and 2 have already been addressed. The automatic execution of multiple tests and configurations using CDRouter is discussed in detail first.

Setting up a simple script to execute multiple tests using BuddyWeb

Perhaps the simplest way to execute multiple test and configuration scenarios with CDRouter is to use BuddyWeb. If you do not have the BuddyWeb add-on for CDRouter, skip ahead to the next section. Although BuddyWeb is primarily a web-based graphical user interface for CDRouter, it does include a few nifty options for executing test packages from the command line. These command line options are separate from CDRouter’s standard command line interface, known as buddy.

When a test package is executed via command line using BuddyWeb, it will look, feel, and behave like any other test package that has been executed using the standard BuddyWeb graphical interface. The basic command for executing a test package from the command line is:

  # buddyweb -package <test package name>

BuddyWeb's command line capability makes it very easy to schedule and execute multiple tests using a bash script. Here's a simple example script that will execute two pre-defined test packages using the "buddyweb -package" command:

-------------- START OF BUDDYWEB REGRESSION SCRIPT --------------

#!/bin/bash

# Example test automation script using BuddyWeb
# Created August 29, 2007, QA Cafe
# This script must be run as root.

LOGFILE=~qa/Desktop/scripts/regressionTest.log

# Redirect all output to LOGFILE
exec >>$LOGFILE 2>&1

echo “##########################################”
echo “Test script started: $(date +%F-%k:%M)”

# Make sure that we are running as root
if [ $(whoami) != "root" ]; then
    echo "Only root can run this script!"
    exit 1
fi

########## Starting test run 1
echo “Starting test run 1 at $(date +%F-%k:%M)”

# Restart DriverLoader
echo “Restarting DriverLoader”
/sbin/rmmod driverloader
/sbin/modprove driverloader

# Executing test package 1
echo “Executing test package 1”
buddyweb –package regression_test_1 –wait

########## Starting test run 2
echo “Starting test run 2 at $(date +%F-%k:%M)”

# Restart DriverLoader
echo “Restarting DriverLoader”
/sbin/rmmod driverloader
/sbin/modprove driverloader

# Executing test package 2
echo “Executing test package 2”
buddyweb –package regression_test_2 –wait

########## Done
echo -e “Test script finished at $(date +%F-%k:%M)\n”

-------------- END OF BUDDYWEB REGRESSION SCRIPT -------------- 

This script is fairly basic. Starting at the top, the first few lines set up a log file (the location and name are determined by the variable LOGFILE) where all standard output and standard error are redirected to. If you intend to use this as a template for your own script(s), be sure to edit the LOGFILE variable accordingly. Next is a check to verify that the script is being run by "root"; if this isn't true, the script will exit. Following this check, the wireless driver (handled by Linuxant's DriverLoader) is restarted to avoid any potential wireless failures. The first test package is then executed using the buddyweb -package command and the -wait option. The -wait option forces the system to wait until the current test package completes before continuing. Once the first test package has completed, the wireless driver is reloaded and the second test package is executed. That's basically it. Any number of test packages can be executed in this fashion by simply appending them to this script. The wireless driver should be reloaded before each test package is executed, if you plan to use wireless.

It is important to note that the notion of a test package is specific to BuddyWeb. A test package is a collection of tests, a configuration file, and various runtime options that together completely define all aspects of a single test run. Test packages and configuration files must be created ahead of time manually using BuddyWeb's graphical interface before the buddyweb -package command can be used. The notion of a test package does not exist when using buddy, as discussed later in this article.

As mentioned previously, test packages that are executed at the command line are in all other respects exactly the same as those that are executed using BuddyWeb directly. You will be able to view the results of your script in BuddyWeb. Should you need to access the data via command line or a script, all BuddyWeb data is located in the /usr/buddyweb directory, including configuration files, test packages, and results (in the /usr/buddyweb/configs, /usr/buddyweb/packages, and /usr/buddyweb/results directories, respectively).

Let’s use buddy instead...

If you don’t have the BuddyWeb add-on, or if you just want to try something different, you can also use buddy (CDRouter's built in command line interface) to execute multiple tests via a simple bash script. Before continuing you may want to check out the CDRouter Quick Start Guide which has a good primer on buddy:

http://www.qacafe.com/static/pdf/cdrouter-quickstart.pdf

A complete listing of buddy's options can be found by invoking the -help command line argument.

   # buddy -help

You'll notice that buddy is a full featured command line interface. In fact, BuddyWeb is built on top of buddy, and simply calls buddy commands with various options whenever it is used to execute a test package. Anything that is possible with BuddyWeb is also possible buddy. For the immediate task at hand however, there are only a handful of buddy options that are required to write a simple script that executes multiple tests. Namely, the -config, -testpath, and -module options. Unlike BuddyWeb which relies on test packages, with buddy you must explicitly specify both the configuration file and the test modules that you want to perform.

This is where the -config and -module options come into play. The -testpath option simply tells buddy where to look for the test modules that you are interested in running. In most cases you will be running CDRouter's standard test modules which are located in the /usr/share/doc/cdrouter directory (which is what the -testpath option should be set to). There are a few other useful optional options that we recommend using:

   -trace    Enable protocol trace messages
   -pt       Enable a one line summary of sent and received packets
   -capture  Create capture files for the LAN and WAN interface
   -logdir   Capture all test output to a result directory

These options will collect various information during each test that can be helpful when investigating strange behavior or debugging issues. Pulling this all together, the basic command for executing a single module using buddy is:

   # buddy -config myConfig1.conf -testpath /usr/share/doc/cdrouter -module
        nat.tcl -trace -pt -capture -logdir myDir  

The output of any buddy commands using the –logdir option will be placed in the specified directory. Each test will have its own log and capture files which will also be placed in the specified directory. We recommend that you create a separate directory for all buddy-related configurations and reference this directory explicitly when using the buddy -config option. Here's a simple bash script for executing the nat.tcl module on two different test configurations using buddy:

-------------- START OF BUDDY REGRESSION SCRIPT --------------  

#!/bin/bash

# Example test automation script using buddy
# Created August 29, 2007, QA Cafe
# This script must be run as root.

MYDIR=~qa/Desktop/myTests
LOGFILE=$MYDIR/regressionTest.log

# Redirect all output to LOGFILE
exec >>$LOGFILE 2>&1

echo “##########################################”
echo “Test script started: $(date +%F-%k:%M)”

# Make sure that we are running as root
if [ $(whoami) != "root" ]; then
    echo "Only root can run this script!"
    exit 1
fi

########## Starting test run 1
echo “Starting test run 1 at $(date +%F-%k:%M)”
DATE=$(date +%Y%m%d%H%M%S)

# Restart DriverLoader
echo “Restarting DriverLoader”
/sbin/rmmod driverloader
/sbin/modprove driverloader

# Executing test run 1
Echo “Executing test run 1”
buddy –config $MYDIR/myConfig1.conf –testpath /usr/share/doc/cdrouter
   –module nat.tcl –pt –capture –trace –logdir $MYDIR/$DATE 

########## Starting test run 2
echo “Starting test run 2 at $(date +%F-%k:%M)”
DATE=$(date +%Y%m%d%H%M%S)

# Restart DriverLoader
echo “Restarting DriverLoader”
/sbin/rmmod driverloader
/sbin/modprove driverloader

# Executing test run 2
echo “Executing test run 2”
buddy –config $MYDIR/myConfig2.conf –testpath /usr/share/doc/cdrouter
   –module nat.tcl –pt –capture –trace –logdir $MYDIR/$DATE 

########## Done
echo -e “Test script finished at $(date +%F-%k:%M)\n”

-------------- END OF BUDDY REGRESSION SCRIPT --------------  

Much like the BuddyWeb script discussed earlier, this script creates a log file for all standard output and standard error messages, and also verifies that it is being run by root (exiting if it is not). Tests are also executed in a very similar fashion with each test run occupying a specific directory that is given a unique name based on the date and time the test run was started (for example, 20070830103021 for 2007-08-30 10:30:21). All capture and log files associated with a particular test run are stored in the corresponding directory - if separate directories are not used for each test run, the capture and log files will simply be overwritten.

Make it automatic . . .

Regardless of which approach you take (BuddyWeb or buddy), we recommend setting either script up to execute automatically using cron. The exact use of cron will vary a little depending on your Linux distribution, but is usually very straightforward. If your system is running Anacron, just verify that the script is executable and move or copy it to the /etc/cron.daily directory:

   # chmod 777 backup
   # cp backup /etc/cron.daily

The /etc/cron.daily directory is a container for scripts that are executed automatically by Anacron each day. Anacron executes all scripts as root, so you don’t have to worry about permissions. All scripts in the /etc/cron.daily directory will be executed automatically at the time specified in the /etc/crontab file as uptime permits (/etc/crontab is the main configuration file for Anacron). We do not recommend editing the /etc/crontab file. If you would like to have the backup script executed at a specific time, we recommend that you manually edit the crontab file for root (the backup script must be run as root, which is why we need to edit root's crontab file). To do this, enter the following command when logged in as root (the -e option opens crontab with the default system editor):

   # crontab -e

Next just add a line to the crontab file in the standard cron format:

        # m h  dom mon dow   command

        00 20 * * * ~qa/Desktop/scripts/backup

The above entry will execute the script ‘backup’ located n the ~qa/Desktop/scripts directory every day at 8:00 pm. Refer to the cron man pages for additional information.

Once set up using Anacron or individual crontabs, the backup utility should run once per day. We recommend that you periodically analyze the backup process log file for errors that may go unnoticed otherwise.

Pulling it all together – setting up a regression test system

You’re probably starting to see the power in what we’ve done in this article to this point. Creating a simple bash script to run multiple configurations and test combinations and then setting it up to run automatically via cron is incredibly useful in and of itself. The next step towards the ultimate goal is to set up a system that will allow you to utilize these automated test scripts to efficiently and conveniently test multiple devices on the same dedicated physical hardware. The hardware is a standard CDRouter host loaded with multiple wired and/or wireless interfaces allowing the connection of more than one device. We refer to this system as a ‘pod’.

When setting up a pod, we typically use off the shelf PCs with as many expansion slots as possible. For Ethernet interfaces we like to use standard dual-port server style NICs (we recommend Intel PRO brand dual port server NICs). Dual port Ethernet NICs are useful because they double the total port density of the pod for a given number of expansion slots. This allows twice the number of devices to be connected to the pod when compared to traditional single port NICs[1]. In addition to Ethernet, we also like to have at least one wireless interface available on the pod (refer to our list of recommended network interfaces on our website: http://www.qacafe.com/kb/articles/show/162 ). One benefit of wireless is that multiple devices can use a single wireless interface for their LAN connections, thus reducing the number of dedicated Ethernet ports required (or allowing more devices to be connected).

A typical pod constructed using readily available off-the-shelf hardware may have seven Ethernet interfaces (one onboard and perhaps three expansion slots for up to three dual-port NICs) and one wireless interface. There are a number of different options for connecting routers to the pod with this configuration. One option would be to connect the maximum of six routers, which is possible if all are using wireless (and only wireless) for the LAN connections. This may not be ideal however, as certain test cases are automatically skipped when using wireless as the only LAN interface. Alternatively, you could connect three routers guaranteeing that each has a dedicated Ethernet LAN and WAN connection. A combination of the two options above would be to connect four routers, two with dedicated Ethernet interfaces for both LAN and WAN, and two more relying on wireless as the only LAN connection. Here's an example of you could allocate interfaces in such a system:

Note that the interface labeled eth0 (typically the interface integrated onto the motherboard of the PC) is reserved for connectivity to the network and is not used for testing. Also note that if you are using the CDRouter-Multiport add-on you can configure each router with multiple LAN (wired or wireless) and/or WAN interfaces. In the scheme above, Routers 1 and 2 are configured to have two LAN interfaces - one Ethernet and one wireless.

Once you have determined how many devices you want to connect to the pod and which interfaces they will be using, you can begin creating the configuration file(s) for each device. Each device will have its own configuration file which will specify the correct LAN and WAN interfaces based on your pre-determined interface allocation. Once the configuration files have been created, you will have to choose which modules to run on each device. This depends largely on your particular regression test strategy. As mentioned earlier there are a number of different strategies, including:

• Testing test the same make/model/version of a router in different modes
• Testing a particular software build on multiple platforms in the same product family (same make and version, but different models)
• Testing multiple versions of the same product (same make and model, but different versions)

Regardless of what your particular strategy happens to be, we recommend keeping track of your regression test setup so that you can quickly identify the purpose of each of the routers connected to the pod. Here's an example of how you could organize everything:

This is just a simple example – you can obviously choose any naming and bookkeeping convention that you like. Just be sure to update your script file(s) accordingly. Remember that if you are using a BuddyWeb based script you need only execute a test package, since a test package defines the configuration file and modules to be used.

We recommend an automatic power cycling system for the devices connected to your pod. This ensures that each time a test is run, the devices are in known initial states which should result in consistent and repeatable results from one test run to the next. There are a number of commercially available solutions – check out this article from our Knowledge Base for more information: http://www.qacafe.com/kb/articles/show/181.

In conclusion, your strategy may be completely different than any of those listed above – there are indeed many different ways to accomplish the ultimate goal using CDRouter. You will have to determine the optimal solution for your needs. In many cases a full regression testing system will involve periodically updating the routers connected to the pod with development code as it is released. This allows development code to be automatically tested in a consistent manner, which allows you to verify that:

1. It is stable (over long and/or short periods of time)
2. New features and/or functionality are working properly
3. Bug fixes are indeed fixing bugs
4. Existing features and functionality still work as expected

Doing this allows you to realize the full potential of CDRouter as a regression test tool. Any issues that are identified can be quickly fed back to the development team and re-tested quickly and consistently when fixed.

Wow, this system rocks! How do I protect all of this valuable test data?

You should back it all up. It would be a shame to lose any of the data you've collected due to a catastrophic failure. Fortunately backing up data in the Linux world is actually very straightforward. We've even written an article on it &emdash; check it out: Easy, efficient and reliable data backups for CDRouter. The most important directories to back up from CDRouter's perspective are those that you use to store results and config files. If you are a BuddyWeb user all configs, packages, and results are located in the /usr/buddyweb directory. You may also find that you want to backup the /home directory, especially if you have custom test scripts located there.

Footnotes

Questions or comments about this article?

Please contact QA Cafe Support: support@qacafe.com
www.qacafe.com
© 2007 QA Cafe