QA Cafe - Automated Regression Testing with CDRouter

Application Notes > Automated Regression Testing with CDRouter

Automated Regression Testing with CDRouter

BuddyWeb 2.0 doesn't just serve up fancy HTML reports for human consumption. The same data is provided in strict, machine parsable language that is ideal for testing automation. In this article, we will discuss those formats, and even provide a complete, working demonstration you are free to implement for your BuddyWeb testing needs.

Introduction to BuddyWeb Data Formats

If you're familiar with BuddyWeb, you already know that it stores configurations, packages, and test results. These are resources and are displayed by default as HTML, because BuddyWeb is most commonly accessed by a human with a web browser. However, you can access these resources in other formats.

There are several formats available: HTML, CSV, JSON, YAML, and XML. Some resources have a restricted selection of formats. Please see the chart detailing this information below. Most of these formats are easily parsable and are intended for automation.

HTML is the default format. If you request a resource with no file extension, or with .html, it is returned as HTML designed for manual interaction.
XML is extremely popular and easy to integrate into custom applications
CSV is data separated by commas. BuddyWeb provides this format for only the test result data. All of the other data provided is meta-data, or "information about information", and is better suited by the other formats mentioned here.
YAML is a markup language designed to be human readable. Parsing it can be more computationally expensive because of this. Language support for YAML is often not as strong as other formats that BuddyWeb provides.
JSON is a lightweight alternative to XML. Its language support tends to be stronger, with robust modules that can handle the subtle differences between various JSON data sources. Internally at QA Cafe we try to use the JSON interface because of the excellent support provided by the various language we use.

Data Format Examples

Let's see some examples. One resource BuddyWeb provides is /live. The /live resource tells us what the system is doing. The default HTML view is well known:

We could burden ourselves with parsing through the HTML source code to automate learning the BuddyWeb status, but it would be much easier to query the JSON format instead. We'll switch to viewing source code using the curl web HTTP client:

# curl -s http://lime.lan:8015/live.js
{
"duration":14,
"pass":0,
"paused":false,
"started_at":1262879982,
"active":true,
"progress":"4%",
"id":"20100107105938",
"package_name":"dos-tests",
"fail":0
}

In this example, a BuddyWeb package was executed a moment before capturing the JSON view of its /live resource. We can immediately understand how much easier it is to parse this format instead of the HTML format.

When you request configurations, packages, or results, you can specify a specific ID to retrieve the complete record. If you omit an ID, you will retrieve a complete list of every record. If you request /packages, you get the HTML view of every package. When you click on the first package, it loads /packages/1 in HTML. If you request /packages.js you get the configuration of every package in JSON, and if you request /packages/1.js you get only the configuration of the first package in JSON.

Meta-Information vs Actual Data

Changing the extension from .js to .yaml or .xml provides the same information in those formats. The one exception is for results. If you request the .js, .xml, or .yaml formats for a result, you will retrieve meta-information about the results, such as the number of passes and failures, the duration, et cetera. If you use the .csv format, you will retrieve the complete results for each test case of the package in CSV format, one test case per line. Users of the Buddy template system will be familiar here as the templates can output identical information.

Available Formats per Resource

Table of Available BuddyWeb 2.0 Data Formats
	.html	.xml	.csv	.js (JSON)	.yaml
/live	*	-	-	*	*
/configurations	*	*	-	*	*
/configurations/<ID>	*	*	-	*	*
/packages	*	*	-	*	*
/packages/<ID>	*	*	-	*	*
/results	meta	meta	data	meta	meta
/results/<ID>	meta	meta	data	meta	meta
Note: 'meta' indicates meta-information about a data set, and 'data' indicates the data set itself. Please see the above paragraphs for further explanation.

An Example Implementation

QACafe uses a fleet of servers which we call 'pods' to perform nearly constant regression testing. Running the tests by hand would be very inefficient, but it is trivial when we automate a program which utilizes the BuddyWeb JSON and CSV resource formats.

The example program we provide requires Perl and a handful of CPAN modules. The program will detect any missing modules and give you instructions on how to perform their one time installation using the CPAN module installer.

Installation Instructions:

mkdir -p /usr/local/pod/perl
cd /usr/local/pod/perl
wget http://www.qacafe.com/other/qacafe_pod_demo.tar
tar xvf pod-demo.tar
./report.pl

This CLI program will let the user execute a batch of BuddyWeb packages on a local or remote BuddyWeb system, and receive a rich email report with links to each failure encountered.

Note: The methods used by this program are supported by the BuddyWeb API reference. This is a safe way of interfacing BuddyWeb and allows future-proof data retrieval from a BuddyWeb system. Please refer to the BuddyWeb Advanced Scripting API Reference for more information. (You must first log into the QA Cafe Customer Lounge to retrieve this PDF document).