CDRouter 5.0 includes the second generation of our comprehensive web-based graphical user interface also known as BuddyWeb. BuddyWebv2 is a significant upgrade which utilizes a powerful SQL database and Ruby web-server to deliver greater scalability, better management of results and test data, and an improved user experience.
When utilized in conjunction with our NTA1000 Network Test Appliance, CDRouter 5.0 is capable of executing test packages that contain millions of test cases and storing test data for millions of individual test runs. These metrics represent significant improvements over previous releases and allow CDRouter 5.0 to be used for extensive testing durations.
Sorting through a large set of test results to locate patterns can be very difficult. CDRouter 5.0 provides a number of intuitive and powerful data management features that dramatically simplify this process. Some of these key features include:
are user-defined labels which can be applied to configuration files, test packages, and test results. Filtering by tags allows a user to sort pages of test results quickly with any metric imaginable.
the test results in the result summary, to show significance
test cases of particular interestThis Application Note provides an overview of the advanced data management features included with CDRouter 5.0. In addition, a few examples are provided which highlight ways in which these features can be used to streamline and optimize your testing process.
With CDRouter 5.0 any number of tags can be independently applied to configuration files, test packages, and test runs. Tags are essentially arbitrary labels that can be used to identify any number of unique characteristics associated with a particular test run or device. For example, information such as model numbers, firmware versions, feature sets, or operating modes can all be applied as individual tags to configuration files and test packages. Likewise, tags can be applied to individual test runs to help organize them.
Individual test runs automatically inherit the tags applied to the associated configuration file and test package that were used. This makes it very easy to filter test runs based on any of the characteristics that have been associated with the particular device being tested.
Tags can also be used to identify specific test runs that may be interesting or that may warrant further investigation. For example, if a user found that the results of a recent test run imply a new bug, that test run could be marked with the tag “bug”. Later, other users can search for known bugs by simply clicking the "bug" tag filter in on the CDRouter Results page.
Before proceeding, it is important to clearly define some common terminology used throughout this Application Note. A test package consists of a configuration file, a list of test cases, and any selected run time options. The results of a test package that have been executed are referred to as a test run. All test runs are displayed on CDRouter's Results page. Clicking on any test run will display the individual test case results for all test cases included in that test run.
As mentioned earlier, tags can be applied independently to configuration files, test packages, and test runs. Furthermore, all tags applied to configuration files and test packages are automatically inherited by any test runs utilizing them. Applying tags is very easy and can be done a number of ways, as discussed below.
Tagging Configuration Files
Configuration files can be tagged at any time by simply navigating to CDRouter's Configurations page. To tag a configuration file, just select it and click the Edit button. In the Config Editor window there is a Tags box that can be used to specify a comma separated list of tags for that particular configuration file. Once saved, any tags will be visible on the Configurations page as blue highlighted text boxes appended to the Description of each configuration file. On the left side of the Configurations page a list of popular tags is also displayed. This list of tags can be used to filter configuration files by tag. Clicking the All Configs link above the list of popular tags will reset the tag filter and display all of the configuration files.
Tags can also be applied to configuration files directly when they are initially created using either the Config Wizard or New Config options. Note that copying a configuration file or files preserves the original tags as well. CDRouter's generic import mechanism also allows imported configuration files to be uniquely tagged during the import process, which is useful for differentiating imported files from locally generated files. Note that by default CDRouter 5.0 applies the tag “import” to all imported files, although this can be easily changed during the import process.
Tagging Test Packages
All of the mechanisms for tagging and filtering configuration files are also available for test packages. To tag test packages simply navigate to CDRouter's Packages page and create, edit, or copy a package to apply tags. Remember that individual tags must be comma separated.
Tagging Test Results
Test runs can be tagged at run-time from the Job Queue which is available on the Utilities page or by using the Launch button which is available when a test package is selected on the Packages page. In either case tags can be applied to uniquely identify a particular test run, ie "Mike's nightly smoke test."
Individual test runs can also be tagged from CDRouter's Results page. To tag one or more test runs, select them and click the Add Tags... button. This will produce a pop-up box that accepts a comma separated list of tags. Once saved, the specified tags will be applied to all test runs that were selected.
Another way to tag a test run is to simply click on its row in CDRouter's Results page to open its detailed view in the current window, or check off as many results as desired and load them into new windows with the View button. Once the detailed view page is displayed, a user may load the expanded view by clicking the middle bar. Additional information about the test results is stored here, including an editable tags list.
Test runs can be starred 
to signify importance or uniqueness. Individual test runs can be starred directly from the Results page or from the detailed view by clicking the star icon. Once clicked, the star icon will turn yellow indicating that it has been applied to the selected run. Starred test runs can be quickly filtered by clicking the Starred link on the left-hand side of the Results page.
Likewise, one or more runs can be archived by selecting them and clicking the Archive button. Once archived, test runs will disappear from the main Results page. Archived test runs can be displayed by clicking the Archived filter link on the left-hand side of the Results page. Archiving old test runs is a great way to clean up the main Results page without actually deleting anything. Note that test runs can also be Unarchived which returns them to the main Results page.
Note that all tags applied to a particular configuration file, test package, or test run, including star and archive status, are preserved when exported. One or more test runs can be exported to a .zip or .tgz file by selecting them and clicking the Download .zip [or .tgz] button on the Results page. The default archive type, .zip or .tgz, can be configured on the Preferences page. Like configuration files and packages, test runs can also be tagged when imported.
Test runs can also be tagged from the command line. This is useful when executing test packages via script as part of a larger automated test environment utilizing CDRouter. To tag a particular test run from the command line add the -tags option to the buddyweb -package command in any existing scripts. For example, to apply the tags “CPE 36b” and “firmware rev 7.8.2-00.1” to a particular test run, add this -tag string:
buddyweb -tags CPE 36b,firmware rev 7.8.2-00.1 -package nightlySmokeTest -wait
CDRouter 5.0 provides global tag management as well by clicking on the Manage tags link on the Utilities page. This will produce a "tag cloud" revealing all of the tags that are currently in use on the system. Tags that are used more frequently appear in larger text whereas tags that are used less frequently appear in smaller text. Clicking on any of the tags in the cloud will display a complete list (with links) of exactly which configurations, packages, and results this tag is currently applied to. From this view the tag can be deleted or renamed. Any changes to the tag will ripple through the system - for example if a particular tag is deleted, all configurations, packages, and results utilizing that particular tag will be automatically updated and the tag will be removed. The tag manager can also be invoked by expanding the Popular Tags window on the left-hand side of the Configurations, Packages, and Results pages and clicking the Manage tags link.
CDRouter 5.0 allows configuration files, test packages, and test runs to be filtered by tag and status (either starred or archived in the case of test runs) and sorted by the attributes which are displayed in the column headers on the Configurations, Packages, and Results pages. Filtering and/or sorting can be useful for quickly locating specific data within CDRouter 5.0.
Filtering Results by DateIn addition to filtering by tag, the Results page in CDRouter 5.0 provides pre-defined filters for all test runs, ie Current Results), Starred results, Archived results, and results occurring on specific dates. Clicking the filter By Date link on the left-hand side opens a utility that includes filters for test runs that were executed today, within the last seven days, within the last 30 days, within this month, and within this year. In addition, all test runs occurring before, on, or after a specific date or within a range of dates can also be filtered.
Flagging Individual Test Results and Applying Test Case Filters
Another tool made available in CDRouter 5.0 is the ability to flag specific test results within a test run. This feature is very useful for identifying a specific test case as warranting further investigation or analysis. One common use of flags is to indicate the exact point during a test run in which a certain unexpected behavior or bug occurred.
Using flags in conjunction with the star functionality discussed earlier can make the process of locating and sharing interesting test results very easy and intuitive. As test data is analyzed, any interesting test case results can be flagged and the test run itself can be starred and updated with more descriptive tags and/or notes. Later, the test run can be located by applying the Starred results filter on the Results page. Note that the results for a particular test run can be starred directly from within the results View mode. Furthermore, additional tags and notes can be added by expanding the result details at the top.
When in the detailed results view a number of test case specific filters are provided. By default CDRouter displays all of the test case results in a particular test run. However, the individual test case results can be filtered by Passes, Failures, and Flagged. CDRouter also includes Next Failure and Prev Failure functions which make navigating for failures through the individual test case results of large test runs even easier. Note that when using the Next Failure and Prev Failure functions, CDRouter will simply jump to the page containing the next or previous failure, respectively.
Another powerful feature of CDRouter 5.0 is the ability to export the Results page as a CSV or XML file by clicking the "Export as .CSV or .XML" links at the bottom of the Results page. Any filtering or sorting applied to the Results page is maintained during the export process as well. CSV files can be easily imported into various spreadsheet applications making it easy to generate graphs and identify trends in device behavior over longer durations. A summary of the test case results for a particular test run can also be exported to CSV directly from the detailed results view page.
There are a number of excellent, real-world applications of the data management features provided in CDRouter 5.0 Here are just a few.
Applying device make, model, and feature tags to each configuration file makes it easy to quickly identify any configuration file(s) associated with a specific device or feature set. Furthermore, any tags that are associated with a configuration file will also be applied to any test runs generated with that configuration file, making it easy to also filter results by device or feature set. Configuration files can be tagged upon creation or at any point in time be simply editing the configuration file.
Once all configuration files have been tagged with device make, model, and feature information, filtering through large sets of stored test runs is easy. Filtering test runs by a single model also allows long term trends in device behavior to be quickly identified. For example, the test runs for a specific device can be filtered using the 'model' tag applied to a configuration file. Looking back over time at the test runs for this device may quickly reveal that certain firmware versions performed better than others or that certain individual test cases periodically failed.
Another great use of tags is to apply device firmware or software version information to a specific test run at run-time. This feature is extremely useful for automated test environments utilizing CDRouter.
For example, a common scenario involves a master test script which is executed on a nightly basis. This master test script first loads firmware onto a specific device and then executes a set of pre-defined CDRouter test cases. With CDRouter 5.0 tags can be applied to test runs at run time, making the process of uniquely associating a test run with a specific firmware or software version very easy. The master script simply needs to apply the firmware or software version as a tag at the command line when calling the CDRouter test packages, as outlined in the Tagging from the Command Line section above.
Also note that individual test runs can be uniquely tagged at run-time by users working with CDRouter manually. This is great for permanently marking a specific test run with some identifying information. For example, a developer may make a quick tweak to a specific line of code and want to quickly test it without having to commit the change and build a new version of firmware. With CDRouter 5.0 this can be easily accomplished by making the tweak and then running the CDRouter test package and applying a unique tag to the test run. This identifies the test run not only for the developer but also for any other users that may be utilizing the CDRouter system.
Some test strategies focus on isolating and testing major functional areas independently. For example, some users may want to test TR-069 independently from multicast. As a result, they may build individual configuration files and test packages that focus on just TR-069 or multicast. Tagging these files with "TR-069" or "multicast" make it easy to quickly identify them and filter test runs by feature set.
Tagging by feature set is also a great way of organizing wireless configurations and test runs. For example, a number of identical wireless capable devices may be connected to a single CDRouter system with the goal of running the same set of baseline test cases against all of the wireless modes and security schemes supported by the device across different versions of firmware. Each device therefore differs only in its wireless configuration. Configuration files for each individual device and wireless configuration can be built and tagged accordingly using tags such as "basic wireless", "WPA-PSK", "AES", "TKIP", "hidden SSID", "802dot11n", or "5GHz". Filtering test runs by the various tags used can reveal interesting results.
Flagging specific test case results is also a good way of highlighting areas when communicating with team members or contacting QA Cafe Support with questions. For example, after a test run has yielded results which could benefit from a QA Cafe Support Team inspection, simply view the test run results and flag the relevant test case results in question. Export the results as a tgz or zip file and include them as an email attachment. The QA Cafe Support Team will be able to easily inspect your results and locate the test cases in question.
Please contact QA Cafe Support: support@qacafe.com
www.qacafe.com
© 2009 QA Cafe
