10 min read
Download this whitepaper (pdf)
There’s no question that broadband is the premier service delivery platform of the 21st century. With the explosion of new devices now aware of the network, and new services looking to exploit them, the devices you produce or deploy are becoming more complex than ever before. This puts pressure on everyone in the supply chain, from chipset vendor to service provider, to ensure their devices are tested well at every stage of the testing lifecycle.
We break this term, testing lifecycle, into four parts. Each part may have a slightly different meaning for vendors as opposed to service providers rolling out a new product or service.
For vendors, the testing lifecycle starts with beta testing of a new product. It may involve a device with newer hardware and more advanced features than the last model, or a product that branches into new territory for the vendor. It may come from customer demand, such a service provider planning a new or innovating service offering. For vendors, at this stage, it may rest entirely in the hands of the system-on-a-chip vendor.
Editor’s note: For incumbent vendors of broadband home devices, examples of the latter that we’ve seen include smart home hubs or cloud managed Wifi systems.
For service providers, cable providers, and other MSOs, this stage usually begins with a plan to upgrade their existing device deployment in order to roll out a new service offering. Requests for Proposals (RFPs) are put out and vendors come forward with proposals for the equipment to be used in the deployment. Providers will often perform testing in their own labs, ask for certification testing to be done, or ask for reports from their vendors before choosing a device or system to deploy.
Editor’s note: Smart Wifi is one of the hottest topics providers are pushing for today - whole-home mesh systems that can be managed to provide the end user with quality Wifi throughout their premises.
In this initial stage, the most important testing involves creating a baseline of successful functionality. This includes testing for:
Nearly every CDRouter test can be used at this stage, but for broadband gateways, we’ve produced a list we call the “top 100”, designed to verify the basic IPv4 functionality of a typical device with the following features:
See the CDRouter Top 100
These top 100 cases demonstrate that a device can boot, connect to a remote WAN termination, allow LAN clients to connect via DHCP, allow basic user traffic requiring DNS and HTTP, etc. Most devices of these types For IPv6 devices, a similar set of tests exist that can be easily built onto a CDRouter test package.
For devices that support TR-069, this is the stage to perform conformance testing with CDRouter’s TR-069 add-on and the CDRouter BBF.069 add-on. Testing TR-069 at this stage will make integration/deployment testing with a real ACS or simulated ACS installation go much more quickly.
Lastly, testing at this stage will probably uncover the most bugs and require several revisions of the baseline code for the device. Using an automated platform like CDRouter in conjunction with continuous integration tools will make sure nothing is missed when establishing this baseline level of validity.
After the initial baseline behavior of a device is established and bugs worked out, the next stage is integration/deployment testing. For both providers and vendors, this involves building a simulated network that mirrors the deployment scenario in which the device will be used, including the expected WAN termination, required network services like DNS or the plethora of DOCSIS related services, and simulating real user behavior on the LAN.
At this stage, we exercise how a device will behave in the face of real-world test setups and user behavior. This includes testing for:
Editor’s note: QA Cafe has an in-depth webinar on the importance of testing higher layer performance and application latency here.
CDRouter is built to simulate a real broadband network on the WAN and home network on the LAN, replicating a number of WAN types and the behavior of LAN hosts. For DOCSIS deployments, CDRouter can simulate the entire DOCSIS environment, including DNS, ToD, DHCP, TFTP, TR-069, SNMP, WebPA, etc., as well as let you build DOCSIS config files as part of your test configuration to examine if the DUT will correctly download and use them, or switch configurations to measure behavior.
CDRouter’s Performance add-on adds some of the biggest value at this stage, letting you perform application layer throughput over TCP and UDP, and most importantly, repeat that testing in a consistent environment over long periods of time to see how your device holds up to constant use.
With CDRouter’s TR-069 and SNMP “scenario” tests, you can build a set of test cases that replicate the exact bootstrap or configuration procedure that your device will undergo in deployment.
Additionally, you can use CDRouter’s ICS add-on to connect to actual real-world network resources required for your device to operate. This is important for devices that rely on cloud or mobile applications in order to properly bootstrap.
When devices are in deployment, testing still continues as either new issues arise or to eliminate known issues that weren’t critical but part of the initial release. No matter how much simulation we do, there is nothing like a production deployment to illuminate the edge cases and unexpected behavior inherent to any complex software.
Testing at this stage focuses on resolving reported issues, building new firmware, and then re-testing baseline to ensure that nothing that changed in the process of updating the firmware broke any older functionality. This includes:
CDRouter shines at this testing phase. Certainly, the thousands of test cases and hundreds of protocols that CDRouter supports are of key value, but the power of its overall automation platform and API integration cannot go overstated.
The key to using CDRouter here is to build test packages that can be run multiple times, overnight, as each new firmware revision is built and pushed to the test environment. When used with continuous integration, tests can be triggered and results collected using the CDRouter API to build a picture of the gains or losses made with each firmware revision.
Besides resolving firmware issues that arise over the course of deployment, vendors and providers must also be ready to add new functionality to existing platforms to avoid the expense of a massive hardware upgrade. They must also be cognizant of security vulnerabilities that are discovered by the networking community that would not have been known a-priori or revealed during normal operation.
This stage uses concepts from all of the previous stages, as developers must balance testing between testing new functions, resolving vulnerabilities, and making sure that these changes don’t affect functionality elsewhere. As such, the key steps here include the rigorous use of baseline testing, performance, and working with continuous integration, but also includes:
In addition to the benefits outlined in previous stages, the QA Cafe team makes a concerted effort to stay on top of the latest vulnerabilities that are specific to home/broadband network devices. As these vulnerabilities arise, our team makes an attempt to build test cases that are able to replicate and reveal the issue and test that it has been patched.
Testing, and particularly quality assurance testing, is often overlooked at various stages of the development process. Testing at every stage will help ensure robust, best-in-class devices, reduce support calls and maintenance, and limit liability from security exploits. It’s also important to make sure that results are fed back to those responsible for previous stages to avoid the “waterfall effect” common to software development.
CDRouter, as a holistic automation platform, is specifically designed to tackle these problems for developers and providers of broadband and consumer networking devices. Keeping your CDRouter system in service, and using it at each stage of the testing lifecycle, will significantly improve your development, procurement, deployment, and troubleshooting processes.