Why You Should Care About Impairment Testing of Internet Protocols


The Internet Is An Imperfect and Hostile Place

The world is an imperfect place.  The internet is no exception.  The internet has its good days and it has its bad days.  Or to be more precise, the internet has its good seconds and its bad seconds.

Blemishes in internet performance arise from many sources.

  • There will always be natural random events that damage internet packets. These events include lightning, solar flares, electrical noise, wind (blowing branches onto wires or across radio paths), and even things as mundane as a squirrel chewing on a cable.
  • Internet performance is strongly affected by congestion within the net. Congestion is common on the relatively skinny links that bring the internet into homes and small businesses.  Congestion is quite common in the more industrial portions of the internet where large capacity “pipes” come together at inter-carrier exchange points and big data centers.  Congestion doesn’t just mean that data flows become sluggish.  Congestion can also cause loss of packets or, somewhat surprisingly, congestion can cause the duplication or re-ordering of data packets (typically as side effects of congestion induced changes in packet routing.)
  • Network devices are often underpowered. Their software is often flawed.  And their configuration settings are often inappropriate for the loads they are expected to handle.    These deficiencies can cause devices to fumble when handling internet traffic, or cause them chatter incorrectly, or even to blither incoherently.
  • There are intentional malicious attacks that intend to disrupt the internet by inducing ill behavior or by simply inducing large traffic overloads.
  • And finally, there are a lot of old devices on the internet. Although many of us think of the internet as something modern we should recognize that today’s internet has gone through many generations of ideas, protocols, and implementations.  These may speak old dialects.  These devices are somewhat like a Shakespearian character dropped into our modern world: they may be able to interact with others, but not well.

The software in many network devices is written as if many of these imperfections do not exist.  This intentional ignorance of risks makes it easy for developers to produce new products quickly and cheaply.  But it also means that once in the hands of customers and consumers these products may wobble in strange ways or simply fail when they encounter network conditions that were not considered or anticipated.

What Is Impairment Testing?

Impairment testing is a method through which developers exercise their network code (including the applications layered onto that code) and their network products without spending time and money traveling about the internet looking for unusual or ill conditions.

Impairment testing uses special network tools to create unusual, odd, bad, or improper network conditions.

Often network impairment tools act in a “man in the middle” role. That means that the impairment tool sits between the device under test (DUT) and the rest of the network.  As a consequence, every packet to and from the DUT has to pass through the impairment tool.

At its most basic level a network impairment tool may do what we call “standard impairments”:

  • Packet Drop
  • Packet Duplication
  • Packet Delay (fixed or variable, the latter being called “jitter”.)
  • Packet Corruption
  • Rate Limitation
  • Changing the sequence in which packets are delivered.

More sophisticated network impairment tools may enrich these standard impairments by adding the ability to define triggers, to create burst effects, or to choreograph patterns that vary the impairments over time.

Some network impairment tools can also alter the contents of packets or create new packets.

And some tools can track the state of protocol handshaking and apply impairments in ways that are affected by that state.  For example, a stateful impairment might reach into a stream of video packets and swap the order of the last packet of a video frame with the first packet of the next frame.

Controlling Chaos

It is broadly acknowledged that the weakest and least tested parts of most software are the parts that handle errors and infrequently occurring conditions.  This under-tested code is usually the place where the bugs and security holes lurk.

The intent of impairment tools is to force software in a device under test (DUT) to exercise those code pathways that are not followed under routine conditions.

But simply throwing packet noise at a DUT is likely to leave a developer dazed and confused.  A developer needs tools that can focus the impairment effects in repeatable ways so that problems be isolated, diagnosed, repaired, and then re-tested to be sure that the repairs actually fixed the problem.

What Are Some Examples of Network Impairment Testing?

A large storm is blowing outside as I write this note.  The rain and wind are creating terrible network conditions.  Many packets are being lost and the packet delay ranges from a few milliseconds to many seconds.  Many of my Apps have become temporarily unusable because of these conditions.

Storms, and these kinds of network conditions are hardly unusual.  And applications that do not work well under these conditions are also hardly unusual.  Some applications degrade nicely and gracefully; but many others fail in awful ways that do not reflect well on their developers or vendors.

A person developing a network-based product who cares about how their application or product behaves under imperfect conditions can use an impairment device to re-create those conditions in their development lab.

Notice that in the testbed shown above that all packets to and from the device under test (DUT) are flowing through the impairment device and are being equally subject to the impairment conditions.

Not All Packets and Flows Need To Be Impaired In the Same Way

In these days of home or company networking, or as the “Internet of Things” becomes more widespread it will become less likely that all packets will be candidates for impairments.  On a typical home or business network a lot of traffic may be handled locally on good quality, high bandwidth paths that do not exit the home or business and thus are not likely to encounter bad conditions.  For this reason it is useful if an impairment device can distinguish between “local” and “non-local” traffic and impose different impairment regimes on each.

Sometimes a developer may want to focus on certain kinds of traffic.  For instance, Voice over IP (VoIP) is often far more sensitive to network conditions than typical web browsing.

For these reasons it can be very useful if impairment tools can differentiate between different kinds of traffic.

What Kind of Results Can One Expect?

Sometimes bad network conditions cause application or operating system code to completely fail; to crash.  From a developer’s point of view these are often the easiest to diagnose and fix.  From a user’s point of view they are certainly inescapably obvious failures.

But more often an application or network stack will degrade more rapidly than it should or begin to exhibit odd side effects.  We have all experienced voice conversations that become incomprehensible or sound like we are in an echo-chamber even when there is no apparent degradation to web browsing.  Without impairment tools developers have a hard time testing and tuning their code so that the user gets the best possible experience.

Because the effects of network impairments on an application are often subtle rather than catastrophic the developer needs to understand the desired behavior of the Device Under Test (DUT) and be sensitive to deviations.  For example, if one is subjecting a streaming video application to impairments the impact may be reflected, rather obviously, as visual blotches on the receiving screen.  Or the impact may be reflected less obviously as a stuttering of the received frame rate – resulting in a distracting video that jerks and sputters along.

What Are the Risks of Failing To Do Impairment Testing

The worst thing that can happen to a product vendor is to discover that the product is failing or misbehaving when it is used by customers.  Problems at that phase are expensive to diagnose and repair, and they damage the reputation of the vendor.

Impairment testing can help you find and repair flaws before your product reaches your customers.

Impairment testing is not a panacea.  Impairment testing is, however, a prudent tool to have in your product testing suite.
If you’re looking for a reliable, secure way to test your business’ critical apps and functionality by simulating various network conditions, check out the Maxwell family of network emulators or download our whitepaper to understand how apps perform on the network, even under adverse network conditions.