Gorilla testing

Nowadays there are many sophisticated testing tools, ranging from unit testing frameworks such as NUnit through to profiling tools such as Compuware’s DevPartner Studio. With the help of these tools and useful techniques such as code reviews, it’s possible to be confident that you’ve found most of the bugs hiding within your application. But one technique that seems to have fallen out of favor among mainstream developers is the idea of random testing.

Although the concept of random testing might conjure up images of a late-night debugging session where you do a passable imitation of an angry gorilla by haphazardly pressing as many keys as possible on your keyboard in order to upset your program, there are better ways of doing it. One random testing tool is called fuzz, a program written by Justin Forrester and Barton Miller in the computer sciences department at the University of Wisconsin. This interesting tool can be used to generate and send a stream of random valid and invalid Win32 messages or random valid mouse/keyboard events to a Windows application.The idea is to test the stability of your application when it’s subjected to a large amount of random input. If, as a result of the random input, your application crashes or hangs in some way, it’s considered to have failed the test.

The first technique of sending a stream of random mouse and keyboard events to your application simulates the behavior of a very erratic end-user who just keeps banging away at your software in a haphazard and tireless fashion. The other technique of sending a stream of random Win32 messages simulates what might happen when your application receives one or more dodgy messages from the Windows kernel, or when it’s subject to an attack by a malicious program. In either case, you want your application to resist the onslaught or at least fail in a graceful manner by saving the user’s work before crashing.

The authors of this clever tool tested many Windows and Unix programs, including some of Microsoft’s most popular applications. The results make sobering reading. Using Windows 2000, 64.3% of the applications tested with random valid mouse and keyboard events failed with either a crash (42.9%) or a hang (21.4%). Some of the programs that failed this test included Access 2000, Netscape 4.7, Paint Shop Pro 5.03, PowerPoint 2000, and Word 2000. When the same applications were tested using random valid and invalid PostMessage calls, a total of 71.4% of them failed. Using random valid and invalid SendMessage calls was even worse—85.7% of the applications failed by crashing or hanging!

If you want to download this freeware tool and try it against your Win32 or .NET applications, you can find fuzz and a short paper discussing it here.