Jul 03, 2004

Software Testing...

Which ever software methodology you choose to partake in, testing is definitely part of it. As Dr. Pressman indicates in his Software Engineering book, testing is done because our failure as humans to communicate and perform with perfection.

Up until this point in my career, I have been rather fortunate as a software developer; the companies that I have worked (and interviewed) with have all taken software quality seriously, and have all had testing resources. Although I sometimes do not appreciate their thoroughness, now that I have read the introduction to Create a Quality Testing Program, I have a new appreciation for them.

As both Dr. Pressman and Tip 49 of The Pragmatic Programmer reminds us, "test your software, or your users will".

There is a certain amount of testing that developers do. When a developer develops a feature or fixes a bug, the developer must ensure that this feature works. The amount of testing that is done depends on a lot of factors, including what the change is and what the developer thinks about testing. The fact is that developing software is a constructive practice, and testing is said to be destructive, and this is where a developer testing the software that he/she creates is sometimes a conflict of interests, as they have to fix the bugs they find.

On the other hand, the developer that works on a project will definitely know it best, and this is why it is important to have communication between testing groups and developers. This is easily forgotten, but the problem is that if you do not know what the product does, it is hard to actually test the product. There are times where the design documents are not entirely accurate or completely detailed, and have communication with the developers definitely clears this up, but also there are parts that the tester may feel were insignificant for whichever reason, but in reality, it was really important (or vice-versa), and developers can also provide some ideas on how to really test the feature and what other functions may be affected.

The goal of testing is to find errors, though. If you are testing something and find nothing wrong with it, chances are that you are not really pushing it. But the question that has always been in my mind is what to test.

We all know about white box testing (writing code to exercise the code) and black box testing (using the product as the user's see it). In the case of black box testing, creating tests is slightly easier, if you have done your design. Within your design steps, you have probably specified some Use Cases of how the user interacts with your system, and therefore, this is a great starting place for your test cases. From this, you can derive some test cases from alternative ways of doing those functions, error paths, going through the documentation to ensure its correct, etc. Black box testing is extremely important, and it can pick up entire bugs that cannot be detected by white box testing, such as missing functionality, errors in the user interface and behaviour (such non-responsiveness), external data structure correctness, and initialization and termination problems. As Dr. Pressman points out, black box testing occurs each time your applications is ran, and therefore, you may want to do a little bit of this yourself.

But it is repetitive and it does not necessarily test every single code path in your code, which could be a rather large task to do. White box testing is meant to fill in the blanks. In order to do black box testing, a certain amount of development must be already underway, however, white box testing starts when you are developing software (to be fair, black box testing also starts at the same time in planning the tests). With a software process like Extreme Programming (XP) or Test Driven Development (TDD), a large portion of these white box tests are done in the form of unit tests, which test individual components alone, then the module, then a set of modules, etc.

There are some great reasons to use unit tests. First, your test program shows how to use the component. This serves as a learning guide for both new developers and developers using your component. The other use for test programs is that it exercises your source code. In other words, it is saying to your users that if you write your code like this test program, I guarantee you that your code will work.

The question always comes back to what to test. Testing frameworks such as JUnit and Boost help in creating tests, and some of their documentation on testing is useful, but there is some code that is always hard to test.

As the Pragmatic Programmer describes, it is easy to test something if you know what it does. This is easily applicable when using the design by contract style of software engineering where you clearly indicate what a function expects (precondition) and what the function will do (postcondition). With design by contract, it is easy to see how to test the code, where you pass the function various correct parameters and ensure that it works, and you pass the function various incorrect parameters and ensure that it does not work (presuming, of course, that the contract is strong). Depending on the function's input parameters, all parameters can be exhaustively tested, or it could take an eternity to test it.

When it seems that something will take an eternity to test, we must consider how useful such a test case is. For example, statistically speaking, most software failures occur at boundary conditions. If the input parameter can be an integer between 1 and 2 billion, most algorithms will treat 100 the same as 1000, therefore we can limit the possibilities to those that are more likely to break.

In the presence of standards, testing is also greatly assisted. Standards generally have some test suite and example processing that can be used. For example, I recently implemented a FedEx Ship Manager API program. The documentation provided with this API provided examples of what the input and output would look like, hence I developed a test program that would populate the input, and I would compare against what the output said it should be; testing for this project was extremely easy, and the program ran out of the box when tested with the actual server.

This actually brings up another point. It is important to keep your test programs around and run them regularly. When you make changes to an application, you might insert a bug. Running your test program will hopefully catch that bug. Testing can give you a false sense of security, though.

When you are fixing a bug, there are two things that should come to mind. First, are there other instances of bugs like this one, and why did my test cases not catch this bug? One method in dealing with bugs is to create a test case that actually captures the bug, and then fix it. Statistics show us that we regularly fix the same bug over and over again, therefore, this will allow us to catch that bug at the beginning. We all know that finding the bug when it is first added is cheaper than when the bug has been in the code base for a while.

There are cases where what the function does is extremely hard to map to a unit test. Sometimes using stubs and mock objects can sometimes enable some automated testing, and ample resources exist for these, including Part 2 of JUnit in Action, and hopefully in Pragmatic Unit Testing in Java with JUnit (which my copy should be coming in shortly). Using these techniques, you can surely test anything.

When and who writes these tests is part of your organization's ideals. XP calls for writing your tests and writing just enough of an interface to run your test program, then adding a bit of core and retesting until the the interface is complete. An off-shoot of this is Test Driven Development, in which your test programs are written before you write your interfaces. What makes TDD so interesting is that it prescribes that you never throw any test programs away, and that you run your test programs regularly. If your test programs fail, everything must stop until the cause of the failure is found and rectified.

Some developers are more comfortable in writing your classes first and then writing their test programs, and others are more comfortable writing the test programs first and then the interfaces. Either scheme works significantly better than a test program either written after the component is already integrated or worst, never written.

Filed In