A traceability matrix can be created using a spreadsheet or any database tool such as MS Access. For a sample traceability matrix you can email me at decarlo@mindavation.com.

Assuming you have a documented (and used!) testing process and you have a traceability matrix in place – you’ve armed yourself with some critical information to determine when you’re done testing. Do all requirements have at least one test case tied to them? If not – you’re probably not testing the solution sufficiently. Conversely, if a given requirement has more than five test cases to evaluate it, that particular requirement is probably being over-tested.

The next characteristic to examine is defects. First, determine the number of defects identified for a given test run (several test cases should be executed during a test run). Next, determine if the number of defects are increasing or decreasing for each test run. The first time the test run is executed, expect to find a high number of defects (because the whole goal of testing is to find defects!). The 2nd run (of the same set of test cases) should have fewer defects. By the time you execute the 3rd run – your test run should be surfacing even fewer errors, and certainly lower severity defects. If the number of defects is going UP per test run, you are obviously not nearing the end of testing. In fact, this may be an indication that when defects are corrected additional errors are being introduced. More than likely the person correcting the original defect did not sufficiently unit test the product prior to returning it to the test run environment.

So, how do you know you’re done testing? Here’s a general rule of thumb. Once each requirement (based on the traceability matrix) has been tested and yields no defects or low severity defects that can be corrected later, the product is ready for “prime time.” Could I test longer and find more defects? You bet – but the time is takes to find those additional defects is usually not worth the team’s time and effort.

If you’re receiving pressure from your management team to reduce the duration of the testing effort to meet schedule constraints (and they are not able to provide you additional resources to perform the testing) plan a meeting with your business management representatives so they can review the traceability matrix and indicate which requirements do not need to be tested prior to implementation. This approach moves the decision making process where it belongs, with the business. The business knows better than you do which requirements are essential for their day-to-day operations. By showing management how the traceability matrix is driving the testing schedule they become better educated on how detailed the testing process can be. Sometimes management will then extend the scheduled implementation date or provide additional testing resources to ensure all test cases are executed as planned. If a decision is made to not test all requirements prior to implementation, a risk strategy should be developed to prepare for the potential impact of having numerous defects post implementation.

continue>>

<<back