Testing New Product Versions

So far, we have simply assumed that a software development project ends when its acceptance tests have been passed and the product has been delivered. However, things look very different in reality. Initial delivery marks only the beginning of a software product’s lifecycle. Many software products are used for years, or even decades, and are usually modified, extended, or repaired multiple times during their lifetimes. See our case study section below for an example:

Case Study: Evaluating the VSR-II hotline tickets

VSR-II has replaced the original VSR system and has been running for a while. To find potential weaknesses in the new system, the end-user support team analyzes the queries it received in the first few months since the system went live. Here are some examples:

1. A few dealerships have been running the system on a non-approved platform with a legacy operating system. The system’s browser cannot display some elements of the user interface and the system sometimes crashes when it accesses the host system.
2. Some customers said that they found selecting extras in the old VSR system complicated, especially when comparing the prices of the various equipment packages. As a result, VSR-II enables the customer to save various configurations and call them up once changes have been made. However, some customers want even more convenience and would like to be able to compare different vehicle models plus equipment packages.
3. Because the corresponding calculation rule was overlooked when programming the insurance module, some rarely requested insurance tariffs cannot be calculated. A similar problem was already known in VSR.
4. In rare cases the order confirmation still hasn’t reached the factory computer after waiting for 15 minutes. However, in order to avoid wasting network capacity, VSR-II cuts the connection at the latest after 15 minutes. This means the dealer has to repeat the order and send the confirmation to the customer separately later on. This causes customer dissatisfaction due to the unnecessary wait for order confirmation.

Issue #1is essentially the dealer’s problem. However, the manufacturer may modify the system to prevent the dealer having to make costly hardware updates.

Issue #2will always crop up, regardless of how comprehensive the original system requirements were. A new system delivers a new user experience, and the customer preferences this generates will only be revealed once the system has been running for a while.

Issue #3could have been found during system testing, although testing is a spot check that doesn’t guarantee freedom from defects, but only a certain probability of finding existing failures. A good test manager would check which tests would have been necessary to reveal this oversight and would modify the test schedule accordingly.

Issue #4was found and remedied using a patch. VSR–II no longer loses the order data, but instead automatically repeats the order process multiple times. However, this still doesn“t prevent the customer having to wait if the order confirmation takes a while. A genuine solution would involve modifying the batch processing system on the company’s host machine—a solution that lies outside the VSR-II remit and thus cannot be remedied in the short term.

These four examples represent typical issues that occur with time, even in the most mature systems:

• The system is used under new, unforeseen, unplanned conditions
• Customers make new, unforeseen requests
• Functionality is required that covers rare (and therefore overlooked) special cases
• Issues evolve and only occur sporadically or once the system has been running for an extended period. These kinds of issues are often caused by external factors.