Non-Functional Tests

Non-functional requirements describe attributes of a system’s functional behavior—i.e., how “well” a system or component should fulfill its function. Its implementation strongly influences customer/user satisfaction and therefore also how well liked the system is. According to [ISO 25010], such characteristics include user satisfaction and efficiency. From the manufacturer’s viewpoint, flexibility and portability are important aspects of a product’s maintainability, as these help to reduce maintenance costs.

The following non-functional system characteristics¹⁴ should be covered and checked by corresponding tests (usually in the course of system testing):

• Load
  Measurement of system behavior under increasing load (for example, number of parallel users, number of transactions)
• Performance
  Measurement of processing speed and response times for specific use cases, usually in conjunction with increasing load
• Data volume
  Observation of system behavior dependent on data volumes (for example, when processing very large files)
• Stress
  Observation of system behavior in overload situations
• (Data) Security
  Combating unauthorized system and/or data access
• Stability/Reliability
  Under constant use (for example, by measuring the number of system malfunctions per hour for specific user profiles)
• Robustness
  When subjected to user errors, programming errors, hardware failure, and similar. Testing exception handling and restart/recovery behavior.
• Compatibility/Data conversion
  Testing compatibility with existing systems, especially during data import/export
• Different configurations
  For example, using different OS versions, languages, or hardware platforms
• Usability
  Testing ease of learning and simplicity of use, including comprehensibility of output for various user groups (see also Acceptance.
• Compliance of system documentation with system behavior
  For example, user manual vs. GUI, or configuration description vs. actual behavior
• Maintainability
  Comprehensibility and up-to-dateness of development documentation, modular structure, and so on

Non-functional requirements are often incompletely or vaguely worded. Attributes such as: “The system needs to be easy to use” or “fast response” cannot be tested in their current form.

Our Tip

• Testers should participate in reviews of the requirements documentation and make sure that all the listed (non-)functional requirements are worded in a way that makes them measurable and therefore testable.

Furthermore, many non-functional requirements are ostensibly so obvious that nobody thinks of mentioning or specifying them. Such “assumed requirements”¹⁵ are often highly relevant, and the system must possess the corresponding implicit attributes.

Case Study: Assumed vs. specified requirements

The original VSR system was designed to run on an operating system provided by a well-known market leader, and has a look and feel that largely matches the OS styles and conventions.

The new, contemporary user interface in VSR-II is browser-based. The marketing department has worked with a web design agency to create a comprehensive style guide that dictates how VSR–II’s interfaces should look on a system-wide basis.

Usability tests use checklists to ensure that the user interface of each module adheres to the specifications stipulated in the style guide. These tests reveal that some of the specified display and interface elements are difficult to read on mobile devices. Although no explicit “easy-to-read” requirement has been formulated, the decision is made to modify the corresponding parts of the style guide so that the assumed requirement of good legibility is fulfilled for all elements of the interface.

Side Note: Testing non-functional attributes using functional tests

It is often expedient to use existing functional tests to verify non-functional attributes. Non-functional tests are usually black-box tests, and can be seen as a kind of “backpack” strapped on to functional tests. An elegant general approach to these kinds of tests is as follows:

The scenarios selected from the functional tests represent a cross-section of the functionality offered by the entire system. The non-functional dimension must be visible in the corresponding test scenario, and is measured during the test run. If the measured value is below a predefined threshold, the test is classified as passed. Practically speaking, the functional test scenario serves as the measurement procedure for determining the non-functional system characteristic you want to test.