In-depth software testing made easier

The software testing paradox

Prevention is better than cure, and many software companies know that the same holds for their products as well. They understand that, in the long run, proper testing ahead of release is much cheaper than fixing software bugs further down the line. In most cases, however, no rigorous testing is performed, because this is a complex process that costs time and money in the short term.

In practice, testers often focus on a ‘happy flow’, testing mainly for the faults they expect. This creates blind spots in the testing process, so that faults that testers fail to take into consideration go unnoticed.

Random input data

Tools for automated testing can speed up the testing process and help avoid such blind spots. Fuzz testing – or fuzzing – is particularly promising. This involves exposing software to unexpected or random input data. As an example, the data may be generated by genetic algorithms, which apply evolutionary principles such as recombination, mutation, and selection to the data. This allows software to be tested for faults quickly and in-depth.

Big tech companies have been using fuzzing for years, but for the average development company, the threshold for using this technology is still quite high. The tools have to be configured manually, which is quite complex. Filtering and interpreting the results is also complicated.

Simpler to implement and interpret

At TNO, we conduct extensive research into the possibilities of automated testing, including fuzzing. We are exploring how the latest scientific insights in this field can be translated into useful, practical solutions. To this end, we are working with academic research groups in the Netherlands and beyond, as well as with development teams and testers from industry players.

We have developed several innovations to open up access to fuzzing, specifically to make it easier to:

• deploy and configure fuzzing tools;
• test software more effectively by harnessing knowledge about that software, for example to have the software modelled automatically;
• filter and categorise fuzz tests results, so they are easier to interpret;
• integrate fuzzing into existing development and testing processes.

This has made fuzzing more applicable and useful as a technique for software developers and manufacturers.

Early fault detection with fuzzing and static testing

​​Fuzzing is a form of dynamic testing, whereby the software is actually run during the test. This is in contrast to static testing, during which only the code is analysed and tested. ​ ​One advantage of static testing is that it can be done early on in the development phase. Detecting faults early can save considerable time and money. ​​​

By contrast, dynamic testing has other benefits; ​ ​it can pick up on functional errors, performance issues, and security vulnerabilities. It can also be used to test usability. At TNO,​​ ​​we are ​investigating how dynamic testing techniques like fuzz testing ​and static testing ​can​ go​ hand in ​hand​.

The many opportunities of API testing

At TNO, we focus mainly on automating testing for application programming interfaces (APIs). These programmes provide an interface between software applications.

Almost all modern software and systems use APIs, and often they are even a crucial component. At the same time, they are vulnerable: faults in APIs can compromise the stability of the programmes they connect. Because APIs provide access to sensitive data and key programme functions, they are a logical target for cyberattacks.

On the other hand, API testing offers many opportunities. As APIs are highly standardised, they allow automated API test innovations to be used for a wide range of software applications.

Work together on future software testing and fuzzing

Together with industry partners and academia, TNO is working on software testing and fuzzing through:

• A software testing and fuzzing workshop;
• Consultancy on testing technologies: strengths, weaknesses, and considerations during implementation;
• A support business case (including a feasibility study or valorisation study);
• Technology innovation and tool development;
• Customisation for a specific client or context.

If you would like to capitalise on the benefits of in-depth automated software testing, or speed up your software testing process with our innovations in fuzzing, please do not hesitate to contact us.