EvoMaster: A Search-Based System Test Generation Tool

Testing web/enterprise applications is complex and expensive when done manually. Often, software testing takes up to half of the development time and cost for a system. So much testing is needed because the cost of software failure is simply too large: for example, in 2017, 304 software failures (reported in the media) impacted 3.6 billion people and $1.7 trillion in assets worldwide (Tricentis, 2017). Unfortunately, due to its high cost, software testing is often left incomplete, and only applied partially. To address this problem, in Software Engineering (SE) research a lot of effort has been spent in trying to design and implement novel techniques aimed at automating several different tasks, where software testing is among the most studied tasks. Search-Based Software Testing (SBST) (Harman et al., 2012) casts the problem of software testing as an optimization problem, aimed at, for example, maximizing code coverage and fault detection. Our SBST tool called EvoMaster addresses these challenges by using evolutionary techniques to automatically generate test cases. It currently focuses on RESTful web services, which are the pillars of modern web and enterprise applications (Allamaraju, 2010; Fielding, 2000). The EvoMaster tool is aimed at:

• Fault detection: EvoMaster can generate tests cases that reveal faults/bugs in the tested applications. Different heuristics are employed, like checking for 500 status codes and mismatches from the API schemas.
• Self-contained tests: the generated tests do start/stop the application, binding to an ephemeral port. This means that the generated tests can be used for regression testing (e.g., added to the Git repository of the application, and run with any build tool such as Maven and Gradle).
• Advanced whitebox heuristics: EvoMaster analyses the bytecode of the tested applications, and uses several heuristics such as testability transformations and taint analysis to be able to generate more effective test cases.
• SQL handling: EvoMaster can intercept and analyse all communications done with SQL databases, and use such information to generate higher code coverage test cases. Furthermore, it can generate data directly into the databases, and have such initialization automatically added in the generated tests. At the moment, EvoMaster supports H2 and Postgres databases.
• Blackbox testing mode: can run on any API (regardless of its programming language), as long as an OpenAPI schema is provided. However, results will be worse than whitebox testing (e.g., due to lack of bytecode analysis).

Published Results
When addressing the testing of real-world web/enterprise applications, there are many challenges. The tested code can for example have complex execution flows, where the boolean predicates in if and loop statements depend on specific input data. Furthermore, the execution flow could depend on interactions with external entities, such as databases, GUIs and remote web services. The search space of all possible test inputs is huge, where only a tiny subset lead to maximize code coverage and detect faults.
To face and overcome those challenges, EvoMaster has been used to experiment with several novel techniques. These techniques are now integrated in EvoMaster, where their best settings (based on empirical studies) are on by default.

Related Work
In the recent years, different techniques have been proposed for black-box testing of RESTful APIs (Atlidakis et al., 2019;Ed-douibi et al., 2018;Karlsson et al., 2020;Viglianisi et al., 2020). Those present different variants of random testing, enhanced with heuristics based on the information provided in the API schemas. As those techniques are black-box, they do not access the source-code/bytecode of the tested APIs, and so cannot exploit any such information to improve the generation of test cases.
At the time of this writing, EvoMaster appears to be the only tool that can do both black-box and white-box testing, and that is also released as open-source. Supporting black-box testing is important, as it is easier to setup and use. However, white-box testing leads to better results (e.g., higher code coverage and fault detection), as it can exploit more information on the system under test.