Since test case design and selection are very time consuming, tedious, difficult and error-prone activities when done manually, there is a need for test automation tools that focus on the automation of these activities but do not exhibit the problems from which existing test generation techniques suffer. Hence, EvoTest will develop and evaluate Evolutionary Testing techniques for complex systems that will allow an automatic test generation in industrial practice. The EvoTest work is distributed into 10 work packages:
The first work package deals with the management of the EvoTest project and the quality assurance of its products. The four work packages WP2, WP3, WP4 and WP5 are the most important work packages for providing the technologies necessary to apply Evolutionary Testing in industrial practice. The three work packages WP6, WP7 and WP8 focus on the evaluation and assessment of the Evolutionary Testing Technologies developed. Therefore, Evolutionary Testing will be applied in three different application domains, namely automotive, telecommunication and accessibility improvement. WP9 merges the results of the different case studies in order to provide an overall assessment of the Evolutionary Testing approach. WP10 concentrates on the dissemination and future exploitation of the EvoTest results.
WP1 comprises all activities related to the project and risk management of the EvoTest project and the quality assurance of its products. The integration of project management, risk management and quality assurance in one work package guarantees a short link between both tasks.
In WP2, starting from an analysis of the current state of the art for Evolutionary Testing, the testing objectives most important in industrial practice will be analysed for being targeted with Evolutionary Testing. Solutions will be developed on how to address these testing objectives. The resulting requirements for setting up an Evolutionary Test and the test generation process behind it will be specified, and will form the input to WP3. It will be defined what kind of input interfaces can be observed for complex systems, and what valid test scenarios have to look like.
Meta-heuristic search techniques are not standalone algorithms in themselves, but rather strategies ready for adoption to specific problems. For test data generation, this involves the transformation of testing objectives to objective functions. Objective functions compare and contrast solutions of the search with respect to the overall search goal. Using this information, the search is directed into potentially promising areas of the search space. WP2 will define the objective functions for the assessment of the generated test scenarios. Objective functions will be designed for the Evolutionary Testing of the functional and non-functional system characteristics ranked with the highest relevance for the testing of complex systems by the industrial partners.
In order to improve test effectiveness, techniques have been developed to focus testing effort towards high risk system components, e.g. those likely to contain most defects. One such technique is reliability engineering. Techniques have been constructed that collect metrics during system development and predict the relative reliability of system components. In doing so, test strategies can be developed that focus testing towards these components. Even though these techniques help focus testing effort they do not aid the construction of tests. EvoTest will attempt to integrate these techniques in the objective function design to provide additional information that can be used to guide the evolutionary test to address the high risk aspects of the system. In doing so, we hope to reduce the risk and search time needed when determining test scenarios for locating faults.
Finally, WP2 will define criteria for the case study selection and the assessment of the Evolutionary Testing approach. It will be decided what the major focus of the evaluation of the evolutionary testing in the case studies should be: (1) a broad evaluation on different test aims and different applications showing that evolutionary testing techniques can in principle be applied to a large variety of different testing problems or (2) a more deep evaluation on a small number of selected testing aims tested in different application domains giving a higher confidence on the success rates evolutionary testing can achieve in industrial practice for the selected test aims. The different assessment criteria identified as relevant will be integrated into a validation framework (described in more detail below). The goal is to ensure a representative evaluation and assessment of Evolutionary Testing for the quality assurance of complex systems.
On the basis of the requirements provided by WP2, WP3 develops the search techniques which will be used in the Evolutionary Testing for finding the test scenarios fulfilling the testing objectives selected. A description language will be developed to describe the input interface for a system under test. From the problem description an Evolutionary Engine will be generated which is able to automatically generate test scenarios corresponding to the input interface of the system under test. To improve the performance of the search, Evolutionary Computation techniques will be developed enabling the seeding of existing test scenarios, and the self-adaptation of the search on the basis of the overall test progress monitored during test execution. Advanced evolutionary techniques about constraint-handling, and hybridization with other search techniques (either deterministic methods or other (meta-)heuristics) will be used to increase effectiveness and efficiency of the search and thus the test.
WP4 is closely linked to WP2 and WP3. The goal of WP4 is to simplify the search problem through the use and adaptation of existing advanced software engineering techniques such as slicing and transformation. Search space structures resulting for the test could show large plateaus of similar objective function values, discontinuous changes and the objective function values, multi-modality, definition gaps resulting from dependencies between different input parameters, and multi-dimensions resulting from a large number of input parameters for complex systems. This problem is addressed in WP4 by modifying the search space before even setting the optimization problem and by reducing the search space size. For this purpose, WP4 has to consider the objective function definitions from WP2 in order to smoothing the search space landscapes resulting for the testing objectives to be targeted by Evolutionary Testing. A smoother landscape improves the abilities of the search to find interesting test scenarios. For the reduction of the search space size WP4 has to interact with WP3 in order to reduce the number of input parameters to be generated and to reduce the resolution required for certain inputs.
WP5 develops the tool framework for the automation of Evolutionary Testing which implements the objective functions defined in WP2 and integrates the components developed in the work packages WP3 and WP4. At first, a system architecture will be designed which defines the components and their technical interactions necessary to implement automated Evolutionary Tests. The components will be divided into those which are generalizable for all testing objectives and those which are test-specific. For example, expected to be generalizable are the Evolutionary Engine generator or tools instrumenting the system under test for the monitoring of the test execution, whereas test driver components which execute the system under test with the generated test scenarios might be domain and test specific. Functional as well as interface specifications will be developed for the different components of the framework. The generalizable components will be implemented in WP5 and integrated with the Evolutionary Engine Generator provided by WP3 and the search space simplification components provided by WP4 resulting in the Automated Evolutionary Testing Framework.
Within WP6, WP7 and WP8 the Automated Evolutionary Testing Framework will be used for the testing of complex industrial systems. The overall goal of the three work packages is to assess the Evolutionary Testing Approach by the results achieved in the case studies. Each work package represents a different application domain. WP6 will evaluate Evolutionary Testing for the testing of complex automotive systems. WP7 will evaluate the Evolutionary Testing in the telecommunication domain and in WP8 Evolutionary Testing will be used to test complex systems which improve the system accessibility for handicapped people. Each of the industrial partners will ensure that appropriate real-life pilot projects are selected in accordance with the selection criteria defined in WP2. The different case studies assure that Evolutionary Testing is judged for a large variety of different testing objectives and different properties of complex systems. The goal is to receive a sound and reliable estimation of the suitability of Evolutionary Testing for improving the test and thus the product quality in industrial practice, and which advancements for industrial testing processes could be achieved with respect to the efficiency of the test. Although the entire Evolutionary Testing Framework will be available in Month 24, the validation phase (that is the case studies in WP6, WP7 and WP8) start applying the evolutionary testing to their systems in months 19, 24 and 27 respectively. Consequently, WP6 starts 5 months before the complete testing framework is available. Although this could be interpreted as an inconsistency in the planning, this has been done on purpose. We have decided to do this in order to send some feedback on the application of the components available in month 19 to WP5 during the development of the whole testing framework.
To run the tests WP6, WP7 and WP8 will have to develop the test specific components required for the test execution and not provided by the framework developed in WP5. The performance of the Evolutionary Testing will be assessed for various criteria defined in WP2 as important for the assessment of the industrial applicability of Evolutionary Testing for the quality assurance of complex systems. Criteria are, for instance, the usability of the framework for developers and testers, the success of the Evolutionary Testing in finding errors or anomalies and the resulting test effort. Results from the case studies will immediately be fed back into the work packages WP3, WP4, and WP5 in order to facilitate fast improvements of the Evolutionary Testing Technologies and framework components.
WP9 will summarize the results of the different case studies to a comprehensive assessment of the industrial applicability of Evolutionary Testing. For this purpose, the validation framework defined in WP2 will be used. The utilization of the validation framework will provide a comprehensive view on the strengths and weaknesses of Evolutionary Testing. Moreover, WP9 will provide a detailed outlook on which other technologies could be integrated or have to be developed in order to further improve the efficiency and effectiveness of Evolutionary Testing.
Since Evolutionary Testing is a very innovative technology, in WP10 the EvoTest project pays high attention to the dissemination of the project results, the close interaction with the scientific community and intensive discussions with practitioners developing complex systems in an industrial context.