Metamorphic testing is a widely used approach in software testing to address the oracle problem. It involves using source test cases as seeds to generate follow-up test cases. The quality of these source test cases plays a crucial role in the effectiveness of metamorphic testing. While previous studies have mainly used random test data or existing test cases as source test cases, there has been limited research on systematic source test case generation for metamorphic testing. In this paper, the authors provide a comprehensive evaluation of different source test case generation techniques and their impact on the fault finding effectiveness of metamorphic testing. They specifically evaluate the effectiveness of line coverage, branch coverage, weak mutation, and random test generation strategies for generating source test cases. The experiments are conducted on 77 methods from 4 open-source code repositories. The results show that by systematically creating source test cases using these strategies, there is a significant increase in the fault finding effectiveness of metamorphic testing. The authors also introduce a simple metamorphic testing tool called "METtester" that they use to conduct metamorphic testing on these methods. Additionally, the paper discusses other related work in this area. For example, Chen et al. proposed partitioning the input domain of the program under test (PUT) into multiple equivalence classes for metamorphic testing and developed an algorithm to generate test cases covering these equivalence classes. Overall, this paper provides valuable insights into the importance of systematic source test case generation for improving the fault detection effectiveness of metamorphic testing. The evaluation results and introduction of METtester contribute to advancing research in this field.
- - Metamorphic testing is a widely used approach in software testing to address the oracle problem.
- - Source test cases are used as seeds to generate follow-up test cases in metamorphic testing.
- - The quality of source test cases is crucial for the effectiveness of metamorphic testing.
- - Previous studies have mainly used random test data or existing test cases as source test cases.
- - Limited research has been done on systematic source test case generation for metamorphic testing.
- - The authors provide a comprehensive evaluation of different source test case generation techniques and their impact on fault finding effectiveness.
- - Line coverage, branch coverage, weak mutation, and random test generation strategies are evaluated for generating source test cases.
- - Experiments are conducted on 77 methods from 4 open-source code repositories.
- - Systematically creating source test cases using these strategies significantly increases fault finding effectiveness.
- - The authors introduce a simple metamorphic testing tool called "METtester" for conducting metamorphic testing.
- - Other related work includes partitioning the input domain of the program under test into multiple equivalence classes and generating test cases covering these classes.
- - This paper provides valuable insights into the importance of systematic source test case generation for improving fault detection effectiveness in metamorphic testing.
Metamorphic testing is a way to test software and solve problems. Test cases are used to create more test cases in metamorphic testing. The quality of the original test cases is very important for this type of testing. Most previous studies have used random or existing test data for the original test cases. Not much research has been done on creating original test cases systematically. The authors of the paper evaluate different ways to create original test cases and how it affects finding mistakes in the software. They tested 77 methods from 4 open-source code repositories. Creating original test cases using these methods helps find mistakes better. The authors also made a tool called "METtester" for doing metamorphic testing. Other related work includes dividing the input domain into different groups and making test cases for each group. This paper shows that creating good original test cases is important for finding mistakes in software.
Definitions- Metamorphic testing: A way to test software by using certain rules or patterns.
- Test case: A set of instructions or actions to check if software works correctly.
- Oracle problem: The challenge of knowing what the correct output should be when testing software.
- Source test case: The initial set of instructions or actions used as a starting point for generating more tests in metamorphic testing.
- Fault finding effectiveness: How well a method can detect mistakes or faults in software.
- Line coverage: How many lines of code are executed during testing.
- Branch coverage: How many branches or choices in the code are taken
Metamorphic Testing: A Comprehensive Evaluation of Source Test Case Generation Strategies
Software testing is an important part of the software development process, as it helps to identify and fix errors in the code. One of the major challenges in software testing is known as the oracle problem, which refers to determining whether a given test case has passed or failed. Metamorphic testing is a widely used approach for addressing this issue, which involves using source test cases as seeds to generate follow-up test cases. The quality of these source test cases plays a crucial role in the effectiveness of metamorphic testing; however, there has been limited research on systematic source test case generation for metamorphic testing.
In this paper, researchers provide a comprehensive evaluation of different source test case generation techniques and their impact on the fault finding effectiveness of metamorphic testing. Specifically, they evaluate line coverage, branch coverage, weak mutation and random test generation strategies for generating source test cases. The experiments are conducted on 77 methods from 4 open-source code repositories and results show that by systematically creating source test cases using these strategies there is a significant increase in the fault finding effectiveness of metamorphic testing. Additionally, they introduce METtester – a simple metamorphic testing tool – that can be used to conduct metamorphic tests on these methods.
Line Coverage
Line coverage involves executing each statement at least once during program execution with one or more input values per statement being tested against multiple outputs (i.e., expected results). This technique ensures that all lines within a program are executed at least once during its lifetime but does not guarantee complete path coverage (i.e., all possible paths through a program).
Branch Coverage
Branch coverage requires executing every branch within a program at least once with one or more input values per branch being tested against multiple outputs (i.e., expected results). This technique guarantees complete path coverage but does not guarantee full functional correctness since some branches may never be executed due to certain conditions not being fulfilled during execution time (e.g., if statements).
Weak Mutation
Weak mutation involves making small changes to existing programs such as changing constants or swapping operators without affecting its functionality significantly while still ensuring sufficient diversity among generated tests so that most faults can be detected by running them through different inputs/outputs combinations over multiple executions (i.e., iterations).
Random Test Generation
Random Test Generation involves randomly selecting inputs from an input domain based on certain criteria such as data type constraints and then running those inputs through various output checks until all possible paths have been explored within the program under consideration (PUT) . This technique provides good fault detection capability but may require large amounts of resources due to its random nature and lack of control over what gets tested when compared with other techniques mentioned above like line/branch coverage etc..
Results & Discussion
The authors evaluated four different approaches for generating source tests: Line Coverage; Branch Coverage; Weak Mutation; Random Test Generation - across 77 methods from 4 open-source code repositories using their proposed tool METtester . Results showed that by systematically creating source tests using these strategies there was an overall improvement in fault finding effectiveness when compared with randomly generated tests or existing ones taken from literature sources . In particular , line/branch coverages provided better performance than weak mutation and random generation techniques respectively while also providing higher levels of confidence regarding functional correctness due to their exhaustive nature . Furthermore , Chen et al.'s work was discussed where they proposed partitioning PUT's input domain into equivalence classes for improved fault detection via metamorphic testing . Finally , METtester was introduced which provides support for conducting automated metamorphic tests across any language supported by LLVM compiler infrastructure .
Conclusion Overall , this paper provides valuable insights into how systematic creation of source tests can improve fault detection capabilities via Metamorphism Testing . The evaluation results along with introduction & discussion about related works contribute towards advancing research in this field while also introducing useful tools like METtester which could prove beneficial towards further development efforts surrounding Metamophism Testing technology .