Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity.
Persistent URL
Author(s)
Lin, Chu-Ti
Tang, Kai-Wei
Wang, Jiun-Shiang
Kapfhammer, Gregory M.
Date Issued
May 24, 2017
Abstract
Test suite reduction is an important approach that decreases the cost of regression testing. A test suite reduction technique operates based on the relationship between the test cases in the regression test suite and the test requirements in the program under test. Thus, its effectiveness should be closely related to the complexity of a regression test suite - the product of the number of test cases and the number of test requirements. Our previous work has shown that cost-aware techniques (i.e., the test suite reduction techniques that aim to decrease the regression test suite's execution cost) generally outperform the others in terms of decreasing the cost of running the regression test suite. However, the previous empirical studies that evaluated cost-aware techniques did not take into account test suite complexity. That is, prior experiments do not reveal if the cost-aware techniques scale and work effectively on test suites with more test cases and more test requirements. This means that prior experiments do not appropriately shed light on how well test suite reduction methods work with large programs or test suites. Therefore, this paper focuses on the Greedy-based techniques and empirically evaluates the additional Greedy and two cost-aware Greedy techniques - at different levels of test suite complexity - from various standpoints including the cost taken to run the regression test suite, the time taken to reduce the test suites, the total regression testing costs, the fault detection capability, the fault detection efficiency, and the common rates of the representative sets. To the best of our knowledge, none of the previous empirical studies classify a considerable number of test suites according to their complexity. Nor do any prior experiments evaluate the test suite reduction techniques, in terms of the aforementioned criteria, at different levels of test suite complexity. This paper represents the first such attempt to carry out this important task. Based on the empirical results, we confirm the strengths and weaknesses of the cost-aware techniques and develop insights into how the cost-aware techniques' effectiveness varies as the test suite complexity increases. (C) 2017 Elsevier B.V. All rights reserved.
Journal
Science of Computer Programming
Department
Computer Science
Citation
Lin, C.T., Tang, K.W., Wang, J.S., and Kapfhammer, G.M. (2017). Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity. Science of Computer Programming 150(1): 1-25. doi: 10.1016/j.scico.2017.05.004
Publisher
Elsevier
Version of Article
Published article
Embargo
The post-print (if available) will be made available to the public after 05/24/2019.
DOI
10.1016/j.scico.2017.05.004
ISSN
0167-6423
Rights
Published articles are subject to copyright, contact the publisher or reprint author for further use of this material. Post-prints will be made available for public use after a 48 month embargo.
File(s)![Thumbnail Image]()
![Thumbnail Image]()
Name
Kapfhammer_2017_SD_testsuites.pdf
Description
Main Document
Size
6.45 MB
Format
Adobe PDF
Checksum (etag)
419af0ed8dd597c601047444a1870171
Name
Kapfhammer_2017_SD_Cover.pdf
Description
Document
Size
329.84 KB
Format
Adobe PDF
Checksum (etag)
d3c872d7aa7a4879240edcb761c79180