Test Selection on Extended Finite State Machines with Provable Guarantees
- 1 Department of Computer Science, University of Nebraska Omaha, Omaha, USA.
- 2 Department of Computer Science, University of Nebraska Omaha, Omaha, USA
Abstract
Building high confidence regression test suites to validate new system versions is a challenging problem. A modelbased approach to build a regression test suite from a given test suite is described. The generated test suite includes every test that will traverse a change performed to produce the new version, and consists of only such tests to reduce the testing costs. Finite state machines extended with typed variables (EFSMs) are used to model systems and system changes are mapped to EFSM transition changes adding/deleting/replacing EFSM transitions and states. Tests are a sequence of input and expected output messages with concrete parameter values over the supported data types. An invariant is formulated to characterize tests whose runtime behavior can be accurately predicted by analyzing their descriptions along with the model. Incremental procedures to efficiently evaluate the invariant and to select tests for regression are developed. Overlaps among the test descriptions are exploited to extend the approach to simultaneously select multiple tests to reduce the test selection costs. Effectiveness of the approach is demonstrated by applying it to several protocols, Web services, and model programs extracted from a popular testing benchmark. Our experimental results show that the proposed approach is economical for regression test selection in all these examples. For all these examples, the proposed approach is able to identify all tests exercising changes more efficiently than brute-force symbolic evaluation.
- M. J. Harrold and A. Orso, “Retesting Software during Development and Maintenance,” Frontiers of Software Maintenance, Beijing, 28 September-4 October 2008, pp. 99-109.
- G. Rothermel and M. J. Harrold, “Analyzing Regression Test Selection Techniques,” IEEE Transactions on Software Engineering, Vol. 22, No. 8, 1996, pp. 529-551. doi:10.1109/32.536955
- S. Yoo and M. Harman, “Regression Testing Minimization, Selection, and Prioritization: A Survey,” Software Testing, Verification, and Reliability, Vol. 22, No. 2, 2010, pp. 67-120. doi:10.1002/stvr.430
- Y. Chen, R. L. Rrobert and H. Ural, “Regression Test Suite Reduction Using Extended Dependence Analysis,” 4th Workshop on Software Quality Assurance, 2007, pp. 62-69.
- B. Korel, L. Tahat and B. Vaysburg, “Model Based Regression Test Reduction Using Dependence Analysis,” International Conference on Software Maintenance, 2002, pp. 214-223.
- D. Detlefs, G. Nelson and J. B. Saxe, “Simplify: A Theorem Prover for Program Checking,” Journal of the ACM, 52, 3, 2005, pp. 365-473. doi:10.1145/1066100.1066102
- D. Kapur and H. Zhang, “An Overview of Rewrite Rule Laboratory (RRL),” Conference on Rewriting Techniques and Applications, Chapel Hill, 3-5 April 1989, pp. 559-563.
- B. Daniel and Z. Pitro, “On Communicating Finite-State Machines,” Journal of the ACM, Vol. 30, No. 2, 1983, pp. 323-342.
- D. Lee and M. Yiannakakis, “Principles and Methods of Testing Finite State Machines-Survey,” IEEE Computers, Vol. 84, No. 8, 1996.
- M. Subramaniam and P. Chundi, “An Approach to Preserve Protocol Consistency and Executability across Updates,” Conference on Formal Engineering Methods, Seattle, 8-12 December 2004, pp. 341-356.
- M. Subramaniam and B. Guo, “A Rewrite-Based Approach for Change Impact Analysis of Communicating Systems Using a Theorem Prover,” Tech Report, Department of Computer Science, University of New Orleans, New Orleans, 2008.
- G. Rothermel and M. J. Harrold, “A Safe, Efficient Regression Test Selection Technique,” ACM Transactions on Software Engineering and Methodology, Vol. 6, No. 2, 1997, pp. 173-210.
- E. Bringmann and A. Kramer, “Model-Based Testing of Automotive Systems,” Conference on Software Testing, Verification and Validation (ICST’08), Lillehammer, 9-11 April 2008, pp. 485-493.
- Y. Chen, R. L. Probert and D. P. Sims, “Specification-Based Regression Test Selection with Risk Analysis,” Conference of the Centre for Advanced Studies on Collaborative Research, 2002.