Customizing applications through program configuration options has been proved by many open-source and commercial projects as one of the best practices in software engineering. However, traditional performance testing is not in synch with this industrial practice. Traditional performance testing techniques consider program inputs as the only external factor. It ignores the performance influence of configuration options. This study aims to stimulate research interest in performance testing in the context of configurable software systems by answering three research questions. That is, why it is necessary to conduct research in performance testing, what are the state-of-the-art techniques, and how do we conduct performance testing research in configurable software systems. In this study, we examine the unique characteristics and challenges of performance testing research in configurable software systems. We review and discuss research topics on the performance bug study, performance anti-patterns, program analysis, and performance testing. We share the research findings from the empirical study and outline the opening opportunities for new and advanced researchers to contribute to the research community.
KeywordsConfigurable Software SystemsPerformance TestingSoftware ConfigurationPerformance Bug Study
Webb, G.K. (2004) Predicting Processor Performance. Issues in Information Systems, 5, 340-346.
Pradel, M., Schuh, P., Necula, G. and Sen, K. (2014) EventBreak: Analyzing the Responsiveness of User Interfaces through Performance-Guided Test Generation. ACM SIGPLAN Notices, 49, 33-47. https://doi.org/10.1145/2714064.2660233
Hollister, S. (2020) Apple Will Pay $113 Million for Throttling Older iPhones in New “Batterygate” Settlement. https://www.theverge.com/2020/11/18/21573710/apple-battery-gate-throttle-iphones-settlement-amount
Han, X. and Yu, T. (2016) An Empirical Study on Performance Bugs for Highly Configurable Software Systems. Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Ciudad Real, September 2016, Article No. 23. https://doi.org/10.1145/2961111.2962602
Dean, D.J., Nguyen, H., Gu, X., Zhang, H., Rhee, J., Arora, N. and Jiang, G. (2014) Perfscope: Practical Online Server Performance Bug Inference in Production Cloud Computing Infrastructures. Proceedings of the ACM Symposium on Cloud Computing, Seattle, 3-5 November 2014, 1-13. https://doi.org/10.1145/2670979.2670987
Han, X., Carroll, D. and Yu, T. (2019) Reproducing Performance Bug Reports in Server Applications: The Researchers’ Experiences. Journal of Systems and Software, 156, 268-282. https://doi.org/10.1016/j.jss.2019.06.100
Attariyan, M., Chow, M. and Flinn, J. (2012) X-Ray: Automating Root-Cause Diagnosis of Performance Anomalies in Production Software. 10th USENIX Symposium on Operating Systems Design and Implementation, Hollywood, 7-10 October 2012, 307-320.
Molyneaux, I. (2009) The Art of Application Performance Testing: Help for Programmers and Quality Assurance. O’Reilly Media, Inc., Sebastopol.
Samsung for Business (2020) Your Phone Is Now More Powerful than Your PC. https://insights.samsung.com/2020/08/07/your-phone-is-now-more-powerful-than-your-pc-2/
Pathak, A., Hu, Y.C. and Zhang, M. (2011) Bootstrapping Energy Debugging on Smartphones: A First Look at Energy Bugs in Mobile devices. Proceedings of the 10th ACM Workshop on Hot Topics in Networks, Cambridge, 14-15 November 2011, Article No. 5. https://doi.org/10.1145/2070562.2070567
Computer Security Resource Center (2016) Automated Combinatorial Testing for Software. http://csrc.nist.gov/groups/SNS/acts/index.html.
Reisner, E., Song, C., Ma, K.-K., Foster, J.S. and Porter, A. (2010) Using Symbolic Evaluation to Understand Behavior in Configurable Software Systems. 2010 ACM/IEEE 32nd International Conference on Software Engineering, Cape Town, 1-8 May 2010, 445-454. https://doi.org/10.1145/1806799.1806864
Jin, G., Song, L., Shi, X., Scherpelz, J. and Lu, S. (2012) Understanding and Detecting Real-World Performance Bugs. ACM SIGPLAN Notices, 47, 77-88. https://doi.org/10.1145/2345156.2254075
Han, X., Yu, T. and Pradel, M. (2021) Confprof: White-Box Performance Profiling of Configuration Options. Proceedings of the ACM/SPEC International Conference on Performance Engineering, Virtual Event, 19-23 April 2021, 1-8. https://doi.org/10.1145/3427921.3450255
Song, L. and Lu, S. (2014) Statistical Debugging for Real-World Performance Problems. Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications, Portland, 20-24 October 2014, 561-578. https://doi.org/10.1145/2660193.2660234
Zaman, S., Adams, B. and Hassan, A.E. (2012) A Qualitative Study on Performance Bugs. 2012 9th IEEE Working Conference on Mining Software Repositories (MSR), Zurich, 2-3 June 2012, 199-208. https://doi.org/10.1109/MSR.2012.6224281
Nistor, A., Jiang, T. and Tan, L. (2013) Discovering, Reporting, and Fixing Performance Bugs. 2013 10th Working Conference on Mining Software Repositories (MSR), San Francisco, 18-19 May 2013, 237-246. https://doi.org/10.1109/MSR.2013.6624035
Koenig, A. (2013) Performance Bugs: Not Just Hard to Detect, but Hard to Define. http://www.drdobbs.com/cpp/performance-bugs-not-just-hard-to-detect/240164448
Foo, K.C., Jiang, Z.M.J., Adams, B., Hassan, A.E., Zou, Y. and Flora, P. (2015) An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments. 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Florence, 16-24 May 2015, 159-168. https://doi.org/10.1109/ICSE.2015.144
Wert, A., Happe, J. and Happe, L. (2013) Supporting Swift Reaction: Automatically Uncovering Performance Problems by Systematic Experiments. 2013 35th International Conference on Software Engineering (ICSE), San Francisco, 18-26 May 2013, 552-561. https://doi.org/10.1109/ICSE.2013.6606601
Alagar, V.S. and Periyasamy, K. (2011) Specification of Software Systems. Springer Science & Business Media, London. https://doi.org/10.1007/978-0-85729-277-3
Nair, V., Menzies, T., Siegmund, N. and Apel, S. (2017) Using Bad Learners to Find Good Configurations. Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, Paderborn, 4-8 September 2017, 257-267. https://doi.org/10.1145/3106237.3106238
Puoskari, E., Vos, T.E.J., Condori-Fernandez, N. and Kruse, P.M. (2013) Evaluating Applicability of Combinatorial Testing in an Industrial Environment: A Case Study. Proceedings of the 2013 International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation, Lugano, 15 July 2013, 7-12. https://doi.org/10.1145/2489280.2489287
Yin, Z., Ma, X., Zheng, J., Zhou, Y., Bairavasundaram, L.N. and Pasupathy, S. (2011) An Empirical Study on Configuration Errors in Commercial and Open Source Systems. Proceedings of the 23rd ACM Symposium on Operating Systems Principles, Cascais, 23-26 October 2011, 159-172. https://doi.org/10.1145/2043556.2043572
Chen, T.-H., Shang, W., Jiang, Z.M., Hassan, A.E., Nasser, M. and Flora, P. (2014) Detecting Performance Anti-Patterns for Applications Developed Using Object-Relational Mapping. Proceedings of the 36th International Conference on Software Engineering, Hyderabad, 31 May-7 June 2014, 1001-1012. https://doi.org/10.1145/2568225.2568259
Smith, C.U. and Williams, L.G. (2000) Software Performance Antipatterns. Proceedings of the 2nd International Workshop on Software and Performance, Ottawa, September 2000, 127-136. https://doi.org/10.1145/350391.350420
Smith, C.U. and Williams, L.G. (2002) New Software Performance Antipatterns: More Ways to Shoot Yourself in the Foot. Proceedings 28th International Conference Computer Measurement Group, Reno, 8-13 December 2002, 667-674.
Smith, C.U. and Williams, L.G. (2003) More New Software Performance Antipatterns: Even More Ways to Shoot Yourself in the Foot. 29th International Computer Measurement Group Conference, Dallas, 7-12 December 2003, 717-725.
Li, Z. and Zhou, Y. (2005) PR-Miner: Automatically Extracting Implicit Programming Rules and Detecting Violations in Large Software Code. ACM SIGSOFT Software Engineering Notes, 30, 306-315. https://doi.org/10.1145/1095430.1081755
Han, X., Yu, T. and Lo, D. (2018) Perflearner: Learning from Bug Reports to Understand and Generate Performance Test Frames. 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE), Montpellier, 3-7 September 2018, 17-28. https://doi.org/10.1145/3238147.3238204
Yu, T., Wen, W., Han, X. and Hayes, J.H. (2016) Predicting Testability of Concurrent Programs. 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), Chicago, 11-15 April 2016, 168-179. https://doi.org/10.1109/ICST.2016.39
Wert, A. (2013) Performance Problem Diagnostics by Systematic Experimentation. Proceedings of the 18th International Doctoral Symposium on Components and Architecture, Vancouver, 17 June 2013, 1-6. https://doi.org/10.1145/2465498.2465499
Hovemeyer, D. and Pugh, W. (2004) Finding Bugs Is Easy. ACM SIGPLAN Notices, 39, 92-106. https://doi.org/10.1145/1052883.1052895
(2016) Codesurfer, a Code Browser That Understands Pointers, Indirect Function Calls, and whole-Program Effects. https://www.grammatech.com/products/codesurfer
Xiao, X., Han, S., Zhang, D. and Xie, T. (2013) Context-Sensitive Delta Inference for Identifying Workload-Dependent Performance Bottlenecks. Proceedings of the 2013 International Symposium on Software Testing and Analysis, Lugano, 15-20 July 2013, 90-100. https://doi.org/10.1145/2483760.2483784
Nistor, A., Song, L., Marinov, D. and Lu, S. (2013) Toddler: Detecting Performance Problems via Similar Memory-Access Patterns. 2013 35th International Conference on Software Engineering (ICSE), San Francisco, 18-26 May 2013, 562-571. https://doi.org/10.1109/ICSE.2013.6606602
Altman, E., Arnold, M., Fink, S. and Mitchell, N. (2010) Performance Analysis of idle Programs. Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, Reno, 17-21 October 2010, 739-753. https://doi.org/10.1145/1869459.1869519
Barna, C., Litoiu, M. and Ghanbari, H. (2011) Model-Based Performance Testing (Nier Track). Proceedings of the 33rd International Conference on Software Engineering, Waikiki, May 2011, 872-875. https://doi.org/10.1145/1985793.1985930
Luo, Q., Nair, A., Grechanik, M. and Poshyvanyk, D. (2016) FOREPOST: Finding Performance Problems Automatically with Feedback-Directed Learning Software Testing. Proceedings of the 38th International Conference on Software Engineering Companion, Austin, 14-22 May 2016 593-596. https://doi.org/10.1145/2889160.2889164
Huang, P., Ma, X., Shen, D. and Zhou, Y. (2014) Performance Regression Testing Target Prioritization via Performance Risk Analysis. Proceedings of the 36th International Conference on Software Engineering, Hyderabad, 31 May 2014-7 June 2014, 60-71. https://doi.org/10.1145/2568225.2568232
Smaalders, B. (2006) Performance Anti-Patterns. Queue, 4, 44-50. https://doi.org/10.1145/1117389.1117403
Anthopoulos, L., Reddick, C.G., Giannakidou, I. and Mavridis, N. (2016) Why E-Government Projects Fail? An Analysis of the Healthcare.gov Website. Government Information Quarterly, 33, 161-173. https://doi.org/10.1016/j.giq.2015.07.003
Weiser, M. (1984) Program Slicing. IEEE Transactions on Software Engineering, SE-10, 352-357. https://doi.org/10.1109/TSE.1984.5010248
Luk, C.-K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J. and Hazelwood, K. (2005) Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation. ACM SIGPLAN Notices, 40, 190-200. https://doi.org/10.1145/1064978.1065034
Schwartz, E.J., Avgerinos, T. and Brumley, D. (2010) All You Ever Wanted to Know about Dynamic Taint Analysis and Forward Symbolic Execution (but Might Have Been Afraid to Ask). 2010 IEEE Symposium on Security and Privacy, Oakland, 16-19 May 2010, 317-331. https://doi.org/10.1109/SP.2010.26
Anvik, J., Hiew, L. and Murphy, G.C. (2006) Who Should Fix This Bug? Proceedings of the 28th International Conference on Software Engineering, Shanghai, 20-28 May 2006, 361-370. https://doi.org/10.1145/1134285.1134336
Siegmund, N., Grebhahn, A., Apel, S. and Kästner, C. (2015) Performance-Influence Models For highly Configurable Systems. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, Bergamo, 30 August-4 September 2015, 284-294. https://doi.org/10.1145/2786805.2786845
Song, C., Porter, A. and Foster, J.S. (2014) Itree: Efficiently Discovering High-Coverage Configurations Using Interaction trees. IEEE Transactions on Software Engineering, 40, 251-265. https://doi.org/10.1109/TSE.2013.55
Sitaraman, M., Kulczycki, G., Krone, J., Ogden, W.F. and Reddy, A.N. (2001) Performance Specification of software Components. ACM SIGSOFT Software Engineering Notes, 26, 3-10. https://doi.org/10.1145/379377.375223
Wong, W.E., Horgan, J.R., London, S. and Agrawal, H. (1997) A Study of Effective Regression Testing in Practice. Proceedings of the 8th International Symposium on Software Reliability Engineering, Albuquerque, 2-5 November 1997, 264-274. https://doi.org/10.1109/ISSRE.1997.630875
Kwon, Y., Lee, S., Yi, H., Kwon, D., Yang, S., Chun, B.-G., Huang, L., Maniatis, P., Naik, M. and Paek, Y. (2013) Mantis: Automatic Performance Prediction for Smartphone Applications. 2013 USENIX Annual Technical Conference, San Jose, 26-28 June 2013, 297-308.
Huang, L., Jia, J., Yu, B., Chun, B.-G., Maniatis, P. and Naik, M. (2010) Predicting Execution Time of Computer Programs Using Sparse Polynomial Regression. Proceedings of the 23rd International Conference on Neural Information Processing Systems, Vol. 1, Vancouver, 6-9 December 2010, 883-891.
Guo, J., Czarnecki, K., Apel, S., Siegmund, N. and Wasowski, A. (2013) Variability-Aware Performance Prediction: A Statistical Learning Approach. 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), Silicon Valley, 11-15 November 2013, 301-311. https://doi.org/10.1109/ASE.2013.6693089