The post-release maintenance is usually the most expensive phase in the software product lifecycle from the first design concepts to the end of product support. To reduce the costs related to post-release maintenance, we propose a run-time framework for measuring software quality characteristics applying the ISO/IEC 25000 software quality and software quality in use models as the starting point. Measurement probes are linked into the software during the development phase and used to collect quality information during the run time. As a proof-of-concept, we implemented measurements in an open-source software project to demonstrate the utility of the framework. As a result, this paper presents a framework for collecting runtime metrics and measuring software quality-in-use with a systematic interface. Additionally, examples of measurement scenarios are presented.
The Four Laws of Application, Total Cost of Ownership (2012) Gartner, Inc., Stamford, CT.
Kasurinen, J., Maglyas, A. and Smolander, K. (2014) Is Requirements Engineering Useless in Game Development? In: Salinesi, C. and van de Weerd, I., Eds., Requirements Engineering: Foundation for Software Quality, REFSQ 2014, Lecture Notes in Computer Science, Vol. 8396, Springer, Cham, 1-16. https://doi.org/10.1007/978-3-319-05843-6_1
Alha, K., Koskinen, E., Paavilainen, J., Hamari, J. and Kinnunen, J. (2014) Free-to-Play Games: Professionals’ Perspectives. Proceedings of Nordic Digra 2014, Gotland, 29 May 2014.
Humble, J. and Farley, D. (2010) Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Pearson Education. https://books.google.fi/books?id=6ADDuzere-YC
Roche, J. (2013) Adopting DevOps Practices in Quality Assurance. Communications of the ACM, 56, 38-43. https://doi.org/10.1145/2524713.2524721
Garvin, D.A. (1984) What Does “Product Quality” Really Mean? Sloan Management Review, 4, 25-43.
ISO/IEC (2011) ISO/IEC 25000: Systems and Software Quality Requirements and Evaluation (SQuaRE)—Guide to SQuaRE.
Motogna, S., Vescan, A., Serban, C. and Tirban, P. (2016) An Approach to Assess Maintainability Change. 2016 IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR), Cluj-Napoca, 19-21 May 2016, 1-6. https://doi.org/10.1109/AQTR.2016.7501279
Bautista, L., Abran, A. and April, A. (2012) Design of a Performance Measurement Framework for Cloud Computing. Journal of Software Engineering and Applications, 5, 69-75. https://doi.org/10.4236/jsea.2012.52011
Hegedus, P. (2013) Revealing the Effect of Coding Practices on Software Maintainability. 2013 29th IEEE International Conference on Software Maintenance (ICSM), Eindhoven, 22-28 September 2013, 578-581. https://doi.org/10.1109/ICSM.2013.99
Kasurinen, J., Taipale, O., Vanhanen, J. and Smolander, K. (2012) Exploring the Perceived End-Product Quality in Software-Developing Organizations. International Journal of Information System Modeling and Design, 3, 1-32. https://doi.org/10.4018/jismd.2012040101
Honglei, T., Wei, S. and Yanan, Z. (2009) The Research on Software Metrics and Software Complexity Metrics. International Forum on Computer Science-Technology and Applications, Chongqing, 25-27 December 2009, Vol. 1, 131-136. https://doi.org/10.1109/IFCSTA.2009.39
Herzig, K., Greiler, M., Czerwonka, J. and Murphy, B. (2015) The Art of Testing Less without Sacrificing Quality. Proceedings of the 37th International Conference on Software Engineering, Vol. 1, Florence, 16-24 May 2015, 483-493. https://doi.org/10.1109/ICSE.2015.66
Fontana, F.A. and Zanoni, M. (2011) On Investigating Code Smells Correlations. IEEE 4th International Conference on Software Testing, Verification and Validation Workshops, Berlin, 21-25 March 2011, 474-475. https://doi.org/10.1109/ICSTW.2011.14
Schrettner, L., Fülöp, L.J., Beszédes, á., Kiss, á. and Gyimóthy, T. (2012) Software Quality Model and Framework with Applications in Industrial Context. 16th European Conference on Software Maintenance and Reengineering, Szeged, 27-30 March 2012, 453-456. https://doi.org/10.1109/CSMR.2012.57
Kasurinen, J., Taipale, O. and Smolander, K. (2010) Test Case Selection and Prioritization: Risk-Based or Design-Based? Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, Bolzano-Bozen, 16-17 September 2010, Article No. 10. https://doi.org/10.1145/1852786.1852800
Rompaey, B.V., Bois, B.D., Demeyer, S. and Rieger, M. (2007) On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test. IEEE Transactions on Software Engineering, 33, 800-817. https://doi.org/10.1109/TSE.2007.70745
Consortium for IT Software Quality. http://it-cisq.org/
Goues, C.L. and Weimer, W. (2012) Measuring Code Quality to Improve Specification Mining. IEEE Transactions on Software Engineering, 38, 175-190. https://doi.org/10.1109/TSE.2011.5
Black, P.E. (2006) Software Assurance during Maintenance. 22nd IEEE International Conference on Software Maintenance, Philadelphia, 24-27 September 2006, 70-72. https://doi.org/10.1109/ICSM.2006.58
Lincke, R., Gutzmann, T. and Löwe, W. (2010) Software Quality Prediction Models Compared. 10th International Conference on Quality Software, Zhangjiajie, 14-15 July 2010, 82-91.
Darcy, D.P. and Kemerer, C.F. (2005) OO Metrics in Practice. IEEE Software, 22, 17-19. https://doi.org/10.1109/MS.2005.160
ISO/IEC (2007) ISO/IEC 15939: Systems and Software Engineering—Measurement Process.
Hynninen, T., Kasurinen, J., Knutas, A. and Taipale, O. (2017) Testing Practices in the Finnish Software Industry. IEEE Conference on Software Engineering Education and Training, Savannah, 7-9 November 2017.
NetBeans IDE. https://netbeans.org/
Bourque, P. and Fairley, R.E. (2014) Guide to the Software Engineering Body of Knowledge, Version 3.0. IEEE Computer Society, Washington DC.
Koomen, T. and Pol, M. (1999) Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing. Addison-Wesley Longman Publishing Co., Inc., Bostong.
van Veenendaal, E. and Wells, B. (2012) Test Maturity Model Integration TMMi. Uitgeverij Tutein Nolthenius, Hertogenbosch.
Horstmann, C.S. and de Pellegrin, A. Violet UML Editor. http://violet.sourceforge.net
McCabe, T.J. (1976) A Complexity Measure. IEEE Transactions on Software Engineering, SE-2, 308-320. https://doi.org/10.1109/TSE.1976.233837
Halstead, M.H. (1977) Elements of Software Science. Vol. 7, Elsevier, New York.
Chidamber, S.R. and Kemerer, C.F. (1994) A Metrics Suite for Object Oriented Design. IEEE Transactions on Software Engineering, 20, 476-493. https://doi.org/10.1109/32.295895
Code Analysis Team (2007) Maintainability Index Range and Meaning—Code Analysis Team Blog. https://blogs.msdn.microsoft.com/codeanalysis/2007/11/20/maintainability-index-range-and-meaning
Oliveira, P., Lima, F.P., Valente, M.T. and Serebrenik, A. (2014) RTTool: A Tool for Extracting Relative Thresholds for Source Code Metrics. IEEE International Conference on Software Maintenance and Evolution, Victoria, 29 September-3 October 2014, 629-632. https://doi.org/10.1109/ICSME.2014.112
Antinyan, V., Staron, M. and Sandberg, A. (2017) Evaluating Code Complexity Triggers, Use of Complexity Measures and the Influence of Code Complexity on Maintenance Time. Empirical Software Engineering, 22, 3057-3087. https://doi.org/10.1007/s10664-017-9508-2
Catal, C. (2011) Software Fault Prediction: A Literature Review and Current Trends. Expert Systems with Applications, 38, 4626-4636. https://doi.org/10.1016/j.eswa.2010.10.024