Reproducibility is a key aspect of the scientific method as it provides evidence for research claims. It is essential to promote openness, accessibility, and collaboration within the scientific community. This article aims to provide an introduction to best practices in reproducibility that are relevant to the transportation research community, to discuss issues and barriers to reproducibility, and to describe methods for addressing these issues. This article starts by discussing openness and transparency, then discusses several key best practices for reproducibility in transportation engineering, highlighting common methods and techniques, as well as the associated benefits. The paper concludes with a discussion of the key barriers to implementing reproducibility practices in transportation research and potential solutions. The barriers include existing culture and attitudes, data sensitivity, insufficient methodological detail, lack of code sharing, limited validation, additional time and research burden, and skill and knowledge gaps. Discussing each of these items provides an opportunity for the transportation research community to evolve to become one that embraces the openness and transparency of reproducibility.
KeywordsReproducibilityOpennessTransparencyScientific MethodResponsible Research
Socol, Y., Shaki, Y.Y. and Yanovskiy, M. (2019) Interests, Bias, and Consensus in Science and Regulation. Dose - Response , 17. https://doi.org/10.1177/1559325819853669
Open Science Collaboration (2015) Estimating the Reproducibility of Psychological Science. Science , 349, aac4716. https://doi.org/10.1126/science.aac4716
Gundersen, O.E. (2021) The Fundamental Principles of Reproducibility. Philosophical Transactions of the Royal Society A : Mathematical , Physical and Engineering Sciences , 379, Article ID: 20200210. https://doi.org/10.1098/rsta.2020.0210
Nissen, S.B., Magidson, T., Gross, K. and Bergstrom, C.T. (2016) Publication Bias and the Canonization of False Facts. eLife , 5, e21451. https://doi.org/10.7554/elife.21451
Landis, S.C., Amara, S.G., Asadullah, K., Austin, C.P., Blumenstein, R., Bradley, E.W., et al . (2012) A Call for Transparent Reporting to Optimize the Predictive Value of Preclinical Research. Nature , 490, 187-191. https://doi.org/10.1038/nature11556
Pusztai, L., Hatzis, C. and Andre, F. (2013) Reproducibility of Research and Preclinical Validation: Problems and Solutions. Nature Reviews Clinical Oncology , 10, 720-724. https://doi.org/10.1038/nrclinonc.2013.171
Rooney, A.A., Cooper, G.S., Jahnke, G.D., Lam, J., Morgan, R.L., Boyles, A.L., et al . (2016) How Credible Are the Study Results? Evaluating and Applying Internal Validity Tools to Literature-Based Assessments of Environmental Health Hazards. Environment International , 92, 617-629. https://doi.org/10.1016/j.envint.2016.01.005
McNutt, M. (2014) Journals Unite for Reproducibility. Science , 346, 679-679. https://doi.org/10.1126/science.aaa1724
Vazire, S. (2018) Implications of the Credibility Revolution for Productivity, Creativity, and Progress. Perspectives on Psychological Science , 13, 411-417. https://doi.org/10.1177/1745691617751884
Engineering National Academies of Sciences (2019) Reproducibility and Replicability in Science. National Academies Press.
Grant, S., Wendt, K.E., Leadbeater, B.J., Supplee, L.H., Mayo-Wilson, E., Gardner, F., et al . (2022) Transparent, Open, and Reproducible Prevention Science. Prevention Science , 23, 701-722. https://doi.org/10.1007/s11121-022-01336-w
Resnik, D.B. and Shamoo, A.E. (2016) Reproducibility and Research Integrity. Accountability in Research , 24, 116-123. https://doi.org/10.1080/08989621.2016.1257387
Loeb, A. (2021) To Qualify as “Scientific”, Evidence Has to Be Reproducible. https://www.scientificamerican.com/article/to-qualify-as-scientific-evidence-has-to-be-reproducible/
Stark, P.B. (2018) Before Reproducibility Must Come Preproducibility. Nature , 557, 613-613. https://doi.org/10.1038/d41586-018-05256-0
Zheng, Z. (2021) Reasons, Challenges, and Some Tools for Doing Reproducible Transportation Research. Communications in Transportation Research , 1, Article ID: 100004. https://doi.org/10.1016/j.commtr.2021.100004
Diaba-Nuhoho, P. and Amponsah-Offeh, M. (2021) Reproducibility and Research Integrity: The Role of Scientists and Institutions. BMC Research Notes , 14, Article No. 451. https://doi.org/10.1186/s13104-021-05875-3
Begley, C.G. and Ellis, L.M. (2012) Raise Standards for Preclinical Cancer Research. Nature , 483, 531-533. https://doi.org/10.1038/483531a
Popper, K.R. (2002) Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge.
Nielsen, M.A. (2014) Reinventing Discovery: The New Era of Networked Science. Princeton University Press。
Munafò, M.R., Nosek, B.A., Bishop, D.V.M., Button, K.S., Chambers, C.D., Percie du Sert, N., et al . (2017) A Manifesto for Reproducible Science. Nature Human Behaviour , 1, Article No. 21. https://doi.org/10.1038/s41562-016-0021
Saltelli, A. and Giampietro, M. (2017) What Is Wrong with Evidence Based Policy, and How Can It Be Improved? Futures , 91, 62-71. https://doi.org/10.1016/j.futures.2016.11.012
Woelfle, M., Olliaro, P. and Todd, M.H. (2011) Open Science Is a Research Accelerator. Nature Chemistry , 3, 745-748. https://doi.org/10.1038/nchem.1149
Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., et al . (2015) Promoting an Open Research Culture. Science , 348, 1422-1425. https://doi.org/10.1126/science.aab2374
Kidwell, M.C., Lazarević, L.B., Baranski, E., Hardwicke, T.E., Piechowski, S., Falkenberg, L., et al . (2016) Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency. PLOS Biology , 14, e1002456. https://doi.org/10.1371/journal.pbio.1002456
McKiernan, E.C., Bourne, P.E., Brown, C.T., Buck, S., Kenall, A., Lin, J., et al . (2016) How Open Science Helps Researchers Succeed. eLife , 5, e16800. https://doi.org/10.7554/elife.16800
Stodden, V., Seiler, J. and Ma, Z. (2018) An Empirical Analysis of Journal Policy Effectiveness for Computational Reproducibility. Proceedings of the National Academy of Sciences , 115, 2584-2589. https://doi.org/10.1073/pnas.1708290115
Leonelli, S. (2016) Data-Centric Biology. University of Chicago Press. https://doi.org/10.7208/chicago/9780226416502.001.0001
Hardwicke, T.E. and Ioannidis, J.P.A. (2018) Populating the Data Ark: An Attempt to Retrieve, Preserve, and Liberate Data from the Most Highly-Cited Psychology and Psychiatry Articles. PLOS ONE , 13, e0201856. https://doi.org/10.1371/journal.pone.0201856
Peng, R.D. (2011) Reproducible Research in Computational Science. Science , 334, 1226-1227. https://doi.org/10.1126/science.1213847
Figshare (2023) About Figshare. https://knowledge.figshare.com/about
(2023) Mendeley Data. https://data.mendeley.com/
Dryad (2023) Home-Publish and Preserve Your Data. https://datadryad.org/stash
Harvard Dataverse (2020) For Researchers—Harvard Dataverse Support. https://support.dataverse.harvard.edu/researchers
(2023) OSF: Open Science Framework. https://osf.io/
Zenodo (2013) European Organization for Nuclear Research and Open AIRE. https://www.zenodo.org/
ScienceDB (2023) Science Data Bank. https://www.scidb.cn/en/introduction
McMaster University (2023) Open Access Data Repositories. https://mira.mcmaster.ca/research/open-access-data-repositories
Piwowar, H.A., Day, R.S. and Fridsma, D.B. (2007) Sharing Detailed Research Data Is Associated with Increased Citation Rate. PLOS ONE , 2, e308. https://doi.org/10.1371/journal.pone.0000308
Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., et al . (2016) The FAIR Guiding Principles for Scientific Data Management and Stewardship. Scientific Data , 3, Article No. 160018. https://doi.org/10.1038/sdata.2016.18
Michener, W.K. (2015) Ten Simple Rules for Creating a Good Data Management Plan. PLOS Computational Biology , 11, e1004525. https://doi.org/10.1371/journal.pcbi.1004525
Simons, D.J. (2014) The Value of Direct Replication. Perspectives on Psychological Science , 9, 76-80. https://doi.org/10.1177/1745691613514755
Baker, M. (2016) 1, 500 Scientists Lift the Lid on Reproducibility. Nature , 533, 452-454. https://doi.org/10.1038/533452a
Matosin, N., Frank, E., Engel, M., Lum, J.S. and Newell, K.A. (2014) Negativity Towards Negative Results: A Discussion of the Disconnect between Scientific Worth and Scientific Culture. Disease Models & Mechanisms , 7, 171-173. https://doi.org/10.1242/dmm.015123
Stodden, V., Guo, P. and Ma, Z. (2013) Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals. PLOS ONE , 8, e67111. https://doi.org/10.1371/journal.pone.0067111
Stodden, V., McNutt, M., Bailey, D.H., Deelman, E., Gil, Y., Hanson, B., et al . (2016) Enhancing Reproducibility for Computational Methods. Science , 354, 1240-1241. https://doi.org/10.1126/science.aah6168
Scacchi, W. (2010) The Future of Research in Free/Open Source Software Development. Proceedings of the FSE / SDP Workshop on Future of Software Engineering Research , Santa Fe New, 7-8 November 2010, 315-319.
Ince, D.C., Hatton, L. and Graham-Cumming, J. (2012) The Case for Open Computer Programs. Nature , 482, 485-488. https://doi.org/10.1038/nature10836
Sonnenburg, S., Braun, M.L., Ong, C.S., et al . (2007) The Need for Open Source Software in Machine Learning. Journal of Machine Learning Research , 8, 2443-2466.
Blischak, J.D., Davenport, E.R. and Wilson, G. (2016) A Quick Introduction to Version Control with Git and Github. PLOS Computational Biology , 12, e1004668. https://doi.org/10.1371/journal.pcbi.1004668
Ram, K. (2013) Git Can Facilitate Greater Reproducibility and Increased Transparency in Science. Source Code for Biology and Medicine , 8, Article No. 7. https://doi.org/10.1186/1751-0473-8-7
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L. and Teal, T.K. (2017) Good Enough Practices in Scientific Computing. PLOS Computational Biology , 13, e1005510. https://doi.org/10.1371/journal.pcbi.1005510
(2017) Challenges in Irreproducible Research. Nature , 546, 583.
Teixeira da Silva, J.A. (2015) Negative Results: Negative Perceptions Limit Their Potential for Increasing Reproducibility. Journal of Negative Results in BioMedicine , 14, Article No. 12. https://doi.org/10.1186/s12952-015-0033-9
Bos, N., Lodi, S., Meyer, M., et al . (2012) The Importance of Systems Thinking to Address Obesity. Nutrition Reviews , 75, 94-106.
Whitmire, A.L., Baldwin, M.K., Desselle, B.C., et al . (2016) Neuroscience Data Integration through the Brokering of Standards. Neuroinformatics , 14, 273-287.
Cuzzocrea, A. and Shahriar, H. (2017) Data Masking Techniques for Nosql Database Security: A Systematic Review. 2017 IEEE International Conference on Big Data ( Big Data ), Boston, 11-14 December 2017, 4467-4473. https://doi.org/10.1109/bigdata.2017.8258486
Sarada, G., Abitha, N., Manikandan, G. and Sairam, N. (2015) A Few New Approaches for Data Masking. 2015 International Conference on Circuits , Power and Computing Technologies [ ICCPCT -2015], Nagercoil, 19-20 March 2015, 1-4. https://doi.org/10.1109/iccpct.2015.7159301
Cobb, M. (2022) What Is Data Masking? Techniques, Types and Best Practices. https://www.techtarget.com/searchsecurity/definition/data-masking
Mivule, K. (2013) Utilizing Noise Addition for Data Privacy, an Overview. https://arxiv.org/pdf/1309.3958
He, J., Cai, L. and Guan, X. (2018) Preserving Data-Privacy with Added Noises: Optimal Estimation and Privacy Analysis. IEEE Transactions on Information Theory , 64, 5677-5690. https://doi.org/10.1109/tit.2018.2842221
Kadampur, M.A. and Somayajulu, D. (2010) A Noise Addition Scheme in Decision Tree for Privacy Preserving Data Mining. http://arxiv.org/abs/1001.3504.
Slijepčević, D., Henzl, M., Klausner, L.D., Dam, T., Kieseberg, P. and Zeppelzauer, M. (2021) k -Anonymity in Practice: How Generalisation and Suppression Affect Machine Learning Classifiers. Computers & Security , 111, Article ID: 102488. https://doi.org/10.1016/j.cose.2021.102488
De Capitani di Vimercati, S., Foresti, S., Livraga, G. and Samarati. P. (2023) k -Anonymity: From Theory to Applications. Transactions on Data Privacy , 16, 25-49.
Templ, M., Meindl, B., Kowarik, A. and Gussenbauer, J. (2023) SdcMicro: Statistical Disclosure Control Methods for Anonymization of Data and Risk Estimation. R Package Version 5.7.5.
Fujita, T. (2022) Anonypy: Anonymization Library for Python. Protect the Privacy of Individuals. https://github.com/glassonion1/anonypy
Machanavajjhala, A., Kifer, D., Gehrke, J. and Venkitasubramaniam, M. (2007) L-Diversity: Privacy beyond k -Anonymity. ACM Transactions on Knowledge Discovery from Data , 1, 3. https://doi.org/10.1145/1217299.1217302
Dwork, C. (2008) Differential Privacy: A Survey of Results. In: Agrawal, M., Du, D., Duan, Z. and Li, A., Eds., Theory and Applications of Models of Computation . TAMC 2008, Springer, 1-19. https://doi.org/10.1007/978-3-540-79228-4_1
Apple Differential Privacy Team (2017) Learning with Privacy at Scale. https://machinelearning.apple.com/research/learning-with-privacy-at-scale
Wood, J. and Basulto-Elias, G. (2024) ERD-Builder: Entity Relationship Diagrams Builder, 2024. R Package Version 1.0.0. https://CRAN.R-project.org/package=ERDbuilder
Birch, K., Cochrane, D. and Ward, C. (2021) Data as Asset? The Measurement, Governance, and Valuation of Digital Personal Data by Big Tech. Big Data & Society , 8. https://doi.org/10.1177/20539517211017308
Asswad, J. and Marx Gómez, J. (2021) Data Ownership: A Survey. Information , 12, Article 465. https://doi.org/10.3390/info12110465
Padova, Y. (2021) Data Ownership versus Data Sharing: And What about Privacy? Lex Electronica , 26, 38-73. https://heinonline.org/HOL/P?h=hein.journals/lexel26i=33
Bezuidenhout, L.M., Leonelli, S., Kelly, A.H. and Rappert, B. (2017) Beyond the Digital Divide: Towards a Situated Approach to Open Data. Science and Public Policy , 44, 464-475. https://doi.org/10.1093/scipol/scw036
Leek, J.T. and Peng, R.D. (2015) Reproducible Research Can Still Be Wrong: Adopting a Prevention Approach. Proceedings of the National Academy of Sciences of the United States of America , 112, 1645-1646. https://doi.org/10.1073/pnas.1421412111
Hartley, M. and Olsson, T.S.G. (2020) Dtoolai: Reproducibility for Deep Learning. Patterns , 1, Article ID: 100073. https://doi.org/10.1016/j.patter.2020.100073
Alahmari, S.S., Goldgof, D.B., Mouton, P.R. and Hall, L.O. (2020) Challenges for the Repeatability of Deep Learning Models. IEEE Access , 8, 211860-211868. https://doi.org/10.1109/access.2020.3039833
Perez-Riverol, Y., Gatto, L., Wang, R., Sachsenberg, T., Uszkoreit, J., Leprevost, F.d.V., et al . (2016) Ten Simple Rules for Taking Advantage of Git and Github. PLOS Computational Biology , 12, e1004947. https://doi.org/10.1371/journal.pcbi.1004947
Prlić, A. and Procter, J.B. (2012) Ten Simple Rules for the Open Development of Scientific Software. PLOS Computational Biology , 8, e1002802. https://doi.org/10.1371/journal.pcbi.1002802
Bokhari, E. and Hubert, L. (2018) The Lack of Cross-Validation Can Lead to Inflated Results and Spurious Conclusions: A Re-Analysis of the Macarthur Violence Risk Assessment Study. Journal of Classification , 35, 147-171. https://doi.org/10.1007/s00357-018-9252-3
Teixeira da Silva, J.A. (2013) The Need for Post-Publication Peer Review in Plant Science Publishing. Frontiers in Plant Science , 4, Article 485. https://doi.org/10.3389/fpls.2013.00485