Advancements in technology have allowed for more efficient methods of testing and assessment. In particular, remotely delivered assessments can be taken on mobile or nonmobile devices in addition to traditional pencil and paper tests. This has led to an increased interest in the comparability of mobile and nonmobile devices on performance outcomes. A variable to consider in performance outcomes on a mobile or nonmobile device is proctoring. There is evidence for both proctored and unproctored conditions leading to better performance outcomes. The present study compared performance on a remotely delivered assessment across mobile and nonmobile devices in proctored and unproctored conditions. Participants were randomly assigned to take a remotely delivered cognitive ability test on either a mobile or nonmobile device in a proctored or unproctored condition. Results indicated that participants tended to perform similarly regardless of the device type or proctoring. Implications are that organizations should consider testing job applicants via mobile devices because performance on a high stakes assessment tends to be similar to testing on a traditional desktop or laptop. Further validation of these results could allow companies to reduce hiring costs by remotely delivering assessments to applicants’ own devices.
KeywordsRemote AssessmentProctoringMobile Devices
Amrein, A. L., & Berliner, D. C. (2002). High-Stakes Testing, Uncertainty, and Student Learning. Education Policy Analysis Archives, 10, 1-74. https://doi.org/10.14507/epaa.v10n18.2002
Arthur Jr., W., Doverspike, D., Muñoz, G. J., Taylor, J. E., & Carr, A. E. (2014). The Use of Mobile Devices in High-Stakes Remotely Delivered Assessments and Testing. International Journal of Selection and Assessment, 22, 113-123. https://doi.org/10.1111/ijsa.12062
Arthur Jr., W., Glaze, R. M., Villado, A. J., & Taylor, J. E. (2010). The Magnitude and Extent of Cheating and Response Distortion Effects on Unproctored Internet-Based Tests of Cognitive Ability and Personality. International Journal of Selection and Assessment, 18, 1-16. https://doi.org/10.1111/j.1468-2389.2010.00476.x
Carstairs, J., & Myors, B. (2009). Internet Testing: A Natural Experiment Reveals Test Score Inflation on a High-Stakes, Unproctored Cognitive Test. Computers in Human Behavior, 25, 738-742.
Coyne, I., Warszta, T., Beadle, S., & Sheehan, N. (2005). The Impact of Mode of Administration on the Equivalence of a Test Battery: A Quasi-Experimental Design. International Journal of Selection and Assessment, 13, 220-224. https://doi.org/10.1111/j.1468-2389.2005.00318.x
Fallaw, S. S., Kantrowitz, T. M., & Dawson, C. R. (2012). 2012 Global Assessment Trends Report. http://www.Shl.com/assets/GATR_2012_US.pdf
Huff, K. C. (2015). The Comparison of Mobile Devices to Computers for Web-Based Assessments. Computers in Human Behavior, 49, 208-212.
Illingworth, A. J., Morelli, N. A., Scott, J. C., & Boyd, S. L. (2015). Internet-Based, Unproctored Assessments on Mobile and Non-Mobile Devices: Usage, Measurement Equivalence, and Outcomes. Journal of Business Psychology, 30, 325-343. https://doi.org/10.1007/s10869-014-9363-8
Jackson, W. (2013). Just What Does NIST Consider a Mobile Device? National Institute of Standards and Technology Special Publication, 800, 124. https://gcn.com/articles/2013/06/27/nist-mobile-device-definition.aspx?admgarea=TC_Mobile
JobTestPrep (2017). Wonderlic Sample Practice Test. TestPrep-Online. https://www.jobtestprep.com/wonderlic-sample-test
Kazmier, L. J., & Browne, C. G. (1959). Comparability of Wonderlic Test Forms in Industrial Testing. Journal of Applied Psychology, 43, 129-132. https://doi.org/10.1037/h0045688
Levine, T. R., & Hullett, C. R. (2002). Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research. Human Communication Research, 28, 612-625. https://doi.org/10.1111/j.1468-2958.2002.tb00828.x
McKelvie, S. J. (1989). The Wonderlic Personnel Test: Reliability and Validity in an Academic Setting. Psychological Reports, 65, 161-162. https://doi.org/10.2466/pr0.1989.65.1.161
McKelvie, S. J. (1994). Validity and Reliability for an Experimental Short Form of the Wonderlic Personnel Test in an Academic Setting. Psychological Reports, 75, 907-910. https://doi.org/10.2466/pr0.1994.75.2.907
Mueller-Hanson, R., Heggestad, E. D., & Thornton, G. C. (2003). Faking and Selection: Considering the Use of Personality from Select-In and Select-Out Perspectives. Journal of Applied Psychology, 88, 348-355. https://doi.org/10.1037/0021-9010.88.2.348
Murphy, K. R., & Myors, B. (2004). Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests (2nd ed.). Mahwah, NJ: Erlbaum.
Nye, C. D., Do, B., Drasgow, F., & Fine, S. (2008). Two-Step Testing in Employee Selection: Is Score Inflation a Problem? International Journal of Selection and Assessment, 16, 112-120. https://doi.org/10.1111/j.1468-2389.2008.00416.x
Pearlman, K. (2009). Unproctored Internet Testing: Practical, Legal, and Ethical Concerns. Industrial and Organizational Psychology, 2, 14-19. https://doi.org/10.1111/j.1754-9434.2008.01099.x
Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. (2003). Web-Based and Paper-and-Pencil Testing of Applicants in a Proctored Setting: Are Personality, Biodata, and Situational Judgment Tests Comparable? Personnel Psychology, 56, 733-752. https://doi.org/10.1111/j.1744-6570.2003.tb00757.x
Potosky, D., & Bobko, P. (1997). Computer versus Paper-and-Pencil Administration Mode and Response Distortion in Noncognitive Selection Tests. Journal of Applied Psychology, 82, 293-299. https://doi.org/10.1037/0021-9010.82.2.293
Sanchez, C. A., & Branaghan, R. J. (2011). Turning to Learn: Screen Orientation and Reasoning with Small Devices. Computers in Human Behavior, 27, 793-797.
Sanchez, C. A., & Goolsbee, J. Z. (2010). Character Size and Reading to Remember from Small Displays. Computers and Education, 55, 1056-1062.
Schroeders, U., & Wilhelm, O. (2010). Testing Reasoning Ability with Handheld Computers, Notebooks, and Paper and Pencil. European Journal of Psychological Assessment, 26, 284-292. https://doi.org/10.1027/1015-5759/a000038
Tippins, N. T. (2009). Internet Alternatives to Traditional Proctored Testing: Where Are We Now? Industrial and Organizational Psychology, 2, 2-10. https://doi.org/10.1111/j.1754-9434.2008.01097.x
Tippins, N. T., Beaty, J., Drasgow, F., Gibson, W. M., Pearlman, K., & Segall, D. O. (2006). Unproctored Internet Testing in Employment Settings. Personnel Psychology, 59, 189-225. https://doi.org/10.1111/j.1744-6570.2006.00909.x
Weaver, H. B., & Bonneau, C. A. (1956). Equivalence of Forms of the Wonderlic Personnel Test: A Study of Reliability and Interchangeability. Journal of Applied Psychology, 40, 127-129. https://doi.org/10.1037/h0047065
Weiner, J. A., & Morrison Jr., J. D. (2009). Unproctored Online Testing: Environmental Conditions and Validity. Industrial and Organizational Psychology, 2, 27-30. https://doi.org/10.1111/j.1754-9434.2008.01102.x
Wilkerson, J. M., Nagao, D. H., & Martin, C. L. (2002). Socially Desirable Responding in Computerized Questionnaires: When Questionnaire Purpose Matters More Than the Mode. Journal of Applied Social Psychology, 32, 544-559. https://doi.org/10.1111/j.1559-1816.2002.tb00229.x