The application of CFA to Multitrait-Multimethod Matrices (MTMM) is an elaborated method for the evaluation of construct validity in terms of the discriminant and convergent validity as well as method effects. It is implemented to evaluate the acceptability of a questionnaire as a measure of constructs or latent variables, during the structural and external stages of an instrument validation process. CFA can be carried out in MTMM matrices, like in any other covariance matrix, to study the latent variables of traits and methods factors. CFA-MTMM models can distinguish systematic trait or method effects from unsystematic measurement error variance, thus offering the possibility of hypotheses testing on the measurement model, while controlling possible effects. A plethora of CFA-MTMM approaches have been proposed, but the Correlated Methods Models and the Correlated Uniqueness Models are two CFA-MTMM methods most widely used. This work discusses how these two approaches can be parametrized, and how inferences about construct validity and method effects can be drawn on the matrix level and on parameters level. Then, the two methods are briefly compared. Additional CFA MTMM parameterizations are also discussed and their advantages and disadvantages are summarized.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for Educational and Psychological Testing (2nd ed.). Washington, DC: Authors.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing (3rd ed.). Washington, DC: Authors.
Bagozzi, R. P. (1993). Assessing Construct Validity in Personality Research: Applications to Measures of Self-Esteem. Journal of Research in Personality, 27, 49-87.
Bagozzi, R. P., & Yi, Y. (1990). Assessing METHOD variance in Multitrait-Multimethod Matrices: The Case of Self-Reported Affect and Perceptions at Work. Journal of Applied Psychology, 75, 547-560.
Benson, J. (1988). Developing a Strong Program of Construct Validation: A Test Anxiety Example. Educational Measurement: Issues and Practice, 17, 10-17.
Brown, T. A. (2015). Confirmatory Factor Analysis for Applied Research (2nd ed.). New York: The Guilford Press.
Brown, T. A., & Moore, M. T. (2012). Confirmatory Factor Analysis. In R. H. Hoyle (Ed.), Handbook of Structural Equation Modeling (pp. 361-379). New York, NY: Guilford Publications.
Browne, M. W. (1984). The Decomposition of Multitrait-Multimethod Matrices. British Journal of Mathematical and Statistical Psychology, 37, 1-21.
Byrne, B. M. (2012). Structural Equation Modeling with Mplus: Basic Concepts, Applications, and Programming (2nd ed.). New York: Routledge.
Byrne, B. M., & Bazana, P. G. (1996). Investigating the Measurement of Social and Academic Competencies for Early/Late Preadolescents and Adolescents: A Multitrait-Multimethod Analysis. Applied Measurement in Education, 9, 113-132.
Byrne, B. M., & Goffin, R. D. (1993). Modeling MTMM Data from Additive and Multiplicative Covariance Structures: An Audit of Construct Validity Concordance. Multivariate Behavioral Research, 28, 67-96.
Campbell, D. T., & Fiske, D. W (1959). Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix. Psychological Bulletin, 56, 81-105.
Campbell, D. T., & O’Connell, E. J. (1967). Method Factors in Multitrait-Multimethod Matrices: Multiplicative Rather than Additive? Multivariate Behavioral Research, 2, 409-426.
Chen, F. F. (2007). Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance. Structural Equation Modeling, 14, 464-504.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance. Structural Equation Modeling, 9, 233-255.
Coenders, G., & Saris, W. E. (2000). Testing Nested Additive, Multiplicative, and General Multitrait-Multimethod Models. Structural Equation Modeling, 7, 219-250. https://doi.org/10.1207/S15328007SEM0702_5
Cole, D. A. (1987). Utility of Confirmatory Factor Analysis in Test Validation Research. Journal of Consulting and Clinical Psychology, 55, 584-594. https://doi.org/10.1037/0022-006X.55.4.584
Cole, D. A., Ciesla, J. A., & Steiger, J. (2007). The Insidious Effects of Failing to Include Design-Driven Correlated Residuals in Latent-Variable Covariance Structure Analysis. Psychological Methods, 12, 381-398. https://doi.org/10.1037/1082-989X.12.4.381
Cole, D. A., Howard, G. S., & Maxwell, S. E. (1981). Effects of Mono- versus Multiple-Operationalization in Construct Validation Efforts. Journal of Consulting and Clinical Psychology, 49, 395-405. https://doi.org/10.1037/0022-006X.49.3.395
Conway, J. M., Scullen, S. E., Lievens, F., & Lance, C. E. (2004). Bias in the Correlated Uniqueness Model for MTMM Data. Structural Equation Modeling, 11, 535-559. https://doi.org/10.1207/s15328007sem1104_3
Coovert, M. D., Craiger, J. P., & Teachout, M. S. (1997). Effectiveness of the Direct Product versus Confirmatory Factor Model for Reflecting the Structure of Multimethod-Multirater Job Performance Data. Journal of Applied Psychology, 82, 271-280. https://doi.org/10.1037/0021-9010.82.2.271
Corten, I. W., Saris, W. E., Coenders, G., van der Veld, W., Aalberts, C. E., & Kornelis, C. (2002). Fit of Different Models for Multitrait-Multimethod Experiments. Structural Equation Modeling, 9, 213-232. https://doi.org/10.1207/S15328007SEM0902_4
Coulacoglou, C., & Saklofske, D. H. (2017). Psychometrics and Psychological Assessment: Principles and Applications. Oxford, UK: Academic Press.
Courvoisier, D. S., Nussbeck, F. W., Eid, M., Geiser, C., & Cole, D. A. (2008). Analyzing the Convergent and Discriminant Validity of States and Traits: Development and Applications of Multimethod Latent State-Trait Models. Psychological Assessment, 20, 270-280. https://doi.org/10.1037/a0012812
Cronbach, L. J., & Meehl, P. E. (1955). Construct Validity in Psychological Tests. Psychological Bulletin, 52, 281-302. https://doi.org/10.1037/h0040957
Cudeck, R. (1988). Multiplicative Models and MTMM Matrices. Journal of Educational Statistics, 13, 131-147. https://doi.org/10.2307/1164750
DeVellis, R. F. (2017). Scale Development: Theory and Applications (4th ed.). Thousand Oaks, CA: Sage.
Dudgeon, P. (1994). A Reparameterization of the Restricted Factor Analysis Model for Multitrait-Multimethod Matrices. British Journal of Mathematical and Statistical Psychology, 47, 283-308. https://doi.org/10.1111/j.2044-8317.1994.tb01038.x
Dumenci, L. (2000). Multitrait-Multimethod Analysis. In S. D. Brown, & H. E. A. Tinsley (Eds.), Handbook of Applied Multivariate Statistics and Mathematical Modeling (pp. 583-611). San Diego, CA: Academic Press. https://doi.org/10.1016/B978-012691360-6/50021-5
Eid, M. (2000). A Multitrait-Multimethod Model with Minimal Assumptions. Psychometrika, 65, 241-261. https://doi.org/10.1007/BF02294377
Eid, M. (2010). Multitrait-Multimethod-Matrix. In N. Salkind (Ed.), Encyclopedia of Research Design (Vol. 1, pp. 850-855). Thousand Oaks, CA: Sage.
Eid, M., Lischetzke, T., & Nussbeck, F. W. (2006). Structural Equation Models for Multitrait-Multimethod Data. In M. Eid, & E. Diener (Eds.), Handbook of Multimethod Measurement in Psychology (pp. 283-299). Washington DC: American Psychological Association. https://doi.org/10.1037/11383-020
Eid, M., Lischetzke, T., Nussbeck, F. W., & Trierweiler, L. I. (2003). Separating Trait Effects from Trait-Specific Method Effects in Multitrait-Multimethod Models: A Multiple-Indicator CT-C (M-1) Model. Psychological Methods, 8, 38-60. https://doi.org/10.1037/1082-989X.8.1.38
Eid, M., Nussbeck, F. W., Geiser, C., Cole, D. A., Gollwitzer, M., & Lischetzke, T. (2008). Structural Equation Modeling of Multitrait-Multimethod Data: Different Models for Different Types of Methods. Psychological Methods, 13, 230-253. https://doi.org/10.1037/a0013219
Flamer, S. (1983). Assessment of the Multitrait-Multimethod Matrix Validity of Likert Scales via Confirmatory Factor Analysis. Multivariate Behavioral Research, 18, 275-308. https://doi.org/10.1207/s15327906mbr1803_3
Franzen, M. D. (2002). Reliability and Validity in Neuropsychological Assessment (2nd ed.). New York: Springer. https://doi.org/10.1007/978-1-4757-3224-5
Furr, R. M. (2011). Scale Construction and Psychometrics for Social and Personality Psychology. New Delhi, IN: Sage Publications. https://doi.org/10.4135/9781446287866
Geiser, C., Eid, M., & Nussbeck, F. W. (2008). On the Meaning of the Latent Variables in the CTC (M-1) Model: A Comment on Maydeu-Olivares and Coffman (2006). Psychological Methods, 13, 49-57. https://doi.org/10.1037/1082-989X.13.1.49
Geiser, C., Eid, M., West, S. G., Lischetzke, T., & Nussbeck, F. W. (2012). A Comparison of Method Effects in Two Confirmatory Factor Models for Structurally Different Methods. Structural Equation Modeling, 19, 409-436. https://doi.org/10.1080/10705511.2012.687658
Goffin, R. D., & Jackson, D. N. (1992). Analysis of Multitrait-Multirater Performance Appraisal Data: Composite Direct Product Method versus Confirmatory Factor Analysis. Multivariate Behavioral Research, 27, 363-385. https://doi.org/10.1207/s15327906mbr2703_4
Green, D. P., Goldman, S. L., & Salovey, P. (1993). Measurement Error Masks Bipolarity in Affect Ratings. Journal of Personality and Social Psychology, 64, 1029-1041. https://doi.org/10.1037/0022-3514.64.6.1029
Grimm, K. J., Pianta, R. C., & Konold, T. (2009). Longitudinal Multitrait-Multimethod Models for Developmental Research. Multivariate Behavioural Research, 44, 233-258. https://doi.org/10.1080/00273170902794230
Gunnell, K. E., Schellenberg, B. J. I., Wilson, P. M., Crocker, P. R. E., Mack, D. E., & Zumbo, B. D. (2014). A Review of Validity Evidence Presented in the Journal of Sport and Exercise Psychology (2002-2012): Misconceptions and Recommendations for Validation Research. In B. D. Zumbo, & E. K. H. Chan (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences, Social Indicators Research Series 54 (pp. 137-156). Switzerland: Springer International. https://doi.org/10.1007/978-3-319-07794-9_8
Hernández, A., & González-Romá, V. (2002). Analysis of Multitrait-Multioccasion Data: Additive versus Multiplicative Models. Multivariate Behavioral Research, 37, 59-87. https://doi.org/10.1207/S15327906MBR3701_03
Hox, J. J., & Kleiboer, A. M. (2007). Retrospective Questions or a Diary Method? A Two-Level Multitrait-Multimethod Analysis. Structural Equation Modeling, 14, 311-325. https://doi.org/10.1080/10705510709336748
Hu, L. T., & Bentler, P. M. (1999). Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
Jackson, D. N. (1969). Multimethod Factor Analysis in the Evaluation of Convergent and Discriminant Validity. Psychological Bulletin, 72, 30-49. https://doi.org/10.1037/h0027421
Kenny, D. A. (1976). An Empirical Application of Confirmatory Factor Analysis to the Multitrait-Multimethod Matrix. Journal of Experimental Social Psychology, 12, 247-252. https://doi.org/10.1016/0022-1031(76)90055-X
Kenny, D. A. (1979). Correlation and Causality. New York: Wiley.
Kenny, D. A., & Kashy, D. A. (1992). Analysis of the Multitrait-Multimethod Matrix by Confirmatory Factor Analysis. Psychological Bulletin, 112, 165-172. https://doi.org/10.1037/0033-2909.112.1.165
Kline, R. B. (2009). Becoming a Behavioral Science Researcher: A Guide to Producing Research That Matters. New York: Guilford Publications.
Kline, R. B. (2016). Principles and Practice of Structural Equation Modeling (4th ed.). New York: Guilford Publications.
Koch, T., Eid, M., & Lochner, K. (2018). Multitrait-Multimethod-Analysis: The Psychometric Foundation of CFA-MTMM Models. In P. Irwing, T. Booth, & D. J. Hughes (Eds.), The Wiley Handbook of Psychometric Testing: A Multidisciplinary Reference on Survey, Scale and Test Development (VII, pp. 781-846). Hoboken, NJ: Wiley.
Koch, T., Schultze, M., Eid, M., & Geiser, C. (2014). A Longitudinal Multilevel CFA-MTMM Model for Interchangeable and Structurally Different Methods. Frontiers in Psychology, 5, 311. https://doi.org/10.3389/fpsyg.2014.00311
Kyriazos, T. A. (2018). Applied Psychometrics: The 3-Faced Construct Validation Method, a Routine for Evaluating a Factor Structure. Psychology, 9, 2044-2072. https://doi.org/10.4236/psych.2018.98117
Kyriazos, T. A., Stalikas, A., Prassa, K., & Yotsidi, V. (2018a). Can the Depression Anxiety Stress Scales Short Be Shorter? Factor Structure and Measurement Invariance of DASS-21 and DASS-9 in a Greek, Non-Clinical Sample. Psychology, 9, 1095-1127. https://doi.org/10.4236/psych.2018.95069
Kyriazos, T. A., Stalikas, A., Prassa, K., & Yotsidi, V. (2018b). A 3-Faced Construct Validation and a Bifactor Subjective Well-Being Model Using the Scale of Positive and Negative Experience, Greek Version. Psychology, 9, 1143-1175. https://doi.org/10.4236/psych.2018.95071
Kyriazos, T. A., Stalikas, A., Prassa, K., Galanakis, M., Flora, K., & Chatzilia, V. (2018c). The Flow Short Scale (FSS) Dimensionality and What MIMIC Shows on Heterogeneity and Invariance. Psychology, 9, 1357-1382. https://doi.org/10.4236/psych.2018.96083
Kyriazos, T. A., Stalikas, A., Prassa, K., Galanakis, M., Yotsidi, V., & Lakioti, L. (2018d). Psychometric Evidence of the Brief Resilience Scale (BRS) and Modeling Distinctiveness of Resilience from Depression and Stress. Psychology, 9, 1828-1857.
Kyriazos, T. A., Stalikas, A., Prassa, K., Yotsidi, V., Galanakis, M., & Pezirkianidis, C. (2018e). Validation of the Flourishing Scale (FS), Greek Version and Evaluation of Two Well-Being Models. Psychology, 9, 1789-1813. https://doi.org/10.4236/psych.2018.97105
LaGrange, B., & Cole, D. A. (2008). An Expansion of the Trait-State-Occasion Model: Accounting for Shared Method Variance. Structural Equation Modeling, 15, 241-271. https://doi.org/10.1080/10705510801922381
Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A Critique of the Correlated Trait Correlated Method and Correlated Uniqueness Models for Multitrait-Multimethod Data. Psychological Methods, 7, 228-244. https://doi.org/10.1037/1082-989X.7.2.228
Lievens, F., & Conway, J. M. (2001). Dimension and Exercise Variance in Assessment Center Scores: A Large-Scale Evaluation of Multitrait-Multimethod Matrices. Journal of Applied Psychology, 86, 1202-1222. https://doi.org/10.1037/0021-9010.86.6.1202
Loehlin, J. C. (2004). Latent Variable Models (4th ed.). Mahwah, NJ: Erlbaum.
Loehlin, J. C., & Beaujean, A. A. (2017). Latent Variable Models: An Introduction to Factor, Path, and Structural Equation Analysis. New York: Taylor & Francis.
Marsh, H. W. (1988). Multitrait-Multimethod Analyses. In J. P. Keeves (Ed.), Educational Research Methodology, Measurement, and Evaluation: An International Handbook (pp. 570-578). Oxford: Pergamon.
Marsh, H. W. (1989). Confirmatory Factor Analysis of Multitrait-Multimethod Data: Many Problems and a Few Solutions. Applied Psychological Measurement, 13, 335-361. https://doi.org/10.1177/014662168901300402
Marsh, H. W., & Bailey, M. (1991). Confirmatory Factor Analyses of Multitrait-Multimethod Data: A Comparison of Alternative Models. Applied Psychological Measurement, 15, 47-70. https://doi.org/10.1177/014662169101500106
Marsh, H. W., & Byrne, B. M. (1993). Confirmatory Factor Analysis of Multitrait-Multimethod Self-Concept Data: Between-Group and Within-Group Invariance Constraints. Multivariate Behavioral Research, 28, 313-349. https://doi.org/10.1207/s15327906mbr2803_2
Marsh, H. W., & Grayson, D. (1995). Latent Variable Models of Multitrait-Multimethod Data. In R. H. Hoyle (Ed.), Structural Equation Modeling: Concepts, Issues, and Applications (pp. 177-198). Thousand Oaks, CA: Sage.
Marsh, H. W., & Hocevar, D. (1983). Confirmatory Factor Analysis of Multitrait-Multimethod Matrices. Journal of Educational Measurement, 20, 231-248. https://doi.org/10.1111/j.1745-3984.1983.tb00202.x
Marsh, H. W., Byrne, B. M., & Craven, R. (1992). Overcoming Problems in Confirmatory Factor Analyses of MTMM Data: The Correlated Uniqueness Model and Factorial Invariance. Multivariate Behavioral Research, 27, 489-507. https://doi.org/10.1207/s15327906mbr2704_1
Martin, A. J. & Marsh, H. W. (2006). Academic Resilience and Its Psychological and Educational Correlates: A Construct Validity Approach. Psychology in the Schools, 43, 267-281. https://doi.org/10.1002/pits.20149
Maydeu-Olivares, A., & Coffman, D. L. (2006). Random Intercept Item Factor Analysis. Psychological Methods, 11, 344-362. https://doi.org/10.1037/1082-989X.11.4.344
Messick, S. (1980). Test Validity and the Ethics of Assessment. American Psychologist, 35, 1012-1027. https://doi.org/10.1037/0003-066X.35.11.1012
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., (pp. 13-103). New York: Macmillan Publishing Co.
Messick, S. (1995). Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. American Psychologist, 50, 741-749. https://doi.org/10.1037/0003-066X.50.9.741
Meyer, J. F., Frost, R. O., Brown, T. A., Steketee, G., & Tolin, D. F. (2013). A Multitrait-Multimethod Matrix Investigation of Hoarding. Journal of Obsessive-Compulsive and Related Disorders, 2, 273-280. https://doi.org/10.1016/j.jocrd.2013.03.002
Millsap, R. E. (1995a). Measurement Invariance, Predictive Invariance, and the Duality Paradox. Multivariate Behavioral Research, 30, 577-605. https://doi.org/10.1207/s15327906mbr3004_6
Millsap, R. E. (1995b). The Statistical Analysis of Method Effects in Multitrait-Multimethod Data: A Review. In P. E. Shrout, & S. T. Fiske (Eds.), Personality Research, Methods, and Theory: A Festschrift Honoring Donald W. Fiske (pp. 93-109). Hillsdale, NJ: Lawrence Erlbaum Associates.
Pohl, S., & Steyer, R. (2010). Modeling Common Traits and Method Effects in Multitrait-Multimethod Analysis. Multivariate Behavioral Research, 45, 45-72. https://doi.org/10.1080/00273170903504729
Pohl, S., Steyer, R., & Kraus, K. (2008). Modelling Method Effects as Individual Causal Effects. Journal of the Royal Statistical Society Series A, 171, 41-63.
Price, L. R. (2017). Psychometric Methods: Theory into Practice. New York: The Guilford Press.
Revelle, W. (2018). Using the Psych Package to Generate and Test Structural Models. https://personality-project.org/r/psych-manual.pdf
Saris, W. E., & Aalberts, C. (2003). Different Explanations for Correlated Disturbance Terms in MTMM Studies. Structural Equation Modeling, 10, 193-213. https://doi.org/10.1207/S15328007SEM1002_2
Sawilowsky, S. (2002). A Quick Distribution-Free Test Fortrend That Contributes Evidence of Construct Validity. Measurement and Evaluation in Counseling andDevelopment, 35, 78-88.
Sawilowsky, S. S. (2007). Construct Validity. In N. J. Salkind (Ed.), Encyclopedia of Measurement and Statistics (pp. 178-180). Thousand Oaks, CA: Sage.
Schmitt, N., & Stults, D. M. (1986). Methodology Review: Analysis of Multitrait-Multimethod Matrices. Applied Psychological Measurement, 10, 1-22. https://doi.org/10.1177/014662168601000101
Shrout, P. E., & Fiske, S. T. (Eds.) (1995). Personality Research, Methods, and Theory: A Festschrift Honoring Donald W. Fiske. Hillsdale, NJ: Lawrence Erlbaum Associates.
Steyer, R. (1989). Models of Classical Psychometric Test Theory as Stochastic Measurement Models: Representation, Uniqueness, Meaningfulness, Identifiability, and Testability. Methodika, 3, 25-60.
Steyer, R. (2005). Analyzing Individual and Average Causal Effects via Structural Equation Models. Methodology, 1, 39-54. https://doi.org/10.1027/1614-1881.1.1.39
Steyer, R., Partchev, I., & Shanahan, M. J. (2000). Modeling True Intraindividual Change in Structural Equation Models: The Case of Poverty and Children’s Psychosocial Adjustment. In T. D. Little, K. U. Schnabel, & J. Baumert (Eds.), Modeling Longitudinal and Multilevel Data: Practical Issues, Applied Approaches, and Specific Examples (pp. 109-126). Mahwah, NJ: Lawrence Erlbaum.
Tomás, J. M., Hontangas, P. M., & Oliver, A. (2000). Linear Confirmatory Factor Models to Evaluate Multitrait-Multimethod Matrices: The Effects of Number of Indicators and Correlation among Methods. Multivariate Behavioral Research, 35, 469-499. https://doi.org/10.1207/S15327906MBR3504_03
Trochim, W. M. (2002). Research Methods Knowledge Base. Ithaca, NY: Cornell University.
Verhees, J., & Wansbeek, T. J. (1990). A Multimode Direct Product Model for Covariance Structure Analysis. British Journal of Mathematical and Statistical Psychology, 43, 231-240. https://doi.org/10.1111/j.2044-8317.1990.tb00938.x
Widaman, K. F. (1985). Hierarchically Nested Covariance Structure Models for Multitrait-Multimethod Data. Applied Psychological Measurement, 9, 1-26. https://doi.org/10.1177/014662168500900101
Wothke W., & Browne, M. W. (1990). The Direct Product Model for the MTMM Matrix Parameterized as a Second Order Factor Analysis Model. Psychometrika, 55, 255-262. https://doi.org/10.1007/BF02295286
Wothke, W. (1993). Nonpositive Definite Matrices in Structural Modeling. In K. A. Bollen, & J. S. Long (Eds.), Testing Structural Equation Models (pp. 256-293). Newbury Park, CA: Sage.
Wothke, W. (1995). Covariance Components Analysis of the Multitrait-Multimethod Matrix. In P. E. E. A. Shrout (Ed.), Personality Research, Methods, and Theory: A Festschrift Honoring Donald W. Fiske (pp. 125-144). Hillsdale, NJ: Lawrence Erlbaum Associates.
Zimmerman, D. W. (1975). Two Concepts of “True-Score” in Test Theory. Psychological Reports, 36, 795-805.
Zinbarg, R. E., Yovel, I., Revelle, W., & McDonald, R. P. (2006). Estimating Generalizability to a Latent Variable Common to All of a Scale’s Indicators: A Comparison of Estimators for ωh. Applied Psychological Measurement, 30, 121-144. https://doi.org/10.1177/0146621605278814