The severe limitations of CFA in showing accuracy in questionnaire (sub) scale prediction

What is it about?

Predicting the factor structure of a test and comparing this with the factor structure, empirically derived from the item scores, is a powerful test of both the test's construct validity and the theory justifying the prediction. The preferred method for such testing is often confirmatory factor analysis (CFA). CFA expresses the degree of discrepancy between predicted and empirical factor structure in chi-square and indices of "goodness of fit" (GOF), while primary factor loadings and modification indices provide some feedback on item level. However, the latter feedback is very limited, which is a missed opportunity, while chi-square and the GOF-indices appear to be problematic. This will be demonstrated by a selective review of the literature on CFA.

Featured Image

Why is it important?

Current users of the method, working in the applied social sciences, may be less aware of the above limitations of CFA than statisticians are, and may be over-optimistic about the reliability of the method when striving to validate questionnaires or testing their theories by means of questionnaire analysis. The current paper is primarily meant for them. Researchers well initiated in statistics may be served by seeing the problems recapitulated in a systematic way. In addition, this may motivate them to take note of the alternative approach to goodness-of-fit, briefly sketched in the last section (applies only to the case of incorrect allocation of indicators to factors). Also of interest to them may be my explanation of the puzzling finding that the values of chi-square and the GOF indices are affected by the number of within-factor inter-item correlations in proportion to the number of between-factor inter-item correlations.

Perspectives