The Forgotten Dimension: The Information Content of Objective Questions

Abstract

Objective examination questions are used for assessment in many disciplines and levels. They allow extensive coverage, reliability, re-usability and administrative ease, but the amount of information they gather about student knowledge is not widely known. In this paper, I show that small changes in question format make large differences to the information gained, and therefore to the efficiency of the assessment process. The basic objective question (BOQ) can be extended beyond the true/false response to test probabilities in complex situations. When objective questions are used to assess students’ abilities, the information gained should be sufficient in quantity and coverage and at the appropriate DOK (Depth of Knowledge) to give a robust and valid assessment. Objective questions take time and expertise to set and validate, so we should seek methods that meet these criteria with the least administrative load. The basic objective question is a statement followed by two or more responses; one response must be marked as correct. The Type 1 MCQ (Multiple Choice Question), multiple true/false question (MTFQ), extended matched question (EMQ) and others are all variants of the BOQ, but they differ widely in the amount of information they gain. For a given input of statements and responses, the Type 1 MCQ delivers the least information about the student’s knowledge, the MTFQ and the BOQ deliver almost four times as much and the EMQ about six times as much. The expected level of student knowledge also affects the information gained. If questions are set so that most (say 90%) of students get the correct answers, then the information gained, and the discriminatory power of the test is much less than when the test is set for lower expected levels. The format used for objective questions and the expected level of correct responses govern their efficiency in collecting pass/fail information. The quality of that information such as depth of knowledge (DOK) depends on the wording and situations modeled for the questions and this is independent of their format.