Metacognitive Monitoring and Mathematical Abilities: Cognitive Diagnostic Model and Signal Detection Theory Approach

Abstract

Besides various in-class assessments, there exist some standardized assessment tools that are administered in several countries, such as PISA (Programme for International Student Assessment) and TIMMS (Trends in International Mathematics and Science Study). The questions' contents, type of responding, grading, and the analyses in these large-scale tests have been diversified in years. In this study, it was aimed to identify the abilities that are measured at PISA mathematics test in a single testing procedure and by utilizing the methods of analyses of Cognitive Diagnostic Model (CDM) as well as Signal Detection Theory (SDT), which have not been used so far in the assessment of these abilities. Therefore, a randomly selected sample of 6th-grade students (N=230) in Izmir was tested with a PISA-equivalent 12-item mathematics test, where the items are graded dichotomously (correct vs. incorrect). CDM estimates were calculated by using the Deterministic Input Noisy Output and Gate (DINA) Model. The participants were asked to report whether they thought they could solve the question correctly, guess even if they thought they could not solve the question, and then, rate their confidence levels on the correctness of their answers in turn so as to allow us to measure their "metacognitive monitoring performance" with the SDT method, which refers to the ability to differentiate correct and incorrect responses. In short, a better metacognitive monitoring performance was obtained by measuring how well once could differentiate their correct and incorrect responses with the observation of they prefer reporting and then giving high confidence levels to the actually correct responses and prefer passing to give an answer yet rate lower confidence levels to the actually incorrect responses given as pure guesses. The results showed that CDM fits well to the assessment of PISA test and those who were better at the ability of "reasoning and developing strategies" in particular among four possible abilities detected with CDM ("representing and communicating", "mathematization", "reasoning and developing strategies", "using symbolic and technical language") had also better metacognitive monitoring performance. The present study, therefore, contributes to the research that investigates what features the ability of better differentiating correct and incorrect responses are actually linked. Based on the results, it is suggested that a better metacognitive monitoring ability is linked to having a better ability of "reasoning and developing strategies" in particular. Additionally, it is suggested that measuring metacognitive monitoring performance at PISA -or even any other possible tests- with the SDT calculation method, that has a relatively straightforward testing procedure, may yield various estimates for the students' abilities measured at the test as well as their related higher-order abilities.