Tags
A-Level exams in Northern Ireland, Assessment and Testing: A survey of research, Assessment: Problems, CCEA, CCEA Regulation, Council for Curriculum Examinations and Assessment, Covid-19, Developments and Statistical Issues, Dr Hugh Morrison, Dr Hugh Morrison Queen's University Belfast, Harvey Goldstein, Jo-Anne Baird, Joel Michell, Justin Edwards, ludwig Wittgenstein, Mike Cresswell, Niels Bohr, Ofqual, P Newton, Robert, Sharon King, whatever standard of attainment it is judged by the awarders to represent, Wood
Why Centre-based Moderation cannot work
Ofqual and CCEA intend to apply a “moderation” process to teacher-predicted grades in order to prevent, for example, teachers awarding inflated grades to their students. This process – yet to be set out in detail – will focus on the Examination Centre in which each pupil would have taken his or her 2020 GCSE, AS or A2 examinations had Covid-19 not intervened. To simplify matters, consider a Centre in which, for instance, 20% to 24% of pupils have secured B grades in CCEA AS Physics for the past three years. Now suppose that in 2020 the physics teachers associated with that Centre return a B-grade prediction for 67% of their AS pupils. Does a statistical technique, or AI algorithm, or mathematical model (possibly drawing on teachers’ predicted rank orders) exist which can defensibly adjust the predicted grades to bring them into line with the 20% to 24% range of the past? Can one compute a defensible compromise position somewhere between 20% and 67%? The answer is an emphatic No.
It is difficult to escape the conclusion that Ofqual and CCEA simply interpret grades as quantities which are countable and can be assigned to individual pupils. But two of the most influential figures in UK assessment reject these claims. The UK’s examining bodies and researchers in education have been, for years, treating grades as quantifiable entities. This is because the awarding bodies have a very poor track record for in-depth thinking about the nature of the “measurements” in which they engage (see Wood[1] (1991)).
There are few individuals with Mike Cresswell’s understanding of the grading of UK examinations. Cresswell’s definition[2] of a grade as representing “whatever standard of attainment it is judged by the awarders to represent” (p. 224), indicates that counting grades as one might count pencils is indefensible. Let me be clear: I am not suggesting that the process for awarding grades needs to be abandoned. I am simply making the point that grading is not governed by strict scientific principles and that adding or subtracting grades is mathematically impermissible. As Cresswell’s definition makes clear, the grading process is a qualitative process rather than a quantitative one.
Now why is Cresswell forced to this vague qualitative definition of a grade? The reason is that the awarding bodies, education researchers, and the general public think of the grade awarded to a given pupil as a measure of the ability of that pupil. The awarding bodies think of a grade as a property of the particular pupil to whom it is awarded. But this is wrong: a grade is not an intrinsic property of the pupil but rather a joint property of the pupil and the examination from which the grade derives. In his 1996 book Assessment: Problems, Developments and Statistical Issues Harvey Goldstein[3] (a towering figure who contributed much to debates on statistical rigour in UK assessment) cautions: “[T]he object of measurement is expected to interact with the measurement in a way that may alter the state of the individual in a non-trivial fashion” (p. 54).
According to Goldstein, the examination does not merely “check up on” a pre-existing ability that the candidate had when he or she entered the examination hall. This static model is rejected for a more dynamic alternative in which the pupil’s ability is expressed through his or her responses to the questions which make up the examination. For Goldstein, ability changes as the candidate interacts with the examination questions: “Thus, on answering a sequence of test questions about quadratic equations the individual may well become “better” at solving them so that the attribute changes during the course of the test” (p. 54). Cresswell’s grade is not an intrinsic property of the candidate; rather, it’s the property of an interaction. Grades do not lend themselves to simple arithmetic manipulation and therefore quantitative procedures – such as simple regression or neural nets – are indefensible.
One can find unequivocal support for the claims of Cresswell and Goldstein in the writings of Niels Bohr and Ludwig Wittgenstein. Also Joel Michell’s research[4] can be used to establish that grades do not satisfy the seven Hölder axioms[5] and therefore are not quantifiable. There can be little doubt that the claims of Ofqual and CCEA that Centre-based statistical techniques, or algorithms, or mathematical modelling, can be used to moderate predicted grades are without foundation.
Dr Hugh Morrison (The Queen’s University of Belfast [retired])
[1] Wood, R. (1991). Assessment and testing: A survey of research. Cambridge: Cambridge University Press.
[2] Baird, J., Cresswell, M., & Newton, P. (2000). Would the real gold standard please step forward? Research Papers in Education, 15(2), 213-229.
[3] Goldstein, H. (1996). Statistical and psychometric models for assessment. In H. Goldstein & T. Lewis (Eds.), Assessment: problems, developments and statistical issues (pp. 41-55). Chichester: John Wiley & Sons.
[4] Michell, J. (1999). Measurement in psychology. Cambridge: Cambridge University Press.
[5] Michell, J., & Ernst, C. (1996). The axioms of quantity and the theory of measurement, Part 1, An English translation of Hölder (1901), Part 1, Journal of Mathematical Psychology, 40, 235-52.
(1997), The axioms of quantity and the theory of measurement, Part II, An English translation of Hölder (1901), Part II, Journal of Mathematical Psychology, 41, 345-56.
Hölder, O. (1901). Die axiome der quatität und die lehre vom mass, Berichte der Sachsischen Gesellschaft der Wissenschaften, Mathematische-Physicke Klasse, 53, 1-64.