, , , , , , , , , , , , , , , , , , , , , ,

Tuesday December 3rd 2013 brings the latest OECD PISA results. Before reading a single headline or watching dazzling charts designed to mislead, first read this expose. If you have a reasoned response to Dr Morrison’s essay do not hesitate to get in touch.

Why Michael Gove should follow India’s lead and
detach himself from PISA

Just ahead of the publication of PISA league tables on 3rd December, India has withdrawn from the list of countries which will feature in the tables. The Education Secretary, Michael Gove, on the other hand, seems determined to stick with PISA despite recent concerns about the global league table published in the Times Educational Supplement in July of this year.

Mr Gove’s Department reiterated its support for PISA in a recently aired Radio 4 programme entitled “PISA – Global Education Tables Tested.” That programme illustrated the dangers inherent in critiquing PISA in statistical terms.

Statistical modellers have made life too easy for PISA because they simply accept the PISA interpretation of the construct “ability.” PISA lays claim to measure the relative qualities of education systems around the world, and it is only when the focus moves to measurement that the profound difficulties inherent in Pisa become clear.

Niels Bohr is ranked among the ten greatest physicists of all time. The father of quantum measurement taught that “unambiguous communication” was the hallmark of measurement in physics. Importantly, Bohr traced measurement in quantum mechanics and measurement in psychology to a common source, which he referred to as “subject/object holism.” The physicist cannot have direct experience of the atom, just as the teacher cannot have direct experience of the child’s mind. Both are forced to describe what is beyond direct experience using the language of everyday experience.

Bohr demonstrated that measurement in quantum physics and in psychology share a common inescapable constraint, namely, one cannot communicate unambiguously about measurement in either realm without factoring in the measuring instrument. Wittgenstein’s writings also support this argument.

The lesson we learn from Bohr is that in all psychological measurement, the entity measured cannot be divorced from the measuring instrument. When this central tenet of measurement is broken, nonsense always ensues. The so-called Rasch model, which produces the PISA ranks, offends against this central measurement principle and therefore the ranks it generates are suspect at best.

(The Rasch model is a member of a family of models which all treat what is measured as independent of the measuring tool. Given that these models underpin both computer adaptive testing and the navigation systems of the newly developed MOOCS of higher education, the implications of Bohr’s thinking are clearly far-reaching.)

The following simple illustration will help make Bohr’s point. Suppose Einstein and a GCSE pupil both produce a perfect score on a GCSE paper. Surely to claim that the pupil has the same mathematical ability as Einstein is to communicate ambiguously? However, unambiguous communication can be restored if we simply take account of the measuring instrument and say, “Einstein and the pupil have the same mathematical ability relative to this particular GCSE paper.” Mathematical ability, indeed any ability, is not an intrinsic property of the individual; rather, it’s a joint property of the individual and the measuring instrument.

In short, ability isn’t a property of the person being measured; it’s a property of the interaction of the person with the measuring instrument. One is concerned with the between rather than the within. It’s hard to imagine a more stark contrast. Statistical modelling critiques of PISA, however, have missed this conceptual error entirely.

My bookshelves are groaning with books concerned with the wide-ranging debates around the notion of intelligence. All of these debates dissolve away when one eschews the twin notions that intelligence is either a property of the person or is an ensemble property, for the simple definition that intelligence is a property of the interaction between person and intelligence test. To say “John has an IQ of 104” is to communicate ambiguously. Furthermore, clarification of the nature of measurement in psychology and education has implications for the UK’s approach to school inspection and serves as a challenge for those who reject, out of hand, “teaching to the test.”

In closing,

when the PISA critique shifts from statistical modelling to measurement, the profound nature of PISA’s error becomes clear.

I trust this essay will be a comfort to those responsible for removing India from PISA, and hope it will prompt a similar decision in the UK.

Dr Hugh Morrison