, , , ,

Why Michael Gove should follow India’s lead and
detach himself from PISA

Just ahead of the publication of PISA league table on 3rd December, India has withdrawn from the list of countries which will feature in the tables. The Education Secretary, Michael Gove, on the other hand, seems determined to stick with PISA despite recent concerns – published in the Times Educational Supplement in July of this year – about the global league table.

Mr Gove’s Department reiterated its support for PISA in a recently-aired Radio 4 programme entitled “PISA – Global Education Tables Tested.” That programme illustrated the dangers inherent in critiquing PISA in exclusively statistical terms. Statistical modellers have made life too easy for PISA because they simply accept the PISA interpretation of the construct “ability.” It is only when the focus moves to measurement that the profound difficulties inherent in PISA come to the fore with greatest clarity.

Niels Bohr is ranked with Newton and Einstein as one of greatest physicists of all time. The father of atomic physics taught that “unambiguous communication” is the hallmark of measurement in quantum physics. Importantly, Bohr traced measurement in quantum mechanics and measurement in psychology to a common source, which he referred to as “subject/object holism.”

The physicist cannot have direct experience of the atom, just as the teacher cannot have direct experience of the child’s mind. The microworld manifests itself in the measuring instruments of the physicist just as mind is expressed in the child’s responses to test items. Both the physicist and the psychologist are forced to describe what is beyond direct experience using the language of everyday experience. Bohr demonstrated that measurement in quantum physics and in psychology share a common inescapable constraint, namely, one cannot communicate unambiguously about measurement in either realm without factoring in the measuring instrument. In Heisenberg’s words: “what we observe is not nature in itself but nature exposed to our form of questioning.”

The lesson we learn from Bohr is that in all psychological measurement, the entity measured cannot be divorced from the measuring instrument. When this central tenet of measurement (in quantum physics or in psychology/education) is broken, nonsense always ensues. The so-called Rasch model, which produces the PISA ranks, offends against this central measurement principle and therefore the ranks it generates are meaningless. According to Bohr, the entity measured and the measuring instrument cannot be meaningfully separated. According to PISA, they are entirely independent. Who are we to believe, Niels Bohr or Andreas Schleicher?

The following simple illustration will help make Bohr’s point. Suppose Einstein and a 16 year-old pupil both produce a perfect score on a GCSE mathematics paper. Surely to claim that the pupil has the same mathematical ability as Einstein is to communicate ambiguously? However, unambiguous communication can be restored if we simply take account of the measuring instrument and say, “Einstein and the pupil have the same mathematical ability relative to this particular GCSE paper.” Mathematical ability, indeed any ability, is not an intrinsic property of the individual; rather, it’s a joint property of the individual and the measuring instrument.

In short, ability isn’t a property of the person being measured; it’s a property of the interaction of the person with the measuring instrument. One is concerned with the between rather than the within. It’s hard to imagine a more stark contrast between Bohr’s teachings and the PISA approach to measurement. Critiques of PISA by statistical modellers, however, have missed this profound conceptual error entirely.

My bookshelves are groaning with books concerned with the wide-ranging debates around the notion of intelligence. All of these debates dissolve away when one eschews the twin notions that intelligence is either a property of the person or is an ensemble property, for the simple definition that intelligence is a property of the interaction between person and intelligence test. To say “John has an IQ of 104” is to communicate ambiguously. An ocean of ink has been spilt because intelligence researchers have missed the simple truth that intelligence is not something we have.

In closing, it is only when the PISA critique shifts from statistical modelling to measurement, the profound nature of PISA’s error becomes clear. PISA produces nonsense because it misconstrues entirely the nature of ability. I trust this essay will be a comfort to those who had the courage to remove India from PISA, and hope it will prompt a similar decision from Michael Gove.

Dr Hugh Morrison