Why the Knewton Platform can make no claim to scientific measurement principles
Dr Hugh Morrison (email: firstname.lastname@example.org)
This article is a critique of the scientific claims made for the Knewton platform. Knewton’s designers portray the platform as drawing upon a science of adaptive learning to support a high quality personalised learning experience. To paraphrase, Knewton claims that its scientific rigour allows it to adapt to the learner, thereby providing the student with a personalised syllabus that enables him or her to learn every single concept he or she encounters. The scientific model of reality which informs Knewton is the Newtonian model.
“Knewton adaptivity is distinguished by its scientific rigor and tremendous scope”
Particularly damaging for the Knewton’s claims to scientific rigour is the fact that the psychological/educational attribute “learn” is governed by a “first-person/third-person asymmetry” which the Newtonian model simply cannot represent (Suter, 1989, pp. 152-153). Newtonian attributes are first-person/third-person symmetric. One of the twentieth century’s greatest physicists, Niels Bohr, regarded Newtonian models as entirely inappropriate for predicates such as “learn.” Knewton’s choice of scientific model was condemned almost 60 years ago by one of the towering figures of American physics, Robert Oppenheimer. Oppenheimer (1955, p. 134) cautions:
“It seems to me that the worst of all possible misunderstandings would be that psychology be influenced to model itself after a physics which is not there anymore, which has been quite outdated.”
Stapp (1993, p. 192) bemoans the fact that “while psychology has been moving towards the mechanical concepts of nineteenth-century physics, physics itself has moved in just the opposite direction.”
Opponents of the Knewton platform seem to be driven by a deep-rooted sense that there is something unsettling about this project. Surely learning is a human activity and “data science” cannot replace the teacher and the school? The central aim of this article is to demonstrate just how much the Knewton claim to scientific rigour is damaged by its exclusion of the all-important human dimension.
Item Response Theory and ambiguous communication
The ideas of the great physicist and Nobel Laureate, Niels Bohr, feature prominently in this paper. To give a brief sense of his standing as a scientist, on news of his death, the following words appeared in the editorial of the New York Times:
“With the passing of Niels Bohr the world has lost not only one of the greatest scientists of this century but also one of the intellectual giants of all time.”
Bohr defined the hallmark of science to be “unambiguous communication.” In quantum theory one cannot meaningfully separate what is measured from the measurement instrument without communicating ambiguously. The following simple illustration will help make Bohr’s point. Let’s return to the twentieth century when Einstein was alive. Suppose that Einstein and a student at an American high school both produced a perfect score on a high school mathematics examination paper. Surely to claim that the student had the same mathematical ability as Einstein would be to communicate ambiguously. The student has nothing to match Einstein’s contributions to special and general relativity, to say nothing of quantum theory. However, unambiguous communication can be restored if we simply take account of the measuring instrument and say, “Einstein and the student have the same mathematical ability relative to this particular examination paper.” Mathematical ability, indeed any ability, is not an intrinsic property of the individual; rather, it’s a joint property of the individual and the measuring instrument.
In short, ability isn’t a property of the person being measured; it’s a property of the interaction of the person with the measuring instrument. In all of Bohr’s writings one finds repeated reference to a profound conceptual equivalence between measurement in quantum theory and measurement in psychology. Bohr referred to this as “subject/object holism.” If this is accepted then (as illustrated in the case of Einstein and the high school student) one cannot meaningfully divorce what is measured from the measuring instrument in psychology.
But a central tenet of Item Response Theory is that the measuring instrument and the measurement outcome must be viewed as entirely independent. This puts Item Response Theory at odds with Bohr’s teachings. Item Response Theory features prominently in the Knewton Platform, in Computer Adaptive Testing, and underpins the PISA league tables.
One cannot meaningfully divorce what is measured from the measuring instrument in psychology
Why Knewton cannot lay claim to a science of learning
Teaching by pointing is a commonplace classroom device. For example, young students are taught the general concept “colour” by observing the colour of objects pointed out by their teacher. In order that the students grasp the concept “colour,” the teacher might define a range of individual colours – green, red, blue and so on – before later revealing that these are all instances of the general concept “colour.” In this case learning might progress from particular concepts such as “red,” “green” and so on, to the more general concept of colour.
In teaching the concept “green,” the teacher might point to a sample square patch of green cloth and utter the word “green.” This gives rise to a “learning paradox” that dates back to Plato: how is the student to know that the teacher wants him or her to attend to the colour of the patch? Couldn’t the teacher, by pointing at the square patch of green cloth, be attempting to teach the concept “square” or “patch” or “cloth”? Clearly the teacher wants the student to attend to the colour of the square patch of cloth. But the student hasn’t yet “got” the general concept “colour.” It seems that before the student can be taught the sub-concept “green,” he or she must already have the wider concept “colour.”
The learning paradox now becomes clear. If the student does not have the concept “colour” he or she can’t learn the sub-concept “green” because the student can’t grasp that the teacher intends him or her to attend to the colour of the patch rather than its geometric shape or texture. On the other hand, if the student already has the concept “colour” he or she can’t be said to learn the sub-concept green from the teacher’s pointing because the student already has grasped the concept “green,” since this colour is a sub-concept of the wider concept “colour.” It seems that no explanation can be offered for how students learn the simple concept “green.” But we all know from experience that students do learn individual colours by pointing.
Learning is a myth
I feel sure some readers will protest as follows: “You’ve got this wrong – when I taught my child the colour green, I pointed to a green door, a green toy, a green car, a green fence etc. etc.” I’m afraid the paradox still applies with full force. Kripke (1982), for example, devotes the greater part of his celebrated text on rule-following to the case of a student taught to add using hundreds of examples. When a concept such as addition is considered to be attributable to the student, one can only explain the student’s ability to give correct answers to a single novel addition problem by attributing to the student an infinite “look-up list” of all possible addition problems complete with answers!
This forces one to conclude that if concepts are treated as the properties of individuals (as in the Knewton platform), then one has to accept that learning and education are phantasms: “the learning paradox may be viewed as shaking the foundations of educational thought by demonstrating that the supposed role of learning (and hence education) is an illusion” (Bereiter, 1985, p. 221). The central message of the Knewton science of learning would seem to be that learning itself is a myth.
Jose Ferreira CEO Knewton
One can escape the learning paradox entirely by adopting the reasoning of Niels Bohr and Ludwig Wittgenstein. According to their approach – which rejects the idea that concepts can be ascribed to individuals – the concept “fraction,” for example, is a property of interactions between the individual and the established mathematical practice of manipulating fractions. Human practices play a pivotal role in this escape from paradox and the arguments generalise to all concepts.
Avoiding the learning paradox
In classes all over the world, colour concepts are taught successfully by pointing and students master addition through worked examples, and classroom practice. Teachers needn’t worry about the learning paradox, nor adjust their teaching techniques to take account of it. Indeed, I suspect that the vast majority of teachers wouldn’t even recognise a version of the learning paradox in Donald Rumsfeld’s words: “There are known knowns, … .” Why is this? It is because teachers in everyday classrooms have access to an ingredient missing from Knewton’s science of adaptive learning, and that ingredient is human nature.
Wittgenstein‘s point is only that, when such lessons do succeed, as they often do in real life, they draw on a resource that is presupposed by the verbal interpretations and definitions but not mentioned in them. That extra resource, to put it at first quite generally and vaguely, is human nature. (Pears, 2006, p. 19)
How can human nature rescue the situation? The distinguished American physicist David Mermin sets out Bohr’s case:
What does it mean for a property to be real? When you study an object how can you be sure you are learning something about the object itself, and not merely discovering some irrelevant feature of the instrument you used in your study? This is a question that has plagued generations of psychologists. When you measure IQ are you learning something about an inherent quality of a person called “intelligence,” or are you merely acquiring information about how a person responds to something you have fancifully called an IQ Test? Until the advent of the quantum theory in 1925 physicists were above such concerns. But since then, with the discovery that experiments at the atomic level necessarily disturb the object they measure, precisely such reservations have been built into the foundations of physics. (Mermin, 1993, p. 1)
Einstein & Bohr
This passage derives from a famous debate in the history of physics between the two greatest physicists of the twentieth-century: Bohr and Einstein. The debate centred on a fundamental question, namely, “what is it for something to have a property?” The outcome favoured Bohr’s cautious position that the result of a quantum measurement is best thought of as the property of an interaction between the measuring tool and the entity mesured. Bohr believed that this was also true for measurement in psychology. Wittgenstein’s philosophy arrives at precisely the same conclusion. Bohr taught that it was wrong to think that the task of physics was to find out how nature is by which he meant that quantum physics cannot be tasked with investigating the intrinsic properties of electrons, photons, and so on. The cautious position to take is that we must settle for the interaction between the electron and the instrument designed by the experimental physicist. Quantum theory teaches that that which is measured, and the measuring tool form an “indivisible whole.” Heisenberg (1958, p. 58) writes:
This again emphasizes a subjective element in the description of atomic events, since the measuring tool has been constructed by the observer, and we have to remember that what we observe is not nature in itself but nature exposed to our method of questioning. Our scientific work in physics consists in asking questions about nature in the language that we possess and trying to get an answer from experiment by the means at our disposal. In this way quantum theory reminds us, as Bohr has put it, of the old wisdom that when searching for harmony in life one must never forget that in the drama of existence we are ourselves both players and spectators.
Returning to our analysis of the Knewton platform, consider a student who has demonstrated mastery of the concept “multiplication,” for example. One gets the clear impression from the information made available by Knewton that, for the Knewton scientists, the measuring tool is the multiplication question together with its associated scoring rubric. This is only true if one subscribes to a quasi-religious Platonism and this approach isn’t available to the Knewton scientist since it rules out the possibility of learning new things. The calculation “20 times 20 = 400” is only true against a background mathematical practice of multiplication. Without this human background, the notions of “right” and “wrong” in mathematics lose their meaning. According to Bohr and Wittgenstein, it is incorrect to treat mastery of the multiplication concept as a property of the individual. Rather, one should define concept mastery as a joint property of the student and the mathematical practice.
And Wittgenstein’s contention is precisely that, with the demise of Platonism, there can be such a thing as adding correctly – such a thing as a determinate requirement imposed by the rules of addition – only within a framework of extensive institutional activity and agreement in the judgements which participation in those institutions involves us in making. The very existence of our concepts depends on such activity. (Wright, 2001, pp. 155-156).
Recall that the learning paradox arose when concepts like “green” and “colour” were ascribed to the individual. Here the paradox can be avoided because a concept is now the property of an interaction. In short, one escapes the learning paradox by treating what is measured and the measuring tool as an indivisible whole, the all-important measuring tool being founded in human customs and practices.
While there can be little doubt that Knewton has considerable value as another learning resource available to teachers and parents, its claim to be a “science” of adaptive learning has no validity. The Knewton software can lay no claim whatever to function according to a science of learning.
Bereiter, C. (1985). Towards a solution of the learning paradox. Review of Educational Research, 55(2), 210-226.
Heisenberg, W. (1958). Physics and philosophy. New York: Harper & Row.
Kripke, S.A. (1982). Wittgenstein on rules and private language. Oxford: Blackwell.
Mermin, N.D. (1993). Lecture given at the British Association Annual Science Festival. Keele: British Association for the Advancement of Science.
Oppenheimer, R. (1955, September 4). Analogy in science. Paper presented at the 63rd Annual Meeting of the American Psychological Association, San Francisco, CA.
Pears, D. (2006). Paradox and platitude in Wittgenstein’s philosophy. Oxford: Oxford University Press.
Stapp, H.P. (1993). Mind, matter and quantum mechanics. Berlin: Springer-Verlag.
Suter, R. (1989). Interpreting Wittgenstein: A cloud of philosophy, a drop of grammar. Philadelphia: Temple University Press.
Wright, C. (2001). Rails to infinity. Cambridge, MA: Harvard University Press.