Why Peter Tymms’ grasp of the fundamentals of measurement in psychology/education disqualifies him from any role in determining the future of transfer testing in Northern Ireland.
Dr Hugh Morrison
The case that Professor Tymms misunderstands the nature of measurement in psychology and education
Professor Peter Tymms is a long-time proponent of the central role that latent variables play in modern psychometrics. The Item Response model advanced by Georg Rasch has an important place in his research. I will argue in what follows that those who advance Item Response Theory approaches in general, and Rasch modelling in particular, have failed to understand the true complexity of the central predicate “ability.” Wittgenstein stressed that ability is something potential, a capacity rather than a state. Individuals are carriers of potentiality and not states. Psychology is concerned interactions and not the intrinsic properties of the relata involved; relations have definite properties while the relata themselves are indefinite. Peter Tymms has failed to appreciate the indeterminacy of the mental.
Measurement in psychology/education is never a process of “checking up” on what is already in the mind/brain of the individual. Rather, unlike measurement in Newtonian physics, the act of measuring transforms a potentiality to a definite state. Measurement in psychology and education should not be concerned with what ability is, but must settle for what can be said about ability. Michell is right to claim that psychometrics is “pathological science,” and that measurement in psychology is “at best speculation and, at worst, a pretence at science.” Trendler’s (2011) claim that measurement theorists should abandon all attempts to repair psychometrics is surely justified. All proponents of Item Response Theory, including Professor Tymms, subscribe to the “reflective model” in which variation in the latent variables is viewed as prior to variation in the manifest variables. Alas, the reverse is true.
Borsboom, Mellenbergh & van Heerden (2003, p. 217), writing in one of psychology’s most respected journals, highlight the incoherence of this entire approach to measurement: “It will be felt that there are certain tensions in this article. We have not tried to cover these up, because we think they are indicative of some fundamental problems in psychological measurement and require clear articulation. … And although the boxes, circles, and arrows in the graphical representation of the model suggest that the model is dynamic and applies to the individual, on closer scrutiny no such dynamics are to be found. Indeed, this has been pinpointed as one of the major problems of mathematical psychology by Luce (1997): Our theories are formulated in a within-subjects sense, but the models we apply are often based solely on between-subjects comparisons.”
Item Response Theory omits entirely the human practices (reading, arithmetic, and so on) into which the child is enculturated by teachers and parents. This is the all-important “environment” in which the child participates, an environment which Item Response Theory is powerless to represent. Item Response Theorists posit “abilities” hidden in the mind/brain which are the source of the child’s test responses. However, the Nobel laureate Herbert Simon dismissed such reasoning: “Human rational behaviour is shaped by a scissors whose blades are the structure of task environments and the computational capabilities of the actor.” The scissor metaphor is a reference to Alfred Marshall’s puzzlement over which scissor blade actually cuts a piece of cloth – the top blade or the bottom. The lesson for psychometrics is that omitting the environment of academic practices in which the child participates will produce nonsense.
One can find the source of Item Response Theory’s difficulty in Niels Bohr’s 1949 paper entitled Discussion with Einstein on Epistemological Problems in Atomic Physics. Few scientists have made a greater contribution to the study of measurement than the Nobel laureate and founding father of quantum theory, Niels Bohr. Given Bohr’s preoccupation what the scientist can say about aspects of reality which are not visible (electrons, photons, and so on), one can understand his constant references to measurement in psychology; “ability” cannot be seen directly, rather, like the microentities that manifest as tracks in particle accelerators, ability manifests in the individual’s responses to test items. Assessment is concerned with “measuring” something which the measurer cannot experience directly, namely, the ability of the examinee.
Quantum theory and psychology have not shown the same willingness to acknowledge the limitations of measurement in their respective disciplines. While physics has made no attempt to disguise its “measurement problem” (it is acknowledged in every undergraduate textbook), Michell (1997) has accused psychometricians of simply closing down all debate on measurement and suffering from a “methodological thought disorder.” Michell’s concerns about the reluctance of psychometricians to engage in debate about the fundamentals of measurement, when set alongside the clear acknowledgement of a measurement problem in physics, bring to mind the words of the French moralist Joseph Joubert: “It is better to debate a question without settling it than to settle it without debating it.”
Item Response Theory relies on a simple inner/outer picture for its models to function. The inner (a realm of timeless, unobserved latent variables, or abilities) is treated as independent of the outer (here examinees write or speak responses at moments in time). This is often referred to as a “reservoir” model in which timeless (hidden) abilities are treated as the source of the individual’s (public) responses given at specific moments in time.
As early as 1929 Bohr rejected this simplistic thinking in strikingly general terms: “Strictly speaking, the conscious analysis of any concept stands in a relation of exclusion to its immediate application. The necessity of taking recourse to a complementary … mode of description is perhaps most familiar to us from psychological problems.” Now what did Bohr mean by these words? Consider, for example, the concept “quadratic.” It is tempting to adopt a reservoir approach and trace a pupil’s ability to apply that concept in accord with established mathematical practice to his or her having the formula in mind. The guidance offered by the formula in mind (Bohr’s reference to “conscious analysis”) accounts for the successful “application,” for example, to the solution of specific items on an algebra test.
However, this temptingly simplistic model in which the formula is in the unobserved mental realm and written or spoken applications of the concept “quadratic” take place in the observed public realm, contains a fundamental flaw; the two realms cannot be meaningfully linked up. The “inner” formula (in one realm) gets its guidance properties from human practices (in the other realm). A formula as a thing-in-itself cannot guide; one has to be trained in the established practice of using the formula before it has guidance properties. In school mathematics examinations around the world, pupils are routinely issued with a page of formulae relevant to the examination. Alas, it is the experience of mathematics teachers everywhere that simply having access to the formula as a thing-in-itself offers little or no guidance to the inadequately trained pupil. The formula located in one realm cannot connect with the applications in the other.
Wittgenstein teaches that no formula, rule, principle, etc. in itself can ever determine a course of action. The timeless mathematical formula in isolation cannot generate all the complexities of a practice (something which evolves in time); rather, as Michael Oakeshott puts it, a formula is a mere “abridgement” of the practice – the practice is primary, with the formula, rule, precept etc. deriving its “life” from the practice.
Returning to Bohr’s writing, it is instructive to explain his use of the word “complementarity” in respect of psychology and to interpret the meaning of the words: “stands in a relation of exclusion.” Complementarity is the most important concept Bohr bequeathed to physics. It involves a combination of two mutually exclusive facets. In order to see its relevance to the validity of IRT modelling, let’s return to the two distinct realms.
We think of the answers to a quadratic equation (of course, a typical school-level quadratic equation has two distinct answers) as being right or wrong. In the realm of application this is indeed the case; when the examinee is measured, his or her response is pronounced right or wrong dependent upon its relation to established mathematical practice. However, in the unobserved realm, populated by rules, formulae and precepts (as things-in-themselves), any answer to a quadratic equation is simultaneously right and wrong!
A formula as a thing-in-itself cannot separate what accords with it from what conflicts with it, because there will always exist an interpretation of the formula for which a particular answer is correct, and another interpretation for which the same answer can be shown to conflict with the formula. Divorced from human practices, the distinction between right and wrong collapses. (This is a direct consequence of Wittgenstein celebrated “private language” argument.) This explains Bohr’s reference to a “relation of exclusion.” In simplistic terms, the unobserved realm, in which answers are compared with the formula for solving quadratics, responses are right-and-wrong, while in the observed realm, where answers are compared with the established practice, responses are right-or-wrong.
On this reading, ability has two mutually exclusive facets which cannot meaningfully be separated. The distinguished Wittgenstein scholar, Peter Hacker (1997, p. 250), captures this situation as follows: “grasping an explanation of meaning and knowing how to use the word explained are not two independent abilities but two facets of one and the same ability.” Ability, construed according to Bohr’s complementarity, is indefinite when unobserved and definite when observed. Moreover, this definite measure is not an intrinsic property of the examinee, but a property of the examinee’s interaction with the measuring tool. According to complementarity, the “inner” and the outer are not two separate localities which somehow connect. As Herbert A. Simon realized, one cannot dispense with either of Hacker’s facets and hope to construe ability correctly; both are vital to the predicate “ability.” Whitaker’s (1996, p. 184) definition of complementarity captures this situation: “mutual exclusiveness and joint completion.”
Measurement of ability is not a matter of passively checking up on what already exists – a central tenet of Item Response Theory. Bohr teaches that the measurer effects a radical change from indefinite to definite. Pace Item Response Theory, measurers, in effect, participate in what is measured. No item response model can accommodate the “jump” from indefinite to definite occasioned by the measurement process. All IRT models mistakenly treat unmeasured ability as identical to measured ability. What scientific evidence could possibly be adduced in support of that claim? No Item Response model can represent ability’s two facets because all such models report ability as a single real number, construed as an intrinsic property of the measured individual.
Finally, in order to highlight the incoherence of the type of measurement model advocated by Peter Tymms, it is instructive to consider a thought experiment in which a primary school child responds to the addition problem: “68 + 57 = ?” In the appendix below the erroneous thinking of the psychometrician is adopted in that the child’s ability is considered to be a mental state which is the source of his or her response. It is demonstrated that all of the facts about the child (his or her complete history of responses to addition problems and complete information about the contents of his or her mind) are in keeping with the answer “68 + 57 = 125.” Unfortunately, all of the facts are also in keeping with the answer “68 + 57 = p” where p is ANY number; someone with complete information has to conclude that the child is right and wrong at the same time.
Appendix
Setting the Scene
Consider the simplest of measurement situations encountered in psychology and education. How can one establish if a student in the early years of formal schooling understands how to use the “+” sign? The student whose grasp of the “+” sign is being measured has been taught to add, but has not yet encountered the problem “68 + 57 = ?” This problem has been selected at random but the argument generalises to any rule-governed activity (Kripke, 1982). The measurement situation is broken into two phases: the phase immediately before the student answers and the phase during which the answer is spoken or written.
This offers two perspectives on the student’s understanding of the “+” sign. The idea that understanding how to use the “+” sign is an “inner” mental state, activity or process has enormous appeal. The temptation to reason that the student’s first-person perspective on his or her understanding is superior to the measurer’s third-person perspective is almost irresistible, since the measurer has to settle for mere behaviour. It is difficult to escape the impression that the student has privileged access to his or her grasp of the meaning of “+,” because to mean is surely to have something in mind? (Putnam, 1988). On the other hand, the measurer must settle for the mere manifestations of that understanding. The student seems to have first-person direct access to the thing-in-itself, namely, his or her understanding of “+” while the third-person perspective is associated with indirect access. The third-person perspective involves observation of the student exercising his or her understanding rather than the understanding itself; it would seem that the student alone can “observe” understanding because it is a mental process. This enticing simple Cartesian picture of the “inner world” is clearly compelling.
This idealised measurement situation will be used to argue that, before the student answers the problem “68 + 57 = ?” (immediately prior to measurement) the totality of facts about the student are in keeping with the student intending to give the right answer and with the intention to give one of an indefinite number of wrong answers. It is established that the state “understands the ‘+’ sign” and the state “doesn’t understand the ‘+’ sign” both can be simultaneously ascribed to the unmeasured student. In short, it is meaningless to assign a definite grasp of the “+” sign to an unmeasured individual.
In quantum theory, unmeasured quantum entities are characterised by superpositions which are portrayed as being “here” and “there” simultaneously. For unmeasured quantum entities, the notion of a definite location is unintelligible. However, when the quantum entity is measured it assumes a definite position and is characterized as either here or there. Consider the measurement of an individual’s ability to respond to the addition problem: “68 + 57 = ?” The case will be made that when the psychologist focuses on the unmeasured ability of the individual, all the facts about that individual can be shown to be in keeping with the individual being both “right” and “wrong” (with respect to the addition problem) at the same time. It will be demonstrated that, immediately prior to the act of measurement, an individual’s understanding of the “+” sign is entirely indeterminate, with the categories “right” and “wrong” being applicable simultaneously.
It will be argued that someone with complete information about the student’s past achievements in addition, together with complete information about his or her mental states, would find it impossible to use this information to predict the student’s answer to any simple addition problem in an infinity of cases. It will be demonstrated below that before a measurement is made – for example, before a student says or writes the answer to the question “68 + 57 = ?” – at that moment, all of the known facts about the student are in keeping with the correct answer “125” and an incorrect answer – “5,” – for example. A rule as a thing-in-itself can never determine an action. The student’s mathematical ability with respect to the question “68 + 57 = ?” is completely indeterminate prior to the statement of the answer.
At the moment prior to answering the question “68 + 57 = ?” all of the facts about the student accord with a correct response and an infinity of incorrect responses. Before the act of measurement, if one restricts oneself to the totality of facts about the individual (inner and outer), then the notion of accord or conflict with a mathematical rule breaks down entirely. Clearly, in conditions where responses can be deemed simultaneously right and wrong, the very notion of correctness has become unintelligible. While the student is characterised as both right and wrong with respect to the question
“68 + 57 = ?” immediately before he or she responds, at the instant of responding the student is deemed correct if the answer is 125, and incorrect if the student answers 5. Immediately before answering the student is right and wrong. The moment the answer is articulated the student is right or wrong. In short, measurement isn’t a matter of checking up an existing attribute (as in Newtonian physics); measurement effects radical change.
Having set the scene for what is to come, the case will now be made that it is meaningless to ascribe a definite ability to an unmeasured individual; the ascription of a definite ability is only meaningful in a measurement context. The idea that psychological measurement owes more to quantum measurement principles than to Newtonian mechanics depends on this case being made. This is achieved by calling on Wittgenstein’s later philosophy and the remainder of this section is given over entirely to this single task. Wittgenstein’s writings on first-person ascription of ability are essential to developing a measurement model with the individual at its core.
No Rule or Formula can Determine its own Continuation
An idea that Anscombe (1985) traces back to Leibniz (1646-1716) is instructive for preparing the reader for Wittgenstein’s insights into the role rule-following plays in thinking about psychological measurement. Leibniz noticed that no formula or rule can fix its own continuation: any number can be regarded as the correct continuation of a rule on some interpretation. He pointed out that an indefinite number of rules are consistent with any finite segment of a series. Anscombe illustrates Leibniz’s thinking using the extension of a simple series such as ‘2, 4, 6, 8, …’
[A]lthough an intelligence tester may suppose that there is only one possible continuation to the sequence 2, 4, 6, 8, … , mathematical and philosophical sophisticates know that an indefinite number of rules (even rules stated in terms of mathematical functions as conventional as polynomials) are compatible with any such finite initial segment. So, if the tester urges me to respond, after, 2, 4, 6, 8, … , with the unique appropriate next number, the proper response is that no such unique number exists. … The intelligence tester has arbitrarily fixed on one answer as the correct one. (Anscombe, 1985, pp. 342-343)
Consider the series completion problem Anscombe (1985) proposes. The student is presented with the first four terms of an infinite series: 2, 4, 6, 8 … He or she is then required to “go on in the same way” by the teacher. An infinite number of formulations will generate the four numbers 2, 4, 6 and 8 but differ on the fifth term (and all terms thereafter). For example, the formula:
Un = 2n – (1/24)(n – 1)(n – 2)(n – 3)(n – 4)
generates: 2, 4, 6, 8, 9, …
while
Un = 2n + 45(n – 1)(n – 2)(n – 3)(n – 4)
generates 2, 4, 6, 8, 1090, …
and finally
Un = 2n – 3(n – 1)(n – 2)(n – 3)(n – 4)
generates 2, 4, 6, 8, -62, … .
In summary, an indefinite number of different continuations can be shown to accord with any finite segment of an arithmetical series. By careful selection, any number can be offered for the fifth term of the series. One’s immediate reaction to the final series given above is that, in writing -62, the student has made a mistake. In writing 2, 4, 6, 8 the student is following the correct rule but it appears that in writing -62 he or she has erroneously switched to a new rule. But it is also possible that the student acted consistently throughout, always applying the same formula, namely,
Un = 2n – 3(n – 1)(n – 2)(n – 3)(n – 4)
to generate all five terms. This (albeit highly unusual) student could rightly claim to be “going on in the same way” when he or she wrote down -62 as the fifth term.
The student’s claim that he or she was simply continuing the rule exhibited by the first four terms is completely defensible since there are an infinite number of rules which begin ‘2, 4, 6, 8’ but diverge on the next term and all terms thereafter. It can be claimed that the student did continue in the same way but the student’s way was at odds with the teacher’s intention when the teacher instructed the student to “go on in the same way.” Unfortunately, “Finite behaviour cannot constrain its interpretation to within uniqueness” (Wright, 2001, p. 98), so what makes the student’s continuation wrong and the teacher’s right?
Wittgenstein’s writings on rule-following do not, for a moment, imagine that real children in real classrooms extend this series of four even numbers as “2, 4, 6, 8, 1090, …” or “2, 4, 6, 8, -62,” for example.
It is a conspicuous feature of these case-histories that the misunderstandings are often widely improbable, and we may wonder why this is so. Evidently, the reason cannot be that Wittgenstein believed that such extreme misunderstandings are at all likely or that a teacher would need to guard against them in real life. So what is the explanation of his preoccupation with improbable misunderstandings? Wittgenstein’s point is not that such misunderstandings are probable, but only that they are possible. They are possible because, if the lesson only proceeds by examples, there will always be many different specifications of the meaning of the word that are satisfied by any finite sequence of examples, and so the student can always pick a specification that was not intended by the teacher. However, if the lesson has been well designed with carefully chosen examples, there will only be one natural way of interpreting them – or perhaps there will be minor variations, to be excluded by further examples. If, on the other hand, the teacher tries to close the gap by offering a definition of a problematic word, the words used in the definition will present the same problem again. (Pears, 2006, p. 18)
“The idea here is that instructions for following a rule underdetermine the correct way to follow the rule … if we consider instructions and explanations as involving the provision of a finite number of examples then there are indefinitely many compatible functions or ways of going on from those examples” (Panjvani, 2008, p. 307). Schroeder (2006, p. 189) concludes: “So, any rule, even the most explicit one, can be misunderstood; and in endless ways too: whichever way the student continues the series, his writing can always be regarded as in accordance with the rule – on a suitable interpretation.”
This problem extends beyond mathematics to all rule following. Bloor (1997, p. 10) stresses that: “This does not just apply to number sequences. Teaching someone the word ‘red’ is, in a sense, teaching them the rule for using the word. This too involves moving from a finite number of examples to an open-ended, indefinitely large range of future applications”. The problems associated with infinite rules also apply to rules with a finite number of applications. Kripke (1982, p. 7) comments: “Following Wittgenstein, I will develop the problem initially with respect to a mathematical example, though the relevant sceptical paradox applies to all meaningful uses of language.” Finally, McGinn (1997, p. 77) notes: “Nor is this problem restricted to the mathematical case. For any word in my language, we can come up with alternative interpretations of what I mean by it that are compatible with both my past usage and any explicit instruction that I might have given myself.”
Every teacher is aware that students do succeed in moving from a finite set of examples to apply the rule in new cases and so it is tempting to conclude that successful understanding somehow results in the student having in mind a rule which accords with the examples but somehow transcends them. Furthermore, it would seem that the student’s ability to apply the rule beyond the finite example set is explained by positing that the student, when confronted with novel applications, is guided by the rule that he or she has “in mind.” Bloor (1997, p. 11) summarises this seductively simple model of how rules are grasped:
It is tempting to suppose that when a teacher is using examples to convey the meaning of a word, the teacher has something ‘in mind,’ and the finite number of examples are just a fragmentary substitute for what is really meant. If only the student could look directly into the mind of the teacher then how simple life would be: they too would have access to the state of understanding that is the source of the teacher’s ability to follow the rule. Of course, they can’t look into the teacher’s mind but, the argument goes, they will only have established an understanding when they can reach beyond all the examples. (Bloor, 1997, p. 12)
‘Understanding’ is a Vague Concept (Wittgenstein, 1983, VI, §13)
Consider the case of simple addition of two natural numbers. Almost everyone who has been to school for a few years is confident that they have understood the meaning of the sign “+.” They feel sure they understand how to use the rule for the “+” sign. This confidence exists despite the fact that no individual has carried out the infinitely long list of computations associated with the “+” sign. Given that individuals live for only a finite number of years and there are an infinite number of natural numbers x and y for which x + y can be computed, it follows that for any individual it will always be possible to identify values of x and y for which they have not yet computed x + y.
The student in the early-years of primary school, and just coming to terms with addition, may have computed x + y for all values of x and y less than 57, for example, so that the computation “68 + 57”, say, is not part of the student’s short computational history. The calculation “68 + 57” would be a novel computation for this student. The point to be emphasised is that for every individual it will always be possible to identify an addition problem that this individual has not previously encountered.
Kripke (1982) considers the case of a student who has not yet performed the computation “68 + 57.” The student has mastered additions with smaller arguments such as “23 + 34,” “19 + 27” and “50 + 51” but hasn’t carried out the calculation “68 + 57.” Kripke (1982) conjures up a “bizarre sceptic” who suggests to the student that, based on the examples the teacher used when instructing him or her to add and all the addition questions the student has completed to date, he or she should now give the answer “5” in response to the question “68 + 57 = ?” Kripke (1982, p. 8) has his bizarre sceptic pose a simple question:
This sceptic questions my certainty about my answer … Perhaps, he suggests, as I used the term ‘plus’ in the past, the answer I intended for ’68 + 57’ should have been ‘5’! Of course the sceptic’s suggestion is obviously insane. My initial response to such a suggestion might be that the challenger should go back to school and learn to add.
The sceptic’s case is based on the fact that the student (in keeping with the rest of humanity) has only ever carried out a finite number of computations. Needless to say, seeing off the claim that, given the student’s computation history, he or she should answer “5” to the question “68 + 57 = ?” the student will claim that the function he or she associated with “+” sign in all past computations requires that the answer “125” be given. But Kripke (1982, pp. 8-9) points out that the finite set of addition problems completed in the past can be variously interpreted:
But who is to say what function this was? In the past I gave myself only a finite number of examples instantiating this function. All, we have supposed, involved numbers smaller than 57. So perhaps in the past I used ‘plus’ and ‘+’ to denote a function I will call ‘quus’ and symbolise by ‘Å.’ It is defined by:
x Å y = x + y if x, y < 57
x Å y = 5 otherwise
Who is to say that this is not the function I previously meant by ‘+’?
The individual’s difficulty in seeing off the sceptic is that all of his or her past computations are for values of x and y less than 57 and for these values ‘+’ and ‘Å’ yield identical values. It is only for values of x and y greater than or equal to 57 that differences occur. It is only for numbers greater than or equal to 57 that addition and quaddition (someone using the quus function is said to be engaged in quaddition) give different results.
Kripke (1982) points out that the facts of interest to the sceptic are to be found in two distinct realms: the “outer” realm of past computations, and the “inner” realm of the mind. If all the facts from the student’s history of past computations are consistent with the function ‘plus’ and with the function ‘quus,’ then maybe an examination of the student’s mental history (facts about the contents of the student’s mind) will decide whether he or she should answer “125” or “5” in order to be consistent with his or her past history of computation. Kripke (1982) permits the individual responding to the sceptic’s challenge to have unlimited and infallible access to past computations (outer) and past mental states and processes (inner). Kripke (1982) frequently makes reference to what an omnipotent, omniscient, all-seeing God – who has access to every aspect of the individual’s computational history and to his or her thought processes – would see if He were to look into the student’s mind.
The evidence is not to be confined to that available to an external observer, who can observe my overt behaviour but not my internal mental state. It would be interesting if nothing in my external behaviour could show whether I meant plus or quus, but something about my inner state could. But the problem here is more radical. … whatever ‘looking into my mind’ may be, the sceptic asserts that even if God were to do it, he still could not determine that I meant addition by ‘plus.’ (Kripke, 1982, p. 14)
Kripke (1982) argues that all of the facts about the student – all the (outer) facts about the student’s computational history and all the (inner) facts about the student’s mental states and processes – are consistent both with that individual understanding the orthodox addition function and the contrived “quus” function by “+” sign.
The sceptic does not argue that our own limitations of access to the facts prevent us from knowing something hidden. He claims that an omniscient being, with access to all available facts, still would not find any fact that differentiates between the plus and the quus hypotheses. (Kripke, 1982, p. 39)
The temptation, of course, is to accept that while any finite series of computations can be made to accord with both ‘plus’ and ‘quus’ functions, an examination of the individual’s mind would turn up a fact or facts that would discriminate between the functions. It is instructive to illustrate how the sceptic refutes such arguments.
When individuals learn to add, the following notion has considerable appeal: if they grasp the meaning of the “+” sign then this understanding of the addition rule fixes unique responses to addition problems they will encounter in the future. The notion of understanding seems to function in “contractual” terms (McDowell, 1998, p. 221). Given the student’s past grasp of the “+” sign and history of successful problem-solving in respect of that symbol, he or she seems contracted to reply in accord with this understanding. In this model, to understand is to have something in mind which is the source of subsequent behaviour in that it contracts the individual who understands addition to reply “125” to the question “68 + 57 = ?”
But what makes the individual’s understanding with respect to the ‘+’ sign, a grasp of adding rather than quadding? Consider an individual who attaches the aberrant interpretation (quus) to his or her past computations. If the student’s understanding of the “+” sign contracts him or her to use it according to the quaddition rule then the individual should reply “5” when asked to compute “68 + 57.” In this case, to reply “125” is to fail to go on in the same way. But what fact about the student’s understanding could be produced to convince the sceptic that he or she is contracted to follow the quaddition rule rather than the addition rule? What fact about a student’s past grasp of the ‘+’ sign makes his or her present response of “5,” for example, right or wrong. The finite number of additions the individual has completed to date – all involving arguments less than 57 – are consistent with understanding the ‘+’ sign in terms of the addition function and the quaddition function.
For the sceptic can point out that I have only ever given myself a finite number of examples manifesting this function, which all involved numbers less than 57, and that this finite number of examples is compatible with my meaning any one of an infinite number of functions by ‘+.’ … If the sceptic is right, then there is no fact about my past intention, or about my past performance, that establishes, or constitutes, my meaning one function rather than another by ‘+.’ (McGinn, 1997, p. 76)
Having a Formula Before one’s Mind
The common-sense view that understanding is a process or activity which happens in the mind has enormous appeal. The notion that to understand is to have something in mind and that this understanding somehow fixes future behaviour in respect of how that understanding is exercised seems beyond challenge. At the same time, it also seems obvious that the future behaviour referred to is somehow inferior to understanding as a thing-in-itself; understanding is the real thing, whereas behaviour is merely a particular manifestation of that understanding. Understanding is construed as “inner” while behaviour is construed as outer. Few dispute the thesis that while the individual somehow has direct access to his or her understanding, the person measuring that understanding, for example, has to settle for indirect access in the form of the individual’s behaviour.
First-person access to understanding seems superior to mere third-person access. Furthermore, the inner and outer seem to be entirely independent realms; after all, someone who has understood addition has the clear sense that he or she will provide the correct answer to the question “2 + 2 = ?” in advance of writing or saying their answer. This person doesn’t have to wait until they’ve responded in order to confirm their understanding to themselves. Understanding (viewed as a property of the inner), in this instance, at least, seems quite independent of the subsequent behaviour in which that understanding is exercised.
And this seems obvious for, to be sure, other people must rely on my behaviour, on what I do and say, in order to discern what I am feeling or thinking. So, it seems that they know how things are with me indirectly. What they directly perceive is merely outward behaviour. But I have direct access to what is inner, to my own mind. I am conscious of how things are with me. The faculty whereby I have such direct access to mental states, events and processes is introspection – and it is because I can introspect that I can say how things are with me without observing what I do and say. (Bennett & Hacker, 2003, pp. 84-85)
Kripke (1982) considers the case of having a formula in mind which can be introspected. Could this provide the elusive fact that distinguishes the individual’s understanding of “+” as a grasp of the addition function rather than a grasp of the quaddition function? Could this be the “inner fact” that determines “125” as the answer the student should give to “68 + 57 = ?” in order to keep faith with the student’s understanding of “+”? This is an attractive option because one can conceive of a formula as something finite but which nevertheless has the capability of generating an infinity of responses. If God were to look into the individual’s mind and spotted, say, the quus function defined above, He could predict with certainty that the individual is contracted to reply “5” to the question “68 + 57 = ?” in order to keep faith with his or her understanding. One can also appreciate the popular appeal of this approach for one does feel that in solving quadratic equations, for example, one “calls to mind” the quadratic formula. One can almost “see” the formula in one’s mind’s eye when solving quadratic equations.
The case against the notion of a formula as the sought-after fact which distinguishes adding from quadding will now be set out. Alas, having a formula in mind will not satisfy the sceptic because a mental representation of the formula in itself cannot determine the individual’s response to any given problem. A student who has not been instructed in the solution of quadratic equations but who has merely memorised the formula for solving quadratic equations will not be able to use it to solve algebraic problems involving quadratic equations. Merely having the formula in mind is not enough to determine use. The student must be trained in the use of the formula; simply having access to the formula in itself – whether in mind or on paper – doesn’t fix the response the student makes when presented with a problem requiring the use of the quadratic formula.
It is the practice in school mathematics examinations around the world to issue a booklet of formulae to examinees. It is the near universal experience of mathematics teachers that students with inadequate training in the solution of quadratic equations may derive little value from having access to the relevant formula booklet. This is because the formulae in themselves don’t fix behaviour. If the formula in isolation fixed the student’s response, then every student issued with a formula booklet in a mathematics test would answer the quadratic question correctly. To underline the limitations of having formulae in mind, Wittgenstein explores the circumstances under which sign-posts offer guidance. Wittgenstein (1953, §85) states that in the Cartesian picture “A rule stands there like a sign-post.” In this statement he is asking the reader to reflect on the property of a wooden sign-post which enables it to serve as a guide to behaviour.
Considered in itself a sign-post is just a board or something similar, perhaps bearing an inscription, on a post. Something so described does not, as such, sort behaviour into correct and incorrect – behaviour that counts as following the sign-post and behaviour that does not” (McDowell, 1992, p. 41).
One is forced to conclude that, despite having all the facts (inner and outer) about the student, these alone can’t determine in advance that his or her response will be “125” or “5.” Causality has broken down. A defining principle of Newtonian mechanics is that if one has complete information about any system one can always predict what will happen next with certainty. Newtonian determinism fails in respect of elementary rule-following: it seems that there are matters which influence the student’s response which are beyond the totality of inner and outer facts.
A quantum physicist would feel entirely at home with Kripke’s (1982) way of expressing his interpretation of Wittgenstein’s philosophy: “If even God, who can see all the facts about the past (and into your mind), could not know that you meant addition then that doesn’t illustrate limitations on God’s knowledge. It shows that there is in this case no fact for him to know” (Ahmed, 2007, p. 102).
Also, since one can look back upon the individual’s present understanding of “+” from a vantage point in the future, it follows that there is no (inner or outer) fact about what he or she currently understands by the “+” sign. It follows that before this student is measured (before he or she responds to the question “68 + 57 = ?”) the totality of facts (inner and outer) are in keeping with the conclusion that the student’s state is a superposition of right and wrong. The student is in an indefinite state with respect to his or her grasp of “+” because all the facts from the two relevant provinces (the inner and outer) are in keeping with the answers “125” and “5.”
Before continuing, it is important to realise that Wittgenstein does not deny that when solving mathematical problems one often has the sense that the relevant formula is “before one’s mind.” Rather, he’s pointing out that this formula before one’s mind can’t be the source of one’s ability to solve the problems – it’s merely a by-product of one’s instruction in addition. It will become clear in the paragraphs below that an introspected formula cannot fix how one solves problems requiring the use of the formula.
Wittgenstein is not here denying that there are characteristic experiential accompaniments to meaning and understanding – for images and the like do sometimes come before our minds when we utter or understand words – but he is denying that such experiential phenomena could constitute understanding. Experiences are at most a symptom or sign of understanding; they are not the understanding itself. The mistake of the traditional empiricist conception of meaning was thus to take as constitutive what is in reality only symptomatic. (McGinn, 1984, p. 4)
It should be stressed, once again, that this argument generalises to all rule-following: “Of course, these problems apply throughout language and are not confined to mathematical examples, though it is with mathematical examples that they can be most smoothly brought out” (Kripke, 1982, p. 19). Ahmed (2007, p. 103) points out that “there is nothing special about ‘plus’ – if scepticism about ‘plus’ is irrefutable then so is scepticism about any word in any language.”
Having a Formula (and its Interpretation) Before one’s Mind
Kripke (1982) now searches for some mechanism in the student’s mind which will obviate the need to conclude that the unmeasured student is in an indefinite state. He reasons that the formula in mind is just like a sign post – it cannot in itself fix the student’s response. It is neither sufficient nor necessary to the student’s exercise of understanding. As a consequence, it might be argued that having a formula in mind is of little value unless one is equipped to interpret it. McDowell (1992, p. 41) makes the case using Wittgenstein’s sign-post metaphor:
What does sort behaviour into what counts as following the sign-post and what does not is not an inscribed board affixed to a post, considered in itself, but such an object under a certain interpretation – such an object interpreted as a sign-post pointing the way to a certain destination. (McDowell, 1992, p. 41)
Kripke (1982) is trying to explain how students follow the rule for the use of the ‘+’ sign given appropriate teaching and a finite number of illustrations of that rule. The idea that they are guided by a mental image of a rule proves unworkable. Could it be that one needs further evidence, namely, evidence that the student can interpret the rule correctly? It follows that there must be evidence in mind that the student has correctly interpreted the rule. However, this is of scant assistance, for the rule is capable of multiple interpretations. It seems, therefore, that one needs to have a rule in mind for selecting the correct interpretation. In order to explain the student’s ability to follow the rule for the use of the “+” sign, one must invoke a further rule – but this time the rule is in mind – for selecting the correct interpretation. One has now fallen into an infinite regress.
An argument centred on the right interpretation will not work for the student must then have access to the rule for selecting this correct interpretation and one has a circular argument, because it is rule-following one is seeking to explain in the first place. It instructive to remind the reader of the central idea here. Facts are being sought about an individual which would determine in advance what response he or she should make to a novel problem which requires the use of a rule. It has been decided that no outer facts fix what the individual does next because any response can be shown to accord with the rule exemplified by a finite set of illustrative examples offered by way of instruction. The search then switched to facts about the individual’s mind.
If God looked into the individual’s mind and spotted a representation of the “quus” rule identified earlier then, at first sight, it seems He could predict with certainty that if asked the question “68 + 57 = ?” the individual must answer “5.” The introduction of a finite entity in the mind (a mental representation of the formula) explaining the individual’s potentially infinite capacity for applying the rule is appealing until one notes that it is possible to have a formula in mind and yet not know how to apply it. It is clear that the formula in itself is neither sufficient nor necessary. It is “normatively inert” (McDowell, 1992, p. 42) because it cannot be used in isolation to pronounce the individual’s future response to the novel problem to be “125” or “5,” or any other number. As McDowell puts it, the formula just “stands there” in need of interpretation.
McDowell (1992, p. 42) points out that attempting to locate the sought-after facts in the individual’s mind is fraught with problems because this is “a region of reality populated by items that, considered in themselves, just ‘stand there’.’’ Mental representations have to be interpreted. McDowell argues that whatever attaching the correct interpretation to the formula might consist in, it is nevertheless an element of a region of reality (the mind) populated by items that just stand there like sign posts. It follows that the interpretation itself has to be interpreted, and so on, in an infinite regress.
Wright (2001, pp. 162-163) makes such a powerful and pithy argument demonstrating the futility of interpretations that it is quoted here in full:
Suppose I undergo some process of explanation – for instance, a substantial initial segment of some arithmetical series is written out for me – and as a result I come to have the right rule ‘in mind.’ How, when it comes to the crunch – at the nth place which lies beyond the demonstrated initial segment, and which I have previously never thought about – does having the rule ‘in mind’ help? Well, with such an example one tends to think of having the rule ‘in mind’ on the model of imagining a formula, or something of that sort. And so it is natural to respond by conceding that, strictly merely having the rule in mind is no help. For I can have a formula in mind without knowing what it means. So – the response continues – it is necessary in addition to interpret the rule. … An interpretation is of help to me, therefore, in my predicament at the nth place only if it is correct. … So how do I tell which interpretation is correct? Does that, for instance, call for a further rule – a rule for determining the correct interpretation of the original – and if so, why does it not raise the same difficulty again, thereby generating a regress?
Selecting the Simplest Rule
It may strike the reader that a criterion based on simplicity may distinguish understanding the “+” sign in terms of the addition rather the quaddition function. The quus function, with its differing approach for numbers less than 57, and those greater than or equal to 57, seems a particularly unwieldy function when compared with the simple plus function. Its mathematical symbolism would also be alien to any student in the primary phase of education. Could it be that the student simply selects a unique interpretation (the “correct” interpretation) from the infinity on offer by simply choosing the interpretation with the simplest associated function?
Chaitin (2007) has extended Gödel’s incompleteness theorem (1931) and Turing’s halting problem (1950) to develop Algorithmic Information Theory. He demonstrates that the search for the simplest rule (or most “elegant” rule in Chaitin’s parlance) which generates a sequence of numbers is equivalent to the search for the shortest computer program which can generate the sequence. Unfortunately, Chaitin (2007, pp. 120-121) confirms that for any finite sequence of numbers, the identification of the simplest interpretation also presents intractable problems.
Let’s say I have a particular calculation, a particular output, that I’m interested in, and that I have this nice, small computer program that calculates it, and think that it’s the smallest possible program, the most concise one that produces this output. Maybe a few friends of mine and I were trying to do it, and this was the best program that we came up with; nobody did any better. But how can you be sure? Well, the answer is that you can’t be sure. It turns out you can never be sure! You can never be sure that a computer program is what I like to call elegant, namely that it’s the most concise one that produces the output that it does. Never, ever!
This paper details only some of Kripke’s (1982) attempts to escape the conclusion that if one restricts oneself to the totality of facts about the individual – outer facts about past practice and inner facts about mental contents (the two sets of facts treated as separately analysable) – one cannot predict the response the individual will make to the simple addition question: “68 + 57 = ?” Since all the facts are in keeping with an infinity of answers, one correct and the rest incorrect, Kripke (1982, p. 17) is compelled to conclude that, in providing a response to this question, the rule-follower must be characterised as making “an unjustified stab in the dark.” The student has no criterion for preferring 125 over 5; all the facts are in keeping with the correct answer and any incorrect answer. The student is in an indeterminate state with respect to an understanding of the “+” sign.
So it seems that from a first-person perspective, individuals who have been taught to add using a finite number of examples, offer the first answer that comes into their heads when required to extend the addition rule to unseen computations. They have no criterion which guides their selection of 125 as the correct answer to the problem “68 + 57 = ?” There must be an error in the reasoning that produces such a counter-intuitive conclusion. According to the logic presented above a student who, having been taught to add via a series of examples and then instructed to “go on in the same way,” subsequently encounters the addition problem “68 + 57 = ?” and responds by writing “5,” can protest that he or she did go on in the same way; the student just didn’t go on in the same as the teacher who issued the instruction. There exists an interpretation which brings the answer “5” into accord with the teacher’s examples. Indeed, this is true of any answer the student offers. These two answers (125 and 5) and an infinity of other answers, are all in keeping with the totality of facts about the student.
Latent Variables Modelling Misrepresents Ability
The student’s interaction with the sceptic shows that the student has no criterion which can be used to differentiate a correct from an incorrect response. The very concepts of right and wrong don’t seem to apply here. This invites the obvious question: In respect of the problem “68 + 57 = ?” what makes “125” the correct answer and “5” the wrong answer? What makes the teacher right in thinking the student should answer “125” and the student wrong in answering “5”? This issue is resolved by bringing in the human practice of mathematics. When we enter the picture, psychological measurement is bound to lose some of its Newtonian objectivity, an objectivity that quantum theory teaches is unattainable.
Heisenberg (1958, pp. 55-56) stresses that the cost of the participant’s inclusion is reduced objectivity in scientific measurement: The “reference to ourselves” means that “our description is not completely objective.” Objectivity in the Newtonian sense is no longer the hallmark of science because it fails to account for the participative element of measurement. Beyond classical physics, measurement models with no place for human practices are of questionable scientific validity: “When we speak of the picture of nature in the exact science of our age, we do not mean a picture of nature so much as a picture of our relationships with nature. The old division of the world into objective processes in space and time and the mind in which these processes are mirrored … is no longer a suitable starting point for our understanding of modern science” (Heisenberg, 1962, pp. 28-29).
Wittgenstein’s writings make clear that, divorced from human practices, the descriptors “right” and “wrong” lose their meaning, even in disciplines like mathematics and logic. Quantum theoretical “weak objectivity” (d’Espagnat, 1983) has replaced the strong objectivity of Newtonian mechanics because Newtonian objectivity misrepresents the psychometrician’s task.
[A]lmost all of us, after sufficient training, respond with roughly the same procedures to concrete addition problems. We respond unhesitatingly to such problems as ’68 + 57,’ regarding our procedure as the only comprehensible one (see Wittgenstein, 1953, §§219, 231, 238), and we agree in the unhesitating responses we make. On Wittgenstein’s conception, such agreement is essential for our game of ascribing rules and concepts to each other (see Wittgenstein, 1953, §240). (Kripke, 1982, p. 96)
Wittgenstein (1975, p. 58) writes: “The only criterion for his multiplying 113 by 44 in a way analogous to the examples is his doing it in the way in which all of us, who have been trained in a certain way, would do it.” It follows that third-person ascriptions of the ability to add are based on the criteria afforded by the practice of mathematics, a practice into which the teacher has been enculturated. Wittgenstein notes that “Indefinitely many other ways of acting are possible: but we do not call them ‘following the rule’” (Malcolm, 1986, p. 155).
Criteria hover somewhere between deductive and inductive grounds (Grayling, 1977) and their nature can be traced back to the introduction of the participating psychologist. For example, there is an intrinsic “vagueness,” to borrow Wittgenstein’s term, in the accepted number of particular additions one ought to compute correctly before having the ability to add ascribed to one. There is no fixed number of even numbers a student should write down before being regarded as someone who “understands” or “has mastered” or “has grasped the meaning of” the even numbers. This vagueness is a constitutive property of psychological predicates; it isn’t a shortcoming.
Suppose we are teaching a student how to construct different series of numbers according to particular formation rules. When will we say that he has mastered a particular series, say, the series of natural numbers? Clearly, he must be able to produce this series correctly: ‘that is, as we do it’ (Wittgenstein, 1953, §145). Wittgenstein points here to a certain vagueness in our criteria for judging that he has mastered the system, in respect of how often he must get it right and how far he must develop it. This vagueness is something that Wittgenstein sees as a distinctive characteristic of our psychological language game, one that distinguishes it from the language-game in which we describe mechanical systems. (McGinn, 1997, p. 89)
Gilbert Ryle (1949, p. 164) writes: “To settle whether a boy can do long division, we do not require him to try out his hand on a million, a thousand, or even a hundred different problems in long division. We should not be quite satisfied after one success, but we should not remain dissatisfied after twenty, provided that they were judiciously variegated and that he had not done them before. A good teacher, who also watched his procedure in reaching them, would be satisfied much sooner, and he would be satisfied sooner still if he got the boy to describe and justify the constituent operations that he performed.”
Hence, third-person ascriptions of ability are based on criteria while first-person ascriptions are not; in the first-person case there are no criteria for attaching the correct interpretation to the mental image of a rule. In short, first-person and third-person ascriptions of ability are mutually exclusive; the former do not require criteria for an ascription of ability (the individual acts for no reason) but the latter do. First-person ascriptions of ability are associated with being right and wrong, while third-person ascriptions are associated with being right or wrong.
Returning to the counterintuitive conclusion drawn by Kripke (1982), must it be accepted that individuals respond to novel addition problems by offering capricious answers? This conclusion needn’t be drawn because there’s an error in Kripke’s premise, namely, that his analysis treats inner facts (associated with first-person ascriptions of ability) as entirely independent of outer facts (associated with third-person ascriptions of ability). The idea that the inner stands in a deterministic relation to the outer has been challenged earlier. Wittgenstein considered first-person and third person ascriptions as forming an indivisible whole; they cannot be meaningfully separated (Malcolm, 1971, pp. 87-91).
In summary when the measurement process is divided into the situation immediately before the individual responds to the question “68 + 57 = ?” (the individual’s ascription of ability to himself or herself) and the situation immediately afterwards (the ascription of ability to the individual by the measurer), the relation is not one of Newtonian determinism between two independent situations. Rather, it’s one of quantum complementarity where complementarity is the more general concept which replaced Newtonian causality. The difficulties identified by Kripke (1982) in respect of causality at the level of the individual can be seen in a new light by eschewing causality for complementarity.
Polkinghorne (1996, p. 70) defines complementarity as a “combination of apparent opposites” and Whitaker (1996, p. 184) describes it as “mutual exclusion and joint completion.” In psychological predicates first-person ascriptions are made without criteria while third-person ascriptions require criteria. This is the mutual exclusiveness facet of complementarity in respect of psychological predicates. But these two very different ascriptions cannot be separated on pains of accepting Kripke’s conclusion that rule-following in mathematics is capricious. This is the joint completion facet.
While first-person/third-person ascriptions of psychological predicates appear to stand in a complementary relationship, this asymmetry is entirely absent in Newtonian physics. Suter (1989, pp. 152-153) writes: “This asymmetry in the use of psychological and mental predicates – between the first-person present-tense and second- and third-person present-tense – we may take as one of the special features of the mental. Physical predicates display no such asymmetry.”
References
Ahmed, A. (2007). Saul Kripke. London: Continuum.
Anscombe, G.E.M. (1985). Wittgenstein on rules and private language. Ethics, 95, 342-352.
Barrett, P. (2008). The consequence of sustaining a pathology: Scientific stagnation – a commentary on the target article “Is psychometrics a pathological science” by Joel Michell. Measurement, 6, 78-83.
Battig, W.F. (1978). Parsimony or psychology. Presidential Address, Rocky Mountain Psychological Association, Denvir, CO.
Bennett, M.R., & Hacker, P.M.S. (2003). Philosophical foundations of neuroscience. Oxford: Blackwell Publishing.
Blinkhorn, S. (1997). Past imperfect, future conditional: Fifty years of test theory. British Journal of Mathematical and Statistical Psychology, 50(2), 175-186.
Bloor, D. (1997). Wittgenstein: Rules and institutions. London: Routledge.
Bohr, N. (1934/1987). The philosophical writings of Niels Bohr: Volume 1 – Atomic theory and the description of nature. Woodbridge: Ox Bow Press.
Bohr, N. (1958/1987). The philosophical writings of Niels Bohr: Volume 2 – Essays 1933 – 1957 on atomic physics and human knowledge. Woodbridge: Ox Bow Press.
Borsboom, D., Mellenbergh, G.J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110 (2), 203-219.
Borsboom, D. (2005). Measuring the mind. Cambridge: Cambridge University Press.
Bridgman, P. W. (1927). The logic of modern physics. New York: Macmillan.
Bruner, J.S. (1990). Acts of meaning. Cambridge, MA: Harvard University Press.
Byrne, B.M. (1989). A primer of LISREL. New York: Springer-Verlag.
Chaitin, G.J. (2007). Thinking about Gödel and Turing. Hackensack, NJ: World Scientific.
d’Espagnat, B. (1983). In search of reality. New York: Springer-Verlag.
Elliot, C.D., Murray, D., & Pearson, L.S. (1978). The British ability scales. Windsor: National Foundation for Educational Research.
Ellis, J.L., & Van den Wollenberg, A. L. (1993). Local homogeneity in latent trait models: A characterization of the homogeneous monotone IRT model. Psychometrika, 58, 417-429.
Favrholdt, D. (Ed.). (1999). Niels Bohr collected works (Volume 10). Amsterdam: Elsevier Science B.V.
Feynman, R.P. (1985). QED: The strange theory of light and matter. Princeton, NJ: Princeton University Press.
Finch, H.L. (1977). Wittgenstein – the later philosophy. Atlantic Highlands, NJ: Humanities Press.
Gieser, S. (2005). The innermost kernel. Berlin: Springer-Verlag.
Gigerenzer, G. (1987). Probabilistic thinking and the fight against subjectivity. In L. Kruger, G. Gigerenzer, & M.S. Morgan (Eds.), The probabilistic revolution – Volume 2: Ideas in the sciences (pp. 11-33). Cambridge, MA: The Massachusetts Institute of Technology Press.
Gödel, K. (1931). Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I. Monatshefte für Mathematik und Physik, 38, 173-198.
Goldstein, H., & Blinkhorn, S. (1997). Monitoring educational standards – an inappropriate model. Bulletin of the British Psychological Society, 30, 309-311.
Hacker, P.M.S. (1997). Insight and illusion: Themes in the philosophy of Wittgenstein. Bristol: Thoemmes Press.
Heisenberg, W. (1958). Physics and philosophy. New York: Prometheus Books.
Heisenberg, W. (1962). The physicist’s conception of nature. London: The Scientific Book Guild.
Hertz, H. (1956). The principles of mechanics presented in a new form. New York: Dover Publications Inc.
Hood, S.B. (2008). Latent variable realism in psychometrics. Unpublished doctoral dissertation, Indiana University.
Honner, J. (2002). The description of nature: Niels Bohr and the philosophy of quantum physics. Oxford: Clarendon Press.
Jammer, M. (1974). The philosophy of quantum mechanics. New York: John Wiley & Sons.
Jammer, M. (1999). Einstein and religion. Princeton, NJ: Princeton University Press.
Jenkins, J.J. (1979). Four points to remember: a tetrahedral model of memory experiments. In L.S. Cremak, & F.I.M. Craik (Eds.), Levels of processing in human memory (pp. 429-446). Hillsdale, NJ: Lawrence Erlbaum.
Jöreskog, K.G., & Sörbom, D. (1993). LISREL 8 user’s reference guide. Chicago: Scientific Software International.
Kalckar, J. (Ed.). (1985). Niels Bohr collected works (Volume 6). Amsterdam: Elsevier Science B.V.
Kripke, S.A. (1982). Wittgenstein on rules and private language. Oxford: Blackwell.
Luce, R.D. (1997). Several unresolved conceptual problems of mathematical psychology. Journal of Mathematical Psychology, 41, 79-87.
Malcolm, N. (1971). Problems of mind. New York: Harper Torchbooks.
Malcolm, N. (1986). Wittgenstein: Nothing is hidden. Oxford: Blackwell.
McDowell, J. (1992). Meaning and intentionality in Wittgenstein’s later philosophy. In P.A. French, T.E. Uehling, & H.K. Wettstein (Eds.), Midwest Studies in Philosophy Volume XVII: The Wittgenstein legacy (pp. 40-52). Notre Dame, Indiana: University of Notre Dame Press.
McDowell, J. (1998). Mind, value and reality. Cambridge, MA: Harvard University Press.
McGinn, C. (1984). Wittgenstein on meaning: Oxford: Blackwell.
McGinn, M. (1997). Wittgenstein and the Philosophical Investigations. London: Routledge.
Mermin, D. (1993). Lecture given at the British Association Annual Science Festival. London: British Association for the Advancement of Science.
Michell, J. (1990). An introduction to the logic of psychological measurement. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88, 355-383.
Michell, J. (1999). Measurement in psychology: A critical history of a methodological concept. New York: Cambridge University Press.
Michell, J. (2000). Normal science, pathological science, and psychometrics. Theory & Psychology, 10(5), 639-667.
Nagel, T. (1986). The view from nowhere. New York: Oxford University Press.
Omnès, R. (1999a). Understanding quantum mechanics. Princeton, NJ: Princeton University Press.
Omnès, R. (1999b). Quantum philosophy. Princeton, NJ: Princeton University Press.
Oppenheimer, R. (1955, September 4). Analogy in science. Paper presented at the 63rd Annual Meeting of the American Psychological Association, San Francisco, CA.
Panjvani, C. (2008). Rule-following, explanation-transcendence, and private language. Mind, 117, 303-328.
Pears, D. (2006). Paradox and platitude in Wittgenstein’s philosophy. Oxford: Clarendon Press.
Polkinghorne, J. (1996). Beyond science. Cambridge: Cambridge University Press.
Polkinghorne, J. (2002). Quantum theory: A very short introduction. Oxford: Oxford University Press.
Putnam, H. (1988). Representation and reality. Cambridge, MA: The MIT Press.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Paedagogiske Institut.
Richardson, K. (1999). The making of intelligence. London: Weidenfeld & Nicolson.
Roediger III, H.L. (2008). Relativity of remembering: Why the laws of memory vanished. Annual Review of Psychology, 59, 225-254.
Schroeder, S. (2006). Wittgenstein. Cambridge: Polity Press.
Shimony, A. (1997). On mentality, quantum mechanics and the actualization of potentialities. In R. Penrose, The large, the small and the human mind (pp. 144-160). Cambridge: Cambridge University Press.
Sobel, M.E. (1994). Causal inference in latent variable models. In A. von Eye & C.C. Clogg (Eds.), Latent variable analysis (pp. 3-35). Thousand Oakes: Sage.
Stapp, H.P. (1972). The Copenhagen interpretation. American Journal of Physics, 40, 1098-1116.
Stapp, H.P. (1993). Mind, matter, and quantum mechanics. Berlin: Springr-Verlag.
Stent, G.S. (1979). Does God play dice? The sciences, 19, 18-23.
Stevens, S.S. (1946). On the theory of scales of measurement. Science, 103, 667-680.
Suen, H.K. (1990). Principles of test theories. Hillsdale, NJ: Erlbaum.
Suter, R. (1989). Interpreting Wittgenstein: A cloud of philosophy, a drop of grammar. Philadelphia: Temple University Press.
Thorndike, R.L. (1982). Educational measurement: Theory and practice. In D. Spearritt (Ed.), The improvement of measurement in education and psychology: Contributions of latent trait theory (pp. 3-13). Melbourne: Australian Council for Educational Research.
Trendler, G. (2011). Measurement theory, psychology and the revolution that cannot happen. Theory and Psychology, 19(5), 579-599.
Tulving, E. (2007). Are there 256 different kinds of memory? In J.S. Nairne (Ed.), The foundations of remembering: Essays in honour of Henry L. Roediger III (pp. 39-52). New York: Psychological Press.
Turing, A.M. (1950). Computing machinery and intelligence. Mind, 59, 433-460.
Whitaker, A. (1996). Einstein, Bohr and the quantum dilemma. Cambridge: Cambridge University Press.
Wick, D. (1995). The infamous boundary. New York: Copernicus.
Williams, M. (1999). Wittgenstein, mind and meaning. London: Routledge.
Willmott, A.S., & Fowles, D.E. (1974). The objective interpretation of test performance: The Rasch model applied. Windsor: National Foundation for Educational Research.
Wittgenstein, L. (1953). Philosophical Investigations. G.E.M. Anscombe, & R. Rhees (Eds.), G.E.M. Anscombe (Tr.). Oxford: Blackwell.
Wittgenstein, L. (1975). Wittgenstein’s lectures on the foundations of mathematics: Cambridge 1939. Chicago: University of Chicago Press.
Wittgenstein, L. (1980). Cambridge lectures (1930-1932). From the notes of John King and Desmond Lee, edited by Desmond Lee. Totowa, NJ: Rowman and Littlefield.
Wittgenstein, L. (1983). Remarks on the foundations of mathematics. Cambridge, MA: MIT Press.
Wright, B.D. (1997). A history of social science measurement. Educational Measurement: Issues and Practice, 16(4), 33-52
Wright, C. (2001). Rails to infinity. Cambridge, MA: Harvard University Press.
