Why exit polling gets it right and pre-election polling doesn’t
Dr Hugh Morrison (firstname.lastname@example.org)
An explanation for the conflicting messages from polling
I have been a critic of PISA league tables for many years. My central concern with PISA is that its measurement model completely misrepresents the nature of the psychological attributes it purports to measure. Proponents of PISA fail to recognize the implications for measurement of the “first-person/third-person asymmetry” (see Appendix below) which characterizes such attributes. The case against PISA also applies to attempts to measure voting intentions in pre-election surveys.
Intention – like pupil ability in the case of PISA – has no definite value prior to measurement, but adopts a definite value when the voter’s intention is registered as a tick in the relevant box. When intention-to-vote is measured, the indefinite is transformed to the definite; a potentiality is transformed to an actuality. Intention-to-vote in its unmeasured state is very different to intention-to-vote in its “measured” state. This type of measurement contrasts with the measurement of length, for example, in Newtonian physics. One isn’t simply checking up on a definite property as is the case in measuring the length of a line; measurement of intention effects radical change. (In the Appendix below these issues are explained in greater detail in respect of the predicate “ability” but the analysis applies unchanged to intentionality.)
The instruments used by pre-election pollsters (like the tools used by PISA) are incapable of capturing the asymmetry which is a hallmark of intentionality. Rather, they interpret unmeasured intentionality as something definite, carried in the mind of the voter, and the purpose of their survey instruments is to merely “check up” on an intrinsic property of the voter, namely, his or her intention-to vote. The tools used by the pre-election pollster misrepresent unmeasured intention as a definite state rather than a potentiality.
An insight into the complexity of the measurement task facing the pre-election pollster can be had from G.E.M. Anscombe, the author of the definitive book on intention: “All this conspires to make us think that if we want to know a man’s intentions it is into the contents of his mind, and only into these, that we must enquire … what a man actually does is the very last thing we need to consider in our enquiry. Whereas I wish to say it is the first.” Wittgenstein teaches us that it is wrong to conclude that when a voter ticks the box to endorse the unequivocal statement “I intend to vote conservative” that the pollster has thereby measured that voter’s intention. This must strike the reader as highly counterintuitive, even shocking. Intention is a very “slippery” concept. To quote Wittgenstein: “These things are finer spun than crude hands have any inkling of.”
Exit polls suffer from none of these difficulties. These polls lay no claim to measure the predicate “intention.” Unlike the pre-election polls their task is merely to check up on voters who emerge from the voting booth already in a definite state. Exit polls measure according to tried and tested Newtonian classical measurement principles, and are subject to none of the conceptual difficulties afflicting pre-election polls.
In summary, both pre-election polls and exit polls use instruments designed to check up on definite properties of voters. Such instruments are perfectly appropriate for the aims of the exit pollster. But the pre-election pollster is attempting to measure intention-to-vote which involves the transformation of a potentiality to an actuality.
(Extract from a critique addressing PISA)
Wittgenstein’s celebrated rule-following argument (central to his philosophies of mind, mathematics and language), set out in his Philosophical Investigations, makes clear that if one restricts oneself to the totality of facts (inner and outer) about the child, these facts are in accord with the right answer (68 + 57 = 125) and an infinity of wrong answers. Mathematics will be used for illustration but the reasoning applies to all rule-following. The reader interested in an accessible exposition of this claim is directed to the second chapter of Kripke’s (1982) Wittgenstein on Rules and Private Language. (The reader should come to appreciate the power of the rule-following reasoning without being troubled by Kripke’s questionable take on the so-called skeptical argument.) The author will now attempt the barest outlines of Wittgenstein’s writing on rule-following.
By their nature, human beings are destined to complete only a finite number of arithmetical problems over a lifetime. The child who is about to answer the question “68 + 57 = ?” for the first time has, of necessity, a finite computational history in respect of addition. Through mathematical reasoning which dates back to Leibniz, this finite number of completed addition problems can be brought under an infinite number of different rules, only one of which is the rule for addition. In short, any answer the child gives to the problem can be demonstrated to be in accord with a rule which generates that answer and all of the answers the child gave to all of the problems he or she has tackled to date. If one had access to the totality of facts about the child’s achievements in arithmetic, one couldn’t use these facts to predict the answer the child will give to the novel problem “68 + 57 = ?” because one can always derive a rule which generates the child’s entire past problem-solving history and any particular answer to “68 + 57 = ?”
Now what of facts concerned with the contents of the child’s mind? Surely an all-seeing God could peer into the child’s mind and determine which rule was guiding the child’s problem-solving? By substituting the numbers 68 and 57 into the rule, God could predict with certainty the child’s response. Alas, having access to inner facts (about the mind or brain) won’t help because having a rule in mind is neither sufficient nor necessary for responding correctly to mathematical problems. Is having a rule in mind sufficient? Clearly not since all pupils taking GCSE mathematics, for example, have access to the quadratic formula and yet only a fraction of these pupils will provide the correct answer to the examination question requiring the application of that formula. Is having the rule in mind necessary? Once again, clearly not because one can be entirely ignorant of the quadratic formula and yet produce the correct answers to algebraic problems involving quadratics using alternative procedures like “completing the square,” graphical methods, the Newton-Raphson procedure, and so on.
It is important to be clear what is being said here. If one could identify an addition problem beyond the set of problems Einstein had completed during his lifetime, is the claim that one couldn’t predict with certainty Einstein’s response to that problem? Obviously not. But the correct answer and an infinity of incorrect answers are in keeping with all the facts (inner and outer) about Einstein. When one is restricted to these facts, Einstein’s ability to respond correctly is indeterminate. In summary, before the child answers the question “68 + 57 = ?” his or her ability with respect to this question is indeterminate. The moment he or she answers, the child’s ability is determinate with respect to the question (125 is pronounced correct, and all other answers are deemed incorrect). One might portray this as follows: before responding the child is right and wrong and, at the moment of response, he or she is right or wrong.
Ability only becomes determinate in context of a measurement; it’s indeterminate before the act of measurement. The conclusion is inescapable – ability is a relational property rather than something intrinsic to the individual, as psychology’s standard measurement model would have it. A definite ability cannot be ascribed to an individual prior to measurement. Ability is a joint property of the individual and the measurement instrument; take away the instrument and ability becomes indeterminate. It is difficult to escape the conclusion that ability (and intelligence, and self-concept, and so on) is a property of the interaction between individual and measuring instrument rather than an intrinsic property of the individual. If psychological constructs were viewed as joint properties of individuals and measuring instruments, then intractable questions such as “what is intelligence?”, “what is memory?” need no longer trouble the discipline.
In simple terms it can be argued that ability has two facets; it is indeterminate before measurement and determinate immediately afterwards. The single description of the standard measurement model is replaced by two mutually exclusive descriptions. Ability is indeterminate before measurement and only determinate with respect to a measurement context. Neither of these descriptions can be dispensed with. The indeterminate and the determinate are mutually exclusive facets of one and the same ability.
Returning to the child who has been taught to add but hasn’t yet encountered the question “68 + 57 = ?” what can be said of his or her ability with respect to this question? When one ponders ability as a thing-in-itself, it’s tempting to think of it as something inner, something that resides in the child prior to being expressed when the child answers. If ability is to be found anywhere, surely it’s to the unmeasured mind one should look? Isn’t it tempting to think of it as something the child “carries” in his or her mind? When the focus is on ability as a thing-in-itself, it seems the child’s eventual answer to the question is somehow inferior; it’s the mere application of the child’s ability rather than the ability itself.
The concept of causality in classical physics is replaced by the notion of “complementarity” in quantum mechanics. Complementarity treats pre-measurement indeterminism and the determinate outcome of measurement as non-separable. Whitaker (1996, p. 184) portrays complementarity as “mutual exclusion but joint completion.” One cannot meaningfully separate the pre-measurement facet of ability from its measurement-determined counterpart. The analogue of Bohr’s complementarity is what Wittgensteinians refer to as first-person/third-person asymmetry. The first-person facet of ability (characterised by indeterminism) and the third-person measurement perspective cannot be meaningfully separated. Suter (1989, pp. 152-153) distinguished the first-person/third-person symmetry of Newtonian attributes from the first-person/third-person asymmetry of psychological predicates: “This asymmetry in the use of psychological and mental predicates – between the first-person present-tense and second- and third-person present-tense – we may take as one of the special features of the mental.” Nagel (1986, p. 22) notes: “the conditions of first-person and third-person ascription of an experience are inextricably bound together in a single public concept.”
The first-person indeterminism detailed earlier seems to indicate that individuals offer responses entirely at random. After all, the totality of facts is in keeping with an infinity of answers, only one of which is correct. But one need only infer “random variation located within the person” (Borsboom, 2005, p. 55) if one mistakenly treats the first-person facet as separable from the third-person. (The author’s earlier practice of stressing the restriction to the totality of facts about the individual was intended to highlight this taken-for-granted separability.)