• About

Pace N.Ireland Education Weblog

~ Northern Ireland education analysis

Pace N.Ireland Education Weblog

Monthly Archives: August 2017

Why Dweck’s “Mindsets” concept should have no place in schools: large-scale evidence from UK public examinations

23 Wednesday Aug 2017

Posted by paceni in Grammar Schools

≈ Leave a comment

Tags

Belfast Royal Academy, Carol Dweck, Dweck’s self-theory, entity theorists, incremental theorists, Mindsets, Northern Ireland, Stanford University, UK public examinations

Psychologist Carol Dweck invoked her research on children’s beliefs to explain the attainment gap between boys and girls.  A number of small scale studies have been published which divide pupils into two broad categories according to their beliefs: (i) “incremental theorists” who believe their mathematical ability can be developed with effort, and (ii) “entity theorists” who believe that mathematical ability is fixed.  It is then argued that boys, in general, tend to be incremental theorists, while girls in general – and “bright” girls in particular – tend to be entity theorists.  Dweck’s writings leave the reader in no doubt as to which belief system is superior; incremental theorists will always have the edge on their peers who subscribe to an entity theory of ability.  When mathematics becomes challenging, girls will always lag behind boys.  However, it is revealed in what follows that large-scale public examination data can be used to stand Dweck’s thesis on its head.

The following passages are from Dweck’s Self-theories: their role in motivation, personality and development, published by Psychology Press.  These passages illustrate how she traces underachievement among “bright” girls later in their school career (when mathematics questions become really challenging) to their having developed harmful entity-focused beliefs during their grade school mathematics education (when the subject lacks real challenge).

“In our research we have found that the students with the most striking history of success are often the most, rather than the least, vulnerable.  In grade school they are far and away the highest achievers. … These bright girls may look very confident and well put together as they go about their academic work, and teachers probably do not think of them as vulnerable.  This is because these girls can easily master what is asked of them in grade school.  Yet as we will see, they are a group that does not want challenge. … And when they are presented with challenge or obstacles, they are a group that readily blames their ability and falls into a helpless pattern” (p. 53).

“We described earlier how bright girls are the highest achieving group in grade school: They earn the highest grades, they exceed bright boys in reading achievement, and they equal bright boys in math achievement.  Moreover, teachers agree that these girls are the stars.  Nevertheless, when we and others look at bright girls in our studies, we find that they are the group with the greatest vulnerability to helplessness.  They are more likely than boys to hold an entity theory of their intelligence. … Moreover, when school begins to get more challenging, as it does in junior high school, these girls traditionally have begun to fall behind their male counterparts, especially in math and science achievement … These motivational patterns, then, can help us understand why girls, who are the stars in grade school, have not traditionally been the stars in the later world of achievement” (pp. 123-124).

“[W]hen these girls hit junior high or high school, these same desires can cause problems.  The work (especially the math) suddenly gets harder, and immediate mastery is often not possible. … Quite the reverse may happen for boys”(p. 147).

Dweck cites a range of small scale studies to bolster her thesis that pupils’ beliefs can play a central role in explaining the “paradox of bright girls” (p. 123).  Alas, Dweck’s reasoning is contradicted by the public examination system in Northern Ireland.  For many years bright girls have outperformed boys at almost every level in mathematics.  The mathematics involved is highly challenging, incorporating differential and integral calculus, the solution of polynomials equations, complex variables, the application of advanced trigonometry, trajectory problems, motion with variable acceleration, and so on.  The examinations are conducted under carefully supervised conditions over a period of up to six hours.  “Examination boards” are charged with ensuring pupils and their teachers do not have sight of the examination paper prior to their entry to the examination hall.  No analysis of the grades of these thousands of pupils who takes mathematics papers supports Dweck’s reasoning.

Dweck’s explanation of the achievement gap between boys and girls in the USA, therefore, fails completely when tested on a large scale in Northern Ireland.  If this were an isolated refutation of Dweck’s worldview, one could be generous and interpret it as demonstrating that her self-theory/mindset research holds up in some cultures but not in others.  (Even this conservative reading would rule out the mindset approach as a psychological principle.)  However, the most likely explanation of the fact that the Northern Ireland data contradicts Dweck’s thinking is that her reasoning – involving “brainology” and “putting” subjects “into” one mindset or another – is plain incoherent.    For a careful analysis of the deep conceptual flaws in Dweck’s research see The flaw in Dweck & Boaler’s  Mindset research  on this blog.

 

Stephen Elliott

 

Rate this:

Time for Ofqual to take back control

13 Sunday Aug 2017

Posted by paceni in Grammar Schools

≈ Leave a comment

Tags

ability, Cambridge Assessment, Derrick Lawley, Dr Hugh Morrison, GCSE Grade 5, Gerd Gigerenzer, Harvey Goldstein, Herbert Simon, Item Response Theory, ludwig Wittgenstein, Niels Bohr, OECD, Ofqual, Pisa rankings, Svend Kriener, TES, Tim Oates, William Stewart

The GCSE Grade 5 controversy: why it’s time for Ofqual to “take back control”

 Numbers3

Dr Hugh Morrison, The Queen’s University of Belfast (retired)  drhmorrison@gmail.com

 The GCSE Grade 5 Controversy

A highly unusual feature of the new numbered GCSE grade scale is the claim that the new grade 5 will somehow reflect the standards of educational jurisdictions ranked near the top of international league tables.  Given the controversy surrounding such tables it will be possible, for the first time, to raise profound technical concerns about a particular grade on the GCSE grade scale.  Moreover, it will be difficult to make the case that grade 5 has any technical merit if Pisa ranks have any role in its determination.  Pisa is the acronym for the Paris-based “Programme for International Student Assessment” and a glance at the Times Educational Supplement of 26.07.2013 will reveal that Pisa league tables are fraught with technical difficulties.

 Concerns about Item Response Theory

The methodology which underpins Pisa ranks is called “Item Response Theory” (IRT).  IRT software claims to estimate the ability of individuals based on their responses to test items.  However, while the claim that ability is some inner state or “trait” of the individual from which his or her tests responses flow – the so-called reservoir model – is central to IRT, the claim is rarely supported by evidence.  There is very good reason to reject the inner state approach.  Consider, for example, the child solving simple problems in arithmetic.  To explain this everyday behaviour it transpires that one must invoke inner states with talismanic properties in that the state must be timeless, infinite and future-anticipating!

Rasch

Great caution is needed when using the word “ability” – while test evidence can justify us in saying that an individual has ability, that same evidence can never be used to justify the claim that the word “ability” refers to an inner (quantifiable) state of that individual.  Ability is not an intrinsic property of an individual; rather, it is a property of the interaction between individual and test items.  The individual’s responses to the test items are an inseparable part of that ability.  Indeed, divorced from a measurement context, ability is indefinite.  Individuals have definite ability only relative to a measurement context; even here it is incorrect to suggest that individuals have a quantifiable entity called “ability.”  Abandoning IRT’s appealingly simple picture of ability as an inner (quantifiable) state that individuals carry about with them, renders IRT untenable.  Ability is a two-faceted entity governed by first-person/third-person asymmetry: while we ascribe ability to ourselves without criteria, criteria are an essential prerequisite when ascribing ability to others.

The picture central to all IRT modelling – that ability is something intrinsic to the individual which is definite (and quantifiable) at all times – is rejected by the Nobel laureate Herbert Simon and by two giants of 20th century thought, physicist Niels Bohr and philosopher Ludwig Wittgenstein.  Indeed, Wittgenstein described the rationale which underpins IRT modelling – that test responses can be explained by appealing to inner processes – as a “general disease of thinking.”  Psychologists have a name for this error; Gerd Gigerenzer of the Max Plank Institute writes: “The tendency to explain behaviour internally without analysing the environment is known as the ‘Fundamental Attribution Error’.”

Niels Bohr

The criticisms levelled by Bohr and Wittgenstein are particularly damaging because IRT modellers construe ability as something inner which can be measured.  Few philosophers can match Wittgenstein’s contribution to our understanding of what can be said about the “inner”; and few scientists can match Bohr’s contribution to our understanding of measurement, particularly when the object of that measurement lies beyond direct experience.  (Bohr is listed among the top ten physicists of all time in recognition of his research on the quantum measurement problem.)  Both Bohr and Wittgenstein are concerned with the same fundamental question: how can one communicate unambiguously about aspects of reality which are beyond the direct experience of the measurer?  Just as Bohr rejected entirely the existence of definite states within the atom, Wittgenstein also rejected any claim to inner mental states; potentiality replaces actuality for both men.

For the duration of his professional life, Bohr maintained that quantum attributes have a “deep going” relation to psychological attributes in that neither can be represented as quantifiable states hidden in some inner realm.  We will always be limited to talking about ability; we will never be able to answer the question “what is ability?” let alone quantify someone’s ability.  Bohr believed that “Our task is not to penetrate into the essence of things, the meaning of which we don’t know anyway, but rather to develop concepts which allow us to talk in a productive way about phenomena in nature. …The task of physics is not to find out how nature is, but to find out what we can say about nature. … For if we want to say anything at all about nature – and what else does science try to do? – we must somehow pass from mathematical to everyday language” [italics added].

Given that IRT software is designed to measure ability, it may surprise readers that the claim that ability can be construed as a quantifiable inner state is rarely defended in IRT textbooks and journal articles.  In their article “Five decades of item response modelling,” Goldstein and Wood trace the beginnings of IRT to a paper written in 1943 by Derrick Lawley.  They note: “Lawley, a statistician, was not concerned with unpacking what ‘ability’ might mean.”  Little has changed in the interim.

Why Ofqual must protect GCSE pupils from the OECD’s “sophisticated processes”

These profound conceptual difficulties with the model which underpins Pisa rankings must surely undermine the OECD’s claim that one can rank order countries for the quality of their education systems.  In a detailed analysis of the 2006 Pisa rankings, the eminent statistician Svend Kreiner revealed that “Most people don’t know that half of the students taking part in Pisa [2006] do not respond to any reading item at all.  Despite that, Pisa assigns reading scores to these children.”  Given such revelations, why are governments, the media and the general public not more sceptical about Pisa rankings?  Kreiner offers the following explanation: “One of the problems that everybody has with Pisa is that they don’t want to discuss things with people criticising or asking questions concerning the results.  They didn’t want to talk with me at all.  I am sure it is because they can’t defend themselves.”

Given the depth of the conceptual problems which afflict IRT and, as a consequence, Pisa rankings, it seems to me foolhardy in the extreme to predicate the new GCSE grade 5 on Pisa rankings.  Ofqual have announced that grade 5 will be “broadly in line with what the best available evidence tells us is the average PISA performance in countries such as Finland, Canada, the Netherlands and Switzerland.”  In addition to Ofqual, the Department for Education and Tim Oates, director of research at Cambridge Assessment, appear to endorse a role for Pisa in UK public examinations.  The Department for Education have produced a report – PISA 2009 Study: How big is the gap? – which creates the impression that “gaps” between England and high performing Pisa countries can be represented on a GCSE grade scale designed for reporting achievement rather than ability.

Finally, the director of Cambridge Assessment asserts: “I am more optimistic … than most other analysts, I don’t see too many problems in these kinds of international comparisons.”  Indeed, Mr Oates believes that UK assessment has much to learn from involving Pisa staff directly in solving the grade 5 problem: “If we want to do it formally then we ought to have discussions with OECD. … OECD have some pretty sophisticated processes of equating tests which contain different items in different national settings.”  There is an immediate problem with this claim.  Since the psychometric definition of equity begins with the words: “for every group of examinees of identical ability …,” equity itself is founded on the erroneous assumption that ability can be quantified.

For the first time in the history of public examinations in the UK the technical fidelity of a GCSE grade will be linked to Pisa methodology.  Given the concerns surrounding IRT, is it not time for Ofqual to distance itself from the claim that the grade 5 standard is somehow invested with properties which allow it to track international standards in the upper reaches of the Pisa league tables?  The recent introduction of the rather vague term “strong pass” smacks of desperation; couldn’t a grade 6 also be deemed a strong pass?  Why not stop digging, sever the link with Pisa, and simply interpret grade 5 as nothing more than the grade representing a standard somewhere between grade 4 and grade 6?

Rate this:

Subscribe

  • Entries (RSS)
  • Comments (RSS)

Archives

  • May 2019
  • October 2018
  • July 2018
  • June 2018
  • May 2018
  • April 2018
  • February 2018
  • January 2018
  • December 2017
  • September 2017
  • August 2017
  • July 2017
  • June 2017
  • May 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • September 2016
  • August 2016
  • May 2016
  • January 2016
  • December 2015
  • November 2015
  • August 2015
  • May 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • October 2014
  • September 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • February 2013
  • January 2013
  • November 2012
  • October 2012
  • March 2012
  • February 2012
  • January 2012
  • December 2011
  • November 2011
  • July 2011
  • June 2011
  • May 2011
  • March 2011
  • February 2011
  • January 2011
  • December 2010
  • November 2010
  • October 2010
  • September 2010
  • August 2010
  • July 2010
  • June 2010
  • May 2010
  • April 2010
  • March 2010
  • February 2010
  • January 2010
  • December 2009
  • November 2009
  • October 2009
  • September 2009
  • August 2009
  • July 2009
  • June 2009
  • May 2009
  • April 2009
  • March 2009
  • February 2009
  • January 2009
  • December 2008
  • November 2008
  • October 2008
  • September 2008
  • August 2008
  • July 2008
  • June 2008
  • April 2008
  • March 2008
  • February 2008
  • January 2008

Categories

  • 11-plus
  • academic selection
  • Caitriona Ruane
  • General
  • Grammar Schools
  • Numeracy and Literacy
  • The Department of Education N.Ireland
  • Uncategorized

Meta

  • Register
  • Log in

Create a free website or blog at WordPress.com.

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy