Albert Einstein, CalTech, Gravitational Wave, Hanfod Observatory, Intensity-weighted GPA, LIGO, Livingston Observatory, MIT, Neutron Stars, Paul Jump, Phil Baty, Queens University Belfast, REF, Research Intensity, Robert Bowman, Times Higher Education
One of the greatest advances in modern physics – the detection of gravitational waves first postulated, a century ago, by Einstein in his general theory of relativity – was made by physicists at the Laser Interferometer Gravitational-Wave Observatory (LIGO). To paraphrase Richard Feynman, LIGO’s measurement precision can be expressed as follows: If you were to measure the distance between earth and the nearest star with this precision, it would be exact to the thickness of a human hair. Such incredible accuracy alone would more than justify the £150 million construction costs of LIGO as a feat of engineering alone.
It is instructive to contrast the measurement properties of the UK’s Research Excellence Framework (REF), which aims to rank-order the research quality of UK universities, with those of LIGO. While the REF league table has no discernible measurement properties whatsoever, its cost far exceeds LIGO’s construction costs, coming in at a staggering one quarter of a billion pounds.
A recent issue of the Times Higher Education (1 – 7 February 2018) included a booklet published by the Queen’s University of Belfast which illustrates the extremes to which universities are prepared to go in using highly questionable data derived from REF ranks for the purposes of self-promotion. Page 5 of the Queen’s booklet consists of a single statement. At the centre of a black A4 page the words “Ranked in the top 10 in the UK for research intensity” (in white print and large font) sit in isolation. (In a much smaller font the university attributes this ranking to the Times Higher Education.) Pages 6 to 9 offer pen portraits of nine “world-leading academics” employed by Queen’s University. The university has used this “research intensity” claim to market the university ever since the publication of the 2014 REF. One cannot browse the university’s website without encountering the claim at every turn. It has appeared on university billboards, in promotional materials and was central to the university’s ubiquitous claim: “we are exceptional.” Why did no one notice that research intensity is a meaningless concept?
In the Research Excellence Framework, the research quality of journal articles, books etc. is assessed and reported on a four-point scale (five-point if one includes the ‘unclassified’ category). The scale is ordinal in the sense that a 3* article is deemed superior to one rated 1* or 2*, and inferior to an article rated 4*. Any appeal to arithmetic is impermissible because an article rated 4* is not 4 times the quality of a 1* article; a 2* article is not two-thirds the quality of a 3* article, and so on. The rules of arithmetic do not apply. (Needless to say, the challenge of assessing research quality would remain unchanged if numbers were abandoned for the grades A, B, C and D.) These whole-number ratings are then used by the Times Higher Education to compute a university’s all-important “research intensity” (reported to two decimal place accuracy) using simple arithmetic. But, as every sixth-form statistician knows, arithmetical operations are not meaningful when applied to an ordinal scale.
How can it be that no world-class scientist at Queen’s seems to have pointed out to those charged with marketing the university that “research intensity” is a highly questionable measure? Are we to believe that no UK scientist has written to the Times Higher Education pointing out the magazine’s error? One gets the clear impression from those charged with marketing Queen’s University that the Belfast campus is crammed to the rafters with ”world-leading” scientists. How could any scientist worthy of the label endorse the notion of “research intensity” when his or her sixth-form mathematics training would identify the notion as nonsensical. In particular, why have none of the university’s world-leading academics, listed on pages 6 through 9 of the Queen’s promotional booklet, questioned the nonsensical claim printed on page 5?”
The Queen’s University of Belfast and the Times Higher Education must clarify their positions on this matter.