‘Support Europe’s bold vision for responsible research assessment’, Nature editorial, July 27, 2022:
The Agreement on Reforming Research Assessment, announced on 20 July and open for signatures on 28 September, is perhaps the most hopeful sign yet of real change. More than 350 organizations have pooled experience, ideas and evidence to come up with a model agreement to create more-inclusive assessment systems. The initiative, four years in the making, is the work of the European University Association and Science Europe (a network of the continent’s science funders and academies), in concert with predecessor initiatives. It has the blessing of the European Commission, but with an ambition to become global.
Signatories must commit to using metrics responsibly, for example by stopping what the agreement calls “inappropriate” uses of journal and publication-based metrics such as the journal impact factor and the h-index. They also agree to avoid using rankings of universities and research organizations — and where this is unavoidable, to recognize their statistical and methodological limitations.
I’m curious if calling this plan “bold” is a way to caution readers that they must proceed cautiously, with considerable scepticism, instead of embracing such ideas with both arms. The plan itself is not bold but – as the editorial itself acknowledges, ironically – in line with what many accomplished research groups and institutes around the world have already expressed a desire for.
Also relevant here is the fact that the editorial appeared in Nature – a journal that has played up its impact factor and the significance of the various papers it has published (to their respective fields) to play up its prestigious nature. Scientists are seeking to install a new research evaluation process in the first place because of the damage such prestige has wrought to the practice of science, essentially as it has come to substitute rigour and transparency, so overturning the tyranny of prestige will deal a blow to Nature‘s large profit margins.