Tag: scientific publishing

Poonam Pandey and peer-review

One dubious but vigorous narrative that has emerged around Poonam Pandey’s “death” and subsequent return to life is that the mainstream media will publish “anything”.

To be sure, there were broadly two kinds of news reports after the post appeared on her Instagram handle claiming Pandey had died of cervical cancer: one said she’d died and quoted the Instagram post; the other said her management team had said she’d died. That is, the first kind stated her death as a truth and the other stated her team’s statement as a truth. News reports of the latter variety obviously ‘look’ better now that Pandey and her team said she lied (to raise awareness of cervical cancer). But judging the former news reports harshly isn’t fair.

This incident has been evocative of the role of peer-review in scientific publishing. After scientists write up a manuscript describing an experiment and submit it to a journal to consider for publishing, the journal editors farm it out to a group of independent experts on the same topic and ask them if they think the paper is worth publishing. (Pre-publishing) Peer-review has many flaws, including the fact that peer-reviewers are expected to volunteer their time and expertise and that the process is often slow, inconsistent, biased, and opaque.

But for all these concerns, peer-review isn’t designed to reveal deliberately – and increasingly cleverly – concealed fraud. Granted, the journal could be held responsible for missing plagiarism and the journal and peer-reviewers both for clearly duplicated images and entirely bullshit papers. However, pinning the blame for, say, failing to double-check findings because the infrastructure to do so is hard to come by on peer-review would be ridiculous.

Peer-review’s primary function, as far as I understand it, is to check whether the data presented in the study support the conclusions drawn from the study. It works best with some level of trust. Expecting it to respond perfectly to an activity that deliberately and precisely undermines that trust is ridiculous. A better response (to more advanced tools with which to attempt fraud but also to democratise access to scientific knowledge) would be to overhaul the ‘conventional’ publishing process, such as with transparent peer-review and/or paying for the requisite expertise and labour.

(I’m an admirer of the radical strategy eLife adopted in October 2022: to review preprint papers and publicise its reviewers’ findings along with the reviewers’ identities and the paper, share recommendations with the authors to improve it, but not accept or reject the paper per se.)

Equally importantly, we shouldn’t consider a published research paper to be the last word but in fact a work in progress with room for revision, correction or even retraction. Doing otherwise – as much as stigmatising retractions for reasons not related to misconduct or fraud, for that matter – on the other hand, may render peer-review suspect when people find mistakes in a published paper even when the fault lies elsewhere.

Analogously, journalism is required to be sceptical, adversarial even – but of what? Not every claim is worthy of investigative and/or adversarial journalism. In particular, when a claim is publicised that someone has died and a group of people that manages that individual’s public profile “confirms” the claim is true, that’s the end of that. This an important reason why these groups exist, so when they compromise that purpose, blaming journalists is misguided.

And unlike peer-review, the journalistic processes in place (in many but not all newsrooms) to check potentially problematic claims – for example, that “a high-powered committee” is required “for an extensive consideration of the challenges arising from fast population growth” – are perfectly functional, in part because their false-positive rate is lower without having to investigate “confirmed” claims of a person’s death than with.

The identity of scientific papers

This prompt arose in response to Stuart Ritchie’s response to a suggestion in an editorial “first published last year but currently getting some attention on Twitter” – that scientists should write their scientific papers as if they were telling a story, with a beginning, middle and end. The act of storytelling produces something entertaining by definition, but it isn’t the same as when people build stories around what they know. That is, people build stories around what they know but that knowledge, when it is first produced, isn’t and in fact can’t be reliably produced through acts of storytelling. This is Ritchie’s point, and it’s clearly true. As Ash Jogalekar commented on Twitter on Ritchie’s post

(This is different from saying scientific knowledge shouldn’t be associated with stories – or that only it should be, a preference that philosopher of science Robert P. Crease calls “scientific gaslighting”.)

Ritchie’s objection arises from a problematic recommendation in the 2021 editorial, that when writing their papers, scientists present the “take-home messages” first, then “select” the methods and results that produced those messages, and then conclude with an introduction-discussion hybrid. To Ritchie, scientists don’t face much resistance, as they’re writing their papers, other than their own integrity that keeps them from cherry-picking from their data to support predetermined conclusions. This is perfectly reasonable, especially considering the absence of such resistance manifested in science’s sensational replication crisis.

But are scientific papers congruent with science itself?

The 2021 editorial’s authors don’t do themselves any favours in their piece, writing:

“The scientific story has a beginning, a middle, and an end. These three components can, and should, map onto the typical IMRaD structure. However, as editors we see many manuscripts that follow the IMRaD structure but do not tell a good scientific story, even when the underlying data clearly can provide one. For example, many studies present the findings without any synthesis or an effort to place them into a wider context. This limits the reader’s ability to gain knowledge and understanding, hence reducing the papers impact.”

Encouraging scientists to do such things as build tension and release it with a punchline, say, could be a recipe for disaster. The case of Brian Wansink in fact fits Ritchie’s concerns to a T. In the most common mode of scientific publishing today, narrative control is expected to lie beyond scientists – and (coming from a science journalist) lies with science journalists. Or at least: the opportunities to shape science-related narratives are available in large quantities to us.

A charitable interpretation of the editorial is that its authors would like scientists to take a step that they believe to be marginal (“right there,” as they say) in terms of the papers’ narratives but which has extraordinary benefits – but I’m disinclined. Their words hew frustratingly but unsurprisingly close to suggesting that scientists’ work isn’t properly represented in the public imagination. The most common suggestions I’ve encountered in my experience are that science journalists don’t amplify the “right” points and that they dwell on otherwise trivial shortcomings. The criticisms generally disregard the socio-political context in which science operates and to which journalists are required to be attuned.

This said, and as Ritchie also admits, the scientific paper itself is not science – so why can’t it be repurposed to ends that scientists are better off meeting than one that’s widely misguided? Ritchie writes:

“Science isn’t a story – and it isn’t even a scientific paper. The mere act of squeezing a complex process into a few thousand words … is itself a distortion of reality. Every time scientists make a decision about “framing” or “emphasis” or “take-home messages”, they risk distorting reality even further, chipping away at the reliability of what they’re reporting. We all know that many science news articles and science books are over-simplified, poorly-framed, and dumbed-down. Why push scientific papers in the same direction?”

That is, are scientific papers the site of knowledge production? With the advent of preprint papers, research preregistration and open-data and data-sharing protocols, many papers of today are radically different from those a decade or two ago. Especially online, and on the pages of more progressive journals like eLife, papers are accompanied by peer-reviewers’ comments, links to the raw data (code as well as multimedia), ways to contact the authors, a comments section, a ready-reference list of cited papers, and links to other articles that have linked to it. Sometimes some papers deemed to be more notable by a journal’s editors are also published together with commentary by an independent scientist on the papers’ implications for the relevant fields.

Scientific papers may have originated as, and for a long time have been, the ‘first expression’ of a research group’s labour to produce knowledge, and thus perfectly subject to Ritchie’s concerns about transforming them to be more engaging. But today, given the opportunities that are available in some pockets of research assessment and publishing, they’re undeniably the sites of knowledge consumption – and in effect the ‘first expression’ of researchers’ attempts to communicate with other scientists as well as, in many cases, the public at large.

It’s then effectively down to science journalists, and the resistance offered by their integrity to report on papers responsibly – although even then we should beware the “seduction of storytelling”.

I think the 2021 editorial is targetting the ‘site of knowledge consumption’ identity of the contemporaneous scientific paper, and offers ways to engage its audience better. But when the point is to improve it, why continue to work with, in Ritchie’s and the editorial’s words, a “journal-imposed word count” and structure?

A halfway point between the editorial’s recommendations and Ritchie’s objections (in his post, but more in line with his other view that we should do away with scientific papers altogether) is to publish the products of scientific labour taking full advantage of what today’s information and communication technologies allow: without a paper per se but a concise description of the methods and the findings, an explicitly labeled commentary by the researchers, the raw code, multimedia elements with tools to analyse them in real-time, replication studies, even honest (and therefore admirable) retraction reports if they’re warranted. The commentary can, in the words of the editorial, have “a beginning, a middle and an end”; and in this milieu, in the company of various other knowledge ‘blobs’, readers – including independent scientists – should be able to tell straightforwardly if the narrative fits the raw data on offer.

All this said, I must add that what I have set out here are far from where reality is at the moment; in Ritchie’s words,

“Although those of us … who’ve been immersed in this stuff for years might think it’s a bit passé to keep going on about “HARKing” and “researcher degrees of freedom” and “p-hacking” and “publication bias” and “publish-or-perish” and all the rest, the word still hasn’t gotten out to many scientists. At best, they’re vaguely aware that these problems can ruin their research, but don’t take them anywhere near seriously enough.”

I don’t think scientific papers are co-identifiable with science itself, or they certainly needn’t be. The latter is concerned with reliably producing knowledge of increasingly higher quality while the former explains what the researchers did, why, when and how. Their goals are different, and there’s no reason the faults of one should hold the other back. However, a research communication effort that has completely and perfectly transitioned to embodying the identity of the modern research paper (an anachronism) as the site of, among other things, knowledge consumption is a long way away – but it helps to bear it in mind, to talk about it and to improve it.

The Frida Kahlo NFT

Like a Phoenix rising from its ashes, Art is reborn into Eternity.

fridanft.org

In July this year, a Mexican businessman named Martin Mobarak allegedly destroyed a painting by Frida Kahlo in order to liberate it from its physical shackles and unto its “eternal” existence henceforth as an NFT that he is selling for $4,000 apiece. He has said the money will go to charity, but it’s hard to understand how that is relevant considering what has (allegedly) been lost. Art these days is not entirely art: at least a part of its purpose has been subverted by cryptocurrencies into an object for proponents of this technology to con. Also, a fundamental tenet of the NFTs market is that scarcity is always better. Put these things together and you realise Mobarak’s actions were a matter of when, not if. However, the intersection of NFT-centric thinking with the art world has been and continues to be complicated.

In 2021, an artist named Beeple sold a collage of images he’d crafted and pieced together to a cryptocurrency entrepreneur named Metakovan for tokens worth $69 million. Metakovan, and his partner Twobadour, had said at the time that they were democratising art by enabling the cryptocurrency-based public ownership of works of art and by taking advantage of cryptocurrencies’ opportunities to allow non-white, non-western people to acquire high-valued art. It was a poorly conceived proposition in many ways – starting from the fact that the acquisition was a façade for Metakovan to inflate the value of the tokens he owned and going up to the fact that the $69-million moment did everything to uphold the links between art and modern capitalism instead of critiquing them (forget tearing them down).

Fast-forward to Martin Mobarak’s (alleged) destruction of ‘Fantasmones Siniestros’ and the contradictions abound. Contrary to Metakovan’s aspiration to democratise anything, even in principle, Mobarak’s (alleged) action epitomises the private ownership of art – outside history itself, isolated in one’s personal collection and thus – by the premium American logic of private ownership – at the unquestionable mercy of its proprietor, who may even choose to burn it without regard for its thingness as a historical-cultural-political object. In both cases, and the thousands of other instances in which NFT-makers have either championed the cause of paying artists or conceived art of their own, this human endeavour has been far removed from its telos of critiquing capitalist society and has become a commodity per se.

But while this is the long- and well-known effect of capitalism on art, the (alleged) destruction of ‘Fantasmones Siniestros’ also confronts us with a tense three-way contest. On vertex 1: The painting is Kahlo’s and is as such an important part of Mexico’s past, heritage and the aspirations of its people through the ages. On vertex 2: The headlines of most news reports, if not all, make sure to mention that the painting was worth $10 million, in a reminder of its monetary value de facto and the place of art by influential artists as an important bourgeois value-store de jure. And on vertex 3: No matter the legerdemain of businesspeople, NFTs will always and eventually ensure the complete commodification of art, thanks to their foundational premise.

The only thing worth prizing here is that on vertex 1: Kahlo’s work, inasmuch as it captures her spirit and anima and is a reminder of what she did, when and amidst whom. But vertices 2 and 3 seem to hold the means by which this preservation has been achieved before and will be achieved in future, and together prompt us to pay more attention to the delicate strands by which the memories of our pasts dangle. We still await the public ownership of the work of important artists, and many others, but anyone who says cryptocurrencies or blockchains are the way to get there is lying.

A similar problem has assailed the world of scientific knowledge and publishing for many years now, with more scientists becoming more aware today of their actual role in society: not to create new knowledge and improve lives as much as to widen the margins of scientific journals, overlook their glaring flaws and the incentives they have set up to the detriment of good science, and comply unquestioningly with their inexplicable price hikes. And even as many scientists have invented notions like “prestige” and “status” to make their allegiance to journals make sense, Mobarak et al. tell us, and themselves, that they’re doing everyone a favour.

The paradoxical virtues of primacy in science

The question of “Who found it first?” in science is deceptively straightforward. It is largely due to the rewards reserved by those who administer science – funding the ‘right’ people working in the ‘right’ areas at the ‘right’ time to ensure the field’s progress along paths deemed desirable by the state – that primacy in science has become valuable. Otherwise, and in an ideal world (in which rewards are distributed more equitably, such that the quality of research is rewarded a certain amount that is lower than the inordinate rewards that accrue to some privileged scientists today but greater than that which scholars working on ‘neglected’ topics/ideas receive, without regard for gender, race, ethnicity or caste), discovering something first wouldn’t matter to the enterprise of science, just as it doesn’t mean anything to the object of the discovery itself.

Primacy is a virtue imposed by the structures of modern science. There is today privilege in being cited as “Subramaniam 2021” or “Srinivasan 2022” in papers, so much so that there is reason to believe many scientific papers are published only so they may cite the work of others and keep expanding this “citation circus”. The more citations there are, the likelier the corresponding scientist is to receive a promotion, a grant, etc. at their institute.

Across history, the use of such citations has also served to obscure the work of ‘other’ scientists and to attribute a particular finding to a single individual or a group. This typically manifests in one of two forms: by flattening the evolution of a complex discovery by multiple groups of people working around the world, sometimes sharing information with each other, to a single paper authored by one of these groups; or by reinforcing the association of one or some names with particular ideas in the scientific literature, thus overlooking important contributions by less well-known scientists.

The former is a complex phenomenon that is often motivated by ‘prestigious’ awards, including the Nobel Prizes, limiting themselves to a small group of laureates at a time, as well as by the meagre availability of grants for advanced research. Scientists and, especially, the institutes at which they work engage as a result in vociferous media campaigns when an important discovery is at hand, to ensure that opportunities for profit that may arise out of the finding may rest with them alone. This said, it can also be the product of lazy citations, in which scientists cite their friends or peers they like or wish to impress, or collections of papers over the appropriate individual ones, instead of conducting a more exhaustive literature review to cite everyone involved everywhere.

The second variety of improper citations is of course one that has dogged India – and one with which anyone working with or alongside science in India must be familiar. It has also been most famously illustrated by instances of women scientists who were subsequently overlooked for Nobel Prizes that were awarded to the men who worked with them, often against them. (The Nobel Prizes are false gods and we must tear them down; but for their flaws, they remain good, if also absurdly selective, markers of notable scientific work: that is, no prize has thus far been awarded to work that didn’t deserve it.) The stories of Chien-Shiung Wu, Rosalind Franklin and Jocelyn Bell Burnell come to mind.

But also consider the Indian example of Meghnad Saha’s paper about selective radiation pressure (in the field of stellar astrophysics), which predated Irving Langmuir’s paper on the same topic by three years. Saha lost out on the laurels by not being able to afford having his paper published in a more popular journal and had to settle for one with “no circulation worth mentioning” (source). An equation in this theory is today known as the Saha-Langmuir equation, but even this wouldn’t be so without the conscious effort of some scholars to highlight Saha’s work and unravel the circumstances that forced him into the shadows.

I discovered recently that comparable, yet not similar, circumstances had befallen Bibhas De, when the journal Icarus rejected a paper he had submitted twice. The first time, his paper presented his calculations predicting that the planet Uranus had rings; the second time was five years later, shortly after astronomers had found that Uranus indeed had rings. Stephen Brush and Ariel Segal wrote in their 2015 book, “Although he did succeed in getting his paper published in another journal, he rarely gets any credit for this achievement.”

In both these examples, and many others like them, scientists’ attempts to formalise their successes by having their claims detailed in the literature were mediated by scientific journals – whose editors’ descisions had nothing to do with science (costs in the former case and who-knows-what in the latter).

At the same time, because of these two issues, flattening and reinforcing, attribution for primacy is paradoxically more relevant: if used right, it can help reverse these problems, these imprints of colonialism and imperialism in the scientific literature. ‘Right’ here means, to me at least, that everyone is credited or none at all, as an honest reflection of the fact that good science has never been vouchsafed to the Americans or the Europeans. But then this requires more problems to be solved, such as, say, replacing profit-based scientific publishing (and the consequent valorisation of sensational results) with a ‘global scientific record’ managed by the world’s governments through an international treaty.

Axiomatically, perhaps the biggest problem with primacy today is its entrenchment. I’m certain humanities and social science scholars have debated this thoroughly – the choice for the oppressed and the marginalised between beating their oppressors at their own game or transcending the game itself. Obviously the latter seems more englightened, but it is also more labour-intensive, labour that can’t be asked freely of them – our scientists and students who are already fighting to find or keep their places in the community of their peers. Then again, beating them at their own game may not be so easy either.

I was prompted to write this post, in fact, after I stumbled on four seemingly innocuous words in a Wikipedia article about stellarators. (I wrote about these nuclear-fusion devices yesterday in the context of a study about solving an overheating problem.) The article reads that when a solenoid – a coiled wire – is bent around to form a loop, the inner perimeter of the loop has a higher density of wire than the outer perimeter. Surely this is obvious, yet the Wikpedia article phrases it thus (emphasis added):

But, as Fermi pointed out, when the solenoid is bent into a ring, the electrical windings would be closer together on the inside than the outside.

Why does a common-sensical claim, which should strike anyone who can visualise or even see a solenoid made into a loop, be attributed to the celebrated Italian physicist Enrico Fermi? The rest of the paragraph to which this sentence belongs goes on to describe how this winding density affects nuclear fusion reactors; it is an arguably straightforward effect, far removed from the singularity and the sophistication of other claims whose origins continue to be mis- or dis-attributed. Wikipedia articles are also not scientific papers. But taken together, the attribution to Fermi contains the footprints of the fact that he, as part of the Knabenphysik of quantum mechanics, worked on many areas of physics, allowing him to attach his name to a variety of concepts at a time when studies on the same topics were only just catching on in other parts of the world – a body of work enabled, as is usual, by war, conquest and the quest for hegemony.

Maybe fighting over primacy is the tax we must pay today for allowing this to happen.

PeerJ’s peer-review problem

Of all the scientific journals in the wild, there are a few I keep a closer eye on: they publish interesting results but more importantly they have been forward-thinking on matters of scientific publishing and they’ve also displayed a tendency to think out loud (through blog posts, say) and actively consider public feedback. Reading what they publish in these posts, and following the discussions that envelope them, has given me many useful insights into how scientific publishing works and, perhaps more importantly, how the perceptions surrounding this enterprise are shaped and play out.

One such journal is eLife. All their papers are open access, and they also publish the papers’ authors’ notes and reviewers’ comments with each paper. They also have a lively ‘magazine’ section in which they publish articles and essays by working scientists – especially younger ones – relating to the extended social environments in which knowledge-work happens. Now, for some reason, I’d cast PeerJ in similarly progressive light, even though I hadn’t visited their website in a long time. But on August 16, PeerJ published the following tweet:

It struck me as a weird decision (not that anyone cares). Since the article explaining the journal’s decision appears to be available under a Creative Commons Attribution license, I’m reproducing it here in full so that I can annotate my way through it.

Since our launch, PeerJ has worked towards the goal of publishing all “Sound Science”, as cost effectively as possible, for the benefit of the scientific community and society. As a result we have, until now, evaluated articles based only on an objective determination of scientific and methodological soundness, not on subjective determinations of impact, novelty or interest.

At the same time, at the core of our mission has been a promise to give researchers more influence over the publishing process and to listen to community feedback over how peer review  should work and how research should be assessed.

Great.

In recent months we have been thinking long and hard about feedback, from both our Editorial Board and Reviewers, that certain articles should no longer be considered as valid candidates for peer review or formal publication: that whilst the science they present may be “sound”, it is not of enough value to either the scientific record, the scientific community, or society, to justify being peer-reviewed or be considered for publication in a peer-reviewed journal. Our Editorial Board Members have asked us that we do our best to identify such submissions before they enter peer review.

This is the confusing part. To the uninitiated: One type of the scientific publishing process involves scientists writing up a paper and submitting it to a journal for consideration. An editor, or editors, at the journal checks the paper and then commissions a group of independent experts on the same topic to review it. These experts are expected to provide comments to help the journal decide whether it should publish the paper, and if yes, if the paper can be improved. Note that they are usually not paid for their work or time.

Now, if PeerJ’s usual reviewers are unhappy with how many papers the journal’s asking them to review, how does it make sense to impose a new, arbitrary and honestly counterproductive sort of “value” on submissions instead of increasing the number of reviewers the journal works with?

I find the journal’s decision troublesome because some important details are missing – details that encompass borderline-unethical activities by some other journals that have only undermined the integrity and usefulness of the scientific literature. For example, the “high impact factor” journal Nature has asked its reviewers in the past to prioritise sensational results over glamorous ones, overlooking the fact that such results are also likelier to be wrong. For another example, the concept of pre-registration has started to become more recently simply because most journals used to refuse (and still do) negative results. That is, if a group of scientists set out to check if something was true – and it’d be amazing if it was true – and found that it was false instead, they’d have a tough time finding a journal willing to publish their paper.

And third, preprint papers have started to become an acceptable way of publishing research only in the last few years, and that too only in a few branches of science (especially physics). Most grant-giving and research institutions still prefer papers being published in journals, instead of being uploaded on preprint repositories, not to mention a dominant research culture in many countries – including India – still favouring arbitrarily defined “prestigious journals” over others when it comes to picking scientists for promotions, etc.

For these reasons, any decision by a journal that says sound science and methodological rigour alone won’t suffice to ‘admit’ a paper into their pages risks reinforcing – directly or indirectly – a bias in the scientific record that many scientists are working hard to move away from. For example, if PeerJ rejects a solid paper, to speak, because it ‘only’ confirms a previous discovery, improves its accuracy, etc. and doesn’t fill a knowledge gap, per se, in order to ease the burden on its reviewers, the scientific record still stands to lose out on an important submission. (It pays to review journals’ decisions assuming that each journal is the only one around – à la the categorical imperative – and that other journals don’t exist.)

So what are PeerJ‘s new criteria for rejecting papers?

As a result, we have been working with key stakeholders to develop new ways to evaluate submissions and are introducing new pre-review evaluation criteria, which we will initially apply to papers submitted to our new Medical Sections, followed soon after by all subject areas. These evaluation criteria will define clearer standards for the requirements of certain types of articles in those areas. For example, bioinformatic analyses of already published data sets will need to meet more stringent reporting and data analysis requirements, and will need to clearly demonstrate that they are addressing a meaningful knowledge gap in the literature.

We don’t know yet, it seems.

At some level, of course, this means that PeerJ is moving away from the concept of peer reviewing all sound science. To be absolutely clear, this does not mean we have an intention of becoming a highly-selective “glamour” journal publisher that publishes only the most novel breakthroughs. It also does not mean that we will stop publishing negative or null results. However, the feedback we have received is that the definition of what constitutes a valid candidate for publication needs to evolve.

To be honest, this is a laughable position. The journal admits in the first sentence of this paragraph that no matter where it goes from here, it will only recede from an ideal position. In the next sentence it denies (vehemently, considering in the article on its website, this sentence was in bold) its decision is a move that will transform it into a “glamour” journal – like Nature, Science, NEJM, etc. have been – nor, in the third sentence, that it will stop publishing “negative or null results”. Now I’m even more curious what these heuristics could be which specify that a) submissions have to have “sound science”, b) “address a meaningful knowledge gap”, and c) don’t exclude negative/null results. It’s possible to see some overlap between these requirements that some papers will occupy – but it’s also possible to see many papers that won’t tick all three boxes yet still deserve to be published. To echo PeerJ itself, being a “glamour” journal is only one way to be bad.

We are being influenced by the researchers who peer review our research articles. We have heard from so many of our editorial board members and reviewers that they feel swamped by peer review requests and that they – and the system more widely – are close to breaking point. We most regularly hear this frustration when papers that they are reviewing do not, in their expert opinion, make a meaningful contribution to the record and are destined to be rejected; and should, in their view, have been filtered out much sooner in the process.

If you ask me (as an editor), the first sentence’s syntax seems to suggest PeerJ is being forced by its reviewers, and not influenced. More importantly, I haven’t seen these bespoke problematic papers that are “sound” but at the same time don’t make a meaningful contribution. An expert’s opinion that a paper on some topic should be rejected (even though, again, it’s “sound science”) could be rooted either in an “arrogant gatekeeper” attitude or in valid reasons, and PeerJ‘s rules should be good enough to be able to differentiate between the two without simultaneously allowing ‘bad reviewers’ to over-“influence” the selection process.

More broadly, I’m a science journalist looking into science from the outside, seeing a colossal knowledge-producing machine that’s situated on the same continuum on which I see myself to be located. If I receive too many submissions at The Wire Science, I don’t make presumptuous comments about what I think should and shouldn’t belong in the public domain. Instead, I pitch my boss about hiring one more person on my team and, second, I’m honest with each submission’s author about why I’m rejecting it: “I’m sorry, I’m short on time.”

Such submissions, in turn, impact the peer review of articles that do make a very significant contribution to the literature, research and society – the congestion of the peer review process can mean assigning editors and finding peer reviewers takes more time, potentially delaying important additions to the scientific record.

Gatekeeping by another name?

Furthermore, because it can be difficult and in some cases impossible to assign an Academic Editor and/or reviewers, authors can be faced with frustratingly long waits only to receive the bad news that their article has been rejected or, in the worst cases, that we were unable to peer review their paper. We believe that by listening to this feedback from our communities and removing some of the congestion from the peer review process, we will provide a better, more efficient, experience for everyone.

Ultimately, it comes down to the rules by which PeerJ‘s editorial board is going to decide which papers are ‘worth it’ and which aren’t. And admittedly, without knowing these rules, it’s hard to judge PeerJ – except on one count: “sound science” is already a good enough rule by which to determine the quality of a scientist’s work. To say it doesn’t suffice for reasons unrelated to scientific publishing, and the publishing apparatus’s dangerous tendency to gatekeep based on factors that have little to do with science, sounds at least precarious.

Bharat Biotech gets 1/10 for tweet

If I had been Bharat Biotech’s teacher and “Where is your data?” had been an examination question, Bharat Biotech would have received 1 out of 10 marks.

The correct answer to where is your data can take one of two forms: either an update in the form of where the data is in the data-processing pipeline or to actually produce the data. The latter in fact would have deserved a bonus point, if only because the question wasn’t precise enough. The question should really have been a demand – “Submit your data” – instead of allowing the answerer, in its current form, to get away with simply stating where the data currently rests. Bharat Biotech gets 1/10 because it does neither; the 1 is for correct spelling.

In fact, the company’s chest-thumping based on publishing nine papers in 12 months is symptomatic of a larger problem with the student. He fails to understand that only data is data, and that the demand for data is a demand for data per se. It ought not to be confused with a demand for authority. Data accords authority in an object-oriented and democratic sense. With data, everyone else can see for themselves – whether by themselves or through the mouths and minds of independent experts they trust – if the student’s claims hold up. And if they do, they confer the object of the data, the COVID-19 vaccine named Covaxin, with attributes like reliability.

(Why ‘he’? The patriarchal conditions in and with which science has operated around the world, but especially in Europe and the US, in the last century or so have diffused into scientific practice itself, in terms of how the people at large have constituted – as well as have been expected to constitute, by the scientific community – scientific authority, expertise’s immunity to criticism and ownership of knowledge production and dissemination apparatuses, typically through “discrimination, socialisation and the gender division of labour”. Irrespective of the means – although both from the company’s and the government’s sides, very few women have fielded and responded to questions about drug/vaccine approvals – we already see these features in the manner in which ‘conventional’ scientific journals have sought to retain their place in the international knowledge production economy, and their tendency to resort to arguments that they serve an important role in it even as they push for anti-transparent practices, from the scientific papers’ contents to details about why they charge so much money.)

However, the student has confused authority of this kind with authority of a kind we more commonly associate with the conventional scientific publishing paradigm: in which journals are gatekeepers of scientific knowledge – both in terms of what topics they ‘accept’ manuscripts on and what they consider to be ‘good’ results; and in which a paper, once published, is placed behind a steeply priced paywall that keeps both knowledge of the paper’s contents and the terms of its ‘acceptance’ by the journal beyond public scrutiny – even when public money funded the research described therein. As such, his insistence that we be okay with his having published nine papers in 12 months is really his insistence that we vest our faith in scientific journals, and by extension their vaunted decision to ‘approve of’ his work. This confusion on his part is also reflected in what he offers as his explanation for the absence of data in the public domain, but which are really his excuses.

Our scientific commitment as a company stands firm with data generation, data transparency and peer-reviewed publications.

Sharing your data in a secluded channel with government bodies is not data transparency. That’s what the student needs for regulatory approval. Transparency applies when the data is available for everyone else to independently access, understand and check.

Phase 3 final analysis data will be available soon. Final analysis requires efficacy and 2 months safety follow-up data on all subjects. This is mandated by CDSCO and USFDA. Final analysis will first be submitted to CDSCO, followed by submissions to peer reviewed journals and media dissemination.

What is required by CDSCO does not matter to those allowing Bharat Biotech’s vaccines into the bloodstreams, and in fact every Indian on whom the student has inflicted this pseudo-choice. And at this point to invoke what the USFDA requires can only lead to a joke: studies of the vaccines involved in the formal vaccination drive have already been published in the US; even studies of new vaccines as well as follow-ups of existing formulations are being placed in the public domain through preprint papers that describe the data from soup to nuts. All we got from the student vis-à-vis Covaxin this year was interim phase 3 trial data in early March, announced through a press release, and devoid even of error bars for its most salient claims.

So even for an imprecisely worded question, it has done well to elicit a telling answer from the student: that the data does not exist, and the student believes he is too good for us all.

Thanks to Jahnavi Sen for reading the article before it was published.

A non-self-correcting science

While I’m all for a bit of triumphalism when some component of conventional publication vis-à-vis scientific research – like pre-publication anonymous peer review – fails, and fails publicly, I spotted an article in The Conversation earlier today that I thought crossed a line (and not in the way you think). In this article, headlined ‘Retractions and controversies over coronavirus research show that the process of science is working as it should’, the author writes:

Some people are viewing the retractions [by The Lancet and the New England Journal of Medicine] as an indictment of the scientific process. Certainly, the overturning of these papers is bad news, and there is plenty of blame to go around. But despite these short-term setbacks, the scrutiny and subsequent correction of the papers actually show that science is working. Reporting of the pandemic is allowing people to see, many for the first time, the messy business of scientific progress.

The retraction of the hydroxychloroquine paper … drew immediate attention not only because it placed science in a bad light, but also because President Trump had touted the drug as an effective treatment for COVID-19 despite the lack of strong evidence. Responses in the media were harsh. … [Their] headlines may have [had] merit, but perspective is also needed. Retractions are rare – only about 0.04% of published papers are withdrawn – but scrutiny, update and correction are common. It is how science is supposed to work, and it is happening in all areas of research relating to SARS-CoV-2.

If you ask me, this is not science working as it should. This is the journals that published the papers discovering that the mechanisms they’d adopted that they’d said would filter fraudulent papers letting fraudulent papers slip through.

But by the author’s logic, “this is science working as it should” would encompass any mistake that’s later discovered, followed by suitable corrective action. This is neither here nor there – and more importantly it allows broken processes to be subsumed under the logic’s all-encompassing benevolence. If this is scientific publishing as it should be, we wouldn’t have to think deeply about how we can fix anonymous pre-publication peer-review because it wouldn’t be broken. However, we know in reality that it is.

If anything, by advancing his argument, the author has cleverly pressed an argumentative tack that supporters of more progressive scientific publishing models in the service of preserving the status quo. Instead, we need to acknowledge that an important part of science, called science publishing, has evolved into a flawed creature – so that we can set about bending the moral arc towards fixing it. (We already know that if we don’t acknowledge it, we won’t fix it.)

Citations and media coverage

According to a press release accompanying a just-published study in PLOS ONE:

Highly cited papers also tend to receive more media attention, although the cause of the association is unclear.

One reason I can think of is a confounding factor that serves as the hidden cause of both phenomena. Discoverability matters just as much as the quality of a paper, and conventional journals implicated in the sustenance of notions like ‘prestige’ (Nature, Science, Cell, The Lancet, etc.) have been known to prefer more sensational positive results. And among researchers that still value publishing in these journals, these papers are more noticed, which leads to a ‘buzz’ that a reporter can pick up on.

Second, sensational results also easily lend themselves to sensational stories in the press, which has often been addicted to the same ‘positivity bias’ that the scientific literature harboured for many decades. In effect, highly cited papers are simply highly visible, and highly visibilised, papers – both to other scientists and journalists.

The press release continues:

The authors add: “Results from this study confirm the idea that media attention given to scientific research is strongly related to scientific citations for that same research. These results can inform scientists who are considering using popular media to increase awareness concerning their work, both within and outside the scientific community.”

I’m not sure what this comment means (I haven’t gone through the paper and it’s possible the paper’s authors discuss this in more detail), but there is already evidence that studies for which preprints are available receive more citations than those published behind a paywall. So perhaps scientists expecting more media coverage of their work should simply make their research more accessible. (It’s also a testament to the extent to which the methods of ‘conventional’ publishers – including concepts like ‘reader pays’ and the journal impact factor, accentuated by notions like ‘prestige’ – have become entrenched that this common-sensical solution is not so common sense.)

On the flip side, journalists also need to be weaned away from ‘top’ journals – I receive a significantly higher number of pitches offering to cover papers published in Nature journals – and retrained to spot interesting results published in less well-known journals as well as, on a slightly separate note, to situate the results of one study in a larger context instead of hyper-focusing on one context-limited set of results.

The work seems interesting, perhaps one of you will like to give it a comb.

The costs of correction

I was slightly disappointed to read a report in the New York Times this morning. Entitled ‘Two Huge COVID-19 Studies Are Retracted After Scientists Sound Alarms’, it discussed the implications of two large studies of COVID-19 recently being retracted by two leading medical journals they were published in, the New England Journal of Medicine and The Lancet. My sentiment stemmed from the following paragraph and some after:

I don’t know if just these two retractions raise troubling questions as if these questions weren’t already being asked well before these incidents. The suggestion that the lack of peer-review, or any form of peer-review at all in its current form (opaque, unpaid) could be to blame is more frustrating, as is the article’s own focus on the quality of the databases used in the two studies instead of the overarching issue. Perhaps this is yet another manifestation of the NYT’s crisis under Trump?

One of the benefits of the preprint publishing system is that peer-review is substituted with ‘open review’. And one of the purposes of preprints is that the authors of a study can collect feedback and suggestions before publishing in a peer-reviewed journal instead of accruing a significant correction cost post-publication, in the form of corrections or retractions, both of which continue to carry a considerable amount of stigma. So as such, the preprints mode ensures a more complete, a more thoroughly reviewed manuscript enters the peer-review system instead of vesting the entire burden of fact-checking and reviewing a paper on a small group of experts whose names and suggestions most journals don’t reveal, and who are generally unpaid for their time and efforts.

In turn, the state of scientific research is fine. It would simply be even better if we reduced the costs associated with correcting the scientific record instead of heaping more penalties on that one moment, as the conventional system of publishing does. ‘Conventional – which in this sphere seems to be another word for ‘closed-off’ – journals also have an incentive to refuse to publish corrections or perform retractions because they’ve built themselves up on claims of being discerning, thorough and reliable. So retractions are a black mark on their record. Elisabeth Bik has often noted how long journals take to even acknowledge entirely legitimate complaints about papers they’ve published, presumably for this reason.

There really shouldn’t be any debate on which system is better – but sadly there is.

Poor journalism is making it harder for preprints

There have been quite a few statements by various scientists on Twitter who, in pointing to some preprint paper’s untenable claims, point to the manuscript’s identity as a preprint paper as well. This is not fair, as I’ve argued many times before. A big part of the problem here is bad journalism. Bad preprint papers are a problem not because their substance is bad but because people who are not qualified to understand why it is bad read it and internalise its conclusions at face value.

There are dozens of new preprint papers uploaded onto arXiv, medRxiv and bioRxiv every week making controversial arguments and/or arriving at far-fetched conclusions, often patronising to the efforts of the subject’s better exponents. Most of them (at least according to what I know of preprints on arXiv) are debated and laid to rest by scientists familiar with the topics at hand. No non-expert is hitting up arXiv or bioRxiv every morning looking for preprints to go crazy on. The ones that become controversial enough to catch the attention of non-experts have, nine times out of then, been amplified to that effect by a journalist who didn’t suitably qualify the preprint’s claims and simply published it. Suddenly, scores (or more) of non-experts have acquired what they think is refined knowledge, and public opinion thereafter goes against the scientific grain.

Acknowledging that this collection of events is a problem on many levels, which particular event would you say is the deeper one?

Some say it’s the preprint mode of publishing, and when asked for an alternative, demand that the use of preprint servers be discouraged. But this wouldn’t solve the problem. Preprint papers are a relatively new development while ‘bad science’ has been published for a long time. More importantly, preprint papers improve public access to science, and preprints that contain good science do this even better.

To making sweeping statements against the preprint publishing enterprise because some preprints are bad is not fair, especially to non-expert enthusiasts (like journalists, bloggers, students) in developing countries, who typically can’t afford the subscription fees to access paywalled, peer-reviewed papers. (Open-access publishing is a solution too but it doesn’t seem to feature in the present pseudo-debate nor does it address important issues that beset itself as well as paywalled papers.)

Even more, if we admitted that bad journalism is the problem, as it really is, we achieve two things: prevent ‘bad science’ from reaching the larger population and retain access to ‘good science’.

Now, to the finer issue of health- and medicine-related preprints: Yes, acting based on the conclusions of a preprint paper – such as ingesting an untested drug or paying too much attention to an irrelevant symptom – during a health crisis in a country with insufficient hospitals and doctors can prove deadlier than usual. But how on Earth could a person have found that preprint paper, read it well enough to understand what it was saying, and act on its conclusions? (Put this way, a bad journalist could be even more to blame for enabling access to a bad study by translating its claims to simpler language.)

Next, a study published in The Lancet claimed – and thus allowed others to claim by reference – that most conversations about the novel coronavirus have been driven by preprint papers. (An article in Ars Technica on May 6 carried this provocative headline, for example: ‘Unvetted science is fuelling COVID-19 misinformation’.) However, the study was based on only 11 papers. In addition, those who invoke this study in support of arguments directed against preprints often fail to mention the following paragraph, drawn from the same paper:

… despite the advantages of speedy information delivery, the lack of peer review can also translate into issues of credibility and misinformation, both intentional and unintentional. This particular drawback has been highlighted during the ongoing outbreak, especially after the high-profile withdrawal of a virology study from the preprint server bioRxiv, which erroneously claimed that COVID-19 contained HIV “insertions”. The very fact that this study was withdrawn showcases the power of open peer-review during emergencies; the withdrawal itself appears to have been prompted by outcry from dozens of scientists from around the globe who had access to the study because it was placed on a public server. Much of this outcry was documented on Twitter and on longer-form popular science blogs, signalling that such fora would serve as rich additional data sources for future work on the impact of preprints on public discourse. However, instances such as this one described showcase the need for caution when acting upon the science put forth by any one preprint.”

The authors, Maimuna Majumder and Kenneth Mandl, have captured the real problem. Lots of preprints are being uploaded every week and quite a few are rotten. Irrespective of how many do or don’t drive public conversations (especially on the social media), it’s disingenuous to assume this risk by itself suffices to cut access.

Instead, as the scientists write, exercise caution. Instead of spoiling a good thing, figure out a way to improve the reporting habits of errant journalists. Otherwise, remember that nothing stops an irresponsible journalist from sensationalising the level-headed conclusions of a peer-reviewed paper either. All it takes is to quote from a grossly exaggerated university press-release and to not consult with an independent expert. Even opposing preprints with peer-reviewed papers only advances a false balance, comparing preprints’ access advantage to peer-review’s gatekeeping advantage (and even that is on shaky ground).