Tag: academic publishing

The BHU Covaxin study and ICMR bait

Earlier this month, a study by a team at Banaras Hindu University (BHU) in Varanasi concluded that fully 1% of Covaxin recipients may suffer severe adverse events. One percent is a large number because the multiplier (x in 1/100 * x) is very large — several million people. The study first hit the headlines for claiming it had the support of the Indian Council of Medical Research (ICMR) and reporting that both Bharat Biotech and the ICMR are yet to publish long-term safety data for Covaxin. The latter is probably moot now, with the COVID-19 pandemic well behind us, but it’s the principle that matters. Let it go this time and who knows what else we’ll be prepared to let go.

But more importantly, as The Hindu reported on May 25, the BHU study is too flawed to claim Covaxin is harmful, or claim anything for that matter. Here’s why (excerpt):

Though the researchers acknowledge all the limitations of the study, which is published in the journal Drug Safety, many of the limitations are so critical that they defeat the very purpose of the study. “Ideally, this paper should have been rejected at the peer-review stage. Simply mentioning the limitations, some of them critical to arrive at any useful conclusion, defeats the whole purpose of undertaking the study,” Dr. Vipin M. Vashishtha, director and pediatrician, Mangla Hospital and Research Center, Bijnor, says in an email to The Hindu. Dr. Gautam Menon, Dean (Research) & Professor, Departments of Physics and Biology, Ashoka University shares the same view. Given the limitations of the study one can “certainly say that the study can’t be used to draw the conclusions it does,” Dr. Menon says in an email.

Just because you’ve admitted your study has limitations doesn’t absolve you of the responsibility to interpret your research data with integrity. In fact, the journal needs to speak up here: why did Drug Safety publish the study manuscript? Too often when news of a controversial or bad study is published, the journal that published it stays out of the limelight. While the proximal cause is likely that journalists don’t think to ask journal editors and/or publishers tough questions about their publishing process, there is also a cultural problem here: when shit hits the fan, only the study’s authors are pulled up, but when things are rosy, the journals are out to take credit for the quality of the papers they publish. In either case, we must ask what they actually bring to the table other than capitalising on other scientists’ tendency to judge papers based on the journals they’re published in instead of their contents.

Of course, it’s also possible to argue that unlike, say, journalistic material, research papers aren’t required to be in the public interest at the time of publication. Yet the BHU paper threatens to undermine public confidence in observational studies, and that can’t be in anyone’s interest. Even at the outset, experts and many health journalists knew observational studies don’t carry the same weight as randomised controlled trials as well as that such studies still serve a legitimate purpose, just not the one to which its conclusions were pressed in the BHU study.

After the paper’s contents hit the headlines, the ICMR shot off a latter to the BHU research team saying it hasn’t “provided any financial or technical support” to the study and that the study is “poorly designed”. Curiously, the BHU team’s repartee to the ICMR’s makes repeated reference to Vivek Agnihotri’s film The Vaccine War. In the same point in which two of these references appear (no. 2), the team writes: “While a study with a control group would certainly be of higher quality, this immediately points to the fact that it is researchers from ICMR who have access to the data with the control group, i.e. the original phase-3 trials of Covaxin – as well publicized in ‘The Vaccine War’ movie. ICMR thus owes it to the people of India, that it publishes the long-term follow-up of phase-3 trials.”

I’m not clear why the team saw fit to appeal to statements made in this of all films. As I’ve written earlier, The Vaccine War — which I haven’t watched but which directly references journalistic work by The Wire during and of the pandemic — is most likely a mix of truths and fictionalisation (and not in the clever, good-faith ways in which screenwriters adopt textual biographies for the big screen), with the fiction designed to serve the BJP’s nationalist political narratives. So when the letter says in its point no. 5 that the ICMR should apologise to a female member of the BHU team for allegedly “spreading a falsehood” about her and offers The Vaccine War as a counterexample (“While ‘The Vaccine War’ movie is celebrating women scientists…”), I can’t but retch.

Together with another odd line in the latter — that the “ICMR owes it to the people of India” — the appeals read less like a debate between scientists on the merits and the demerits of the study and more like they’re trying to bait the ICMR into doing better. I’m not denying the ICMR started it, as a child might say, but saying that this shouldn’t have prevented the BHU team from keeping it dignified. For example, the BHU letter reads: “It is to be noted that interim results of the phase-3 trial, also cited by Dr. Priya Abraham in ‘The Vaccine War’ movie, had a mere 56 days of safety follow-up, much shorter than the one-year follow-up in the IMS-BHU study.” Surely the 56-day period finds mention in a more respectable and reliable medium than a film that confuses you about what’s real and what’s not?

In all, the BHU study seems to have been designed to draw attention to gaps in the safety data for Covaxin — but by adopting such a provocative route, all that took centerstage was its spat with the ICMR plus its own flaws.

What’s with superconductors and peer-review?

Throughout the time I’ve been a commissioning editor for science-related articles for news outlets, I’ve always sought and published articles about academic publishing. It’s the part of the scientific enterprise that seems to have been shaped the least by science’s democratic and introspective impulses. It’s also this long and tall wall erected around the field where scientists are labouring, offering ‘visitors’ guided tours for a hefty fee – or, in many cases, for ‘free’ if the scientists are willing to pay the hefty fees instead. Of late, I’ve spent more time thinking about peer-review, the practice of a journal distributing copies of a manuscript it’s considering for publication to independent experts on the same topic, for their technical inputs.

Most of the peer-review that happens today is voluntary: the scientists who do it aren’t paid. You must’ve come across several articles of late about whether peer-review works. It seems to me that it’s far from perfect. Studies (in July 1998, September 1998, and October 2008, e.g.) have shown that peer-reviewers often don’t catch critical problems in papers. In February 2023, a noted scientist said in a conversation that peer-reviewers go into a paper assuming that the data presented therein hasn’t been tampered with. This statement was eye-opening for me because I can’t think of a more important reason to include technical experts in the publishing process than to wean out problems that only technical experts can catch. Anyway, these flaws with the peer-review system aren’t generalisable, per se: many scientists have also told me that their papers benefited from peer-review, especially review that helped them improve their work.

I personally don’t know how ‘much’ peer-review is of the former variety and how much the latter, but it seems safe to state that when manuscripts are written in good faith by competent scientists and sent to the right journal, and the journal treats its peer-reviewers as well as its mandate well, peer-review works. Otherwise, it tends to not work. This heuristic, so to speak, allows for the fact that ‘prestige’ journals like Nature, Science, NEJM, and Cell – which have made a name for themselves by publishing papers that were milestones in their respective fields – have also published and then had to retract many papers that made exciting claims that were subsequently found to be untenable. These journals’ ‘prestige’ is closely related to their taste for sensational results.

All these thoughts were recently brought into focus by the ongoing hoopla, especially on Twitter, about the preprint papers from a South Korean research group claiming the discovery of a room-temperature superconductor in a material called LK-99 (this is the main paper). This work has caught the imagination of users on the platform unlike any other paper about room-temperature superconductivity in recent times. I believe this is because the preprints contain some charts and data that were absent in similar work in the past, and which strongly indicate the presence of a superconducting state at ambient temperature and pressure, and because the preprints include instructions on the material’s synthesis and composition, which means other scientists can produce and check for themselves. Personally, I’m holding the stance advised by Prof. Vijay B. Shenoy of IISc:

Many research groups around the world will attempt to reproduce these results; there are already some rumours that independent scientists have done so. We will have to wait for the results of their studies.

Curiously, the preprints have caught the attention of a not insignificant number of techbros, who, alongside the typically naïve displays of their newfound expertise, have also called for the peer-review system to be abolished because it’s too slow and opaque.

Peer-review has a storied relationship with superconductivity. In the early 2000s, a slew of papers coauthored by the German physicist Jan Hendrik Schön, working at a Bell Labs facility in the US, were retracted after independent investigations found that he had fabricated data to support claims that certain organic molecules, called fullerenes, were superconducting. The Guardian wrote in September 2002:

The Schön affair has besmirched the peer review process in physics as never before. Why didn’t the peer review system catch the discrepancies in his work? A referee in a new field doesn’t want to “be the bad guy on the block,” says Dutch physicist Teun Klapwijk, so he generally gives the author the benefit of the doubt. But physicists did become irritated after a while, says Klapwijk, “that Schön’s flurry of papers continued without increased detail, and with the same sloppiness and inconsistencies.”

Some critics hold the journals responsible. The editors of Science and Nature have stoutly defended their review process in interviews with the London Times Higher Education Supplement. Karl Ziemelis, one of Nature’s physical science editors, complained of scapegoating, while Donald Kennedy, who edits Science, asserted that “There is little journals can do about detecting scientific misconduct.”

Maybe not, responds Nobel prize-winning physicist Philip Anderson of Princeton, but the way that Science and Nature compete for cutting-edge work “compromised the review process in this instance.” These two industry-leading publications “decide for themselves what is good science – or good-selling science,” says Anderson (who is also a former Bell Labs director), and their market consciousness “encourages people to push into print with shoddy results.” Such urgency would presumably lead to hasty review practices. Klapwijk, a superconductivity specialist, said that he had raised objections to a Schön paper sent to him for review, but that it was published anyway.

A similar claim by a group at IISc in 2019 generated a lot of excitement then, but today almost no one has any idea what happened to it. It seems reasonable to assume that the findings didn’t pan out in further testing and/or that the peer-review, following the manuscript being submitted to Nature, found problems in the group’s data. Last month, the South Korean group uploaded its papers to the arXiv preprint repository and has presumably submitted them to a journal: for a finding this momentous, that seems like the obvious next step. And the journal is presumably conducting peer-review at this point.

But in both instances (IISc 2019 and today), the claims were also accompanied by independent attempts to replicate the data as well as journalistic articles that assimilated the various public narratives and their social relevance into a cogent whole. One of the first signs that there was a problem with the IISc preprint was another preprint by Brian Skinner, a physicist then with the Massachusetts Institute of Technology, who found the noise in two graphs plotting the results of two distinct tests to be the same – which is impossible. Independent scientists also told The Wire (where I worked then) that they lacked some information required to make sense of the results as well as expressed concerns with the magnetic susceptibility data.

Peer-review may not be designed to check whether the experiments in question produced the data in question but whether the data in question supports the conclusions. For example, in March this year, Nature published a study led by Ranga P. Dias in which he and his team claimed that nitrogen-doped lutetium hydride becomes a room-temperature superconductor under a pressure of 1,000 atm, considerably lower than the pressure required to produce a superconducting state in other similar materials. After it was published, many independent scientists raised concerns about some data and analytical methods presented in the paper – as well as its failure to specify how the material could be synthesised. These problems, it seems, didn’t prevent the paper from clearing peer-review. Yet on August 3, Martin M. Bauer, a particle physicist at Durham University, published a tweet defending peer-review in the context of the South Korean work thus:

The problem seems to me to be the belief – held by many pro- as well as anti-peer-review actors – that peer-review is the ultimate check capable of filtering out all forms of bad science. It just can’t, and maybe that’s okay. Contrary to what Dr. Bauer has said, and as the example of Dr. Dias’s paper suggests, peer-reviewers won’t attempt to replicate the South Korean study. That task, thanks to the level of detail in the South Korean preprint and the fact that preprints are freely accessible, is already being undertaken by a panoply of labs around the world, both inside and outside universities. So abolishing peer-review won’t be as bad as Dr. Bauer makes it sound. As I said, peer-review is, or ought to be, one of many checks.

It’s also the sole check that a journal undertakes, and maybe that’s the bigger problem. That is, scientific journals may well be a pit of papers of unpredictable quality without peer-review in the picture – but that would only be because journal editors and scientists are separate functional groups, rather than having a group of scientists take direct charge of the publishing process (akin to how arXiv currently operates). In the existing publishing model, peer-review is as important as it is because scientists aren’t involved in any other part of the publishing pipeline.

An alternative model comes to mind, one that closes the gaps of “isn’t designed to check whether the experiments in question produced the data in question” and “the sole check that a journal undertakes”: scientists conduct their experiments, write them up in a manuscript and upload them to a preprint repository; other scientists attempt to replicate the results; if the latter are successful, both groups update the preprint paper and submit that to a journal (with the lion’s share of the credit going to the former group); journal editors have this document peer-reviewed (to check whether the data presented supports the conclusions), edited, and polished[1]; and finally publish it.

Obviously this would require a significant reorganisation of incentives: for one, researchers will need to be able to apportion time and resources to replicate others’ experiments for less than half of the credit. A second problem is that this is a (probably non-novel) reimagination of the publishing workflow that doesn’t consider the business model – the other major problem in academic publishing. Third: I have in my mind only condensed-matter physics; I don’t know much about the challenges to replicating results in, say, genomics, computer science or astrophysics. My point overall is that if journals look like a car crash without peer-review, it’s only because the crashes were a matter of time and that peer-review was doing the bare minimum to keep them from happening. (And Twitter was always a car crash anyway.)

[1] I hope readers won’t underestimate this the importance of editorial and language assistance that a journal can provide. Last month, researchers in Australia, Germany, Nepal, Spain, the UK, and the US had a paper published in which they reported, based on surveys, that “non-native English speakers, especially early in their careers, spend more effort than native English speakers in conducting scientific activities, from reading and writing papers and preparing presentations in English, to disseminating research in multiple languages. Language barriers can also cause them not to attend, or give oral presentations at, international conferences conducted in English.”

The language in the South Korean group’s preprints indicate that its authors’ first language isn’t English. According to Springer, which later became Springer Nature, the publisher of the Nature journals, “Editorial reasons for rejection include … poor language quality such that it cannot be understood by readers”. An undated article on Elsevier’s ‘Author Services’ page has this line: “For Marco [Casola, managing editor of Water Research], poor language can indicate further issues with a paper. ‘Language errors can sometimes ring a bell as a link to quality. If a manuscript is written in poor English the science behind it may not be amazing. This isn’t always the case, but it can be an indication.'”

But instead of palming the responsibility off to scientists, journals have an opportunity to distinguish themselves by helping researchers write better papers.

Must war have consequences for scientists?

The Journal of Molecular Structure has temporarily banned manuscript submissions from scientists working at state science institutes in Russia. The decision extends the consequences of war beyond the realm of politics, albeit to persons who have played no role in Putin’s invasion and might even have opposed it at great risk to themselves. Such reactions have been common in sports, for example, but much less so in science.

The SESAME synchrotron radiation facility in Jordan, operated by CERN and the Jordan atomic energy agency and with support from UNESCO, takes pride in promoting peace among its founding members (Bahrain, Cyprus, Egypt, Iran, Israel, Pakistan, the Palestinian Authority and Turkey). CERN in Europe, born in the aftermath of World War II, has a similar goal.

In fact, in the science-adjacent enterprise of spaceflight, the corresponding US and Russian agencies have cooperated against the shared backdrop of the International Space Station even when their respective heads of state have been at odds with each other on other issues. But as Pradeep Mohandas wrote recently, Roscosmos’s response to sanctions against Russia have disrupted space science to an unprecedented degree, including the ExoMars and the Venera D missions. Update, March 8, 2022, 7:14 pm: CERN also seems to have suspended Russia’s ‘observer’ status in the organisation and has said it will cooperate with international sanctions against the country.

Such virtues are in line with contemporary science’s aspiration to be ‘apolitical’, irrespective of whether that is humanitarian, and ‘objective’ in all respects. This is of course misguided, yet the aspiration itself persists and is often considered desirable. In this context, the decision of the editor of the Journal of Molecular Structure, Rui Fausto, to impose sanctions on scientists working at institutions funded by the Russian government for Russia’s invasion of Ukraine comes across as enlightened (even though Fausto himself calls his decision “apolitical”). But it is not.

Science in the 21st century is of course a reason of state. In various conflicts around the world, both communities and nation-states have frequently but not explicitly appropriated the fruits of civilian enterprise, especially science, to fuel and/or sustain conflicts. Nation-states have done this by vouchsafing the outcomes of scientific innovation to certain sections of the population to directly deploying such innovation on battlefields. Certain communities, such as the casteist Brahmins of Silicon Valley, misogynistic academics in big universities and even those united by their latent queerphobia, have used the structural privileges that come with participating in the scientific, or the adjacent technological, enterprise to perpetrate violence against members of “lower” castes, female students and genderqueer persons, for reasons that have nothing to do with the latter’s academic credentials.

However, the decision of the Journal of Molecular Structure is undermined by two problems with Fausto’s reasoning. First, the Russia-Ukraine conflict may be the most prominent in the world right now but it isn’t the only one. Others include the conflict in the Kashmir Valley, Israel’s occupation of Palestine, the Yemeni civil war and the oppression of Uyghur and Rohingya Muslims in South and Southeast Asia. Why haven’t Fausto et al. banned submissions from scientists working at state-sponsored institutes in India, Israel, Saudi Arabia and China? The journal’s editorial board doesn’t include any scientists affiliated with institutes in Russia or Ukraine – which suggests both that there was no nationalistic stake to ban scientists in Russia alone and that there could have been a nationalistic stake that kept the board from extending the ban to other hegemons around the world. Either way, this glaring oversight reduces the journal’s decision to grandstanding.

The second reason, and also really why Fausto’s decision shouldn’t be extended to scientists labouring in other aggressor nations, is that Russia’s president Vladimir Putin is an autocrat – as are the political leaders of the countries listed above (with the exception of Israel). As I wrote recently in an (unpublished) essay:

… we have all come across many stories in the last two  years in which reporters quoted unnamed healthcare workers and government officials to uncover important details of the Government of India’s response to the country’s COVID-19 epidemic. Without presuming to know the nature of relationships between these ‘sources’ and the respective reporters, we can say they all likely share a conflict of ethics: they are on the frontline and they are needed there, but if they speak up, they may lose their ability to stay there.

Indeed, India’s Narendra Modi government itself has refused to listen to experts or expertise, and has in fact often preempted or sought to punish scientists whom it perceives to be capable of contradicting the government’s narratives. Modi’s BJP enjoys an absolute majority in Parliament, allowing it a free hand in lawmaking, and as an authoritarian state it has also progressively weakened the country’s democratic institutions. In all, the party has absolute power in the country, which it often uses to roll over the rights of minorities and health and ecological safeguards based on science as much as to enable industrial development and public administration on its own terms. In this milieu, speaking up and out is important, but we shouldn’t kid ourselves about how much we can expect our comments to achieve.

Similarly, in Putin’s Russia, more than 4,700 scientists and science journalists recently signed an open letter protesting the invasion of Ukraine, potentially opening themselves up to persecution (the Russian government has already arrested more than 5,000 protestors). But how much of a damn does Putin give for scientists studying molecular structure in the country’s state-funded research facilities? In an ideal scenario, pinching the careers of certain people only makes sense if the country’s leader can be expected to heed their words. Otherwise, sanctions such as that being imposed by the Journal of Molecular Chemistry will have no effect except on the scientists’ work – scientists who are now caught between a despot and an inconsiderate journal.

Ultimately, Fausto’s decision would seem to be apolitical, but in a bad way. Would that it had been political, it would also have been good.Modern science surely has a difficult place in society. But in autocratic setups, there arises a pronounced difference between a science practised by the élite and the powerful, in proximity to the state and with privileged access to political power, and which would deserve sanctions such as those extended by the Journal of Molecular Structure. Then there is the science more removed from that power, still potentially being a reason of state but at the same time less “open to co-optation by the powerful and the wealthy” (source).