Retraction Watch – Close Read

Why it’s important to address plagiarism

Plagiarism is a tricky issue. If it’s straightforward to you, ask yourself if you’re assuming that the plagiariser (plagiarist?) is fluent in reading and writing, but especially writing, English. The answer’s probably ‘yes’. This is because for someone entering into an English-using universe for the first time, certain turns of phrase and certain ways to articulate complicated concepts stick with you the first time you read them, and when the time comes for you to spell out the same ideas and concepts, you passively, inadvertently recall them and reuse them. You don’t think – at least at first – that they’re someone else’s words, more so if you haven’t been taught, for no fault of yours, what academic plagiarism is and/or that it’s bad.

This is also why there’s a hierarchy of plagiarism. For example, if you’re writing a scientific paper and you copy another paper’s results, that’s worse than if you copy verbatim the explanation of a certain well-known idea. This is why former University Grants Commission chairman Praveen Chaddah wrote in 2014:

There are worse offences than text plagiarism — such as taking credit for someone else’s research ideas and lifting their results. These are harder to detect than copy-and-pasted text, so receive less attention. This should change. To help, academic journals could, for instance, change the ways in which they police and deal with such cases.

But if you’re fluent with writing English, if you know what plagiarism and plagiarise anyway (without seeking resources to help you beat its temptation), and/or if you’re stealing someone else’s idea and calling it your own, you deserve the flak and (proportionate) sanctions coming your way. In this context, a new Retraction Watch article by David Sanders makes for interesting reading. According to Sanders, in 2018, he wrote to the editors of a journal that had published a paper in 2011 with lots of plagiarised text. After a back-and-forth, the editors told Sanders they’d look into it. He asked them again in 2019 and May 2021 and received the same reply on both occasions. Then on July 26 the journal published a correction to the 2011 article. Sanders wasn’t happy and wrote back to the editors, one of whom replied thus:

Thank you for your email. We went through this case again, and discussed whether we may have made the wrong decision. We did follow the COPE guidelines step by step and used several case studies for further information. This process confirmed that an article should be retracted when it is misleading for the reader, either because the information within is incorrect, or when an author induces the reader to think that the data presented is his own. As this is a Review, copied from other Reviews, the information within does not per se mislead the reader, as the primary literature is still properly cited. We agree that this Review was not written in a desirable way, and that the authors plagiarised a large amount of text, but according to the guidelines the literature must be considered from the point of view of the reader, and retractions should not be used as a tool to punish authors. We therefore concluded that a corrigendum was the best way forward. Hence, we confirm our decision on this case.
Thank you again for flagging this case in the first place, which allowed us to correct the record and gain deeper insights into publishing ethics, even though this led to a solution we do not necessarily like.

Sanders wasn’t happy: he wrote on Retraction Watch that “the logic of [the editor’s] message is troubling. The authors engaged in what is defined by COPE (the Committee on Publication Ethics) as ‘Major Plagiarism’ for which the prescribed action is retraction of the published article and contacting the institution of the authors. And yet the journal did not retract.” The COPE guidelines summarise the differences between minor and major plagiarism this way:

Source: https://publicationethics.org/files/COPE_plagiarism_disc%20doc_26%20Apr%2011.pdf

Not being fluent in English could render the decisions made using this table less than fair, for example because an author could plagiarise several paragraphs but honestly have no intention to deceive – simply because they didn’t think they needed to be that careful. I know this might sound laughable to a scientist operating in the US or Europe, out of a better-run, better-organised and better-funded institute, and who has been properly in the ins and outs of academic ethics. But it’s true: the bulk of India’s scientists work outside the IITs, IISERs, DAE/DBT/DST-funded institutes and the more progressive private universities (although only one – Ashoka – comes to mind). Their teachers before them worked in the same resource-constrained environments, and for most of whom the purpose of scientific work wasn’t science as much as an income. Most of them probably never used plagiarism-checking tools either, at least not until they got into trouble one time and then found out about such things.

I myself found out about the latter in an interesting way – when I reported that Appa Rao Podile, the former vice-chancellor of the University of Hyderabad, had plagiarised in some of his papers, around the time students at the university were protesting the university’s response to the death of Rohith Vemula. When I emailed Podile for his response, he told me he would like my help with the tools with which he could spot plagiarism. I thought he was joking, but after a series of unofficial enquiries over the next year or so, I learnt that plagiarism-checking software was not at all the norm, even if solutions like Copyscape were relatively cheap, in state-funded colleges and second-tier universities around the country. I had no reason to leave Podile off the hook – but not because he hadn’t used plagiarism-checking software but because he was a vice-chancellor of a major university and had to have done better than claim ignorance.

(I also highly recommend this November 2019 article in The Point, asking whether plagiarism is wrong.)

According to Sanders, the editor who replied didn’t retract the paper because he thought it wasn’t ‘major plagiarism’, according to COPE – whereas Sanders thought it was. The editor appears to have reasoned his way out of the allegation, in the editor’s view at least, by saying that the material printed in the paper wasn’t misleading because it had been copied from non-misleading original material and that the supposedly lesser issue was that while it had been cited, it hadn’t been syntactically attributed as such (placed between double quotes, for example). The issue for Sanders, with whom I agree here, is that the authors had copied the material and presented it in a way that indicated they were its original creators. The lengths to which journal editors can go to avoid retracting papers, and therefore protect their journal’s reputation, ranking or whatever, is astounding. I also agree with Sanders when he says that by refusing to retract the article, the editors are practically encouraging misconduct.

I’d like to go a step further and ask: when journal editors think like this, where does that leave Indian scientists of the sort I’ve described above – who are likely to do better with the right help and guidance? In 2018, Rashmi Raniwala and Sudhir Raniwala wrote in The Wire Science that the term ‘predatory’, in ‘predatory journals’, was a misnomer:

… it is incorrect to call them ‘predatory’ journals because the term predatory suggests that there is a predator and a victim. The academicians who publish in these journals are not victims; most often, they are self-serving participants. The measure of success is the number of articles received by these journals. The journals provide a space to those who wanted easy credit. And a large number of us wanted this easy credit because we were, to begin with, not suitable for the academic profession and were there for the job. In essence, these journals could not have succeeded without an active participation and the connivance of some of us.

It was a good article at the time, especially in the immediate context of the Raniwalas’ fight to have known defaulters suitably punished. There are many bad-faith actors in the Indian scientific community and what the Raniwalas write about applies to them without reservation (ref. the cases of Chandra Krishnamurthy, R.A. Mashelkar, Deepak Pental, B.S. Rajput, V. Ramakrishnan, C.N.R. Rao, etc.). But I’m also confident enough to say now that predatory journals exist, typified by editors who place the journal before the authors of the articles that constitute it, who won’t make good-faith efforts to catch and correct mistakes at the time they’re pointed out. It’s marginally more disappointing that the editor who replied to Sanders replied at all; most don’t, as Elisabeth Bik has repeatedly reminded us. He bothered enough to engage – but not enough to give a real damn.

NCBS fracas: In defence of celebrating retractions

Continuing from here…

Irrespective of Arati Ramesh’s words and actions, I find every retraction worth celebrating because how hard-won retractions in general have been, in India and abroad. I don’t know how often papers coauthored by Indian scientists are retracted and how high or low that rate is compared to the international average. But I know that the quality of scientific work emerging from India is grossly disproportionate (in the negative sense) to the size of the country’s scientific workforce, which is to say most of the papers published from India, irrespective of the journal, contain low-quality science (if they contain science at all). It’s not for nothing that Retraction Watch has a category called ‘India retractions’, with 196 posts.

Second, it’s only recently that the global scientific community’s attitude towards retractions started changing, and even now most of it is localised to the US and Europe. And even there, there is a distinction: between retractions for honest mistakes and those for dishonest mistakes. Our attitudes towards retractions for honest mistakes have been changing. Retractions for dishonest conduct, or misconduct, have in fact been harder to secure, and continue to be.

The work of science integrity consultant Elisabeth Bik allows us a quick take: the rate at which sleuths are spotting research fraud is far higher than the rate at which journals are retracting the corresponding papers. Bik herself has often said on Twitter and in interviews how most journals editors simply don’t respond to complaints, or quash them with weak excuses and zero accountability. Between 2015 and 2019, a group of researchers identified papers that had been published in violation of the CONSORT guidelines in journals that endorsed the same guidelines, and wrote to those editors. From The Wire Science‘s report:

… of the 58 letters sent to the editors, 32 were rejected for different reasons. The BMJ and Annals published all of those addressed to them. The Lancet accepted 80% of them. The NEJM and JAMA turned down every single letter.

According to JAMA, the letters did not include all the details it required to challenge the reports. When the researchers pointed out that JAMA’s word limit for the letter precluded that, they never heard back from the journal.

On the other hand, NEJM stated that the authors of reports it published were not required to abide by the CONSORT guidelines. However, NEJM itself endorses CONSORT.

The point is that bad science is hard enough to spot, and getting stakeholders to act on them is even harder. It shouldn’t have to be, but it is. In this context, every retraction is a commendable thing – no matter how obviously warranted it is. It’s also commendable when a paper ‘destined’ for retraction is retracted sooner (than the corresponding average) because we already have some evidence that “papers that scientists couldn’t replicate are cited more”. Even if a paper in the scientific literature dies, other scientists don’t seem to be able to immediately recognise that it is dead and cite it in their own work as evidence of this or that thesis. These are called zombie citations. Retracting such papers is a step in the right direction – insufficient to prevent all sorts of problems associated with endeavours to maintain the quality of the literature, but necessary.

As for the specific case of Arati Ramesh: she defended her group’s paper on PubPeer in two comments that offered more raw data and seemed to be founded on a conviction that the images in the paper were real, not doctored. Some commentators have said that her attitude is a sign that she didn’t know the images had been doctored while some others have said (and I tend to agree) that this defence of Ramesh is baffling considering both of her comments succeeded detailed descriptions of forgery. Members of the latter group have also said that, in effect, Ramesh tried to defend her paper until it was impossible to do so, at which point she published her controversial personal statement in which she threw one of her lab’s students under the bus.

There are a lot of missing pieces here towards ascertaining the scope and depth of Ramesh’s culpability – given also that she is the lab’s principal investigator (PI), that she is the lab’s PI who has since started to claim that her lab doesn’t have access to the experiments’ raw data, and that the now-retracted paper says that she “conceived the experiments, performed the initial bioinformatic search for Sensei RNAs, supervised the work and wrote the manuscript”.

[Edit, July 11, 2021, 6:28 pm: After a conversation with Priyanka Pulla, I edited the following paragraph. The previous version appears below, struck through.]

Against this messy background, are we setting a low bar by giving Arati Ramesh brownie points for retracting the paper? Yes and no… Even if it were the case that someone defended the indefensible to an irrational degree, and at the moment of realisation offered to take the blame while also explicitly blaming someone else, the paper was retracted. This is the ‘no’ part. The ‘yes’ arises from Ramesh’s actions on PubPeer, to ‘keep going until one can go no longer’, so to speak, which suggests, among other things – and I’m shooting in the dark here – that she somehow couldn’t spot the problem right away. So giving her credit for the retraction would set a low, if also weird, bar; I think credit belongs on this count with the fastidious commenters of PubPeer. Ramesh would still have had to sign off on a document saying “we’ve agreed to have the paper retracted”, as journals typically require, but perhaps we can also speculate as to whom we should really thank for this outcome – anyone/anything from Ramesh herself to the looming threat of public pressure.

Against this messy background, are we setting a low bar by giving Arati Ramesh brownie points for retracting the paper? No. Even if it were the case that someone defended the indefensible to an irrational degree, and at the moment of realisation offered to take the blame while also explicitly blaming someone else, the paper was retracted. Perhaps we can speculate as to whom we should thank for this outcome – Arati Ramesh herself, someone else in her lab, members of the internal inquiry committee that NCBS set up, some others members of the institute or even the looming threat of public pressure. We don’t have to give Ramesh credit here beyond her signing off on the decision (as journals typically require) – and we still need answers on all the other pieces of this puzzle, as well as accountability.

A final point: I hope that the intense focus that the NCBS fracas has commanded – and could continue to considering Bik has flagged one more paper coauthored by Ramesh and others have flagged two coauthored by her partner Sunil Laxman (published in 2005 and 2006), both on PubPeer for potential image manipulation – will widen to encompass the many instances of misconduct popping up every week across the country.

NCBS, as we all know, is an elite institute as India’s centres of research go: it is well-funded (by the Department of Atomic Energy, a government body relatively free from bureaucratic intervention), staffed by more-than-competent researchers and students, has published commendable research (I’m told), has a functional outreach office, and whose scientists often feature in press reports commenting on this or that other study. As such, it is overrepresented in the public imagination and easily gets attention. However, the problems assailing NCBS vis-à-vis the reports on PubPeer are not unique to the institute, and should in fact force us to rethink our tendency (mine included) to give such impressive institutes – often, and by no coincidence, Brahmin strongholds – the benefit of the doubt.

(1. I have no idea how things are at India’s poorly funded state and smaller private universities, but even there, and in fact at the overall less-elite and but still “up there” in terms of fortunes, institutes, like the IISERs, Brahmins have been known to dominate the teaching and professorial staff, if not the students, and still have been found guilty of misconduct, often sans accountability. 2. There’s a point to be made here about plagiarism, the graded way in which it is ‘offensive’, access to good quality English education to people of different castes in India, a resulting access to plus inheritance of cultural and social capital, and the funneling of students with such capital into elite institutes.)

As I mentioned earlier, Retraction Watch has an ‘India retractions’ category (although to be fair, there are also similar categories for China, Italy, Japan and the UK, but not for France, Russia, South Korea or the US. These countries ranked 1-10 on the list of countries with the most scientific and technical journal publications in 2018.) Its database lists 1,349 papers with at least one author affiliated with an Indian institute that have been retracted – and five papers since the NCBS one met its fate. The latest one was retracted on July 7, 2021 (after being published on October 16, 2012). Again, these are just instances in which a paper was retracted. Further up the funnel, we have retractions that Retraction Watch missed, papers that editors are deliberating on, complaints that editors have rejected, complaints that editors have ignored, complaints that editors haven’t yet received, and journals that don’t care.

So, retractions ~~– and retractors –~~ deserve brownie points.

Dealing with plagiarism? Look at thy neighbour

Four doctors affiliated with Kathmandu University (KU) in Nepal are going to be fired because they plagiarised data in two papers. The papers were retracted last year from the Bali Medical Journal, where they had been published. A dean at the university, Dipak Shrestha, told a media outlet that the matter will be settled within two weeks. A total of six doctors, including the two above, are also going to be blacklisted by the journal. This is remarkably swift and decisive action against a problem that refuses to go away in India for many reasons. But I’m not an apologist; one of those reasons is that many teachers at colleges and universities seem to think “plagiarism is okay”. And for as long as that attitude persists, academicians are going to be able to plagiarise and flourish in the country.

One of the other reasons plagiarism is rampant in India is the language problem. As Praveen Chaddah, a former chairman of the University Grants Commission, has written, there is a form of plagiarism that can be forgiven – the form at play when a paper’s authors find it difficult to articulate themselves in English but have original ideas all the same. The unforgivable form is when the ideas are plagiarised as well. According to a retraction notice supplied by the Bali Medical Journal, the KU doctors indulged in plagiarism of the unforgivable kind, and were duly punished. In India, however, I’m yet to hear of an instance where researchers found to have been engaging in such acts were pulled up as swiftly as their Nepali counterparts were, or had sanctions imposed on their work within a finite period and in a transparent manner.

The production and dissemination of scientific knowledge should not have to suffer because some scientists aren’t fluent with a language. Who knows, India might already be the ‘science superpower’ everyone wants it to be if we’re able to account for information and knowledge produced in all its languages. But this does not mean India’s diversity affords it the license to challenge the use of English as the de facto language of science; that would be stupid. English is prevalent, dominant, even hegemonic (as K. VijayRaghavan has written). So if India is to make it to the Big League, then officials must consider doing these things:

Inculcate the importance of communicating science. Writing a paper is also a form of communication. Teach how to do it along with technical skills.
Set aside money – as some Australian and European institutions do¹ – to help those for whom English isn’t their first, or even second, language write papers that will be appreciated for their science instead of rejected for their language (unfair though this may be).
DO WHAT NEPAL IS DOING – Define reasonable consequences for plagiarising (especially of the unforgivable kind), enumerate them in clear and cogent language, ensure these sanctions are easily accessible by scientists as well as the public, and enforce them regularly.

Researchers ought to know better – especially the more prominent, more influential ones. The more well-known a researcher is, the less forgivable their offence should be, at least because they set important precedents that others will follow. And to be able to remind them effectively when they act carelessly, an independent body should be set up at the national level, particularly for institutions funded by the central government, instead of expecting the offender’s host institution to be able to effectively punish someone well-embedded in the hierarchy of the institution itself.

1. Hat-tip to Chitralekha Manohar.

Featured image credit: xmex/Flickr, CC BY 2.0.

A conference’s peer-review was found to be sort of random, but whose fault is it?

It’s not a good time for peer-review. Sure, if you’ve been a regular reader of Retraction Watch, it’s never been a good time for peer-review. But aside from that, the process has increasingly been taking the brunt for not being able to stem the publishing of results that – after publication – have been found to be the product of bad research practices.

The problem may be that the reviewers are letting the ‘bad’ papers through but the bigger issue is that, while the system itself has been shown to have many flaws – not excluding personal biases – journals rely on the reviewers and naught else to stamp accepted papers with their approval. And some of those stamps, especially from Nature or Science, are weighty indeed. Now add to this muddle the NIPS wrangle, where researchers may have found that some peer-reviews are just arbitrary.

NIPS stands for the Neural Information Processing Systems (Foundation), whose annual conference was held in the second week of December 2014, in Montreal. It’s considered one of the few main conferences in the field of machine-learning. Around the time, two attendees – Corinna Cortes and Neil Lawrence – performed an experiment to judge how arbitrary the conference’s peer-review could get.

Their modus operandi was simple. All the papers submitted to the conference were peer-reviewed before they were accepted. Cortes and Lawrence then routed a tenth of all submitted papers through a second peer-review stage, and observed which papers were accepted or rejected in the second stage (According to Eric Price, NIPS ultimately accepted a paper if either group of reviewers accepted it). Their findings were distressing.

About 57%* of all papers accepted in the first review were rejected during the second review. To be sure, each stage of the review was presumably equally competent – it wasn’t as if the second stage was more stringent than the first. That said, 57% is a very big number. More than five times out of 10, peer-reviewers disagreed on what could be published. In other words, in an alternate universe, the same conference but with only the second group of reviewers in place was generating different knowledge.

Lawrence was also able to eliminate a possibly redeeming confounding factor, which he described in a Facebook discussion on this experiment:

… we had a look through the split decisions and didn’t find an example where the reject decision had found a ‘critical error’ that was missed by the accept. It seems that there is quite a lot of subjectivity in these things, which I suppose isn’t that surprising.

It doesn’t bode well that the NIPS conference is held in some esteem among its attendees for having one of the better reviewing processes. Including the 90% of the papers that did not go through a second peer-review, the total predetermined acceptance rate was 22%, i.e. reviewers were tasked with accepting 22 papers out of every 100 submitted. Put another way, the reviewers were rejecting 78%. And this sheds light on the more troubling perspective of their actions.

If the reviewers had been randomly rejecting a paper, they would’ve done so at the tasked rate of 78%. At NIPS, one can only hope that they weren’t – so the second group was purposefully rejecting 57% of the papers that the first group had accepted. In an absolutely non-random, logical world, this number should have been 0%. So, that 57% is closer to 78% than is 0% implies some of the rejection was random. Hmm.

While this is definitely cause for concern, forging ahead on the basis of arbitrariness – which machine-learning theorist John Langford defines as the probability that the second group rejects a paper that the first group has accepted – wouldn’t be the right way to go about it. This is similar to the case with A/B-testing: we have a test whose outcome can be used to inform our consequent actions, but using the test itself as a basis for the solution wouldn’t be right. For example, the arbitrariness can be reduced to 0% simply by having both groups accept every n^th paper – a meaningless exercise.

Is our goal to reduce the arbitrariness to 0% at all? You’d say ‘Yes’, but consider the volume of papers being submitted to important conferences like NIPS and the number of reviewer-hours being available to evaluate them. In the history of conferences, surely some judgments must have been arbitrary for the reviewer to have fulfilled his/her responsibilities to his/her employer. So you see the bigger issue: it’s not all the reviewer as much as it’s also the so-called system that’s flawed.

Langford’s piece raises a similarly confounding topic:

Perhaps this means that NIPS is a very broad conference with substantial disagreement by reviewers (and attendees) about what is important? Maybe. This even seems plausible to me, given anecdotal personal experience. Perhaps small highly-focused conferences have a smaller arbitrariness?

Problems like these are necessarily difficult to solve because of the number of players involved. In fact, it wouldn’t be entirely surprising if we found that nobody or no institution was at fault except how they were all interacting with each other, and not just in fields like machine-learning. A study conducted in January 2015 found that minor biases during peer-review could result in massive changes in funding outcomes if the acceptance rate was low – such as with the annual awarding of grants by the National Institutes of Health. Even Nature is wary about the ability of its double-blind peer-review to solve the problems ailing normal ‘peer-review’.

Perhaps for the near future, the only takeaway is likely going to be that ambitious young scientists are going to have to remember that, first, acceptance – just as well as rejection – can be arbitrary and, second, that the impact factor isn’t everything. On the other hand, it doesn’t seem possible in the interim to keep from lowering our expectations of peer-reviewing itself.

*The number of papers routed to the second group after the first was 166. The overall disagreement rate was 26%, so they would have disagreed on the fates of 43. And because they were tasked with accepting 22% – which is 37 or 38 – group 1 could be said to have accepted 21 that group 2 rejected, and group 2 could be said to have accepted 22 that group 1 rejected. Between 21/37 (56.7%) and 22/38 (57.8%) is 57%.

Hat-tip: Akshat Rathi.