Tag: replication crisis

The identity of scientific papers

This prompt arose in response to Stuart Ritchie’s response to a suggestion in an editorial “first published last year but currently getting some attention on Twitter” – that scientists should write their scientific papers as if they were telling a story, with a beginning, middle and end. The act of storytelling produces something entertaining by definition, but it isn’t the same as when people build stories around what they know. That is, people build stories around what they know but that knowledge, when it is first produced, isn’t and in fact can’t be reliably produced through acts of storytelling. This is Ritchie’s point, and it’s clearly true. As Ash Jogalekar commented on Twitter on Ritchie’s post

(This is different from saying scientific knowledge shouldn’t be associated with stories – or that only it should be, a preference that philosopher of science Robert P. Crease calls “scientific gaslighting”.)

Ritchie’s objection arises from a problematic recommendation in the 2021 editorial, that when writing their papers, scientists present the “take-home messages” first, then “select” the methods and results that produced those messages, and then conclude with an introduction-discussion hybrid. To Ritchie, scientists don’t face much resistance, as they’re writing their papers, other than their own integrity that keeps them from cherry-picking from their data to support predetermined conclusions. This is perfectly reasonable, especially considering the absence of such resistance manifested in science’s sensational replication crisis.

But are scientific papers congruent with science itself?

The 2021 editorial’s authors don’t do themselves any favours in their piece, writing:

“The scientific story has a beginning, a middle, and an end. These three components can, and should, map onto the typical IMRaD structure. However, as editors we see many manuscripts that follow the IMRaD structure but do not tell a good scientific story, even when the underlying data clearly can provide one. For example, many studies present the findings without any synthesis or an effort to place them into a wider context. This limits the reader’s ability to gain knowledge and understanding, hence reducing the papers impact.”

Encouraging scientists to do such things as build tension and release it with a punchline, say, could be a recipe for disaster. The case of Brian Wansink in fact fits Ritchie’s concerns to a T. In the most common mode of scientific publishing today, narrative control is expected to lie beyond scientists – and (coming from a science journalist) lies with science journalists. Or at least: the opportunities to shape science-related narratives are available in large quantities to us.

A charitable interpretation of the editorial is that its authors would like scientists to take a step that they believe to be marginal (“right there,” as they say) in terms of the papers’ narratives but which has extraordinary benefits – but I’m disinclined. Their words hew frustratingly but unsurprisingly close to suggesting that scientists’ work isn’t properly represented in the public imagination. The most common suggestions I’ve encountered in my experience are that science journalists don’t amplify the “right” points and that they dwell on otherwise trivial shortcomings. The criticisms generally disregard the socio-political context in which science operates and to which journalists are required to be attuned.

This said, and as Ritchie also admits, the scientific paper itself is not science – so why can’t it be repurposed to ends that scientists are better off meeting than one that’s widely misguided? Ritchie writes:

“Science isn’t a story – and it isn’t even a scientific paper. The mere act of squeezing a complex process into a few thousand words … is itself a distortion of reality. Every time scientists make a decision about “framing” or “emphasis” or “take-home messages”, they risk distorting reality even further, chipping away at the reliability of what they’re reporting. We all know that many science news articles and science books are over-simplified, poorly-framed, and dumbed-down. Why push scientific papers in the same direction?”

That is, are scientific papers the site of knowledge production? With the advent of preprint papers, research preregistration and open-data and data-sharing protocols, many papers of today are radically different from those a decade or two ago. Especially online, and on the pages of more progressive journals like eLife, papers are accompanied by peer-reviewers’ comments, links to the raw data (code as well as multimedia), ways to contact the authors, a comments section, a ready-reference list of cited papers, and links to other articles that have linked to it. Sometimes some papers deemed to be more notable by a journal’s editors are also published together with commentary by an independent scientist on the papers’ implications for the relevant fields.

Scientific papers may have originated as, and for a long time have been, the ‘first expression’ of a research group’s labour to produce knowledge, and thus perfectly subject to Ritchie’s concerns about transforming them to be more engaging. But today, given the opportunities that are available in some pockets of research assessment and publishing, they’re undeniably the sites of knowledge consumption – and in effect the ‘first expression’ of researchers’ attempts to communicate with other scientists as well as, in many cases, the public at large.

It’s then effectively down to science journalists, and the resistance offered by their integrity to report on papers responsibly – although even then we should beware the “seduction of storytelling”.

I think the 2021 editorial is targetting the ‘site of knowledge consumption’ identity of the contemporaneous scientific paper, and offers ways to engage its audience better. But when the point is to improve it, why continue to work with, in Ritchie’s and the editorial’s words, a “journal-imposed word count” and structure?

A halfway point between the editorial’s recommendations and Ritchie’s objections (in his post, but more in line with his other view that we should do away with scientific papers altogether) is to publish the products of scientific labour taking full advantage of what today’s information and communication technologies allow: without a paper per se but a concise description of the methods and the findings, an explicitly labeled commentary by the researchers, the raw code, multimedia elements with tools to analyse them in real-time, replication studies, even honest (and therefore admirable) retraction reports if they’re warranted. The commentary can, in the words of the editorial, have “a beginning, a middle and an end”; and in this milieu, in the company of various other knowledge ‘blobs’, readers – including independent scientists – should be able to tell straightforwardly if the narrative fits the raw data on offer.

All this said, I must add that what I have set out here are far from where reality is at the moment; in Ritchie’s words,

“Although those of us … who’ve been immersed in this stuff for years might think it’s a bit passé to keep going on about “HARKing” and “researcher degrees of freedom” and “p-hacking” and “publication bias” and “publish-or-perish” and all the rest, the word still hasn’t gotten out to many scientists. At best, they’re vaguely aware that these problems can ruin their research, but don’t take them anywhere near seriously enough.”

I don’t think scientific papers are co-identifiable with science itself, or they certainly needn’t be. The latter is concerned with reliably producing knowledge of increasingly higher quality while the former explains what the researchers did, why, when and how. Their goals are different, and there’s no reason the faults of one should hold the other back. However, a research communication effort that has completely and perfectly transitioned to embodying the identity of the modern research paper (an anachronism) as the site of, among other things, knowledge consumption is a long way away – but it helps to bear it in mind, to talk about it and to improve it.

The calculus of creative discipline

Every moment of a science fiction story must represent the triumph of writing over world-building. World-building is dull. World-building literalises the urge to invent. World-building gives an unnecessary permission for acts of writing (indeed, for acts of reading). World-building numbs the reader’s ability to fulfil their part of the bargain, because it believes that it has to do everything around here if anything is going to get done. Above all, world-building is not technically necessary. It is the great clomping foot of nerdism.

Once I’m awake and have had my mug of tea, and once I’m done checking Twitter, I can quote these words of M. John Harrison from memory: not because they’re true – I don’t believe they are – but because they rankle. I haven’t read any writing of Harrison’s, I can’t remember the names of any of his books. Sometimes I don’t remember his name even, only that there was this man who uttered these words. Perhaps it is to Harrison’s credit that he’s clearly touched a nerve but I’m reluctant to concede anymore than this.

His (partial) quote reflects a narrow view of a wider world, and it bothers me because I remain unable to extend the conviction that he’s seeing only a part of the picture to the conclusion that he lacks imagination; as a writer of not inconsiderable repute, at least according to Wikipedia, I doubt he has any trouble imagining things.

I’ve written about the virtues of world-building before (notably here), and I intend to make another attempt in this post; I should mention what both attempts, both defences, have in common is that they’re not prescriptive. They’re not recommendations to others, they’re non-generalisable. They’re my personal reasons to champion the act, even art, of world-building; my specific loci of resistance to Harrison’s contention. But at the same time, I don’t view them – and neither should you – as inviolable or as immune to criticism, although I suspect this display of a willingness to reason may not go far in terms of eliminating subjective positions from this exercise, so make of it what you will.

There’s an idea in mathematical analysis called smoothness. Let’s say you’ve got a curve drawn on a graph, between the x- and y-axes, shaped like the letter ‘S’. Let’s say you’ve got another curve drawn on a second graph, shaped like the letter ‘Z’. According to one definition, the S-curve is smoother than the Z-curve because it has fewer sharp edges. A diligent high-schooler might take recourse through differential calculus to explain the idea. Say the Z-curve on the graph is the result of a function Z(x) = y. If you differentiate Z(x) where ‘x’ is the point on the x-axis where the Z-curve makes a sharp turn, the derivative Z'(x) has a value of zero. Such points are called critical points. The S-curve doesn’t have any critical points (except at the ends, but let’s ignore them); L-, and T-curves have one critical point each; P- and D-curves have two critical points each; and an E-curve has three critical points.

With the help of a loose analogy, you could say a well-written story is smooth à la an S-curve (excluding the terminal points): it it has an unambiguous beginning and an ending, and it flows smoothly in between the two. While I admire Steven Erikson’s Malazan Book of the Fallen series for many reasons, its first instalment is like a T-curve, where three broad plot-lines abruptly end at a point in the climax that the reader has been given no reason to expect. The curves of the first three books of J.K. Rowling’s Harry Potter series resemble the tangent function (from trigonometry: tan(x) = sin(x)/cosine(x)): they’re individually somewhat self-consistent but the reader is resigned to the hope that their beginnings and endings must be connected at infinity.

You could even say Donald Trump’s presidency hasn’t been smooth at all because there have been so many critical points.

Where world-building “literalises the urge to invent” to Harrison, it spatialises the narrative to me, and automatically spotlights the importance of the narrative smoothness it harbours. World-building can be just as susceptible to non-sequiturs and deus ex machinae as writing itself, all the way to the hubris Harrison noticed, of assuming it gives the reader anything to do, even enjoy themselves. Where he sees the “clomping foot of nerdism”, I see critical points in a curve some clumsy world-builder invented as they went along. World-building can be “dull” – or it can choose to reveal the hand-prints of a cave-dwelling people preserved for thousands of years, and the now-dry channels of once-heaving rivers that nurtured an ancient civilisation.

My principal objection to Harrison’s view is directed at the false dichotomy of writing and world-building, and which he seems to want to impose instead of the more fundamental and more consequential need for creative discipline. Let me borrow here from philosophy of science 101, specifically of the particular importance of contending with contradictory experimental results. You’ve probably heard of the replication crisis: when researchers tried to reproduce the results of older psychology studies, their efforts came a cropper. Many – if not most – studies didn’t replicate, and scientists are currently grappling with the consequences of overturning decades’ worth of research and research practices.

This is on the face of it an important reality check but to a philosopher with a deeper view of the history of science, the replication crisis also recalls the different ways in which the practitioners of science have responded to evidence their theories aren’t prepared to accommodate. The stories of Niels Bohr v. classical mechanics, Dan Shechtman v. Linus Pauling and the EPR paradox come first to mind. Heck, the philosophers Karl Popper, Thomas Kuhn, Imre Lakatos and Paul Feyerabend are known for their criticisms of each other’s ideas on different ways to rationalise the transition from one moment containing multiple answers to the moment where one emerges as the favourite.

In much the same way, the disciplined writer should challenge themself instead of presuming the liberty to totter over the landscape of possibilities, zig-zagging between one critical point and the next until they topple over the edge. And if they can’t, they should – like the practitioners of good science – ask for help from others, pressing the conflict between competing results into the service of scouring the rust away to expose the metal.

For example, since June this year, I’ve been participating on my friend Thomas Manuel’s initiative in his effort to compose an underwater ‘monsters’ manual’. It’s effectively a collaborative world-building exercise where we take turns to populate different parts of a large planet with sizeable oceans, seas, lakes and numerous rivers with creatures, habitats and ecosystems. We broadly follow the same laws of physics and harbour substantially overlapping views of magic, but we enjoy the things we invent because they’re forced through the grinding wheels of each other’s doubts and curiosities, and the implicit expectation of one creator to make adequate room for the creations of the other.

I see it as the intersection of two functions: at first, their curves will criss-cross at a point, and the writers must then fashion a blending curve so a particle moving along one can switch to the other without any abruptness, without any of the tired melodrama often used to mask criticality. So the Kularu people are reminded by their oral traditions to fight for their rivers, so the archaeologists see through the invading Gezmin’s benevolence and into the heart of their imperialist ambitions.