In the opening of my most recent post where I wrote about the essential role in science of challenging conclusions, I quoted Carl Sagan from The Demon-Haunted World (1996) about certitude being applicable only to pure mathematics. I had only read the excerpt from Sagan's book that Richard Dawkins included in his always rewarding The Oxford Book of Modern Science Writing (2008). I decided to read the book as a bit of after-the-fact due diligence. What if I'd been wrong about what Sagan was saying? (I wasn't.)
This explains why one day, recently, I had Sagan's book with me when I placed a carryout order at a nearby cafe and sat down at a table with someone else also waiting for his order to be filled. He noticed the book and asked what it was. I showed him the cover and, as I did so, someone else came and sat down, also waiting for her order.
The man said, with a relatively heavy accent, "Ah, Carl Sagan, I know of him. He was a cosmo . . . ." He struggled to complete the word.
"A cosmologist. Yes." I said.
At that moment the newcomer to our talk interjected forcefully, "No, he wasn't!!"
As I stared at this aggressive interloper to our conversation, I could feel that I was starting to blush and was at a loss for words. My first thought was that I'd butchered the word and said something like "cosmetologist" (which Sagan certainly wasn't). Then I recognized that, apparently, I had been wrong, seemingly stupid in calling him a cosmologist.
"He was an astronomer, very different from a cosmologist," she told us, with a tone that brooked no argument.
"Cosmology," I began , "... about the origins of the universe."
"Yes, a completely different field," she schooled me.
There are several explanations (excuses?) for why I said what I said, but, at that moment, I was, to use a word that will crop up later in this post, mortified by being wrong. I was carrying the man's book, for God's sake, and had not immediately placed him in the correct scientific field. What kind of poser was I?
This episode has stayed with me partly because the memory is painful, but also because I was already thinking about the scientific method where the search for wrongness apparently is unrelenting. We humans hate to be wrong and are wrong much more often than we know. We are also very bad at acknowledging when we have been wrong. Clearly, this is true for me (see episode above), but is it true for scientists whose careers are forged in the crucible of the scientific method?
The ubiquity and challenges of being wrong are described nicely by journalist and writer Kathryn Schulz in her exhaustive and perceptive Being Wrong: Adventures in the Margins of Error (2010). She explores the factors that contribute to our being wrong, the ways in which we respond to being wrong, and, surprisingly, why being wrong is a good thing. She also considers the scientific method.
Though her book is helpful in considering my minor instance of being wrong, her primary focus is on being wrong to the point of realizing a core belief of yours (scientific or otherwise) no longer holds, or finding that your world view or sense of self has come undone.
The experience of having been wrong can be excruciating. Schulz writes, "Certain mistakes can actually kill us, but, many, many more of them just make us want to die. That's why the word 'mortify' comes up so often when people talk about their errors." (p. 26)
But she goes to great lengths to convince the reader that being wrong is actually "far from being a sign of intellectual inferiority, the capacity to err is crucial to human cognition." (p. 9) It's how we learn (yes, I did learn a little something from the episode above). Further, she states, "Ultimately, then, we are transformed by error through accepting it." (p. 252) This leads to the salutary act of embracing doubt, that is, being aware of the possibility we could be wrong.
Schulz on the scientific method is quite interesting.
As an ideal of intellectual inquiry and a strategy for the advancement of knowledge, the scientific method is essentially a monument to the utility of error. Most of us gravitate toward trying to verify our beliefs, to the extent that we bother investigating their validity at all. But scientists gravitate toward falsification: as a community, if not as individuals, they seek to disprove their beliefs. Thus, the defining feature of a hypotheses is that it has the potential to be proven wrong (which is why it must be both testable and tested), and the defining feature of a theory is that it hasn't been proven wrong yet. (p. 35)
For me, the key portion of this passage regarding scientists is: "as a community, if not as individuals, they seek to disprove their beliefs." (emphasis added) That distinction is important and speaks directly to the question I posed above. Do scientists, as practitioners of the scientific method, respond differently to being found to be wrong because that is always recognized as a possibility? Schulz is positing that it's the scientific community that moves forward on this constant testing, not necessarily the individual scientist who might have been wrong.
Of this constant challenge in science, Sagan in The Demon-Haunted World writes:
If you examine science in its everyday aspect, of course you find that scientists run the gamut of human emotion, personality, and character. But there's one facet that is really striking to the outside, and that is the gauntlet of criticism considered acceptable or even desirable. (p. 31)
He then asks:
Why do we put up with it? Do we like to be criticized? No, no scientist enjoys it. Every scientist feels a proprietary affection for his or her ideas and findings. . . . [T]he hard but just rule is that if the ideas don't work, you must throw them away. (p. 32)
I take Sagan as suggesting that, given their experience in the constant testing of the scientific method, scientists, after being knocked down by finding they erred, will pick themselves up, dust themselves off, and move forward. Is that true? Is it mostly true? As I've discovered, possibly not.
My interest here isn't how scientists respond when a major scientific belief system falls apart and is replaced with another, i.e., a paradigm shift as described by scientific philosopher Thomas S. Kuhn in The Structure of Scientific Revolutions (1970). Rather, it's when a scientist finds they've been wrong about a finding or an idea, something important to them, but not upsetting of a scientific belief system. Are they just like the rest of us in their responses to this kind of error, or are they different, having been living within the embrace of the scientific method which schools them in understanding that they could be wrong?
Turns out that, upon learning they've been wrong, many scientists don't behave gracefully. Consider my favorite story about a scientist getting something very wrong. It involves paleontologists Edward Drinker Cope (1840-1897) and Othniel Charles Marsh (1831-1899), who, in the beginning, worked together amicably. Then they had a falling out and things turned nasty. (I have written about the Cope-Marsh feud several times, including the post here.)
Contributing particularly to the animosity that Cope felt toward Marsh was a specific event that, in some ways, was simply part of what happens in science. In 1868, Cope, at the time a young scientist making his way, received a jumbled mix of fossil bones of a plesiosaur (an ancient marine reptile) from a contact out west. He spent a year or more cleaning and piecing together the reptile's skeleton. He followed an established method for assembling the creature and, when finished, was convinced this was a new plesiosaur taxon, one distinguished from all others by its distinctive, long neck topped with a small skull. He named his fossil and wrote it up for publication by the American Philosophical Society.
When Marsh took a look at what Cope had put together, he let him know that he had it wrong (perhaps not gently, though that's how Marsh characterized it). He told Cope that he had placed the animal's skull on the animal's tail. Cope, according to Marsh, had things ass-backwards, literally.
Cope's first response was to be defensive and he sought to prove Marsh wrong. Neither man gave in, until the renowned and respected paleontologist Joseph Leidy (1823-1891) was brought in to settle matters. Leidy concurred with Marsh. As Mark Jaffe writes in The Gilded Dinosaur: The Fossil War Between E.D. Cope and O.C. Marsh and the Rise of American Science:
Cope was mortified. He quickly set about buying up all the copies he could find of the transactions of the American Philosophical Society, although some remained rattling around town. (p. 14)
So, Cope's first response was denial and argument, then a digging in, until it was incontrovertibly shown that he had erred. This was followed by a desperate effort to cover things up, make it seem like the error had never happened. Though this particular incident was of very low stakes for paleontology, it was of high stakes for Cope. This paper was to have been a major breakthrough for him, an establishing event. Instead, he'd been wrong and, after being called out on it, knew it.
That scientists don't always readily admit to being wrong, even if the evidence seems overwhelming that they are, is the subject of Michael D. Lemonick's article titled When Scientists Screw Up (Slate, October 15, 2012) He recounts several incidents in which scientists made significant claims only for their evidence to be shown to be wrong. Admit defeat? Hardly, some stick to their disproven guns for many years.
The admission in science of one's wrongness before being called out on it is apparently rare enough that it's a unicorn. Lemonick cites astronomer Andrew Lyne who, in a presentation about a major discovery he'd announced several months earlier, proceeded to tell his colleagues that he'd made a mistake in his calculations. The reaction? A standing ovation! Perhaps these scientists realized that what Lyne had done in admitting his error was rarer than what he originally claimed to have discovered.
Lemonick's article doesn't describe Lyne's reaction when he first realized his mistake. For a first hand account of that by someone who went on to own up to their mistake, I recommending reading the article that psychologist Julia Strand penned about being wrong. (Julia Strand, Scientists Makes Mistakes. I Made a Big One, Elemental, March 24, 2020.)
In 2019, she was a young academic up for tenure. A year earlier, she had made an exciting discovery of an intervention that significantly reduced the time needed (listening effort) by test subjects to detect speech in a noisy environment, compared to that required by control subjects without the intervention. (I have a tenuous grasp on what she did and what it might have meant.) She'd published her results and her findings prompted an NIH grant and became the basis for an app in development.
But when difficulties replicating her findings arose, she went in search for an explanation. Ultimately, she figured out that she had incorrectly programmed the timing of the testing for the control subjects: the clock had started before their testing actually began. As a result, it only appeared that the test subjects were able to detect speech quicker than the control subjects; in effect, they'd gotten a head start.
Here's how Strand describes the moment of finding she was wrong and her reactions, including one worthy of Edward Drinker Cope:
When I found the mistake, I was home alone on my laptop - working late in the evening. While I sat in the dark (crying), I briefly consider what would happen if I never told anyone. The bug was hard for me to identify; maybe no one else would ever find it. I could just go on with other research and nobody would ever know.
But clearly she doesn't do that, though the consequences she foresaw were dire.
I felt deeply embarrassed to have made such a stupid mistake, disappointed that my finding was junk, guilty for wasting everyone's time and polluting the literature, and worried that admitting the error and retracting the paper would jeopardize my job, my grant funding, and my professional reputation.
The next day, she owned up to the mistake, contacted everyone involved, including the journal that published her study and the chair of her tenure committee. Turns out that Strand received the equivalent of Lyne's standing ovation: her reputation wasn't trashed, people she contacted were supportive and comforting, and, yes, she received tenure.
Just how rare this self-correction of research results might be is suggested by the data from the Loss-of-Confidence Project which, roughly a decade ago, queried the psychology research community on the extent to which its scientists had lost confidence in the validity of one or more of the findings of their prior research and what they'd done about that, if anything. (Julia M. Rohrer, et al., Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project, Perspectives on Psychological Science, Volume 16, Number 6, 2021.)
Psychology researchers were invited to submit descriptions of their own research in which they had lost confidence. The project ultimately received just 13 statements describing research from scientists, among them one from Julia Strand. In essence, the scientists admitted publicly, perhaps for the first time, there were problems with these research findings. In addition, an online, anonymous survey garnered 316 responses about loss of confidence (not just from research scientists in psychology fields).
According to Rohrer et al., the researchers submitting statements admitted to committing methodological errors; making invalid inferences; and resorting to so-called p-hacking (in effect, cherry-picking results from multiple statistical analyses).
Of the researchers responding to the online, anonymous questionnaire, almost half had experienced a loss of confidence in one or more of their findings. The predominant reason for the loss of confidence was the recognition that they had engaged in "questionable research practices." Tellingly, only a distinct minority (just 17%) had made their loss of confidence public (and when made public, it was not often done very aggressively). Rohrer et al. acknowledge that their respondents are not statistically representative of scientific research in general, but suggest that their finding may mean that "thousands of articles published each year . . . will eventually lose the confidence of their authors - often because of known errors yet typically without any public disclosure." (p.1263)
In the article's abstract, the authors are quite blunt:
Far from evaluating new evidence dispassionately and infallibly, individual scientists often cling stubbornly to prior findings. (p. 1255)
That's only human of them.
Posts
