Wrong By DAVID H. FREEDMAN, Introduction

Wrong By DAVID H. FREEDMAN

Introduction

Success consists of going from failure to failure without loss of enthusiasm.
— Winston Churchill

     I’m sitting in a coffee shop in a pediatric hospital in Boston, hard by a nine-foot-tall bronze teddy bear, with a man who is going to perform a surprising trick. I’m thinking of an article recently published in a prestigious medical journal, an article that reports the results of a research study, and he will tell me whether or not the study is likely to turn out to be right or wrong. It’s the sort of study that your doctor might read about, and that you might learn about from a newspaper, website, or morning TV news show. It may well be that the results of this study will change your life — they might convince you to start eating or avoiding certain foods to lower your risk of heart disease, or to take a certain drug to help you beat cancer, or to learn whether or not you are carrying a gene linked to vulnerability to a mental illness. But this man won’t need to hear any of the particulars of the study to perform his feat. All he needs to know is that it was a study published in a top journal.

     His prediction: it’s wrong. It’s a prediction that strikes at the foundation of expertise and our trust in it.

     The man is John Ioannidis, a doctor and researcher whose specialty is calculating the chances that studies’ results are false. For someone dedicated to spotlighting the inadequacies of his colleagues’ lifework, Ioannidis is pleasant, polite, and soft-spoken, even if he discreetly radiates the fidgety energy of someone who habitually packs too much into his day. He looks young for a man heading into his midforties, with a slight build, a wavy mop of fine, dark hair, and a thin mustache. Also a bit surprising about Ioannidis is that he is highly regarded by his peers. Communities usually find ways to marginalize those who expose their flaws, but the world of medical research, in which extraordinary talent and effort are prerequisites for attaining even the lowest rungs of recognition, has kept Ioannidis in demand via the field’s standard trappings of success: prestigious appointments, including one at the world-class Tufts–New England Medical Center and another at the University of Ioannina Medical School in his native Greece; frequent citations by colleagues of his work, some of which has been published in the field’s top journals; and a stream of invitations to speak at conferences, where he is generally a big draw.

     There’s no standard career path to becoming a deconstructor of wrongness, and Ioannidis took a roundabout route to it. Born in 1965 in the United States to parents who were both physicians, he was raised in Athens, where he showed unusual aptitude in mathematics and snagged Greece’s top student math prize. By the end of college, he seemed on track for a career as a mathematician. But he had come to feel the family pull of medicine and, not wanting to turn his back on math, decided to combine the two and become a medical mathematician. “I didn’t know exactly what such a thing might be,” he says, “but I felt sure there was some important component of medicine that was mathematical.” He graduated first in his class at the University of Athens Medical School, then shipped off to Harvard for his residency in internal medicine, followed by a research and clinical appointment at Tufts in infectious diseases. The math had to this point remained in the background, but in 1993, while at Tufts, he saw his chance to even

things up a bit. There was growing interest in the new field of “evidence-based medicine” — that is, trying to equip physicians to do not merely what they had been taught to assume would help patients but what had been rigorously proven in studies would help patients. “Amazingly, most medical treatment simply isn’t backed up by good, quantitative evidence,” says Ioannidis — news that would likely come as a surprise to most patients. Distilling this sort of knowledge out of a chaos of patient data often requires more statistical-analysis firepower than clinical researchers bear, providing an opening for Ioannidis to make a mark.

     Carrying his new interest to joint appointments at the National Institutes of Health and Johns Hopkins in the mid-1990s, Ioannidis began to look for interesting patterns in those medical-journal studies that explore how patients fare with certain treatments. Such studies are essentially the coin of the realm when it comes to communicating solid evidence of treatment effectiveness to physicians. A good doctor, it is presumed, scans the journals for the results of these studies to see what works and what doesn’t on which patients, and how well and with what risks, modifying her practices accordingly. Does it make sense to prescribe an antibiotic to a child with an ear infection? Should middle-aged men with no signs of heart disease be told to take a small, daily dose of aspirin? Do the potential benefits of a particular surgical intervention outweigh the risks? Studies presumably provide the answers. In examining hundreds of these studies, Ioannidis did indeed spot a pattern — a disturbing one. When a study was published, often it was only a matter of months, and at most a few years, before other studies came out to either fully refute the findings or declare that the results were “exaggerated” in the sense that later papers revealed significantly lesser benefits to the treatment under study. Results that held up were outweighed two-to-one by results destined to be labeled “never mind.”

     What was going on here? The whole point of carrying out a study was to rigorously examine a question using tools and techniques that would yield solid data, allowing a careful and conclusive analysis that would replace the conjecture, assumptions, and sloppy assessments that had preceded it. The data were supposed to be the path to truth. And yet these studies, and most types of studies Ioannidis looked at, were far more often than not driving to wrong answers. They exhibited the sort of wrongness rate you would associate more with fad-diet tips, celebrity gossip, or political punditry than with state-of-the-art medical research.

     The two-out-of-three wrongness rate Ioannidis found is worse than it sounds. He had been examining only the less than one-tenth of one percent of published medical research that makes it to the most prestigious medical journals. In other words, in determining that two-thirds of published medical research is wrong, Ioannidis is offering what can easily be seen as an extremely optimistic assessment. Throw in the presumably less careful work from lesser journals, and take into account the way the results end up being spun and misinterpreted by university and industrial PR departments and by journalists, and it’s clear that whatever it was about expert wrongness that Ioannidis had stumbled on in these journals, the wrongness rate would only worsen from there.

     Ioannidis felt he was confronting a mystery that spoke to the very foundation of medical wisdom. How can the research community claim to know what it’s doing, and to be making significant progress, if it can’t bring out studies in its top journals that correctly prove anything, or lead to better patient care? It was as if he had set out to improve the battle effectiveness of a navy and immediately discovered that most of its boats didn’t float. Nor did the problems appear to be unique to medicine: looking at other branches of science, including chemistry, physics, and psychology, he found much the same. “The facts suggest that for many, if not the majority, of fields, the majority of published studies are likely to be wrong,” he says. Probably, he adds, “the vast majority.”

     Medical and other scientific expertise aren’t exactly the bottom of the barrel when it comes to expert wisdom. Yes, much-heralded drugs get yanked off the market, we get conflicting advice about what to eat, and toxic chemicals make their way into our homes. But you don’t have to dig far in pretty much any other field to see similar, or worse, arrays of screwups. I could fill this entire book, and several more, with examples of expertise gone wrong — not only in medicine but in physics, finance, child raising, the government, sports, entertainment, and on and on. (Just for fun, I’ve stuck a small sampling in Appendix 1.) The fact is, expert wisdom usually turns out to be at best highly contested and ephemeral, and at worst flat-out wrong.

     Of course, compiling anecdotes and quoting experts about expertise doesn’t prove that experts usually mislead us. Actually, proving expert wrongness isn’t really the point of this book. I’ve found that most people don’t need much convincing that experts are usually wrong. How could we not suspect that to be the case? We constantly hear experts contradict one another and even themselves on a vast range of issues, whether they’re spouting off on diets, hurricane preparedness, the secrets to being a great manager, the stock market, cholesterol-lowering drugs, getting kids to sleep through the night, the inevitability of presidential candidates, the direction of home values, the key to strong marriages, vitamins, the benefits of alcohol or aspirin or fish, the existence of weapons of mass destruction, and so on. As the world watched its financial institutions and economies teetering and in some cases collapsing in 2008 and 2009, many found it maddening that the great majority of financial experts, from those who advise heads of state to those who advise working stiffs, not only failed to foresee the trouble but in many cases specifically took to the airwaves to counsel that there wasn’t much to worry about, and in general failed to have anything consistent and helpful to say about the problems. We can all agree that there is a growing obesity epidemic, but it sometimes seems as if no two experts agree on what works when it comes to losing the excess weight. And those of us who hope to see our children’s schools improve can choose between experts who say that the curricula need to be less rigid and test-oriented, and experts who say precisely the opposite. If anything, we live in a time of acute frustration with experts, even as many of us remain dependent on them and continue to heed their advice.

     Putting trust in experts who are probably wrong is only part of the problem. The other side of the coin is that many people have all but given up on getting good advice from experts. The total effect of all the contradicting and shifting pronouncements is to make expert conclusions at times sound like so much blather — a background noise of modern life. I think by now most of us have at some point caught ourselves thinking, or at least have heard from people around us, something along these lines: Experts! One day they say vitamin X / coffee / wine / drug Y / a big mortgage / baby learning videos / Six Sigma / multitasking / clean homes / arguing / investment Z is a good thing, and the next they say it’s a bad thing. Why bother paying attention? I might as well just do what I feel like doing. Do we really want to just give up on expertise in this way? Even if experts usually fail to give us the clear, reliable guidance we need, there are still situations, as we’ll see, where failing to follow their advice can be self-defeating and even deadly.

     So I’m not going to spend much time trying to convince you that experts are often, and possibly usually, wrong. Instead, this book is about why expertise goes wrong and how we may be able to do a better job of seeking out more trustworthy expert advice. To that end, we’re going to look at how experts — including scientists, business gurus, and our other highly trusted sources of wisdom — fall prey to a range of measurement errors, how they come to have deep biases that lead them into gamesmanship and even outright dishonesty, and how interactions among them tend to worsen rather than correct for these problems. We’re also going to examine the ways in which the media sort through the flow of dubious expert pronouncements and further distort them, as well as how we ourselves are drawn to the worst of this shoddy output, and how we end up being even more misled on the Internet. Finally, we’ll try to extract from everything we’ve discovered a set of rough guidelines that can help to separate the most suspect expert advice from the stuff that has a better chance of holding up.

     As I said, most people are quite comfortable with the notion that there’s a real problem with experts. But some — mostly experts — do in fact take objection to that claim. Here are the three objections I encountered the most often, along with quick responses.

     (1) If experts are so wrong, why are we so much better off now than we were fifty or a hundred years ago? One distinguished professor put it to me this way in an e-mail note: “Our life expectancy has almost doubled in the past seventy-five years, and that’s because of experts.” Actually, the vast majority of that gain came earlier in the twentieth century from a very few sharp improvements, and especially from the antismoking movement. As for all of the drugs, diagnostic tools, surgical techniques, medical devices, lists of foods to eat and avoid, and impressive breakthrough procedures and technologies that fill medical journals and trickle down into media reports, consider this: between 1978 and 2001, according to one highly regarded study, U.S. life spans increased fewer than three years on average — when the drop in smoking rates slowed around 1990, so did life-expectancy gains. It’s hard to claim we’re floating on an ocean of marvelously effective advice from a range of experts when we’ve been skirting the edges of a new depression, the divorce rate is around 50 percent, energy prices occasionally skyrocket, obesity rates are climbing, children’s test scores are declining, we’re forced to worry about terrorist and even nuclear attacks, 118 million prescriptions for antidepressants are written annually in the United States, chunks of our food supply periodically become tainted, and, well, you get the idea. Perhaps a reasonable model for expert advice is one I might call “punctuated wrongness” — that is, experts usually mislead us, but every once in a while they come up with truly helpful advice.

     (2) Sure, experts have been mostly wrong in the past, but now they’re on top of things. In mid-2008 experts were standing in line to talk about the extensive, foolproof controls protecting our banks and other financial institutions that weren’t in place in the late 1920s — just before those institutions started collapsing. Cancer experts shake their heads today over the ways in which generations of predecessors wasted decades hunting down the mythical environmental or viral roots of most cancers, before pronouncing as a sure thing the more recent theory of how cancer is caused by mutations in a small number of genes — a theory that, as we’ll see, has yielded almost no benefits to patients after two decades. Most everyone missed what was happening to our climate, or even spoke of a global cooling crisis, until we came to today’s absolutely certain understanding of global warming and its man-made causes — well, we’ll see how that turns out. How could we have been so foolish before? And what sort of fool would question today’s experts’ beliefs? In any case, the claim that we’ve come from wrong ideas to right ideas suggests that there’s a consensus of experts today on what the right ideas are. But there is often nothing close to such a consensus. When experts’ beliefs clash, somebody has to be wrong — hardly a sign of an imminent convergence on truth.

     And, finally, (3) So what if experts are usually wrong? That’s the nature of expert knowledge — it progresses slowly as it feels its way through difficult questions. Well, sure, we live in a complex world without easy answers, so we might well expect to see our experts make plenty of missteps as they steadily chip away at the truth. I’m not saying that experts don’t make any progress, or that they ought to have figured it all out long ago. I’m suggesting three things: we ought to be fully aware of how large a percentage of expert advice is flawed; we should find out if there are perhaps much more disconcerting reasons why experts so frequently get off track other than “that’s just the nature of the beast”; and we ought to take the trouble to see if we can come up with clues that will help distinguish better expert advice from fishier stuff. And, by the way, if experts are so comfortable with the notion that their efforts ought to be expected to spit out mostly wrong answers, why don’t they work a little harder to get this useful piece of information across to us when they’re interviewed on morning news shows or in newspaper articles, and not just when they’re confronted with their errors?

     Given that I’ve already started throwing the term “expert” around left and right, I suppose I ought to make sure you know what I mean by the word. Academics study “expertise” in pianists, athletes, burglars, birds, infants, computers, trial witnesses, and captains of industry, to name just a few examples. But when I say “expert,” I’m mostly thinking of someone whom the mass media might quote as a credible authority on some topic — the sorts of people we’re usually referring to when we say things like “According to experts . . .” These are what I would call “mass” or “public” experts, people in a position to render opinions or findings that a large number of us might hear about and choose to take into account in making decisions that could affect our lives. Scientists are an especially important example, but I’m also interested in, for example, business, parenting, and sports experts who gain some public recognition for their experience and insight. I’ll also have some things to say about pop gurus, celebrity advice givers, and media pundits, as well as about what I call “local” experts — everyday practitioners such as non-research-oriented doctors, stockbrokers, and auto mechanics.

     I’ve heard it said, half kiddingly, that meteorologists are the only people who get paid to be wrong. I would argue that in that sense most of our experts are paid to be wrong, and are probably wrong a much higher percentage of the time than are meteorologists. I’m going to show that although the process of wringing useful insights and advice from complex subjects may indeed be an inherently slow and erratic one, there are many other, less benign reasons why experts go astray. In fact, we’ll see that expert pronouncements are pushed toward wrongness so strongly that in the end it’s harder, I think, to explain why they’re sometimes right. But that doesn’t mean we’re hopelessly mired in this swamp of bad advice. With a decent compass, we can find our way out. Let’s start by exploring some of the muck.

     From “Wrong: Why Experts Keep Failing Us — And How to Know When Not to Trust Them” by David H. Freedman. Excerpt courtesy of Little, Brown and Company, the publisher. All rights reserved. Except as permitted under the U.S. Copyright Act of 1976, no part of this publication may be reproduced, distributed, or transmitted in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher.

http://www.nytimes.com/2010/06/11/books/excerpt-wrong.html?ref=books&pagewanted=print