AUTHOR: Ben Goldacre – find him at his official website.
This book will answer to all the questions you have about your neighbour relying on homeopaths when he is sick, nutritionists having such an ubiquitous appeal and media coverage, how a drug enters the market, or simply why clever people can believe stupid things.
This book has been particularly enlightening for me: it’s been several years since I met CAM. Never been really convinced about its claims, but it had many warm people among that movement, so I made some friends there. Since I really do my best to give a positive contribution to the world, I thought for some time that boycotting everything which was related to big pharma and profit within the health sector was the best thing I could do. I sank into CAM, and was lucky, because my reasoning has been lucid enough to save me from ever agreeing with it. You’ll find some great thought about what could be done about this issue at the end of the article.
I know how harmful it could be to buy into CAM. I haven’t experienced it directly on health issues, but even thinking about the amount of mental resources that are diverted from solving real problems into some infantile…bullshit, makes me mad. As you will learn, it is not easy to be immune, everyone is somehow influenced by heuristics. Science can save us, and Goldacre makes a superb job to illustrate what’s behind the world of bad science.
“Why we do science: to prevent ourselves from being misled by our own atomised experiences and prejudices.” p.xi
The author digs into the reasons that bring people believe in “detox” programs and purification rituals, especially so claimed “new” ones. New rituals are attached with complicated scientific terminology and acquire a scientific aura, when in fact there is no science whatsoever backing those processes.
“Purification and redemption are such recurrent themes in ritual because there is a clear and ubiquitous need for them; we all do regrettable things as a result of our own circumstances, and new rituals are frequently invented in response to new circumstances.” p.13
“The presentation of these purification diets and rituals [detox footpaths, ear candles, detox patches] has always been a product of their time and place, and now tat science is our dominant explanatory framework for the natural and moral world, for right or wrong, it’s natural that we should bolt a bastardised pseudoscientific justification onto our redemption.
Like so much of the nonsense in bad science, “detox” pseudoscience isn’t something done to us, by venal and exploitative outsiders; it is a cultural product, a recurring theme, and we do it to ourselves.” p.14
Brain Gym is an American organization that aims “to enhance the experience of whole brain learning”, and entered into many schools. Here you have some ‘smart’ advice: “Drink a glass of water before Brain Gym activities. As it is a major component of blood. water is vital for transporting oxygen to the brain.” “Make a ‘C’ shape with your thumb and forefinger and place on either side of the breastbone just below the collarbone. Gently rub for twenty or thirty seconds whilst placing your other hand over your navel. Change hands and repeat. This exercise stimulates the flow of oxygen carrying blood through the carotid arteries to the brain to awaken it and increase concentration and relaxation.” Want more? Visit them at http://www.braingym.org
“People will buy into bogus explanations much more readily when they are dressed up with a few technical words from the world of neuroscience.” p.17
“More clues can be found in the extensive literature in irrationality. People tend, for example, to rate longer explanations as being more similar to “experts’ explanations”. There is also the “seductive details” effect: if you present related (but logically irrelevant) details to people as part of an argument, this seems to make more difficult for them to encode, and later recall, the main argument of a text, because their attention is diverted.” p.19
“[The] process of professionalising the obvious fosters a sense of mystery around science and health advice that is unnecessary and destructive.
All to often this spurious privatisation of common sense is happening in areas where we could be taking control,
doing it ourselves, feeling our own potency and our ability to make sensible decisions; instead we are fostering our dependence on expensive outside systems and people.” p.21
The progenium XY complex
“Classically, cosmetics companies will take highly theoretical, textbookish information about the way that cells work – the components at a molecular level or the behaviour of cells in a glass dish – and then pretend it’s the same as the ultimate issue of whether something makes you look nice. “This molecular component”, they say with a flourish, “is crucial for collagen formation”. And that will be perfect true (along with many other amino acids which are used by your body to assemble protein in joints, akin, and everywhere else), but there is no reason to believe that anyone is deficient in it or that smearing it on your face will make any difference to your appearance. In general, you don’t absorb things very well through your skin, because its purpose is to be relatively impermeable. When you sit in a bath of baked beans for charity, you do not get fat, nor do you start farting.” p.26“The simple theme running through all these products is that you can hoodwink your body, when in reality there a re finely tuned “homeostatic” mechanisms, huge, elaborate systems with feedback and measuring devices, constantly calibrating and recalibrating the amounts of various different chemical constituents being sent to different parts of your body. If anything, interfering with that system is likely to have the opposite of the simplistic effects claimed.” p.26
“I can happily view fancy cosmetics – and other forms of quackery – as a special, self-administered, voluntary tax on people who don’t understand science properly.
I would also be the first to agree that people don’t buy expensive cosmetics simply because they have a belief in their efficacy, because it’s “a bit more complicated than that”: these are luxury goods, status items, and they are bought for all kinds of interesting reasons.
But it’s not entirely morally neutral. First, the manufacturers of these products sell shortcuts to smokers and the obese; they sell the idea that a healthy body can be attained by using expensive potions, rather than by simple old-fashioned exercise and eating your greens. […]
More than that,
these ads sell a dubious worldview. They sell the idea that science is not about the delicate relationship between evidence and theory.
They suggest, instead, with all the might of their international advertising budgets, their Microcellular Complexes, their Neutrilium XY, their Tenseur Peptidique Végétal, and the rest, that science is about impenetrable nonsense involving equations, molecules, sciencey diagrams, sweeping didactic statements from authority figures in white coats, and that this sciencey-sounding stuff might just as well be made up, concocted, confabulated out of thin air, in order to make money.” p.28
Just in case you need a quick refresh about how homeopathy should work, visit Wikipedia.
“The real question with homeopathy is very simple: Does it work? In fact, how do we know if any given treatment is working?
Symptoms are a very subjective thing, so almost every conceivable way of establishing the benefits of any treatment must start with he individual and his or her experience, building from there. Let’s imagine we are talking . maybe even arguing – with someone who thinks that homeopathy works, someone who feels it is a positive experience, and who feels he gets better, quicker, with homeopathy. They would say: “All I know is, I feel as if it works. I get better when I take homeopathy.” It seems obvious to them, and to an extent it is. This statement’s power, and its flaws, lie in its simplicity. Whatever happens, the statement stands as true. p.38-39
But you could pop up and say: “Well, perhaps that was the placebo effect.” Because the placebo effect is far more complex and interesting than most people suspect, going way beyond a mere sugar pill; it’s about the whole cultural experience of a treatment, your expectations beforehand, the consultation process you go through while receiving the treatment, and much more. […]
So when our homeopathy fan says that homeopathic treatment makes them feel better, we might reply: “I accept that, but perhaps your improvement is because of the placebo effect,” and they cannot answer no, because they have no possible way of knowing whether they got better through the placebo effect or not. They cannot tell. The most they can do is restate, is response to your query, their original statement: “All I know is, I feel as if it works. I get better when I take homeopathy.” p.39
Next, you might say: “OK, I accept that, but perhaps, also, you feel you’re getting better because of ‘regression to the mean’”. [It] is basically another phrase for the phenomenon whereby, as alternative therapists like today, all things have a natural cycle.” p.39-40 […]
“Many illnesses have what is called a natural history: they are bad, and then they get better. As Voltaire said,
“The art of medicine consists in amusing the patient while nature cures the disease.”
Let’s say you have a cold. It’s going to get better after a few days, but at the moment you feel miserable. It’s quite natural that when your symptoms are at their very worst, you will do things to try to get better. You might take a homeopathic remedy. You might sacrifice a goat and dangle its entrails around your neck. You might bully your physician into giving you antibiotics. (I’ve listed these in order of increasing ridiculousness.)” p.40
We can accept people are sincere about their feelings (“All I know is, I feel as if it works. I get better when I take homeopathy.”), but we cannot accept the underlying flaws: we would feed reasonings that don’t resonate with what we as mankind have agreed to call “reality”.
“[…] Amazing things simply happen sometimes; people can survive, despite all the odds, for no apparent reasons. As the researchers made clear in their own description, claims for miracle cures should be treated with caution, because “miracles”, occur routinely, in 1 percent of cases by their definition, and without any specific intervention. The lesson of this paper is that
we cannot reason from one individual’s experience or even that of a handful, selected out to make a point.” p.43
Because “Extraordinary claims require the support of extraordinarily convincing data”; but miraculous claims aren’t that extraordinary, after all, and they suffice simple explanations as pointed out above.
“So how do we move on? The answer is that we take lots of individuals, a sample of patients who represent the people we hope to treat, with all of their individual experiences, and count them all up. This is clinical academic medical research, in a nutshell.” p.43
“Speaking with homeopaths, I have encountered a great deal of angst about the idea of measuring, as if this were somehow not a transparent process, as if it were forcing a square peg into a round hole, because “measuring” sounds scientific and mathematical. We should pause for just a moment and think about this clearly. Measuring involves no mystery and no special devices. We ask people if they feel better and count up the answers.” p.44
“If antiauthoritarian rhetoric is your thing, then bear this in mind:
perpetrating a placebo-controlled trial of an accepted treatment – whether it’s an alternative therapy or any form of medicine – is an inherently subversive act. You undermine false certainty, and you deprive doctors, patients, and therapists of treatments that previously pleased them.” p.45
“The alternative therapy literature’s certainly riddled with incompetence, but flaws in trials are actually very common throughout medicine. In fact, it would be fair to say that
all research has some flaws, simply because every trial will involve a compromise between what would be ideal and what is practical or cheap.” p.46
“One important feature of a good trial is that neither the experimenters nor the patients know if they got the homeopathy sugar pill or the simple placebo sugar pill, because we want to be sure that any difference we measure is the result of the difference between the pills and not of people’s expectations or biases.” p.47
This procedure is called blinding.
“Our greatest mistake would be to forget that data is used for serious decisions in the very real world, and bad information causes suffering and death.” p.48
“Some of the biggest figures in evidence-based medicine got together and did a review of blinding in all kinds of trials of medical drugs and found that trials with inadequate blinding exaggerated the benefits of the treatments being studied by 17 percent.” p.48
“Does randomisation matter? As with blinding, people have studied the effect of randomisation in huge reviews of large numbers of trials and found that the ones with dodgy methods of randomisation overestimate treatments effects by 41 percent. In reality, the biggest problem with poor-quality trials is not that they’ve used an inadequate method of randomisation; it’s that they don’t tell you how they randomised the patients at all. This is a classic warning sign and often means the trial has been performed badly. […]
As a general rule it’s always worth worrying when people don’t give you sufficient details about their methods and results.
Transparency and detail are everything is science.” p.52
Further details about experimental flaws:
“Ignoring dropouts tends to exaggerate the benefits of the treatment being tested, and a high dropout rate is always a warning sign.” p.53
“Academics conventionally measure the quality of a study using standardised tools like the Jadad score, a seven-point tick list.” p.54
The next paragraphs would be dedicated to the role of meta-analysis. Reporting its basic functioning here would have been boring. Watch The Cochrane Collaboration explanatory video instead. The Cochrane Collaboration is a non-profit organization which gathers and summarises the best evidence from research in order to make informed choices about treatment.
“[Meta-analysis] is particularly useful when there have been lots of trials, each too small to give a conclusive answer, but all looking at the same topic.” p.56
“Meta-analyses have shown that a treatment previously believed to be ineffective is in fact rather good, but because the trials that had been done were each too small, individually, to detect the real benefit, nobody had been able to spot it.” p.56-57
“You may say what research is missing, on whom, how, measuring what, and why you want to do it, but the hand-waving, superficially open-minded call for “more research” is meaningless and unhelpful.” p.59
“People do experience that homeopathy is positive for them, but the action is likely to be in the whole process of going to see n homeopath, of being listened to, having some kind of explanation for your symptoms, and all the other collateral benefits of old-fashioned, paternalistic, reassuring medicine. (Oh, and regression to the mean)” p.59
“So we should measure that, and here is the final super lesson in evidence-based medicine that homeopathy can teach us: sometimes you need to be imaginative about what kinds of research you do, compromise, and be driven by the questions that need answering, rather than by the tools available to you.It is very common for researchers to research the things that interest them in all areas of medicine, but they can be interested in quite different things from patients.” p.60
Evidently, those people don’t really care about how to have a greater impact in the world. It is true, sometimes we associate human wanderings with the necessary creative freedom to invent astounding, beneficial products. But what if these rambles divert precious resources from focusing on major, popular problems? Wisdom of crowds proves that people know what that need, but occasionally specialists set themselves aside and lose the contact with real-world problems.
“In the case of homeopathy, similarly, homeopaths want to believe that the power is in the pill, rather than in the whole process of going to visit an homeopath, having a chat, and so on. It is crucially important to their professional identity.” p.60
The placebo effect
“For all the dangers of complementary and alternative medicine, to me the greatest disappointment is the way it distorts our understanding of our bodies. Just as the big bang theory is far more interesting than the creation story in Genesis, so the story that science can tell us about the natural world is far more interesting than any fable about magic pills concocted by an alternative therapist.” p.65
For that matter, engagement is actually strictly related with the pattern of storytelling. So the big bang theory, for example, besides fascinating anyone who wants detailed, complex and inter-related stories, might be less interesting for those who want straightforward, simple stories. We need people who can break complex issues into digestible pieces, without losing its overall meaning. Check as an example the vintage children’s book on the science of weight and weightlessness.
“In most studies, we don’t have a “no treatment” group to compare both the placebo and the drug with, and for a very good ethical reason: if your patients are ill, you shouldn’t be leaving them untreated simply because of your own mawkish interest in the placebo effect. In fact, in most cases today it is considered wrong even to use a placebo in a trial; whenever possible you should compare your new treatment with the best preexisting, current treatment. […]
Placebo-controlled trials are also frowned up by the evidence-based medicine community, because it known it’s an easy way to cook the books and get positive trial data to support your company’s big investment. […] Patients and doctors aren’t so interested in whether a new drug works better than nothing; they’re interested in whether it works better than the best treatment they already have. […]
If we don’t want to do unethical scientific experiments with “no treatment” groups on sick people, […] we can compare one placebo with another.” p.67
Main listed findings about the placebo effect:
- four placebo pills are better than two;
- placebo pills colour have an effect on the outcome;
- placebo is more effective in capsule form than pill form;
- injection is a far more dramatic intervention than taking a pill: in fact, the former is more effective that the latter;
- elaborate placebo rituals (like acupuncture) are more effective than simple sugar pills;
- expensive-packaged, brand-name medicines are more effective than others;
- placebo operations have shown to be beneficial in knee pain, angina and post-operational recovery.
“Both what the doctor says and what the doctor believes have an effect on healing. If that sound obvious, I should say they have an effect that has been measured, elegantly, an carefully designed trials.” p.74
Goldacres cites Gracely and al. (1985) as an example. You can find a nice experiment’s explanation here.
“In 1987, Thomas showed that simply giving a diagnosis – even a “fake” placebo diagnosis – improved patient outcomes. […]
This raises the specter of something way beyond the placebo effect, and cuts even further into the work of alternative therapists, because we should remember that alternative therapists don’t just give placebo treatments; they also give what we might call placebo explanations or placebo diagnosis: ungrounded, unevidenced, often fantastical assertions about the nature of the patient’s disease, […] which the therapist claims uniquely to understand.” p.76“And here it seems that the placebo explanation – even if grounded in sheer fantasy – can be beneficial to a patient, although interestingly, perhaps not without collateral damage, and it must be done delicately; assertively and authoritatively giving someone access to the sick role can also reinforce destructive illness beliefs and behaviours, unnecessarily medicalise symptoms like aching muscles […], and militate against people’s getting on with life and getting better. It’s a very tricky area.” p.76-77“Although the authoritative and paternalistic reassurance of the Victorian doctor who “blinds with science” is a thing of the past in medicine, the success of the alternative therapy movement – practitioners mislead, mystify, and blind their patients with sciencey-sounding “authoritative” explanations, like the most patronising Victorian doctor imaginable – suggests that there may still be a market for that kind of approach.” p.78
“Dr. Steward Wolf too the placebo effect to the limit. He took two women who were suffering from nausea and vomiting, one of them pregnant, and told them he had a treatment that would improve their symptoms. In fact, he passed a tube down into their stomachs (so that they would’t taste the revolting bitterness) and administered ipecac, a drug that should actually induce nausea and vomiting.
Not only did the patients’ symptoms improve, but their gastric contractions, which ipecac should worsen, were reduced. His results suggest – albeit it in a very small sample – that
a drug could be made to have the opposite effect from what you would predict from the pharmacology, simply by manipulating people’s expectations.” p.80
The reported sample is really tiny, yet results are impressive!
“We must remember, […] that the placebo effect – or the meaning effect – is culturally specific.
[…] if homeopathy has any benefits at all for AIDS suffers in Botswana, it may be through its implicit association with the white-coat Western medicine that so many African countries desperately need.” p.83
“When [alternative therapists] claim their treatments are having a specific and measurable effect on the body, through specific technical mechanisms rather than ritual, they are championing a very old-fashioned and naive form of biological reductionism, in which the mechanics of their interventions, rather than the relationship and the ceremony, have the positive effect on healing.” p.84
Again, this is a matter of professional identity. Those practitioners aren’t interested in pursuing the truth – if any doubt was left – rather in preserving their business.
“There are often situations where people want treatment, but medicine has little to offer – lots of back pain, stress at work, medically unexplained fatigue, and most common colds, to give just a few examples. Going through a cheater of medical treatment, and trying every medication in the book, will give you only side effects. A sugar pill in these circumstances seems a very sensible option, as long as it can be administered cautiously, and ideally with a minimum of deceit.” p.85
We shouldn’t forget about the negative side effects of believing in things that have no evidence. In addition to the points reported above at pag. 77, we should mention how medicalisation can lead people into believing that
“a pill is an appropriate response to a social problem, or a modest viral illness.” p.85
“Survey data shows that a disappointing experience with mainstream medicine is almost the only factor that regularly correlates with choosing alternative therapies.” p.85
As to say: the pizza I had at Albert’s was terrible. I’ll never eat pizza again!
The nonsense du jour
“[Nutritionists] extrapolate widely from “laboratory bench data” to make claims about humans; they extrapolate from “observational data” to make “intervention claims”; they “cherry-pick”; and last, they quote published scientific research evidence that seems, as far as one can tell, not to exist.” p.87
“Media nutritionists speak beyond the evidence. Often it is about selling pills, sometimes it is about selling dietary fads, or new diagnoses, or fostering dependence, but it is always driven by their desire to create a market for themselves, in which they are the experts, whereas you are merely bamboozled and ignorant.” p.88
“In defence of almost all nutritionists, I would argue that they lack the academic experience, the ill will, and perhaps even the intellectual horsepower to be fairly derided as liars.” p.89
“It is impossible for someone to lie unless he thinks he knows the truth. Producing bullshit requires no such conviction… When an honest man speaks, he says only what he believes to be true; and for the liar, it is correspondingly indispensable that he considers his statements to be false. For the bullshitter, however, all these bets are off: he is neither on ye side of the true nor on the side of the false. His eye is not on the facts at all, as the eyes of the honest man and of the liar are, except insofar as they may be pertinent to his interest in getting away with what he says. He does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose.” Henry Frankfurt, “On Bullshit”, 1986, p.90
One of popular nutritional advice’s major negligences is omitting fundamental confounding variables. A classical example occurs when newspapers report that “moderate alcohol intake” is associated with improved health conditions, whereas background characteristics of teetotallers – belonging to a specific, generally less healthy ethnic group, either indulging in chocolate and chips rather than alcohol, either recovering alcoholics who don’t drink anymore, but have especially poor health conditions – are ignored and therefore fail to point at the real, not-so-surprising findings of the study.
“You have to be very cautious about how you extrapolate from what happens to some cells in a dish on a laboratory bench to the complex system of a living human being, where things can work inn completely the opposite way from what laboratory work would suggest. […] This is just another example of how nutritionism, despite the alternative medicine rhetoric and phrases like “holistic”, is actually a crude, unsophisticated, old-fashioned, and above all, reductionist tradition.” p.94
Goldacre pointed out this striking about turn of alternative medicine, claiming to be holistic but actually taking into account just a small and imprecise number of variables to conceive their theories; this finding is worth be repeated.
“These are the cornerstone of evidence-based medicine,
and often worth having at the back of your mind: it needs to be a strong association, which is consistent, and specific to the thing you are studying, where the putative cause comes before the supposed effect in time; ideally there should be a biological gradient, such as a dose-response effect; it should be consistent or at least not completely at odds with what is already known (because extraordinary claims require extraordinary evidence); and it should be biologically plausible.” p.94-95
“A surrogate outcome [“It improves blood test results; it must be protective against heart attacks!”] can, of course, be used to generate and examine hypotheses about a real disease in a real person, but it needs to be very carefully validated. Does it show a clear dose-response relationship? Is it a true predictor of disease or merely a “covariable”, something that is related t the disease in a different way (e.g., caused by it rather than involved in causing it)?” p.95-96
Because, in the end, “I think you’ll find it’s a little more complicated than that”.
“Just because something can upregulate or downregulate something in a model doesn’t mean it will have the effect you expect in a person.” p.97
“The idea is to try and give all the information to help others to judge the value of your contribution; not just the information that leads to judgement in one particular direction or another.” Richard Feynman, p.97
“Many of [medical academic journals] will contain contradictory claims; picking out what’s relevant – and what’s not – is a gargantuan task. […] We rely on review articles, or on meta-analyses, or textbooks, or hearsay, or chatty journalistic reviews of a subject.
That’s if your interest is getting to the truth of the matter. What if you’ve got just a point to prove?
There are few opinions so absurd that you couldn’t find at least one person with a Ph. D. somewhere in the world to endorse them for you;
and similarly, there are few propositions in medicine so ridiculous that you couldn’t conjure up some kind of published experimental evidence somewhere to support them, if you didn’t mind its being a tenuous relationship and cherry-picked the literature, quoting only the studies that were in your favour.” p.97-98
“Today cherry-picking is one of the most dubious practices in alternative therapies, particularly in nutritionism, where it seems to be accepted essentially as normal practice (it is this cherry picking, in reality, that helps characterise what alternative therapists conceive of, rather grandly, as their alternative paradigm).” p.98
“As with all enormous interlocking systems – societies, for example, or businesses – an intervention in one place can have quite unexpected consequences; there are feedback mechanisms, compensatory mechanisms. Rates of change in one localised area can be limited by quite unexpected factors that are entirely remote from what you are altering, and excess of one thing in one place can distort the usual pathways and flows, to give counterintuitive results.” p.101
Now let’s apply this new level of awareness to the next theory: antioxidants.
“The theory that antioxidants are protective is separate from, but build upon, the free radical theory of disease. If free radicals are dangerous, the argument goes, and antioxidants on the big diagrams are involved in neutralising them, then eating more antioxidants should be good for you. […]
There are a number of problems with this as a theory. First, who says free radicals are always bad? If you’re going to reason just from theory, and from diagrams, then you can hook all kinds of things together and make it seem as if you’re talking sense. Free radicals are vital for your body to kill off bacteria in phagocytic immune cells, so should you set yourself up in business and market an antioxidant-free diet for people with bacterial infections?
Second, just because antioxidants are involved in doing something good, why should eating more of them necessarily make that process more efficient? I know it makes sense superficially; but so do a lot of things, and that’s what’s really interesting about science: sometimes the results aren’t quite what you might expect. Perhaps an excess in antioxidants is simply excreted or turned into something else. Perhaps it just sits there doing nothing because it’s not needed. […] Perhaps, if you have an unusually enormous amount of antioxidant lying around in your body doing nothing, it doesn’t just do nothing. Perhaps it does something actively harmful.” p.102-103
Here I report a funny anecdote to tone down a bit the issue:
“Carrots were the source of one of the great disinformation coups of World War II, when the Germans couldn’t understand how our [the British] pilots could see their planes coming from huge distances, even in the dark. To stop them from trying to work out if we’d invented anything clever like radar (as we had), the British instead started an elaborate and entirely made-up nutritionist rumour. Carotenes in carrots, they explained, are transported to the eye and converted to retinal, which is the molecule that detects light in the eye (this is basically true and is a plausible mechanism, like those we’ve already dealt with), so, went the story, doubtless with much chortling behind their excellent RAF moustaches, we have been feeding our chaps huge plates f carrots, to jolly good effect.” p.106
“[…] Placebo-controlled trial data on antioxidant vitamin supplements has continued to give negative results. The most up-to-date Cochrane reviews of the literature […] show that antioxidant supplements are either ineffective or perhaps even actively harmful.” p.107
Haven’t checked out yet how Cochrane reviews work? Do it now!
“Things that work in theory often do not work in practice, and in such cases we need to revise our theories, even if it is painful.”
This is not an issue that competes to medicine alone; it is rather a wide-spread dangerous habit that permeates the whole human activity, as revising theories is hard and doesn’t gain much appraise.
“It’s a chilling thought that when we think we are doing good, we may actually be doing harm, but it is one we must always be alive to, even in the most innocuous situations.” p.108
If you enjoy this kind of conversations, I suggest you check out the Doing Good Better review, where you’ll find counterintuitive examples about presumably doing good, such as the rise of ethical consumerism, which might actually be harmful. (p.144)
“Doubt is our product,” [a cigarette executive once observed], “since it is the best means of competing with the ‘body of fact’ that exists in the minds of the general public. It is also the means of establishing a controversy.” p.109
There’s a whole book about the topic. You might be brave enough to check it out and dig deeper, if you wish.
“Nobody in the media dares challenge these tactics, where lobbyists raise sciencey-sounding defences of their products, because they feel intimidated and lack the skills to do so.
Even if they did, there would simply be a confusing and technical discussion on the radio, which everyone would switch off, and at most the consumer would hear only “controversy”: job done. p.110
The antioxidant story is an excellent example of how wary we should be of blindly following hunches based on laboratory-level and theoretical data, and naively assuming, in a reductionist manner, that this must automatically map onto dietary and supplement advice, as the media nutritionists would have us do. […]
More than anything it illustrates how this atomised, overcomplicated view of diet can be used to mislead and oversell.” p.110
What I was used to buy into, was a vision of science as a reductionist framework. I was wrong. Real science strives to pull together as many variables as it can, to build less incomplete picture of the world. I got here because mindfulness has the same aim: to include everything you can into the realm of your consciousness, to properly seize and take care of every thought.
After we’ve been hit by this debunking flood, is there anything left to improve our health?
“If I were writing a lifestyle book, it would have the same advice on every page, and you’d know it already.
Eat lots of fruit and vegetables, and live your whole life in every way as well as you can:
exercise regularly as part of your daily routine, avoid obesity, don’t drink to much, don’t smoke, and don’t get distracted from the real, basic, simple causes of ill health.” p.111
“Like our modern food gurus, the historical figures of nutritionism were mostly enthusiastic laypeople, and they all claimed to understand nutritional science, evidence, and medicine better than the scientists and doctors of their era. The advice and the products may have shifted with prevailing religious and moral notions, but they have always played to the market, be it puritan or liberal, New Age or Christian.” p.112
“Anyone who tells you to eat more fresh fruits and vegetables is all right by me. If that were the end of it, I’d be nutritionists’ biggest fan, because I’m all in favour of “evidence-based interventions to improve the nation’s health,” as they used to say to us in medical school. […]
[Nutritionists] have to manufacture complication, to justify the existence of their profession. These new nutritionists have a major commercial problem with the evidence. There’s nothing very professional or proprietary about “Eat your greens,” so they have had to push things further. But unfortunately for them, the technical, confusing, overcomplicated, tinkering interventions that they promote – the enzymes, the exotic berries – are very frequently not supported by convincing evidence.” p.126-127
“The most important take-home message with diet and health is that
anyone who ever expresses anything with certainty is basically wrong, because the evidence for cause and effect in this area is almost always weak and circumstantial,
and changing an individual person’s diet may not even be where the action is.” p.127
“One of the most significant “lifestyle” causes of death and disease is social class. […]
In the Bronx of NYC, 25 percent of the population is obese and 27 percent have serious health problems. Just across the East River in Manhattan, […] just 15 percent are obese and 20 percent have serious health problems.” p.129
“The reason for this phenomenal disparity in health is not that the people in Manhattan are careful to eat goji berries and a handful of Brazil nuts everyday, thus ensuring they’re not deficient in selenium, as per nutritionists’ advice. That’s a fantasy and in some respect one of the most destructive features of the whole nutritionist project;
it’s distraction from the real causes of ill health,
but also – do stop me if I’m pushing this too far – in some respects,
a manifesto or right-wing individualism.
You are what you eat, and people die young because they deserve it. They choose death, through ignorance and laziness, but you choose life, fresh fish, olive oil, and that’s why you are healthy. You’re going to see eighty. You deserve it. Not like them.
Back in the real world, genuine health interventions to address the social and lifestyle causes of disease are far less lucrative, and far less of a spectacle, than anything a vitamin pill peddler, or a nutritionist, would care to engage with.” p.130
Kudos for this noble author, who embraces logic to address common flaws and underlines their intimate connection with major problems that affect our whole society. I never thought nutritionism could represent the long tail of right-wing politics. I’m shocked.
The doctor will sue you now
Chapter 8 illustrates the harm of anti-AIDS campaigners in South Africa. Here is the brief story and some data to seize an enormous humanitarian, scientific and sociological oversight you might have never know about.
He then walked into South Africa with the claim that vitamin pills would cure AIDS, buying full-page newspaper ads. Tragically, the president of South Africa Thabo Mbeki was a well know AIDS dissident, and immediately supported Rath’s claims about AIDS not existing, pretentiously arguing that antiretroviral medication does more harm than good.
A study estimated that such damaging governmental demagogy, which actualised in refusing free donations of drugs in addition to impede existent treatment programs, caused 330,000 unnecessary deaths between 1999 and 2007. The 2000 International AIDS Conference made the world aware of such murderous policies: the official presentation claimed that antiretroviral medication was killing babies and children, not AIDS. Despite international pressures, the government’s behaviour didn’t change. Health’s minister Manto Tshabalala-Msimang repeatedly stated publicly that the remedies for AIDS were “beetroot, garlic, lemons, and African potatoes.”
“Alternative therapists like to suggest that their treatments and ideas have not been sufficiently researched. As you know now, this is often untrue.” p.138
“This [Africa] is a continent that has been brutally exploited by the developed world, first by empire and then by globalised capital. Conspiracy theories about AIDS and Western medicine are not entirely absurd in this context. The pharmaceutical industry has indeed been caught performing drug trials in Africa that would be impossible anywhere in the developed world. Many find it suspicious that black Africans seem to be the biggest victims of AIDS and point to the biological warfare programs set up by the apartheid governments; there have also been suspicions that the scientific discourse of HIV/AIDS might be a device, a Trojan horse for spreading even more exploitative Western political and economic agendas around a problem that is simply one of poverty.
And these are new countries, for which independence and self-rule are recent developments, which are struggling to find their commercial feet and true cultural identity after centuries of colonisation. Traditional medicine represents an important link with an autonomous past, besides which, antiretroviral medications have been unnecessarily – offensively, absurdly – expensive, and until moves to challenge this became partially successful, many African were effectively denied access to medical treatment as a result.” p.139
The previous paragraph partly exonerates Africans from supporting this murderous theory. But I should add that Matthias Rath is a product of our, Western culture. Despite the harm he has caused, alternative therapists continue to support his work and his vitamin pills.
Is mainstream medicine evil?
“Terrible things happen in medicine, when it goes right as well when it goes wrong. Everybody agrees that we should work to minimise the errors, everybody agrees that doctors are sometimes terrible. […]
Doctors can be awful, and mistakes can be murderous, but the philosophy driving evidence based medicine is not.
How well does it work? One thing you could measure is how much medical practice is evidence based. This is not easy. From the state of current knowledge, around 13 percent of all treatments have good evidence, and a further 21 percent are likely to be beneficial. This sounds low, but it seems the more common treatments tend to have a better evidence base. […] Between 50 and 80 percent of all medical activity is “evidence-based”. It’s still not great, and if you have any ideas on how to improve that, do please write about it.” p.148
I’ve always thought that everything medicine implements is evidence-based, and that the alternative movement was pushing for practices that weren’t supported by evidence and needed to be studied further. As I could learn from this book, such claims are false and do not give space to vast meta-analyses that have already shown clear results. Anyway, within medicine itself there is a fairly large number of treatments that are not evidence-based. I’ve always thought of medicine as a solid body of confirmed practices, with no idiosyncrasies. I was wrong, but it is good to know that every human application has room for significant improvement.
“It seems to me that a lot of the stranger ideas people have about medicine derive from an emotional struggle with the very notion of a pharmaceutical industry. Whatever our political leanings, we all feel nervous about profit taking any role in the caring professions, but that feeling has nowhere to go. Big pharma is evil; I would agree with that premise. But
because people don’t understand exactly how big pharma is evil, their anger gets diverted away from valid criticisms
– its role in distorting data, for example, or withholding lifesaving AIDS drugs from the developing world –
and channeled into infantile fantasies.
“Big pharma is evil,” goes the line of reasoning; “therefore homeopathy works and the MMR vaccine causes autism.” This is probably not helpful.”
Health issues have a strong emotional baggage. Emotion, or System 1 according to Daniel Kahneman terminology, takes control over reasoning and oversimplifies the necessary, painful, rational discussion about such sensible topics. This example evidences very well when heuristics don’t help humanity make better decisions.
The drug journey in brief: take any molecule you think might serve your purpose. Test it in animals. Then test it in a small number of healthy, young men who need money; if it works out well, test it on a couple of hundred people with the relevant illness, to work out the dose and the drug effectiveness.
“A lot of drugs fail a this point, which is a shame; bringing a drug to market costs around five hundred million dollars in total.” p.152
Last trial consists in a randomised, blinded experiment over thousands of patients, which could be replicated many times before applying for the drug license.
“Drug trials are expensive, so an astonishing 90 percent of clinical drug trials, and 70 percent of trials reported in major medical journals, are conducted or commissioned by the pharmaceutical industry.
A key feature of science is that findings should be replicated, but if only one organisation is doing the funding, then this feature is lost.” p.152
“Drug companies have a large influence over what gets researched, how it is researched, how the results are reported, how they are analysed, and how they are interpreted.
Sometimes whole areas can be orphaned because of a lack of money and corporate interests. Homeopaths and vitamin pill quacks would tell you that their pills are good examples of this phenomenon. That is moral affront to the better examples. […] The Global Forum for Health Research estimates that only 10 percent of the world’s health burden receives 90 percent of total biomedical research funding.” p.153
“Put yourself in the shoes of a big pharma researcher.
You have a pill. It’s OK, maybe not that brilliant, but a lot of money is riding on it. You need a positive result, but your audience aren’t homeopaths, journalists, or the public; they are doctors and academics, who have been trained in spotting the obvious tricks, like “no blinding”, or “inadequate randomisation”. Your sleight of hand will have to be much more elegant, much more subtle, but every bit as powerful.
What can you do?
You could study it on winners.” p.154
Study your drug only on those people who show an improvement.
“You can compare your drug against a useless control.
Almost all drugs will be compared against placebo at some stage of their lives, and drug reps, the people employed by big pharma to bamboozle doctors, love the unambiguous positivity of the graphs these studies can produce.”
- If you do have to compare your drug with one produced by a competitor, […] you could use an inadequate dose of the competing drug, either give it in the wrong way or even too much of it to produce stronger side effects. (p.155)
- You can not ask for side effects, either ask for them in unspecified, chaotic ways or even exclude them from the reports, aiming to publish only the “most relevant” side effects.
- Instead of a real-world outcome, you can use a “surrogate outcome”. (p.156)
- If your trial has been good overall, but has thrown out a few negative results, you can abstain from putting them on the graph.
- If your results are completely negative, don’t publish them at all, or publish them only after a long delay. (p.157)
Here are the classic tricks to play in your statistical analysis to make sure your trial has a positive result:
- Ignore the protocol entirely. Always assume that any correlation proves causation.
- Play with the baseline. (p.157)
- Ignore dropouts.
- Clean up the data. Delete unhelpful data and keep spurious ones that make your research look good.
- Change the trial period to meet required positive results: shorten it, either extend it.
- Torture the data. Make your computer track any relevant subgroup behaviour and use it to your advantage.
- Try every button on the computer. (p.158)
“And when you are finished, the most important thing is to publish wisely.
If you have a positive trial, but it was a completely unfair test, put it in an obscure journal (published, written, and edited entirely by the industry). […]
If your finding is really embarrassing, hide it away somewhere, and cite “data on file”.” p.159
“Overall, studies funded by a pharmaceutical company were found to be four times more likely to give results that were favourable to the company than were independent studies.” p.159
“How can we explain the apparent fact that industry-funded trials are often so glowing? How can all the drugs simultaneously be better than all the others? The crucial kludge may happen after the trial is finished.” p.160
“Positive trials are more likely to get published than negative ones.
When you get a negative result, it feels as if its all been a bit of a waste of time.
It’s easy to convince yourself that you found nothing when in fact you discovered a very useful piece of information: that the thing you were testing doesn’t work.” p.161
“Generally the influence of publication bias is more subtle, and you can get a hint that publication bas exists in a field by doing something very clever called a funnel plot.
If there is publication bias, the results will be skewed.” p.162
“The smaller, more rubbish negative trials may miss, because they have been ignored.
Not only has publication bias been demonstrated in many fields of medicine, but a paper has even found evidence of publication bias in studies of publication bias.” p.163
“Drug companies can go one better than neglecting negative studies. Sometimes, when they get positive results, instead of just publishing them once, they publish them several times, in different places, in different forms, so that it looks as if there were lots of different positive trials.” p.164
The author proposes a global, simple solution to eradicate this huge big pharma problem: to establish a public, open clinical trials register. Find out more about his project, All Trials, and eventually sign the petition for its implementation at http://www.alltrials.net/petition/.
Why clever people believe stupid things
“Why do we have statistics, why do we measure things, and why do we count? If the scientific method has any authority – or, as I prefer to think of it, value – it is because it represents a systematic approach, but this is valuable only because the alternatives can be misleading.” p. 172
Here you have a list of bias that damage good decision making, or, as the author elegantly descried in the chapter title, why clever people believe stupid things.
I will go much deeper in cognitive bias in the review of the classic Thinking, Fast and Slow, which would be posted very soon.
We see patterns where there is only random noise (Randomness);
We see casual relationships where there are none (Regression to the mean); p.176
“These are two very good reasons to measure things formally. It’s bad news for intuition already. Can get it much worse?” p.177
We overvalue confirmatory information for any given hypothesis;
We seek out confirmatory information for any given hypothesis;
“This tendency is dangerous, because if you ask only questions that confirm your hypothesis, you will be more likely to elicit information that conforms it, giving a spurious sense of confirmation. It also means – if we think more broadly – that
the people who pose the questions already have a head start in popular discourse.” p.178
- “Our assessment of the quality of new evidence is biased by our previous beliefs.
It is important that we have clear strategies available to us to appraise evidence, regardless of its conclusions, and this is the major strength of science.” p.180
“Our attention is always drawn to the exceptional and the interesting, and if you have something to sell, it makes sense to guide people’s attention to the features you most want them to notice.” p.181
Recognising novelty – an evolutionary strategy, or what else? Our eye catches the fast movement over the slow one, the bright colours over the faded ones, and it all seems a rooted biological issue, built to save us from dangerous situations and to guide us toward a better partner. Given this, not being able to recognise small, important details could lead to harmful actions.
“No matter what you do with statistics about risk or recovery, your numbers will always have inherently low psychological availability, unlike miracle cures, scare stories, and distressed parents.” p.181
“It feels almost too obvious to mention, but our values are reinforced by conformity and by the company we keep. We are selectively exposed to information that revalidates our beliefs, partly because we expose ourselves to situations in which those beliefs are apparently confirmed; partly because we ask questions that will – by their very nature – give validating answers; and partly because we selectively expose ourselves to people who validate our beliefs.” p.182
“Communal reinforcement goes a long way toward explaining how religious beliefs can be passed on in communities from generation to generation.It also explains how testimonials within communities of therapists, psychologists, celebrities, theologians, politicians, talk show hosts, and so on can supplant and become more powerful than scientific evidence.” p.183
“This is something humans are very good at – filtering irrelevant information – but that skill comes at cost of ascribing disproportionate bias to some contextual data.
We tend to assume, for example, that positive characteristics cluster: people who are attractive must also be good; people who seem kind must also be intelligent and well informed.” p.184
“The challenge, perhaps, is to work out which tools to use where. Because trying to be “scientific” about your relationship with your partner is as stupid as following your intuitions about casualty.” p.185
“[Risk managers] are much more likely to make the right decisions when information about risk is presented as natural frequencies, rather than as probabilities or percentages.” p.188
“Probabilistic reasoning is difficult for everyone, but everyone understands normal numbers. This is why natural frequencies are the only sensible way to communicate risk.” p.189
Next, a bonus right from the statistics class, to help you remember – either learn about – basic statistical concepts which are very useful to evaluate the exact results of any research, and bypass all the journalistic, sensational bullshit. These methods will be described as statistical means to adjust the significance of results, which would otherwise show scary, but often unrooted claims about drugs’ effects you’ll find in any popular journal.
“What does “statistically significant” mean? It’s just a way of expressing the likelihood that the result you got was attributable merely to chance.” […] p.194
“The standard cutoff point for statistical significance is a p-value of 0.05, which is just another way of saying, ‘If I did this experiment a hundred times, I’d expect a spurious positive result on five occasions, just by chance.'” p.195
The American Statistical Association has recently made specific recommendation for a statistical practice, and stated that p-value is an overestimated metric, which should never be intended as a surrogate of scientific reasoning. Read the full article here.
“Correct for clustering”: this is an operation that statisticians make to level differences between studies with bigger sample sizes, where results are probably more significant, from studies with smaller samples with less significance. Correcting for clustering therefore reduces the overall significance of results.
“Correction for multiple comparisons” is run every time there is too much data to choose from, with the risk of pairing data and randomly produce their significance, and would not have happened if other comparisons had occurred.
“[Correction for multiple comparisons], like correcting for clustering, is particularly brutal on the data and often reduces the significance of findings dramatically.” p.196
“The interesting thing abut statistics is not the tricky math, but what the numbers mean. […]
Before you go to your data with your statistical tool, you have to have a specific hypothesis to test. If your hypothesis comes from analysing the data. then there is no sense in analysing the same data again to confirm it.” p.205
I think I’m going to study some statistics, after this reading.
The media MMR’s hoax
This chapter illustrates the MMR vaccine scare that began in the UK in 1998, which went on for several years; it involved excellent figures as the Blairs and had huge resonance on every media. It represents a very good example of bad science, and how it can be harmful to the whole society. I reported some highlights that particularly evidence the link between general bad science tricks as described earlier with a case that is internationally known, to provide an easy comparison to situations you may know already happened in your country.
“The diversity and isolation of these anti vaccination panics help illustrate the way in which they reflect local political and social concerns more than a genuine appraisal of the risk data:
because if the vaccine for hepatitis B, or MMR, or polio is dangerous in one country, it should be equally dangerous everywhere on the planet, and if those concerns were genuinely grounded in the evidence, especially in an age of the rapid propagation of information, you would expect the concerns to be expressed by journalists everywhere. They’re not.” p.215
“Unlikely combinations of events will always happen, somewhere, to some people, entirely by chance. Drawing a target around them after the fact tells us nothing at all.” p.217
“If there is one thing that has adversely affected communication among scientists, journalists and the public, it is the fact that science journalists simply do not cover major science news stories.” p.228
“Journalists are used to listening with a critical ear to briefings from press officers, politicians, PR executives, salespeople, lobbyists, celebrities, and gossipmongers, and they generally display a healthy natural skepticism, but in the case of science, they don’t have the skills to critically appraise a piece of scientific evidence on its merits. At best the evidence of these “experts” will be examined only in terms of who they are as people or perhaps whom they have worked for. Journalists – and many campaigners – think that this is what it means to critically appraise a scientific argument and seem rather proud of themselves when they do it.
The scientific content of stories – the actual experimental evidence – is brushed over and replaced with didactic statements from authority figures on either side of the debate which contributes to a pervasive sense that scientific advice is somehow arbitrary and dedicated upon social role – the “expert” – rather than on transparent and readily understandable empirical evidence.” p.229
Read it again, please. What a marvellous reasoning. And it applies not only to science, but to many other areas of knowledge as well.
“Once a scare is running, perhaps every refutation can seem like an admission of guilt, drawing attention to the scare.” p.233
“A systematic review from the Cochrane Collaboration found five studies looking at the use of specific heath interventions before and after media coverage of specific stories, and each found that favourable publicity was associated with greater use, and unfavourable coverage with lower use.
It’s not just the public: medical practice is influenced by the media too, and so are academics.” p.244
Bottomline: even if we know that media coverage of scientific trials is more than usually flawed and imprecise, it yet influences our behaviour. We can’t screen ourselves from such a harm. We have to act and eradicate it.
And another thing
“Economists and doctors talk about “opportunity costs”, the things you could have done but didn’t, because you were distracted by doing something less useful. To my mind,
the greatest harm posed by the avalanche of nonsense we have seen in this book is best conceived as the ‘opportunity cost of bullshit.'” p.256
I chose to report the following paragraphs at their entire length, because I consider them a spectacular conclusion for the whole book. I could not edit it, so enjoy!
“On a larger scale, many people are angry about the evils of the pharmaceutical industry and nervous about the role of profit in health care; but these are formless and uncalibrated institutions, so the valuable political energy that comes from this outrage is funnelled – wasted – through infantile issues like the miraculous properties of vitamin pills or the evils of MMR. Just because big pharma can behave badly, that does not mean that star pills work better than placebo, nor does it mean that MMR causes autism. Whatever the wealthy pill peddlers try to tell you, with their brand-building conspiracy theories, big dharma isn’t afraid of the food supplement pill industry; it is the food supplement pill industry. Similarly, big pharma isn’t frightened for its profits because popular opinion turned against MMR; if they have any sense, these companies are relieved that the public is obsessed with MMR and thus distracted from the other far more complex and real issues connected with the pharmaceutical business and its inadequate regulation.” p.255-256
“To engage meaningfully in a political process of managing the evils of big pharma, we need to understand a little about the business of evidence; only then can we understand why transparency is so important in pharmaceutical research, for example, or the details of how it can be made to work or concoct new and imaginative solutions.
But the greatest opportunity cost comes, of course, in the media, who have failed science so spectacularly; getting stuff wrong and dumbing down. No amount of training will ever improve the wildly inaccurate stories, because newspapers already have specialists health and science correspondents who understand science. Editors will always – cynically – sideline those people and give stupid stories to generalists, for the simple reason that they want stupid stories. Science is beyond their intellectual horizon, so they assume you can just make it up anyway. In an ear when mainstream media are in fear for their lives, their claims to act as effective gatekeepers to information are somewhat undermined by the content of pretty much every column or blog entry I’ve ever written.
To academics, and scientists of all shades, I would say this: you cannot ever possibly prevent newspapers from printing nonsense, but you can add your own sense into the mix.” p.257
“There are intelligent people about there who want to be pushed, to keep their knowledge and passion for science alive, and neglecting them comes at a serious cost to society. Institutions have failed in this regard. The indulgent and well-financed “public engagement with science” community has been worse than useless, because it too is obsessed with making the message to everyone, rarely offering stimulating content to the people who are already interested.
Now you don’t need these people. Start a blog. Not everyone will care, but some will, and they will find your work.
Unmediated access to ache expertise is the future, and you know, science isn’t hard
– academics around the world explain hugely complicated ideas to ignorant eighteen-year-old every September –
it just requires motivation.
[…] There is no money in it, but you knew that when you started on this path. You will do it because you know that knowledge is beautiful, and because if only a hundred people share your passion, that is enough.” p.258