For me, the takeaway from this affair is that there is no one-size-fits-all solution to make statistics impossible to hack. Getting rid of p-values is appropriate sometimes, but not other times. Demanding large sample sizes is appropriate sometimes, but not other times. Not trusting silly conclusions like “chocolate causes weight loss” works sometimes but not other times. At the end of the day, you have to actually know what you’re doing. Also, try to read more than one study.
Another good indicator is, that in Germany Bild and the Huffington Post reported about it. You generally want to avoid to write a story based on these two. Everybody knows, that Bild consists of sex, hate, tits and the weather forecast as the Ärzte put it in a song mocking people that get their information from that paper.
For me the take away is that non-trivial independent replication of results still stands as the gold standard for experimentation. P-hacking shows up clearly when the experiment is replicated. P-values in my experience are only a starting point for further experiments.
Take the cold-fusion debacle. When independent groups attempted to replicate the results they were unable to do so.
Of course the other big take away is that people really don't understand statistics at a basic level, even the "scientific reporters".
For me the take away is that non-trivial independent replication of results still stands as the gold standard for experimentation.
Agreed. Too bad funding agencies rarely if ever give you the money to do it :(.
The takeaway here is that statistics is arguably one of the most nuanced quantitative fields out there. It's really easy to shoot yourself in the foot, particularly with p-values.
I think every statistical test has its place, but my personal favorite lambasting of p-value testing is Steiger and Fouladi's 1997 paper on non-centrality interval estimation [1].
As an aside, Steiger was my graduate statistics professor several years ago, and probably the primary reason I know this paper even exists. If you enjoy the harsh treatment of significance testing in the paper, just imagine hearing it straight from the horse's mouth during lecture :).
Even if I had the time to read several studies and expertise to understand them, I couldn't afford it. Out of curiosity I just went to the Journal of Nutrition (I know nothing about it, picked that journal randomly) and tried to download one random issue. It was $273. Going to multiple original sources isn't practical for most of us.
I doubt that the typical mainstream journalist writing a popular story on a scientific subject could justify this expense to her boss either.
If you need regular access to the scientific literature you have or make an arrangement with an institution. You don't pay per article or issue. It is a real problem for unaffiliated or freelance folks, though.
Even as a freelance person it's not a huge problem for me. It can be annoying, but if I'm doing real work rather than just looking up one article on a whim, it's not a showstopper. Mostly I just use the local university library. Non-affiliates can't check out books, but anyone can walk in and get on the wifi. And once you're on their wifi, you get whatever database access they've negotiated for their IP range. Not every university library allows this, but quite a lot do.
I'd hope that news outlets covering any kind of science pay for access to at least some major journals as a matter of course. The subscription prices are probably much lower per issue than the price to buy an issue individually.
Most journals also show you abstracts for free, which are overviews of what each paper says. If you find a paper that you want to read in full, email the author - most scientists are very happy to know that people are interested in their work, so they'll give out copies to anyone who asks.
This isn't to defend the paywalls - I think open access is important. But I don't think the paywalls are a valid excuse for journalists.
I think the prices for most of these journals individually is enormous - they are usually sold in bundles through individually negotiated, confidential pricing agreements that run into the hundreds of thousands of dollars. Far cheaper per journal, but probably including a bunch or journals that the institution normally wouldn't be interested in, and a ton of garbage.
Not to say that many/most papers can't be found with a little bit of effort, and I can't really imagine a situation where an establishment journalist emails authors about one of their papers and doesn't get it.
It is very often possible for journalists to get papers for free. In many cases it's easy to email the author and they'll happily provide you with a copy within short timeframes. Some archives provide free access to journalists, e.g. the cochrane library.
I'm all for open access, because I think everyone should have the right to see publicly funded scientific research. But from my experience as a journalist who's written a number of pieces on scientific works it often isn't a big problem. I never had to pay for a journal article to do my work.
Its a serious problem even in academia. I have had to email authors of papers or go begging on reddit.com/r/scholar more than once, even when I had very expensive university subscriptions available to me.
Why are calories not counted on any of the individuals?
Free reign is not a way to go about doing a study. They might very well have, we don't know. We all know cocoa has weight loss properties (Theobromine), but to do such a poorly constructed study was pointless.
It's difficult to understand why anyone would ever pay any attention to nutrition "science". Everything is wrong. TFA complimented the meatheads on some bodybuilding site for their skepticism. People who pay very close attention to their own habits and results are probably going to be more right about their own nutrition than all the scientists in the world put together.
I upvoted you because even though you're too far in outright rejecting empiricism about nutrition, the state of nutrition science is indeed quite poor and shrouded in mythology.
Of course one can say basic things like "average daily calories should be between 800 and 3000", but even a range that covers 95% of the population isn't right for everyone, and isn't precise enough to really help anyone. Individuals differ not only in terms of lifestyle, which conceivably could be studied, but also in terms of genetics, which no study to date has examined in more than a rudimentary way. The various statistics sins that we complain about on HN are exacerbated when important factors like genetic background are left unstudied. And we haven't even mentioned the fact that most published (and unpublished) nutrition studies are funded by the food and supplement industries!
A rejection of a particularly poor practice of science is not a rejection of science. I doubt we'll see decent nutrition science until we're all equipped with a network of bloodstream nanobots that measures the actual physical response we each display to nutritional input.
Yeah it wasn't meant as a put-down. Those meatheads know more about their own nutrition, with respect to their focus on muscle formation, than anyone else does. "Meathead" is a literal term for that.
I wouldn't go that far... Nutrition 'science' is quite valid, the only problem is the amount of people in the nutrition space who aren't actually doing 'science'.
People who pay very close attention to their own habits and results are probably going to be more right about their own nutrition than all the scientists in the world put together.
Doubt it. People are often quite irrational about these things. Which is why there's so many people who follow fad diets (and buy the books).
Ah the critical distinction is paying attention to your body is not the same thing as buying a book. As per OPs comment, I used to lift, its good exercise, and I saw plenty of demonstrations of data gathering technique and analysis.
It's difficult to understand why anyone would ever pay any attention to nutrition "science"
It's easy to understand, actually (I take your statement literally).
Being in shape is not rocket science: you eat clean, you do sports, and you're done. Although that sounds ambiguous, everybody does know what clean food is, and what sports mean (ahem, walking doesn't count).
The thing is that it's a damn hard lifestyle, if one doesn't have the right mindset/(self)education/upbring.
Nutrition "science" is nothing else that a stream of silver bullets, which are easy to apply (avoid this, avoid that), so the public wants to believe in it, because it makes life much easier.
It's obviously a deception; but truth isn't the point - feeling good about the idea is.
The thing is that it's a damn hard lifestyle, if one doesn't have the right mindset/(self)education/upbring.
And budget. To do properly requires a pretty good budget for good quality protein sources, vegetables, etc.
Although that sounds ambiguous, everybody does know what clean food is
Not at all. You project too much. Many people truly believe that healthy food is what the supermarkets & TV tell you is healthy. The popularity of the "NutriBullet" is proof alone - "turn any food into a super food"... Uhh huh.
The real story here is that, even after being shown proof of that it was a hoax all along, a chunk of the same population will continue to hold that eating chocolate helps to lose weight.
I laughed. But more seriously, I wonder if Bohannon's media experiment would have succeeded if you change a detail. E.g. if it was about something truly gross, like drinking curdled milk to lose weight. Journalists want to tell people what they want to hear, because people will tend to buy the paper that says what they are already thinking (weird, I know).
It's human nature that people are more skeptical about claims they don't like.
Imagine how mad the people in the study would be after they found out it was a sham. "I drank curdled milk for 3 weeks so you could get your big break!"
That's an interesting angle even for this study - how do the original participants feel about it I wonder? I'd guess some mixture of amused and cynical.
Bohannon tested a system, it failed, and he reported exactly how he tested the system and how it failed. 'The system' in this case is the global food science and health journalism industry.
He found that ignorance and laziness is rampant, that journal peer review is non-existent, and skepticism about results and methods is non-existent.
He has done a tremendous public service exposing the flaws in a system. With any luck consumers of these information sources will take everything with an enormous, skeptical grain of salt, because the last and best line of defense is your own skepticism critical thinking.
He did not find that "journal peer review is non-existent". He targeted a low-quality journal that does not send out papers for review to make sure he could claim that the research was "published" when sending out his press releases. That's explicit in the article, which you might find an interesting read
This isn't about peer review, because the paper was knowingly submitted only to journals known not to conduct peer review. Quoting from the article:
We needed to get our study published pronto, but since it was such bad science, we needed to skip peer review altogether. Conveniently, there are lists of fake journal publishers.
And later:
The new publisher’s CEO, Carlos Vasquez, emailed Johannes to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”
This is really more about science journalism than about science.
These kinds of journals are difficult for a lay person to tell apart from actual, scientific journals, but distinguishing between the two is exactly the kind of job a competent science journalist would do. Inept science journalists, on the other hand, give these fake journals a reason to exist by citing them unchallenged.
And his experiment reveals that the Internet gets its nutrition science from inept science journalists. Think of all the arguments won by people searching google and having this show up. I once won an argument by changing Wikipedia during the argument and showing it to him on his computer.
[Your post is flagged/dead, so I'm replying here. I hope you see it.]
The relationship between counting calories and weight loss is not quite as simple as the formula [Calories Injested] - [Calories Burned] = [Delta body fat] might indicate.
To be honest, chocolate may help in an indirect way...
If someone's on a diet, and never gets any 'reward', they might be tempted to cheat or give up.
If they're told a little chocolate every day is good, they'll stick with their diet, have their chocolate, and forego worse 'cheat' items (even though chocolate has plenty of fat and sugar, the amount you consume before you're content is relatively small compared to say, ice-cream, cookies, etc...).
Studies (and anecdotes) have shown that chocolate makes people happy, so that little happiness at the end might just help people stay on their diet and not stress out (stress can cause the body to store fat). Maybe it's something like the French paradox...
The study was pretty funny though, and I appreciate their honesty. And it goes to show why pretty much all research on fad diets is nonsense. Still though, if you choose between drinking a gallon of pop per day or chocolate, chocolate wins. So maybe their 'research' didn't go to waste.
It's not news that the peer-review system isn't safe from exploits. People have been using Markov chains to generate studies which have then been published. And journalists uncritically picks up everything they can. All functions in society are based on trust and are easily exploitable if a person is not behaving trustworthy.
So making a fake study about chocolate and getting lots of magazines to publish it I think is a little unethical. It's not news, and the reason most people don't do it is because they want to act trustworthy. For example, someone could, if they wanted, submit patches to some free software projects which would introduce awful security holes and get them merged or just write shit on Wikipedia.
I disagree. The study was fake because the intent was to deceive. He used flawed methods, but could have used 10 times as sophisticated methods and still would have been able to publish completely invalid results.
To continue this line of though, distributed computation is hard problem, and that's just computers. Here, people have to do real work to statically verify a paper's procedure, which is very cognitive overloading. Imagine reviewing a PR and deciding with certainty if it will work or not. Even then they should also "recompute" the experiment to completely verify it. Of course add human error on top of all this.
It seems there's a problem here, the article made it seem anyone can produce a bad study for lulz, and get everyone's attention. What is more scary is people doing it for vested interests. Maybe there's something that could be learned from projects like BIONC, Wikipedia, or Bitcoin, but better science is going to cost more money, and require more cultural attention. Not sure how to fix those things :)
On the flip side, you have some well-heeled corporations and industry groups working to discredit good science that's bad for them. Would be interesting to read a piece on how that's done.
I don't think you need to go that far, you just need a PR person with a conscience to spill the beans on what they did (or dissect what others have done).
http://slatestarcodex.com/2015/05/30/that-chocolate-study/
Another good indicator is, that in Germany Bild and the Huffington Post reported about it. You generally want to avoid to write a story based on these two. Everybody knows, that Bild consists of sex, hate, tits and the weather forecast as the Ärzte put it in a song mocking people that get their information from that paper.
For me the take away is that non-trivial independent replication of results still stands as the gold standard for experimentation. P-hacking shows up clearly when the experiment is replicated. P-values in my experience are only a starting point for further experiments.
Take the cold-fusion debacle. When independent groups attempted to replicate the results they were unable to do so.
Of course the other big take away is that people really don't understand statistics at a basic level, even the "scientific reporters".
In essence "science" reporters have made it their business to not understand statistics.
That way they can turn weak correlations into strong causations, rake in the eyeballs, and then go "oops" whenever someone points out problems.
Replication is hard and expensive, particularly in the medical field, and does not get the researcher any real science points.
Besides, it's more fun to do a meta-analysis on a flock of sorta-vaguely-similar studies.
Agreed. Too bad funding agencies rarely if ever give you the money to do it :(.
The takeaway here is that statistics is arguably one of the most nuanced quantitative fields out there. It's really easy to shoot yourself in the foot, particularly with p-values.
I think every statistical test has its place, but my personal favorite lambasting of p-value testing is Steiger and Fouladi's 1997 paper on non-centrality interval estimation [1].
As an aside, Steiger was my graduate statistics professor several years ago, and probably the primary reason I know this paper even exists. If you enjoy the harsh treatment of significance testing in the paper, just imagine hearing it straight from the horse's mouth during lecture :).
[1]: http://www.statpower.net/Steiger%20Biblio/Steiger&Fouladi97....
Even if I had the time to read several studies and expertise to understand them, I couldn't afford it. Out of curiosity I just went to the Journal of Nutrition (I know nothing about it, picked that journal randomly) and tried to download one random issue. It was $273. Going to multiple original sources isn't practical for most of us.
I doubt that the typical mainstream journalist writing a popular story on a scientific subject could justify this expense to her boss either.
If you need regular access to the scientific literature you have or make an arrangement with an institution. You don't pay per article or issue. It is a real problem for unaffiliated or freelance folks, though.
Even as a freelance person it's not a huge problem for me. It can be annoying, but if I'm doing real work rather than just looking up one article on a whim, it's not a showstopper. Mostly I just use the local university library. Non-affiliates can't check out books, but anyone can walk in and get on the wifi. And once you're on their wifi, you get whatever database access they've negotiated for their IP range. Not every university library allows this, but quite a lot do.
I'd hope that news outlets covering any kind of science pay for access to at least some major journals as a matter of course. The subscription prices are probably much lower per issue than the price to buy an issue individually.
Most journals also show you abstracts for free, which are overviews of what each paper says. If you find a paper that you want to read in full, email the author - most scientists are very happy to know that people are interested in their work, so they'll give out copies to anyone who asks.
This isn't to defend the paywalls - I think open access is important. But I don't think the paywalls are a valid excuse for journalists.
I think the prices for most of these journals individually is enormous - they are usually sold in bundles through individually negotiated, confidential pricing agreements that run into the hundreds of thousands of dollars. Far cheaper per journal, but probably including a bunch or journals that the institution normally wouldn't be interested in, and a ton of garbage.
http://www.pnas.org/content/111/26/9425.full
Not to say that many/most papers can't be found with a little bit of effort, and I can't really imagine a situation where an establishment journalist emails authors about one of their papers and doesn't get it.
It is very often possible for journalists to get papers for free. In many cases it's easy to email the author and they'll happily provide you with a copy within short timeframes. Some archives provide free access to journalists, e.g. the cochrane library.
I'm all for open access, because I think everyone should have the right to see publicly funded scientific research. But from my experience as a journalist who's written a number of pieces on scientific works it often isn't a big problem. I never had to pay for a journal article to do my work.
Its a serious problem even in academia. I have had to email authors of papers or go begging on reddit.com/r/scholar more than once, even when I had very expensive university subscriptions available to me.
By the way, this is the documentary for which this was done:
In French: http://www.arte.tv/guide/fr/052711-000/pour-maigrir-mangez-d...
And in German: http://www.arte.tv/guide/de/052711-000/schlank-durch-schokol...
Finally, the most accurate reactions come from the web itself: http://forum.bodybuilding.com/showthread.php?t=166912491
'everyone'?
I don't recall being fooled into thinking this.
Slightly less flippantly - the assertion, conclusion, and title are wrong.
It's difficult to understand why anyone would ever pay any attention to nutrition "science". Everything is wrong. TFA complimented the meatheads on some bodybuilding site for their skepticism. People who pay very close attention to their own habits and results are probably going to be more right about their own nutrition than all the scientists in the world put together.
I upvoted you because even though you're too far in outright rejecting empiricism about nutrition, the state of nutrition science is indeed quite poor and shrouded in mythology.
Of course one can say basic things like "average daily calories should be between 800 and 3000", but even a range that covers 95% of the population isn't right for everyone, and isn't precise enough to really help anyone. Individuals differ not only in terms of lifestyle, which conceivably could be studied, but also in terms of genetics, which no study to date has examined in more than a rudimentary way. The various statistics sins that we complain about on HN are exacerbated when important factors like genetic background are left unstudied. And we haven't even mentioned the fact that most published (and unpublished) nutrition studies are funded by the food and supplement industries!
A rejection of a particularly poor practice of science is not a rejection of science. I doubt we'll see decent nutrition science until we're all equipped with a network of bloodstream nanobots that measures the actual physical response we each display to nutritional input.
that's not very nice. a lot of people on HN lift weights.
I wouldn't have a problem if posters on said bodybuilding forum referred to HN as "the nerds on some programming site". It's the truth, isn't it?
The word you are looking for is "denizens". No need to slur your speech.
Yeah it wasn't meant as a put-down. Those meatheads know more about their own nutrition, with respect to their focus on muscle formation, than anyone else does. "Meathead" is a literal term for that.
I wouldn't go that far... Nutrition 'science' is quite valid, the only problem is the amount of people in the nutrition space who aren't actually doing 'science'.
Doubt it. People are often quite irrational about these things. Which is why there's so many people who follow fad diets (and buy the books).
Ah the critical distinction is paying attention to your body is not the same thing as buying a book. As per OPs comment, I used to lift, its good exercise, and I saw plenty of demonstrations of data gathering technique and analysis.
It's easy to understand, actually (I take your statement literally).
Being in shape is not rocket science: you eat clean, you do sports, and you're done. Although that sounds ambiguous, everybody does know what clean food is, and what sports mean (ahem, walking doesn't count).
The thing is that it's a damn hard lifestyle, if one doesn't have the right mindset/(self)education/upbring. Nutrition "science" is nothing else that a stream of silver bullets, which are easy to apply (avoid this, avoid that), so the public wants to believe in it, because it makes life much easier.
It's obviously a deception; but truth isn't the point - feeling good about the idea is.
And budget. To do properly requires a pretty good budget for good quality protein sources, vegetables, etc.
Not at all. You project too much. Many people truly believe that healthy food is what the supermarkets & TV tell you is healthy. The popularity of the "NutriBullet" is proof alone - "turn any food into a super food"... Uhh huh.
The real story here is that, even after being shown proof of that it was a hoax all along, a chunk of the same population will continue to hold that eating chocolate helps to lose weight.
I laughed. But more seriously, I wonder if Bohannon's media experiment would have succeeded if you change a detail. E.g. if it was about something truly gross, like drinking curdled milk to lose weight. Journalists want to tell people what they want to hear, because people will tend to buy the paper that says what they are already thinking (weird, I know).
It's human nature that people are more skeptical about claims they don't like.
Imagine how mad the people in the study would be after they found out it was a sham. "I drank curdled milk for 3 weeks so you could get your big break!"
That's an interesting angle even for this study - how do the original participants feel about it I wonder? I'd guess some mixture of amused and cynical.
It is a lot easier to convince someone of something they want to believe than of something they don't.
When I read this, I nodded. But now you've got me wondering if I just want to believe you.
Actually, the takeaway for me is that John Bohannon behaved unethically and made a bunch of invalid conclusions.
Bohannon tested a system, it failed, and he reported exactly how he tested the system and how it failed. 'The system' in this case is the global food science and health journalism industry.
He found that ignorance and laziness is rampant, that journal peer review is non-existent, and skepticism about results and methods is non-existent.
He has done a tremendous public service exposing the flaws in a system. With any luck consumers of these information sources will take everything with an enormous, skeptical grain of salt, because the last and best line of defense is your own skepticism critical thinking.
You can't delegate thinking.
He did not find that "journal peer review is non-existent". He targeted a low-quality journal that does not send out papers for review to make sure he could claim that the research was "published" when sending out his press releases. That's explicit in the article, which you might find an interesting read
This isn't about peer review, because the paper was knowingly submitted only to journals known not to conduct peer review. Quoting from the article:
And later:
This is really more about science journalism than about science.
These kinds of journals are difficult for a lay person to tell apart from actual, scientific journals, but distinguishing between the two is exactly the kind of job a competent science journalist would do. Inept science journalists, on the other hand, give these fake journals a reason to exist by citing them unchallenged.
And his experiment reveals that the Internet gets its nutrition science from inept science journalists. Think of all the arguments won by people searching google and having this show up. I once won an argument by changing Wikipedia during the argument and showing it to him on his computer.
If anything I applaud John's work, this is a great social experiment.
And this attitude is one of the bigger reasons he was able to do it.
@reamworks
[Your post is flagged/dead, so I'm replying here. I hope you see it.]
The relationship between counting calories and weight loss is not quite as simple as the formula [Calories Injested] - [Calories Burned] = [Delta body fat] might indicate.
Details here: http://slatestarcodex.com/2015/01/12/the-physics-diet/
To be honest, chocolate may help in an indirect way...
If someone's on a diet, and never gets any 'reward', they might be tempted to cheat or give up.
If they're told a little chocolate every day is good, they'll stick with their diet, have their chocolate, and forego worse 'cheat' items (even though chocolate has plenty of fat and sugar, the amount you consume before you're content is relatively small compared to say, ice-cream, cookies, etc...).
Studies (and anecdotes) have shown that chocolate makes people happy, so that little happiness at the end might just help people stay on their diet and not stress out (stress can cause the body to store fat). Maybe it's something like the French paradox...
The study was pretty funny though, and I appreciate their honesty. And it goes to show why pretty much all research on fad diets is nonsense. Still though, if you choose between drinking a gallon of pop per day or chocolate, chocolate wins. So maybe their 'research' didn't go to waste.
My takeaway is you can always sell people something they want to buy.
Did any major news organizations retract their articles or apologize?
Do they ever? This is likely one of many reasons for eroding confidence in those same organizations.
It's not news that the peer-review system isn't safe from exploits. People have been using Markov chains to generate studies which have then been published. And journalists uncritically picks up everything they can. All functions in society are based on trust and are easily exploitable if a person is not behaving trustworthy.
So making a fake study about chocolate and getting lots of magazines to publish it I think is a little unethical. It's not news, and the reason most people don't do it is because they want to act trustworthy. For example, someone could, if they wanted, submit patches to some free software projects which would introduce awful security holes and get them merged or just write shit on Wikipedia.
It wasn't actually a fake study. That's the point.
They used the same statistical methods as too many 'honest' groups out there, and got results thereafter.
I disagree. The study was fake because the intent was to deceive. He used flawed methods, but could have used 10 times as sophisticated methods and still would have been able to publish completely invalid results.
I wasn't a fake study, it was a dishonest one.
To continue this line of though, distributed computation is hard problem, and that's just computers. Here, people have to do real work to statically verify a paper's procedure, which is very cognitive overloading. Imagine reviewing a PR and deciding with certainty if it will work or not. Even then they should also "recompute" the experiment to completely verify it. Of course add human error on top of all this.
It seems there's a problem here, the article made it seem anyone can produce a bad study for lulz, and get everyone's attention. What is more scary is people doing it for vested interests. Maybe there's something that could be learned from projects like BIONC, Wikipedia, or Bitcoin, but better science is going to cost more money, and require more cultural attention. Not sure how to fix those things :)
On the flip side, you have some well-heeled corporations and industry groups working to discredit good science that's bad for them. Would be interesting to read a piece on how that's done.
Unfortunately the investment to make good science just to get it discredited by interested parties isn't as cost effective.
I don't think you need to go that far, you just need a PR person with a conscience to spill the beans on what they did (or dissect what others have done).