Saturday, February 25, 2017

Fake News and Fake? Science

      When I was working on a Ph.D. in Physics my work was to reproduce the findings of another researcher.  It was an extremely difficult experiment that required measurements on ultra-pure sodium metal at near absolute zero temperatures using microwave sensors to pull faint signals barely at the edge of noise levels.  The problem was made worse by the fact that duplicating another physicist's results no matter how impossible the experiment does not qualify one for a Ph.D.  I needed to make my own discoveries.  So when I failed to duplicate his results, I had no problem.  I was not going to be able to publish either way.  Little incentive exists in science to reproduce research.  
     One should not be surprised by a recent report in Nature that 
More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research.
http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970
The data reveal sometimes-contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant 'crisis' of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature.
When researchers were asked why this problem exists most than 60% responded
pressure to publish and selective reporting — always or often contributed. More than half pointed to insufficient replication in the lab, poor oversight or low statistical power. A smaller proportion pointed to obstacles such as variability in reagents or the use of specialized techniques that are difficult to repeat.
In other words a rush to get a scoop and confirmation bias. I would add failure to adequate share the data, ALL OF IT, good and bad, and adequate documentation of technique.

       Nature has decided to take action against this problem.  They are introducing a checklist intended to prompt authors to disclose technical and statistical information in their submissions, and to encourage referees to consider aspects important for research reproducibility. 

http://www.nature.com/news/announcement-reducing-our-irreproducibility-1.12852

No comments:

Post a Comment