Study concludes Psychology research can be questionable
Scientific studies about how people act or think can be replicated less than half the time by outside experts, according to a study that’s raising new questions about the seriousness of psychological research.
A team of 270 scientists tried reproducing 100 psychology and social science studies that had been published in three top peer-reviewed journals.
Just 39 percent came out with the same results as the initial reports, according to the findings, published in an August 2015 issue of the journal Science.
The study topics ranged from people’s social lives and interactions with others to research involving perception, attention and memory.
No medical therapies were called into question as a result of the study, although a separate effort is underway to evaluate cancer biology studies.
“It’s important to note that this somewhat disappointing outcome does not speak directly to the validity or the falsity of the theories,” said Gilbert Chin, a psychologist and senior editor at Science. “What it does say is that we should be less confident about many of the original experimental results.”
Study co-author Brian Nosek of the University of Virginia said the research shows the need for scientists to continually question themselves.
“A scientific claim doesn’t become believable because of the status or authority of the person that generated it,” Nosek told reporters. “Credibility of the claim depends in part on the repeatability of its supporting evidence.”
Problems can arise when scientists cherry-pick their data to include only what is deemed “significant,” or when study sizes are so small that false negatives or false positives arise. Some experts believe the problem may be even worse than the new study suggested.
John Ioannidis, a biologist at Stanford University in California, told Science magazine that he suspects about 25 percent of psychology papers would hold up under scrutiny, about the same “as what we see in many biomedical disciplines.”
One study author who participated in the project as both a reviewer and reviewee was E.J. Masicampo, assistant professor at Wake Forest University in North Carolina.
She was part of a team that was able to replicate a study that found that people who are faced with a confrontational task, such as having to play a violent video game, prefer to listen to angry music and think about negative experiences beforehand.
When outside researchers, however, tried to replicate Masicampo’s own study — which hypothesized that a sugary drink can help college students do better at making a complicated decision — they were not successful.
Masicampo expressed no bitterness, chalking up the differences to geographical factors and stressing that the experiment showed how complicated it can be to do a high-quality replication of a study.
There are ways to fix the process so that findings are more likely to hold up under scrutiny, according to Dorothy Bishop, professor of developmental neuropsychology at the University of Oxford.
She urged mandatory registration of research methods beforehand to prevent scientists from picking only the most favorable data for analysis, as well as requiring adequate sample sizes and wider reporting of studies that show “null result” — or in other words, results that do not support the hypothesis that was initially put forward.
Scientists could also publish their methods and data in detail so that others could try to replicate their experiments more easily. These are “simply ways of ensuring that we are doing science as well as we can,” Bishop said.