After eight years, a project that tried to reproduce the results of key cancer biology studies has finally concluded. And its findings suggest that like research in the social sciences, cancer research has a replication problem.
Researchers with the Reproducibility Project: Cancer Biology aimed to replicate 193 experiments from 53 top cancer papers published from 2010 to 2012. But only a quarter of those experiments were able to be reproduced, the team reports in two papers published December 7 in eLife.
The researchers couldn’t complete the majority of experiments because the team couldn’t gather enough information from the original papers or their authors about methods used, or obtain the necessary materials needed to attempt replication.
What’s more, of the 50 experiments from 23 papers that were reproduced, effect sizes were, on average, 85 percent lower than those reported in the original experiments. Effect sizes indicate how big the effect found in a study is. For example, two studies might find that a certain chemical kills cancer cells, but the chemical kills 30 percent of cells in one experiment and 80 percent of cells in a different experiment. The first experiment has less than half the effect size seen in the second one.
Sign Up For the Latest from Science News
Headlines and summaries of the latest Science News articles, delivered to your inbox
Thank you for signing up!
There was a problem signing you up.
The team also measured if a replication was successful using five criteria. Four focused on effect sizes, and the fifth looked at whether both the original and replicated experiments had similarly positive or negative results, and if both sets of results were statistically significant. The researchers were able to apply those criteria to 112 tested effects from the experiments they could reproduce. Ultimately, just 46 percent, or 51, met more criteria than they failed, the researchers report.
“The report tells us a lot about the culture and realities of the way cancer biology works, and it’s not a flattering picture at all,” says Jonathan Kimmelman, a bioethicist at McGill University in Montreal. He coauthored a commentary on the project exploring the ethical aspects of the findings.
It’s worrisome if experiments that cannot be reproduced are used to launch clinical trials or drug development efforts, Kimmelman says. If it turns out that the science on which a drug is based is not reliable, “it means that patients are needlessly exposed to drugs that are unsafe and that really don’t even have a shot at making an impact on cancer,” he says.
At the same time, Kimmelman cautions against overinterpreting the findings as suggesting that the current cancer research system is broken. “We actually don’t know how well the system is working,” he says. One of the many questions left unresolved by the project is what an appropriate rate of replication is in cancer research, since replicating all studies perfectly isn’t possible. “That’s a moral question,” he says. “That’s a policy question. That’s not really a scientific question.”
The overarching lessons of the project suggest that substantial inefficiency in preclinical research may be hampering the drug development pipeline later on, says Tim Errington, who led the project. He is the director of research at the Center for Open Science in Charlottesville, Va., which cosponsored the research.
As many as 14 out of 15 cancer drugs that enter clinical trials never receive approval from the U.S. Food and Drug Administration. Sometimes that’s because the drugs lack commercial potential, but more often it is because they do not show the level of safety and effectiveness needed for licensure.
Much of that failure is expected. “We’re humans trying to understand complex disease, we’re never going to get it right,” Errington says. But given the cancer reproducibility project’s findings, perhaps “we should have known that we were failing earlier, or maybe we don’t understand actually what’s causing [an] exciting finding,” he says.
Still, it’s not that failure to replicate means that a study was wrong or that replicating it means that the findings are correct, says Shirley Wang, an epidemiologist at Brigham and Women’s Hospital in Boston and Harvard Medical School. “It just means that you’re able to reproduce,” she says, a point that the reproducibility project also stresses.
Scientists still have to evaluate whether a study’s methods are unbiased and rigorous, says Wang, who was not involved in the project but reviewed its findings. And if the results of original experiments and their replications do differ, it’s a learning opportunity to find out why and the implications, she adds.
Errington and his colleagues have reported on subsets of the cancer reproducibility project’s findings before, but this is the first time that the effort’s entire analysis has been released (SN: 1/18/17).
During the project, the researchers faced a number of obstacles, particularly that none of the original experiments included enough details in their published studies about methods to attempt reproduction. So the reproducibility researchers contacted the studies’ authors for additional information.
While authors for 41 percent of the experiments were extremely or very helpful, authors for another third of the experiments did not reply to requests for more information or were not otherwise helpful, the project found. For example, one of the experiments that the group was unable to replicate required the use of a mouse model specifically bred for the original experiment. Errington says that the scientists who conducted that work refused to share some of these mice with the reproducibility project, and without those rodents, replication was impossible.
Some researchers were outright hostile to the idea that independent scientists wanted to attempt to replicate their work, says Brian Nosek, executive director at the Center for Open Science and a coauthor on both studies. That attitude is a product of a research culture that values innovation over replication, and that prizes the academic publish-or-perish system over cooperation and data sharing, Nosek says.
Some scientists may feel threatened by replication because it is uncommon. “If replication is normal and routine, people wouldn’t see it as a threat,” Nosek says. But replication may also feel intimidating because scientists’ livelihoods and even identities are often so deeply rooted in their findings, he says. “Publication is the currency of advancement, a key reward that turns into chances for funding, chances for a job and chances for keeping that job,” Nosek says. “Replication doesn’t fit neatly into that rewards system.”
Even authors who wanted to help couldn’t always share their data for various reasons, including lost hard drives or intellectual property restrictions or data that only former graduate students had.
Calls from some experts about science’s “reproducibility crisis” have been growing for years, perhaps most notably in psychology (SN: 8/27/18). Then in 2011 and 2012, pharmaceutical companies Bayer and Amgen reported difficulties in replicating findings from preclinical biomedical research.
But not everyone agrees on solutions, including whether replication of key experiments is actually useful or possible, or even what exactly is wrong with the way science is done or what needs to improve (SN: 1/13/15).
At least one clear, actionable conclusion emerged from the new findings, says Yvette Seger, director of science policy at the Federation of American Societies for Experimental Biology. That’s the need to provide scientists with as much opportunity as possible to explain exactly how they conducted their research.
“Scientists should aspire to include as much information about their experimental methods as possible to ensure understanding about results on the other side,” says Seger, who was not involved in the reproducibility project.
Ultimately, if science is to be a self-correcting discipline, there needs to be plenty of opportunities not only for making mistakes but also for discovering those mistakes, including by replicating experiments, the project’s researchers say.
“In general, the public understands science is hard, and I think the public also understands that science is going to make errors,” Nosek says. “The concern is and should be, is science efficient at catching its errors?” The cancer project’s findings don’t necessarily answer that question, but they do highlight the challenges of trying to find out.
FAQs
How many studies Cannot be replicated? ›
It found that about 39% failed to reproduce the original results.
What does it mean if a study cant be replicated? ›Scientists aim for their studies to be replicable — meaning that another researcher could perform a similar investigation and obtain the same basic results. When a study cannot be replicated, it suggests that our current understanding of the study system or our methods of testing are insufficient.
What is one reason why a study may be difficult to replicate? ›A failure to replicate previous results can be due to any number of factors, including the discovery of an unknown effect, inherent variability in the system, inability to control complex variables, substandard research practices, and, quite simply, chance.
Can studies be replicated? ›What is Replication of a Study? Replication of a study is repeating a study's procedure and observing if the prior findings repeat in similar conditions. A study is replicated when the results of original study are as closely related to the newly collected data.
What percentage of studies are replicated? ›In psychology, only 39 percent of the 100 experiments successfully replicated. In economics, 61 percent of the 18 studies replicated as did 62 percent of the 21 studies published in Nature/Science.
Why do we repeat experiments 3 times? ›Repeating an experiment more than once helps determine if the data was a fluke, or represents the normal case. It helps guard against jumping to conclusions without enough evidence. The number of repeats depends on many factors, including the spread of the data and the availability of resources.
What are the main causes of lack of reproducibility? ›- Lack of Access to Raw Data and Methodologies. ...
- Invalidated Biological Materials. ...
- Lack of Knowledge to Analyze Data. ...
- Incorrect Laboratory Practices. ...
- Undervaluing Negative Results. ...
- Open Science Approach. ...
- Use of Validated Biomaterials.
In short, the results of following the same procedure may be different because of inherent variability of materials and chance. The great the experimenter's control over these factors, the more reproducible the results.
What are the disadvantages when studies Cannot be replicated? ›These studies point to how challenging it can be to replicate research. Yet without that replication, the research community may find it hard to trust a study's data or know how to interpret what those data mean.
What are the barriers to replication studies? ›We identify four main barriers to carrying out replication studies: resource availability, publication bias, regulatory constraints, and social factors. We argue that all four barriers are exacerbated when the focus is on threatened species, and that they are likely to persist in the future.
What will most likely be a limitation for other scientists trying to recreate this study? ›
What will most likely be a limitation for other scientists trying to recreate this study? There are ethical controversies over testing human subjects.
Does the research study Cannot be replicated or repeated? ›The research study cannot be replicated or repeated because it is unique in every case. Data are in the form of numbers and analyzed statistically. Data analysis is an on-going process. It can be done at any stage of the process.
Is it ethical to replicate a study? ›Reproducibility is not just a scientific issue; it is also an ethical one. When scientists cannot reproduce a research result, they may suspect data fabrication or falsification. In several well-known cases, reproducibility issues led to allegations of data fabrication or falsification.
Can you replicate an experiment? ›A scientific experiment is replicable if it can be repeated with the same analytical results. Due to all sorts of factors, including random variability, this is not as common as some might think.
Is there a replication crisis? ›The replication crisis, also known as the reproducibility crisis and the replicability crisis, is a crisis that impacts the methodology of scientific research. Over time, it has been realized by several bodies that the results of many scientific studies are hard or almost impossible to accurately reproduce.
Should all studies be replicated? ›One of the most important features of a scientific research paper is that the research must be replicable, which means that the paper gives readers enough detailed information that the research can be repeated (or 'replicated').
How many replicates is enough for an experiment? ›Normally we design experiment with 3 replicates, each replicate has like 10 samples/treatment (so total number of samples n = 30/treatment). Then we average the results of these 10 samples to get 1 number/replicate and use these 3 numbers/treatment to performing statistical analysis.
Does repeating an experiment increase accuracy? ›The accuracy of a measurement is dependent on the quality of the measuring apparatus and the skill of the scientist involved. For data to be considered reliable, any variation in values must be small. Repeating a scientific investigation makes it more reliable.
What is the difference between repeat and replicate? ›Repeat and replicate measurements are both multiple response measurements taken at the same combination of factor settings; but repeat measurements are taken during the same experimental run or consecutive runs, while replicate measurements are taken during identical but different experimental runs, which are often ...
Does increasing the number of trials improve accuracy? ›Repeated trials are where you measure the same thing multiple times to make your data more reliable. This is necessary because in the real world, data tends to vary and nothing is perfect. The more trials you take, the closer your average will get to the true value.
Is science really facing a reproducibility crisis? ›
To summarize, an expanding metaresearch literature suggests that science—while undoubtedly facing old and new challenges—cannot be said to be undergoing a “reproducibility crisis,” at least not in the sense that it is no longer reliable due to a pervasive and growing problem with findings that are fabricated, falsified ...
Is there a reproducibility crisis in science? ›It's the base rate fallacy. The base rate fallacy may help to explain low reproducibility in various fields of science. The replication crisis refers to a longstanding pattern of researchers being unable to replicate the findings of previous studies.
How can we fix the reproducibility crisis? ›Here's the solution: replicate the study in the original publication. Simple. Most original studies you'll read only have one sample and one set of results recorded. Some articles may have multiple variations of the same experiment, but they're variations.
Can scientific results be replicated? ›Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. Two studies may be considered to have replicated if they obtain consistent results given the level of uncertainty inherent in the system under study.
Can research be replicated with the same result? ›Getting the same result when an experiment is repeated is called replication. If research results can be replicated, it means they are more likely to be correct. Repeated replication of investigations may turn a hypothesis into a theory.
What are the pros and cons of Replication? ›- Advantages & Disadvantages of Data Replication. There are following advantages of replication:
- Availability. ...
- Increased parallelism. ...
- Less Data Movement over Network. ...
- Increased overhead on update. ...
- Require more disk space. ...
- Expensive.
The “replication crisis” in psychology, as it is often called, started around 2010, when a paper using completely accepted experimental methods was published purporting to find evidence that people were capable of perceiving the future, which is impossible.
What factors contribute to the replication crisis? ›- Failure to comply with proper practices. Research and scientific practices should be scrupulous and have high attention to detail. ...
- Rush to publish. ...
- Lack of transparency. ...
- Issues with the peer-review system. ...
- Public interest.
There are least 3 general types of replication studies- direct replication, conceptual replication and replication-plus-extension. In direct replication, researchers attempt to conduct research using methods that are as close as they can to those used by original researchers.
What are the three requirements of replication? ›Replication occurs in three major steps: the opening of the double helix and separation of the DNA strands, the priming of the template strand, and the assembly of the new DNA segment.
What are the disadvantages of data replication? ›
DISADVANTAGES OF DATA REPLICATION –
More storage space is needed as storing the replicas of same data at different sites consumes more space. Data Replication becomes expensive when the replicas at all different sites need to be updated. Maintaining Data consistency at all different sites involves complex measures.
- Questions about value.
- Questions of morality[2]
- Questions about the supernatural.
- Questions concerning ultimate reality.
- Science doesn't make moral judgments.
- Science doesn't make aesthetic judgments.
- Science doesn't tell you how to use scientific knowledge.
- Science doesn't draw conclusions about supernatural explanations.
- 1st Limitation. Science deals with only things that can be observed.
- 2nd Limitation. Scientific observations may be faulty.
- 3rd Limitation. Scientists can be bias.
- 4th Limitation. Science cannot make value judgments.
- 5th Limitation. Science cannot provide universal statements.
- 6th Limitation.
Replicability is a necessary factor in the knowledge production process in the natural sciences because it facilitates the independent verification of data and is also considered a significant principle of scientific research.
Why do scientists need to replicate a study? ›Research must be repeated before a finding can be accepted as well-established. Findings obtained at one time might not hold true at another time with different researchers or different experimental subjects. To check the reliability of a finding, one must replicate the research.
What does it mean if a study Cannot be replicated? ›Scientists aim for their studies to be replicable — meaning that another researcher could perform a similar investigation and obtain the same basic results. When a study cannot be replicated, it suggests that our current understanding of the study system or our methods of testing are insufficient.
What is lack of replication in an experiment? ›The term, which originated in the early 2010s, denotes that findings in behavioral science often cannot be replicated: Researchers do not obtain results comparable to the original, peer-reviewed study when repeating that study using similar procedures.
› replication-study ›Replication Study - Repeating a Previous Study
Replication in Psychology Research
Understanding Health Research · Replicability
It is very important that research can be replicated, because it means that other researchers can test the findings of the research. Replicability keeps researchers honest and can give readers confidence in research.
How many replications does an experiment have? ›
Normally we design experiment with 3 replicates, each replicate has like 10 samples/treatment (so total number of samples n = 30/treatment). Then we average the results of these 10 samples to get 1 number/replicate and use these 3 numbers/treatment to performing statistical analysis.
Does the research study Cannot be replicated or repeated? ›The research study cannot be replicated or repeated because it is unique in every case. Data are in the form of numbers and analyzed statistically. Data analysis is an on-going process. It can be done at any stage of the process.
How many times can DNA be replicated? ›Abstract. The preparation for DNA replication initiation is tightly linked to cell-cycle progression, ensuring that replication occurs only once per cycle. The time is ripe for a molecular dissection of the links between the two processes.
Why is research can be replicated any time but its findings can never be duplicated? ›In short, the results of following the same procedure may be different because of inherent variability of materials and chance. The great the experimenter's control over these factors, the more reproducible the results. All the best with your research!
Can you replicate an experiment? ›A scientific experiment is replicable if it can be repeated with the same analytical results. Due to all sorts of factors, including random variability, this is not as common as some might think.
What are the main causes of lack of reproducibility? ›Use of misidentified, cross-contaminated, or over-passaged cell lines and microorganisms. Reproducibility can be complicated and/or invalidated by biological materials that cannot be traced back to their original source, are not thoroughly authenticated, or are not properly maintained.
Why are replications not typically conducted? ›Replications of one's own earlier work, or the work of others, is typically discouraged because it does not represent original thinking. Instead, academics are rewarded for high numbers of publications, and flashy studies are often given prominence in media reports of published studies.
How do you determine how many replicates are enough? ›You can determine the number of experiments you would do by multiplying 3X4X n, where n is the number of replications. Please note that replications should be at least 2. The more you do replications, the more precise results you get.
Do replicates increase accuracy? ›Using replicates offers three major advantages: Replicates can be used to measure variation in the experiment so that statistical tests can be applied to evaluate differences. Averaging across replicates increases the precision of gene expression measurements and allows smaller changes to be detected.
What makes a study replicable? ›Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. Two studies may be considered to have replicated if they obtain consistent results given the level of uncertainty inherent in the system under study.
Can research be replicated with the same result? ›
Getting the same result when an experiment is repeated is called replication. If research results can be replicated, it means they are more likely to be correct. Repeated replication of investigations may turn a hypothesis into a theory.
What is the difference between replication and repeat trials? ›Repeated trials are used within a study to verify the results and enhance the statistical measures. Replication is used to verify the findings of the original group by researchers outside of the original group.
How often is DNA replication wrong? ›Nonetheless, these enzymes do make mistakes at a rate of about 1 per every 100,000 nucleotides. That might not seem like much, until you consider how much DNA a cell has. In humans, with our 6 billion base pairs in each diploid cell, that would amount to about 120,000 mistakes every time a cell divides!
How long would it take you to copy all your DNA? ›An average-sized human chromosome contains a single linear DNA molecule of about 150 million nucleotide pairs. To replicate such a DNA molecule from end to end with a single replication fork moving at a rate of 50 nucleotides per second would require 0.02 × 150 × 106 = 3.0 × 106 seconds (about 800 hours).
How often does DNA replication get it wrong? ›It is estimated that replicative eukaryotic DNA polymerases make errors approximately once every 104 – 105 nucleotides polymerized [58, 59]. Thus, each time a diploid mammalian cell replicates, at least 100,000 and up to 1,000,000 polymerase errors occur.