Retraction Watch
3-30-18
Nature journal retracts controversial CRISPR paper after authors admit results may be wrong
Nature Methods has retracted a 2017 paper suggesting a common gene editing technique may cause widespread collateral damage to the genome.
The notice has a long backstory: After the paper was published, it immediately drew an outcry from critics (including representatives from companies who sell the tool, whose stock fell after publication). Some critics argued that the authors, led by Vinit B. Mahajan at Stanford University, hadn’t employed sufficient controls, so they couldn’t be sure that the observed mutations stemmed from the tool, rather than normal background variation between mice. Only months after the paper appeared, the journal issued an expression of concern about the article. In a new preprint posted on BioRxiv on Monday, the authors concede that their critics may be right.
In the new preprint, Mahanjan and his colleagues acknowledge that the gene editing technique — known as CRISPR-Cas9 — “may not introduce numerous, unintended, off-target mutations.”
However, according to the retraction notice, Mahajan and several of his co-authors object to the retraction:
This paper is being retracted because the genomic variants observed by the authors in two CRISPR-treated mice cannot be conclusively attributed to CRISPR/Cas9. The paper was a peer reviewed Correspondence in the journal. The authors made their observation as part of their work on correction of a gene involved in blindness. The authors used mice of the inbred FVB/NJ strain from the JAX genetic quality control program that were purchased within months of each other and that were not bred in the authors’ laboratory. The assumption was that this design was sufficient to control for genetic variation in an inbred strain. Since publication of the work, however, it has been brought to the journal’s and the authors’ attention that without parental controls or more analysis of genetic background, it is not certain that the variants reported are due to CRISPR treatment (https://www.nature.com/articles/nmeth.4559, https://www.nature.com/articles/nmeth.4552, https://www.nature.com/articles/nmeth.4553, https://www.nature.com/articles/nmeth.4541, https://www.nature.com/articles/nmeth.4554). The study is therefore being retracted to maintain the accuracy of the scientific record.
SHT and WHW agree with the retraction. KAS, DFC, AGB and VBM do not agree with the retraction.
All authors note that there is very little whole genome sequencing data on the effects of CRISPR treatment in vivo. The question of whether or not CRISPR has effects on the in vivo genome will require further study; the authors are carrying out follow up studies using whole genome sequencing.
“Unexpected mutations after CRISPR–Cas9 editing in vivo” has been cited 39 times since it was published last year, according to Clarivate Analytics’ Web of Science — 34 of which are dated after Nature Methods published the expression of concern.
“We regret this omission”
Stephen Tsang of Columbia University, one of the two authors who agreed with the retraction, told us:
First and foremost, I trust you noted that our Nature correspondence was free from any scientific misconduct or misrepresentation. At its core, our correspondence was analogous to a case report, which by definition lacks the degree of scientific rigor expected in a full research article. Nevertheless, such case reports are published in esteemed clinical journals, including NEJM.
Since our Nature submission, it has become open to question whether or not the variants in inbred mice are due to non-clonality or CRISPR. A recent independent analysis of our data by a separate research group reported that our variants were not germline (https://www.biorxiv.org/content/early/2017/09/28/193565), thus agreeing with our correspondence. To our knowledge, however, only 1 FVB/NJ mouse in the entire public database has undergone whole genome sequencing (WGS) at 50x resolution (lower resolution compared to our WGS 60X) . Therefore, we ultimately agree with the retraction given that more WGS should be performed to preserve scientific rigor. It is only then that any definite conclusions can be made.
George Church, a geneticist at Harvard and co-founder of Editas, which sells CRISPR technology, told Retraction Watch:
This seems like a great example of rapidly self-correcting science.
Although stocks of companies selling CRISPR initially tumbled after the paper first appeared, Church told us:
I was never worried. Some investors look for opportunities to sell high then buy back low and then watch the rebound — based more on herd psychology than lab science.
The retraction notice is published alongside five articles that critique the paper’s findings. The journal is also publishing an editorial, which notes that four outside experts reviewed the five articles and the study authors’ responses to them, as part of “multiple rounds of peer review.” The conclusion:
Without a more direct assessment of the background variation in the animals used for the experiment, it is not possible to determine whether the variants reported by Schaefer et al. represent off-target effects of CRISPR or simply reflect variation already present in the background of these mice. The central claim of the paper is therefore not sufficiently supported by the data. This is the reason for retraction of the paper.
The editorial says the journal decided to publish the five critiques of the paper but not the authors’ responses, because:
…the latter did not resolve the central criticism: that the study lacked controls needed to ascribe a causal role to CRISPR.
The editorial adds:
The original paper was peer reviewed, but we should have sought at least one additional referee with expertise in the genetics of inbred mouse strains. We regret this omission. While ensuring appropriate referee expertise is a task we have always taken seriously, and is a central part of the editorial process, we have now put in place further processes to reduce the likelihood that such an error will happen again.
Before we learned about the retraction, we asked Mahajan this week whether the authors plan to issue another notice about the paper, or request a retraction; he told us:
There continues to be great interest in CRISPR/Cas9 off-targeting from our group and others. Our methodology of whole genome sequencing after CRISPR in vivo raised important questions calling for more research about potential adverse effects and model systems. At least one group studied the data and found that many of the mutations were not likely to be inherited in mice (https://www.biorxiv.org/content/early/2017/09/28/193565). New versions of Cas9 have been designed specifically to have less off-target risks, and some other groups, including at Microsoft (https://www.nature.com/articles/s41551-017-0178-6), are using exciting new techniques such as machine learning to help avoid Cas9 off-targeting. Our own group is sequencing many more CRISPR/cas9 treated mice, created with different versions of cas9, and we look forward to other scientists and corporations publishing their own whole genome sequence data and analysis. We are confident that by working together in an open, transparent and unbiased manner, the research community can find safe and effective ways to implement new therapies for our patients.
A spokesperson for Intellia Therapeutics, another gene editing company (which also contributed one of the five critiques published today), told us she thought the retraction was “appropriate:”
As with any new technology, we should be investigating the benefits and risks. At the same time, claims of flaws in the CRISPR system should also be reviewed carefully. We are glad that with rigorous re-examination by Intellia as well as other groups, the scientific record has now been corrected.
Vic Myer, chief technology officer of Editas Medicine (and a co-author on one of the five critiques published today) told us:
Our goal is to make safe and effective genomic medicines for a broad class of diseases, and Cas9 continues to distinguish itself as a powerful, precise genome editing system. We believed that the conclusions drawn from the initial study were unsubstantiated by the disclosed experiments as they were designed and carried out, and thus we engaged in appropriate scientific discourse with the journal. We’re pleased that the scientific process worked to correct the record and that our response was considered during this process.
Lauryl Nutter, associate director at the Centre for Phenogenomics in Toronto (and co-author of one of the five criticisms published today), told us:
As we said in our communication to Nature, I and my co-authors think there were serious underlying flaws in the experimental design and interpretation of data in the original manuscript; so much so that the authors’ conclusions were not supported by the data presented. In the absence of additional data to support their conclusions and given other publications – both peer reviewed and pre-prints on BioRivx – that demonstrate very low off-target rates for Cas9 in mouse zygotes, it seems appropriate to retract the paper.
Nutter added:
While I do applaud Nature in publishing an EOC once the widespread nature of the concerns about this particular paper were recognized, I do wonder why the review process did not raise these same concerns before publication. The flaws were not recognized by a few specialists in the field, but rather many scientists in the broader genetics and biomedical research community.
3-30-18
Nature journal retracts controversial CRISPR paper after authors admit results may be wrong
Nature Methods has retracted a 2017 paper suggesting a common gene editing technique may cause widespread collateral damage to the genome.
The notice has a long backstory: After the paper was published, it immediately drew an outcry from critics (including representatives from companies who sell the tool, whose stock fell after publication). Some critics argued that the authors, led by Vinit B. Mahajan at Stanford University, hadn’t employed sufficient controls, so they couldn’t be sure that the observed mutations stemmed from the tool, rather than normal background variation between mice. Only months after the paper appeared, the journal issued an expression of concern about the article. In a new preprint posted on BioRxiv on Monday, the authors concede that their critics may be right.
In the new preprint, Mahanjan and his colleagues acknowledge that the gene editing technique — known as CRISPR-Cas9 — “may not introduce numerous, unintended, off-target mutations.”
However, according to the retraction notice, Mahajan and several of his co-authors object to the retraction:
This paper is being retracted because the genomic variants observed by the authors in two CRISPR-treated mice cannot be conclusively attributed to CRISPR/Cas9. The paper was a peer reviewed Correspondence in the journal. The authors made their observation as part of their work on correction of a gene involved in blindness. The authors used mice of the inbred FVB/NJ strain from the JAX genetic quality control program that were purchased within months of each other and that were not bred in the authors’ laboratory. The assumption was that this design was sufficient to control for genetic variation in an inbred strain. Since publication of the work, however, it has been brought to the journal’s and the authors’ attention that without parental controls or more analysis of genetic background, it is not certain that the variants reported are due to CRISPR treatment (https://www.nature.com/articles/nmeth.4559, https://www.nature.com/articles/nmeth.4552, https://www.nature.com/articles/nmeth.4553, https://www.nature.com/articles/nmeth.4541, https://www.nature.com/articles/nmeth.4554). The study is therefore being retracted to maintain the accuracy of the scientific record.
SHT and WHW agree with the retraction. KAS, DFC, AGB and VBM do not agree with the retraction.
All authors note that there is very little whole genome sequencing data on the effects of CRISPR treatment in vivo. The question of whether or not CRISPR has effects on the in vivo genome will require further study; the authors are carrying out follow up studies using whole genome sequencing.
“Unexpected mutations after CRISPR–Cas9 editing in vivo” has been cited 39 times since it was published last year, according to Clarivate Analytics’ Web of Science — 34 of which are dated after Nature Methods published the expression of concern.
“We regret this omission”
Stephen Tsang of Columbia University, one of the two authors who agreed with the retraction, told us:
First and foremost, I trust you noted that our Nature correspondence was free from any scientific misconduct or misrepresentation. At its core, our correspondence was analogous to a case report, which by definition lacks the degree of scientific rigor expected in a full research article. Nevertheless, such case reports are published in esteemed clinical journals, including NEJM.
Since our Nature submission, it has become open to question whether or not the variants in inbred mice are due to non-clonality or CRISPR. A recent independent analysis of our data by a separate research group reported that our variants were not germline (https://www.biorxiv.org/content/early/2017/09/28/193565), thus agreeing with our correspondence. To our knowledge, however, only 1 FVB/NJ mouse in the entire public database has undergone whole genome sequencing (WGS) at 50x resolution (lower resolution compared to our WGS 60X) . Therefore, we ultimately agree with the retraction given that more WGS should be performed to preserve scientific rigor. It is only then that any definite conclusions can be made.
George Church, a geneticist at Harvard and co-founder of Editas, which sells CRISPR technology, told Retraction Watch:
This seems like a great example of rapidly self-correcting science.
Although stocks of companies selling CRISPR initially tumbled after the paper first appeared, Church told us:
I was never worried. Some investors look for opportunities to sell high then buy back low and then watch the rebound — based more on herd psychology than lab science.
The retraction notice is published alongside five articles that critique the paper’s findings. The journal is also publishing an editorial, which notes that four outside experts reviewed the five articles and the study authors’ responses to them, as part of “multiple rounds of peer review.” The conclusion:
Without a more direct assessment of the background variation in the animals used for the experiment, it is not possible to determine whether the variants reported by Schaefer et al. represent off-target effects of CRISPR or simply reflect variation already present in the background of these mice. The central claim of the paper is therefore not sufficiently supported by the data. This is the reason for retraction of the paper.
The editorial says the journal decided to publish the five critiques of the paper but not the authors’ responses, because:
…the latter did not resolve the central criticism: that the study lacked controls needed to ascribe a causal role to CRISPR.
The editorial adds:
The original paper was peer reviewed, but we should have sought at least one additional referee with expertise in the genetics of inbred mouse strains. We regret this omission. While ensuring appropriate referee expertise is a task we have always taken seriously, and is a central part of the editorial process, we have now put in place further processes to reduce the likelihood that such an error will happen again.
Before we learned about the retraction, we asked Mahajan this week whether the authors plan to issue another notice about the paper, or request a retraction; he told us:
There continues to be great interest in CRISPR/Cas9 off-targeting from our group and others. Our methodology of whole genome sequencing after CRISPR in vivo raised important questions calling for more research about potential adverse effects and model systems. At least one group studied the data and found that many of the mutations were not likely to be inherited in mice (https://www.biorxiv.org/content/early/2017/09/28/193565). New versions of Cas9 have been designed specifically to have less off-target risks, and some other groups, including at Microsoft (https://www.nature.com/articles/s41551-017-0178-6), are using exciting new techniques such as machine learning to help avoid Cas9 off-targeting. Our own group is sequencing many more CRISPR/cas9 treated mice, created with different versions of cas9, and we look forward to other scientists and corporations publishing their own whole genome sequence data and analysis. We are confident that by working together in an open, transparent and unbiased manner, the research community can find safe and effective ways to implement new therapies for our patients.
A spokesperson for Intellia Therapeutics, another gene editing company (which also contributed one of the five critiques published today), told us she thought the retraction was “appropriate:”
As with any new technology, we should be investigating the benefits and risks. At the same time, claims of flaws in the CRISPR system should also be reviewed carefully. We are glad that with rigorous re-examination by Intellia as well as other groups, the scientific record has now been corrected.
Vic Myer, chief technology officer of Editas Medicine (and a co-author on one of the five critiques published today) told us:
Our goal is to make safe and effective genomic medicines for a broad class of diseases, and Cas9 continues to distinguish itself as a powerful, precise genome editing system. We believed that the conclusions drawn from the initial study were unsubstantiated by the disclosed experiments as they were designed and carried out, and thus we engaged in appropriate scientific discourse with the journal. We’re pleased that the scientific process worked to correct the record and that our response was considered during this process.
Lauryl Nutter, associate director at the Centre for Phenogenomics in Toronto (and co-author of one of the five criticisms published today), told us:
As we said in our communication to Nature, I and my co-authors think there were serious underlying flaws in the experimental design and interpretation of data in the original manuscript; so much so that the authors’ conclusions were not supported by the data presented. In the absence of additional data to support their conclusions and given other publications – both peer reviewed and pre-prints on BioRivx – that demonstrate very low off-target rates for Cas9 in mouse zygotes, it seems appropriate to retract the paper.
Nutter added:
While I do applaud Nature in publishing an EOC once the widespread nature of the concerns about this particular paper were recognized, I do wonder why the review process did not raise these same concerns before publication. The flaws were not recognized by a few specialists in the field, but rather many scientists in the broader genetics and biomedical research community.