Posted by Jef Akst, "The Scientist.com"
20th January 2010
Deciding when to pull a grant for any reason is one of the most difficult tasks any funding agency faces. It is not a decision that is taken lightly, and is usually a last resort. But it happens. Scientists who falsify data or misuse funds or even fail to show satisfactory progress do, from time to time, lose their funding.
The National Institutes of Health (NIH) admits to the occasional termination of basic research grants, emphasizing the rarity of such a drastic measure. But how rare is rare? I wanted numbers. To my surprise, they don't seem to exist.
I started with the obvious -- an email to a press officer with a simple question: How many grants had been terminated? The response (attributed to NIH's Office of Extramural Research, OER): "...enforcement actions are taken on a grant-by-grant basis and are not captured centrally in NIH's electronic systems; therefore, no aggregate data are available on revoked grants."
Unsatisfied with this answer, I decided to do a little more digging. Three FOIA requests and several emails and phone calls later, I was finally convinced -- the NIH system simply does not consistently code this information in the database, and thus has no way of assessing, tracking or analyzing how often grants are pulled.
"It's surprising" and "suspicious," said David Kaplan of Case Western Reserve University in Ohio. "Wouldn't they want to know for their own purposes -- to follow how many times is this happening, what are the circumstances, [and] what are the classifications of grants that get revoked?"
Without tracking these statistics, the NIH would not be able to recognize shifts in the frequency of grant terminations, Kaplan said. If, for example, the number of revoked grants increased fivefold, from 10 one year to 50 the next, it might indicate "an unexpected consequence of a policy change," he said. Or it might be a good thing, if it is a result of the agency catching cases of malfeasance that had previously been overlooked. But to even spot such changes, "you've got to follow this stuff," he said.
To complicate matters further, the lack of a consistent way to indicate in the system when a grant is pulled makes it nearly impossible to tell if a particular grant was carried through to completion. After a long series of emails with the OER trying to determine where in the grant file one might find such information, I finally got the unsettling response that "the depth and complexity of NIH's oversight and compliance activities and the variety of options and outcomes available to ensure compliance and effective project management of the projects NIH supports makes it very difficult to tell through the system what happened to a particular project with just a couple of data points." So basically, the information may or may not be there, but either way, the agency's case-by-case approach to handling incidents of grant revocation makes it extremely difficult to find.
This is a serious concern when it comes to future funding, said Nejat D?zg?nes of the University of the Pacific, who argues that "previous productivity" should be considered when reviewing grant applications. "The application at hand might look outstanding, but if this person has a track record of proposing ideas and not following up on them [or having grants actually revoked], then it should raise some questions," he said.
Additionally, the grant file should also indicate the reason that grant ended prematurely, D?zg?nes added. If a grant was terminated for "legitimate reasons," reviewers should be able to see that information "so it's not held against a new project" put forth by the same investigator, he said.
But beyond the issues of record keeping for the NIH's benefit is the more disconcerting issue of public disclosure. "This is a public institution -- I don't think there is any doubt [that this information] should be public," Kaplan said. "People should know."
"You and I are both taxpayers, so that's our money," David Adams of the Duke University School of Medicine agreed. "I think like any other government [agency], where public disclosure is not going to impact national security, I think NIH should report" when a grant has been terminated. "I don't think they need to make a big announcement, but certainly it should be part of the database."
Because the NIH is seemingly very open about cases of confirmed scientific misconduct, some researchers were surprised by this apparent hole in the NIH's system. "My experience is that NIH wouldn't be sweeping those [cases] under the carpet," said Seth Kalichman of the University of Connecticut, who attributes the lack of records on revoked grants to a lack of communication between the "two different huge bureaucracies" that manage NIH grantees -- the grants management office and the Center for Scientific Review.
Furthermore, Kalichman said he would expect the NIH would want to make this information public, as an example to other researchers. "They don't want investigators thinking no one's watching," he said.
Regardless of the reason the NIH does not keep track of grants that have been terminated, the need for doing so is clear. "It's not the most wonderful thing in the world to keep statistics on, but it is important," Kaplan said. "I certainly would advocate that that particular field be included," Adams agreed. "I'm curious myself as to how prevalent it is."
20th January 2010
Deciding when to pull a grant for any reason is one of the most difficult tasks any funding agency faces. It is not a decision that is taken lightly, and is usually a last resort. But it happens. Scientists who falsify data or misuse funds or even fail to show satisfactory progress do, from time to time, lose their funding.
The National Institutes of Health (NIH) admits to the occasional termination of basic research grants, emphasizing the rarity of such a drastic measure. But how rare is rare? I wanted numbers. To my surprise, they don't seem to exist.
I started with the obvious -- an email to a press officer with a simple question: How many grants had been terminated? The response (attributed to NIH's Office of Extramural Research, OER): "...enforcement actions are taken on a grant-by-grant basis and are not captured centrally in NIH's electronic systems; therefore, no aggregate data are available on revoked grants."
Unsatisfied with this answer, I decided to do a little more digging. Three FOIA requests and several emails and phone calls later, I was finally convinced -- the NIH system simply does not consistently code this information in the database, and thus has no way of assessing, tracking or analyzing how often grants are pulled.
"It's surprising" and "suspicious," said David Kaplan of Case Western Reserve University in Ohio. "Wouldn't they want to know for their own purposes -- to follow how many times is this happening, what are the circumstances, [and] what are the classifications of grants that get revoked?"
Without tracking these statistics, the NIH would not be able to recognize shifts in the frequency of grant terminations, Kaplan said. If, for example, the number of revoked grants increased fivefold, from 10 one year to 50 the next, it might indicate "an unexpected consequence of a policy change," he said. Or it might be a good thing, if it is a result of the agency catching cases of malfeasance that had previously been overlooked. But to even spot such changes, "you've got to follow this stuff," he said.
To complicate matters further, the lack of a consistent way to indicate in the system when a grant is pulled makes it nearly impossible to tell if a particular grant was carried through to completion. After a long series of emails with the OER trying to determine where in the grant file one might find such information, I finally got the unsettling response that "the depth and complexity of NIH's oversight and compliance activities and the variety of options and outcomes available to ensure compliance and effective project management of the projects NIH supports makes it very difficult to tell through the system what happened to a particular project with just a couple of data points." So basically, the information may or may not be there, but either way, the agency's case-by-case approach to handling incidents of grant revocation makes it extremely difficult to find.
This is a serious concern when it comes to future funding, said Nejat D?zg?nes of the University of the Pacific, who argues that "previous productivity" should be considered when reviewing grant applications. "The application at hand might look outstanding, but if this person has a track record of proposing ideas and not following up on them [or having grants actually revoked], then it should raise some questions," he said.
Additionally, the grant file should also indicate the reason that grant ended prematurely, D?zg?nes added. If a grant was terminated for "legitimate reasons," reviewers should be able to see that information "so it's not held against a new project" put forth by the same investigator, he said.
But beyond the issues of record keeping for the NIH's benefit is the more disconcerting issue of public disclosure. "This is a public institution -- I don't think there is any doubt [that this information] should be public," Kaplan said. "People should know."
"You and I are both taxpayers, so that's our money," David Adams of the Duke University School of Medicine agreed. "I think like any other government [agency], where public disclosure is not going to impact national security, I think NIH should report" when a grant has been terminated. "I don't think they need to make a big announcement, but certainly it should be part of the database."
Because the NIH is seemingly very open about cases of confirmed scientific misconduct, some researchers were surprised by this apparent hole in the NIH's system. "My experience is that NIH wouldn't be sweeping those [cases] under the carpet," said Seth Kalichman of the University of Connecticut, who attributes the lack of records on revoked grants to a lack of communication between the "two different huge bureaucracies" that manage NIH grantees -- the grants management office and the Center for Scientific Review.
Furthermore, Kalichman said he would expect the NIH would want to make this information public, as an example to other researchers. "They don't want investigators thinking no one's watching," he said.
Regardless of the reason the NIH does not keep track of grants that have been terminated, the need for doing so is clear. "It's not the most wonderful thing in the world to keep statistics on, but it is important," Kaplan said. "I certainly would advocate that that particular field be included," Adams agreed. "I'm curious myself as to how prevalent it is."