Post Tagged with: "peer-review"

Who knows reviewers’ identities?

A couple of posts ago I shared a pretty unpleasant experience I had after peer-reviewing a grant application. In short my anonymity appeared to have been breached and I received ,what I took to be, a thinly veiled threat from the grant’s author.

Some of the comments that followed thanked me for bring the case to light but were critical because I hadn’t gone far enough and named names. Therefore what had the post achieved? I take the point, but I’m still not prepared to name the persons or organisations.

However, I did contact the funding body involved who were willing to investigate the matter. They also suggested that I first file a freedom-of-information request asking for details on how anonymity is protected and with whom reviewer identities are shared.

For the sake of completion I asked the main science research councils in the UK (not just the one that was involved in my incident) for the same information (details below).

In short most committees (that’s typically upwards of two dozen people) are aware of the identify of reviewers. This is probably not news to most, but I figure its good to know who knows who you are.

In most cases identities are revealed in the committee meeting. Which made me consider how the reviewer knew my identify before the committee sat. Until I remembered that I had reviewed the grant twice (it had been rejected the first time, but a resubmission was requested). So a panel member from the first meeting must have made a note (mentally or otherwise) of my name and then shared it.

It strikes me that its rather too easy for reviewer anonymity to be breached. So what’s to be done? In the short term maybe its worth checking who is on a panel. Then if you know of a relationship between a member and the proposal’s author, that might result in a leak,  refuse to review the grant. In the longer term, should the system change so that reviewers are truly anonymous? And as for me, my next step is to take up the offer of that investigation.

And in case you are interested here are the more detailed responses from the research councils.

The BBSRC website provides a document called “Peer review and freedom of information’. Part of which states that the identify of reviewers is

‘…available to the members of the peer review body.’

I asked for clarification on what constitutes a ‘peer review body’. To which they responded

‘.. peer review body refers to Research Committees and any other ad hoc panels that assess grant applications.’

they added that

Their identity [of reviewers] is revealed to the peer review body only as part of normal business meeting

The other research councils have similar freedom of information documents to the BBSRC, however they are less clear on the policies with regards to reviewer identities. So I asked them directly if identities are revealed to panels/committees and how anonymity is assured.

NERC replied :

Panels see the reviewer names but they are required to keep all the business of the meeting confidential including reviewers names and which members of the panel introduce the proposal and we rely on them to do that. That confidentiality is set out in the Reviewer Protocols. These are available on the NERC website and part of signing up to the Peer Review College. Access to proposal information is via Je-S [ grant submission/review system] and panel members are required to sign up to the Reviewer Protocols before they can see the proposal information in order to review it. Details of the Reviewer Protocols can be found at:

http://www.nerc.ac.uk/funding/application/peerreview/members-details/PRC-reviewer-protocols.pdf

The EPSRC‘s statement was:

Each reviewer has an anonymised reference for each proposal they are sent, but their identity is indicated in the meeting schedule provided to the panel members. Each schedule is customised so that if the panel member has a conflict of interest the reviewers remain anonymous for the affected proposal.

All panel members are sent guidance which includes our code of practice based on the Nolan Committee’s seven principles of public life – for further details please refer to the link below.

http://www.epsrc.ac.uk/funding/peerrev/panels/Pages/protocols.aspx

And from the MRC I received:

At the board/panel meeting itself the names of the reviewers are projected onto a screen for each application in turn.  This information is not provided in hard copy to any of the board/panel members.

At Fellowship panels, the information is provided to the chair of the meeting and the information is securely destroyed immediately following the meeting.  Other panel members may request the information at the meeting and may be provided with this information verbally at the meeting to enable better decision making.

The Biomedical catalyst panel meeting is currently the only exception to this as information is revealed to panel members prior to the meeting. This is because the meeting is jointly run with the Technology Strategy Board.

 

By May 15, 2014 0 comments opinion, science policy

‘Anonymous’ peer-review – a cautionary tale

Some time ago I received a grant proposal to review. Its was from an eminent professor.

I was extremely disappointed, on numerous accounts, with the quality of the proposal. In short I thought it relied very heavily on the authors standing in the community and only paid lip service to the science. And so (naively) safe in the knowledge that my review would remain anonymous, I gave a frank assessment of it.  I advised the panel to reject the proposal.

Satisfied that I had done my job I gave it no more thought until, a few months later, I received an email from the proposal’s author. It started …

I have had a review – see below – for a pending application with [research council/charity]. It is of the “unhelpful” type the community gets, and looks as if it from an unsympathetic, uninformed, prejudiced non-expert – the other reviews were all very positive, but I now have to answer the points.

What followed was a spurious query about some detail in one of my papers, which he/she apparently needed to help address the comments I’d given as an ‘anonymous’ referee.

It seemed to me that the real reason for the email was to tell me off.  But how could this happen? How could the author have found out that I had refereed the proposal? So I sort advice from senior colleagues who had sat on research council/charity committees. And I was shocked by what they told me. It seemed that it wasn’t unusual for the committees to know who the referees are, but even worse committee members were often happy to share this information with eminent grant applicants.

Since this episode I’ve been in a quandary.  What should I do when asked to review a proposal from a high profile scientist? Should I be truthful about what I think, risk being found out and so make enemies? Should I lie and write glowing reviews based on who the applicant is and not what is proposed? Or should I refuse to review applications from the higher echelons of community? To be honest I’m still at a loss.

There’s more. The email went on …

Also, I am getting personal comments from others in the field [lists of other eminent names] to ensure that the [research council/charity name] panel does understand the importance of the approach, to help all of us doing [technique], and not create a negative and highly damaging impression of [technique] at their panel from this kind of review – it playing right into the hands of the [another technique] community which is well represented on this panel.

So it seems that, if you have the connections, its acceptable to seek one’s own referees and lobby the panel with them.

The whole episode has left me with a very bitter taste and a complete lack of confidence in the validity of the  peer-review system.

Finally, this all happened sometime ago, and since then I’ve thought long and hard about whether I should make the incident public. After all, pointing out these goings on is hardly going to make me any friends. But I’m convinced that giving an honest assessment of an application in a peer review has already damaged my chances of getting grants. Therefore I wanted to share this cautionary tale with early career scientists who may still believe in the system.

P.S. The proposal was funded.

By March 28, 2014 24 comments opinion, science policy

ACS – Day 4: Peer-Review Reviewed

Wendy Warr gave a bleak and blistering critique on the current state of chemical peer-review at the recent ACS National Meeting in San Francisco. Her points are even more poignant as she is an associate editor for ACS. The salient features of her critique are listed below.

Problems with Peer-Review:

  • It can delay publications for months.
  • An editor can make or break a paper by sending it to the author’s friends or competitors.
  • Historically biased against women, single authors, etc…
  • It costs reviewers’ time (she gave a statistic that 41% of reviewers would like to be paid).
  • Reviewers tend to favor conservative science and not far-out new ideas.
  • Difficult finding qualified reviewers for multidisciplinary work.
  • Basing the quality of a paper on 2 reviewers, basically just 2-data points, is statistically insignificant.
  • As more papers are being submitted the burden for reviewers is increasing.

Warr did not give many solutions to these problems. However, she did point to resources addressing peer-review.

  • Peer-to-Peer – Nature’s blog specifically focusing on peer-review.
  • Naboj – A website where you can comment on arxiv papers and pubmed papers.
  • Faculty of 1000 – A website that tracks what authoritative people in the field think are the good current papers.

Unfortunately, Naboj and Faculty of 1000 do not really address the problems of peer-review. The former is just a comments hub, and the latter is just an aggregator. At the end of her talk George Purvis asked, why the government simply doesn’t setup a system for peer-review like ebay. Where submitters can have a trust scale associated with them, and people could thumb through the history of a reviewer. The idea of a peer-review ebay is seductive, but I doubt it would be greatly used; I could live without seeing “A+++++ paper will read again!!!!”

Bluntly speaking, you can not expect to have a vibrant peer-review community without a vibrant post-review community, and the chemical community seems decidedly averse to putting their name on-the-line.

Mitch

Post Script: Before Dr. Warr began she made explicit that she was not speaking on behalf of the ACS-Pubs machine but as an independent scientist.

By March 25, 2010 10 comments chem 2.0