Chemistry Blog

«

»

Mar 28

‘Anonymous’ peer-review – a cautionary tale

by Mark | Categories: opinion, science policy | (121708 Views)

Some time ago I received a grant proposal to review. Its was from an eminent professor.

I was extremely disappointed, on numerous accounts, with the quality of the proposal. In short I thought it relied very heavily on the authors standing in the community and only paid lip service to the science. And so (naively) safe in the knowledge that my review would remain anonymous, I gave a frank assessment of it.  I advised the panel to reject the proposal.

Satisfied that I had done my job I gave it no more thought until, a few months later, I received an email from the proposal’s author. It started …

I have had a review – see below – for a pending application with [research council/charity]. It is of the “unhelpful” type the community gets, and looks as if it from an unsympathetic, uninformed, prejudiced non-expert – the other reviews were all very positive, but I now have to answer the points.

What followed was a spurious query about some detail in one of my papers, which he/she apparently needed to help address the comments I’d given as an ‘anonymous’ referee.

It seemed to me that the real reason for the email was to tell me off.  But how could this happen? How could the author have found out that I had refereed the proposal? So I sort advice from senior colleagues who had sat on research council/charity committees. And I was shocked by what they told me. It seemed that it wasn’t unusual for the committees to know who the referees are, but even worse committee members were often happy to share this information with eminent grant applicants.

Since this episode I’ve been in a quandary.  What should I do when asked to review a proposal from a high profile scientist? Should I be truthful about what I think, risk being found out and so make enemies? Should I lie and write glowing reviews based on who the applicant is and not what is proposed? Or should I refuse to review applications from the higher echelons of community? To be honest I’m still at a loss.

There’s more. The email went on …

Also, I am getting personal comments from others in the field [lists of other eminent names] to ensure that the [research council/charity name] panel does understand the importance of the approach, to help all of us doing [technique], and not create a negative and highly damaging impression of [technique] at their panel from this kind of review – it playing right into the hands of the [another technique] community which is well represented on this panel.

So it seems that, if you have the connections, its acceptable to seek one’s own referees and lobby the panel with them.

The whole episode has left me with a very bitter taste and a complete lack of confidence in the validity of the  peer-review system.

Finally, this all happened sometime ago, and since then I’ve thought long and hard about whether I should make the incident public. After all, pointing out these goings on is hardly going to make me any friends. But I’m convinced that giving an honest assessment of an application in a peer review has already damaged my chances of getting grants. Therefore I wanted to share this cautionary tale with early career scientists who may still believe in the system.

P.S. The proposal was funded.

19 comments

5 pings

Skip to comment form

  1. fluorogrol

    Granting ‘junior’ academics confidence to criticise the work of ‘senior’ academics in their field without fear of a professional penalty is one of the key reasons to favour anonymous peer review over an open/non-anonymous system.

    If that’s being regularly circumvented (obviously it’s hard to say how widespread this sort of thing is), then it becomes harder to argue that it outweighs the advantages of greater accountability etc. in other systems.

    1. Mark

      I think this really an argument for real anonymity, not removal of it.

      1. fluorogrol

        I agree, but how do you begin to police that?

        My point is that if this is genuinely widespread, then I struggle to see how you could stamp it out. It’s not a case of education – everyone knows why peer review is anonymous – so you’re left just appealing to people’s better nature. Good luck with that.

        The only other counter-measure I can imagine is the name-and-shame approach mentioned above, and it’s not hard to see why you (or anyone in the same position) wouldn’t want to do that.

  2. sfsf

    Massive corruption of the system and a big problem with UK (in particular) funding
    You should both complain to the funding agency and go public (making sure you highlight the author and the funding agency).
    Mr Big has already screwed you – at least this way it won’t happen to someone else

    1. Mark

      I don’t think naming names will help me or the cause any.

      There needs to be clarity in the system, so people know what really happens and how it works. Which is why I’ve brought this case to people’s attention.

      I’m still considering an approach to the funding body.

      1. Ruey

        I think you should approach the funding body confidentially, but as you said, don’t name names publicly. Can you also forward redacted versions of stuff to Athena Swan (if UK based on research councils) as a case study of the intimidating environment in academia?

        To me the email reads as a thinly veiled threat, not a telling off, and the funding body needs to know that their procedures are resulting in senior academics threatening and bullying younger academics at a more vulnerable stage of their careers. It was the bit where it says “I am getting personal comments from others in the field…” – it may as well say “I have told all my powerful friends that you disagreed with me…”

        But thank you for posting this. I hope everyone who is considering an academic career gets to read it.

  3. Mitch

    I think everyone would agree that it was massively unacceptable for the committee to give out the names of the ‘anonymous’ reviewers. It is also probably very common. These committees are clubs and clicks, it is unfortunately just the way it is and I doubt it’ll change. You did every junior members in science a favor to inform them that this is common in the culture.

  4. Dmitry

    This is very sad. I’ve heard stories (with various flavors of fiction) about people stealing ideas of others by taking an advantage of peer-review process, but this is something new. At first I got saddened by thinking that corruption and low behavior is as common in modern science as in any other human endeavor involving competition for money and prestige. At a second thought I thought that one way to stand against it is to push young scientists with strong ethics and integrity principles into science, as well as educate future scientists about what is good and bad for science. Seriously, avoiding scrutiny by pulling the strings will benefit Mr X short-term, but aggregate effect from many Mr X’s risks to damage science beyond repair long term.

  5. Kristin H

    I can’t tell you the number of times that journal editors have let slip the names of the researchers who have reviewed my manuscripts. I don’t think what happened to you is at all unusual (although it’s unusual that the other researcher actually contacted you rather than silently stewing and planning revenge).

  6. Ethan E.

    Thanks for the blog, I’m glad you had the courage to post it. I think anyone who has the interests of good science at heart would agree that anonymous peer reviews of grant proposals are preferable to the current situation. The fact that the grant author not only contacted you but pointed to flaws in your own papers, as some kind of defense of his own proposal is beyond ridiculous. I wish you had named names, personally.

  7. Ethan E.

    *pointed to perceived flaws(I meant to say)

  8. Jan H.

    no system is ever working without flaws. we are humans and we biologically developed to take any chance to succeed, if we could use it and stay unpunished. we also developed a sense for injustice telling us to punish those who brake agreements. by publicly exposing the eminent professor, you may achieve both, change of the system and his punishment. communication works this way, mutual knowledge serves to change the relationship status, whereas individual conserves it. but if the clique has enough power to use false arguments and have possibility to silence you, that you achieve nothing. it is always a choice – withstand the wrong or fight against it, tertio non datur. i would go fighting as i always do 😉

  9. Fenton

    This entry was a precautionary eye-opener. It would be very helpful to know the name of the funding organization

    1. Mark

      I’d rather not name any names at the moment. However, I have been in contact with the various funding bodies (not just the one involved in this case) to ask for more info on how they protect reviewer anonymity. Once I have responses I’ll be sure to share them.

  10. Mark M.

    Unfortunately, the peer-review and review system (in academic literature, proposal funding and scholarship funding for students) is broken. Stories like these are the reason why many young scientists (such as myself) are discouraged, and losing interest in pursuing careers in the Ol’ Boys Club of academia (ie. too much hassle for too little compensation).

    Good luck out there.

  11. ChemChem

    This is disappointing, but sadly not unusual. All I can say is please keep giving your honest opinion as it is what makes the science better. I’ve heard that arguments that researchers in a certain sub-field should ‘stick together’ or risk being ‘beaten’ by researchers in another sub-field but it’s really poppy-cock. All you are really doing is propping up the established leaders in your field and undermining its long term security (as when the funders do their bean-counting reviews they will look at impact factors and the like and find that the numbers don’t stack up because the field has been funding sup-par science).

    So, for a young scientist, there is absolutely no benefit for you in propping up others’ poor proposals.

    As for people deliberately breaching confidence to divulge names of anonymous peer-reviewers – how deeply unprofessional.

  12. Jeff_Simpson

    Mark,

    I thank you for exposing the corruption you have encountered. Here is the States, we suffer from similar maladies. Over the years, I have found that subjectivity and bias vary widely from one reviewer pool to the next, suggesting that in some areas certain favorite reviewers become entrenched. I imagine they see themselves as noble warriors working hard to ensure that they are able to divert society’s precious resources to their darling niche.

    In politics we call this patronage and lobbying. But in science, this practice can correctly be called subversion. To be a scientist is to be an adherent to the scientific method. How can we perversely accept that objectivity suffers in the peer-review process for the funding of scientific research? It is ironic that fairness should suffer at the hands of those nominally devoted to truth and objectivity.

    I have spent some time reflecting on ways in which the dispensation of extramural funding is accomplished. Three criteria must be met to promote maximum fairness and therefore maximum social benefit.

    (1) The rules must be fair. It is sometimes true that established researchers devote less effort to funding renewals compared with newer members of the community seeking to develop a new research program. Explicit recognition that senior researchers are held to a different standard should be debated and codified if merit is seen in maintaining long-standing funding continuity. On the other hand, allocating funding to new investigators helps encourage the next generation. I can see merit on both sides of the old-versus-new debate, but have no sympathy for the successful senior investigator who bemoans the lack of new talent in his or her area (hint: look in mirror for problem).

    (2) Those that run the programs must enforce the rules. Various program officers will vary in their management styles of the review panels. Those that are not good at it should be educated, given another chance, and removed if they fail to improve. In order for this to happen, there needs to be a formal complaint mechanism to address aggrieved, funding-denied applicants.

    (3) Reviewers should be subject to some evaluation of their reviewing skill. Based on what you wrote, it appears that you did not play along and give the eminent professor a review based on his or her reputation. The beauty of science is that we are to speak the truth. Although our society may popularize the fiction that it values truth in the abstract sense, in practice we seem less grateful to whistleblowers when they point out the flaws in a well-established social edifice.

    This is a classic example of why we need to ask who will watch the watchers?

    I have encountered bias in my efforts to raise extramural funds, but I have also interacted with program officers who seem to be genuinely devoted funding exciting research.

    Here in the States, the trend in research instrument funding is towards merger and consolidation – following several decades of this practice in the business sector. Increasing the size of the pyramid is great for those at the top, but fewer and fewer can occupy the top positions. This is the general trend in NMR (my field). Higher frequency NMR instruments, with their concomitant higher and higher price tags, are being funded to the exclusion of the replacement of instruments that service the needs of many, many synthetic chemists. We are making a few top scientists very happy at the expense of everyone else. Like tax breaks for the rich, I have yet to see the elite sufficiently dazzle the rest of us with their unique reception of society’s largesse.
    Anyone who is contemplating an academic career in science would do well to consider the inhospitable landscape into which one must venture at the outset of one’s career. Many of my Boston-area NMR colleagues are in industry, and they sometimes express to me the opinion that it is much harder to flourish in an academic environment, given the anemic funding outlook and its impact on instrument renewal schedules (I maintain three 16 year old NMR instruments, another is 15 years old). Imagine if your phone or your laptop was 16 years old…

    From others I have heard horrible stories of open reviewer bias, especially in review panels concerned with a branch of science where there are only a few major players. All the funding goes to the alumni of the major groups. The problem with this model is that it favors extant viewpoints to the exclusion of new viewpoints or approaches. It is a recipe for stagnation.

    A few years ago a proposal from multiple contributors that I assembled received egregiously subjective negative reviews. The program officer who discussed the reviews with me was ignorant of some of the rules and was as subjective as the reviewers. He also scolded me for allowing some improperly formatted references to make their way into the final draft. Feeling that I had attained the maximum benefit the program officer could offer to me (zero), I abandoned trying to understand the logic of the system and instead moved on to the deeper realization that the peer-review system is only as good as the people that staff it. After my Freedom of Information Act (FOIA) request for more detail on the pathetically subjective reviews and reviewers was denied, I wrote ten drafts of a complaint letter to the funding agency. As I proceeded with my edits, I distilled my letter down to a list of helpful suggestions to improve the efficacy of the program for the good of all. I have no confidence that my letter did any good, other than to make me feel like I had done my part.

    My final point is this: the problem is real and it is big. Subjectivity has no part in science, unless you are talking about agenda-driven science which is not science at all but rather pedagogy. Like Edward Snowden, you do the right thing when you speak the truth, and for that you will be accordingly punished.

  13. Magnus

    The problem runs to the very core of academia – many folk that have clawed their way to the top of the pile defend their position with little thought for ethics or the wider community. Perhaps its a function of the fact that the way to be successful in science for many is to be completely selfish and to stomp on anyone that starts to compete or challenge ideas they have long guarded.

    A few years ago I had a letter published in that recounts a sorry tale of the gagging of an earnest young PhD student (me) who dared to challenge the scientific establishment over the way they treated hydrothermal vents. These are habitats that are very expensive to get to and often require prestigious grants and international collaboration. The senior scientists involved in this small but expensive world did not like me asking them questions about how they treated their study habitat and the lack of planning around likely impacts of research on a habitat that at the time we knew very little about. I was gagged (I was told later by a senior scientist in the field) because my rabble rousing threatened grant applications. The New Scientist never did publish an article about the problem although did. And I never worked in deep sea science again.

  14. GC

    Sorry to hear what happened to you.

    I have some experience to share with you. I have had more than my share of this kind of mob law.
    More than 20 years ago I made an experiment that was really important in the field. But a senior scientist had decided that the subject should have been the subject of his protegee. He blocked my papers with biased reviews, not just one, but many by using his network. And he had a proposal for beam time of mine rejected on a unique instrument: One of his team mates was on the committee and said the proposal had to be rejected because it had already been done by them. This was not true, but when I protested, I was accused of questioning the integrity of the whole institute and pictured as a b***. Somewhat later I was accidentally in that institute and discovered that they were doing my proposal with my protocol using unofficial beam time. They tried again to make it my fault, but the director of my lab intervened. It was a big scandal, but it did not stop them. They continued mobbing me for more than 10 years to push me out. I could write a book about all the dirty play they unleashed on me. For one thing, if peer review where not anonymous, the conflict with a dishonest person would be visible to anyone. It is the secrecy that empowers the dishonest and disempowers the honest. If there are some honest people left in the field then in the case of an open conflict it would fire back on the aggressors and not last for a long time. When I complained in my case, people would just act as though I was cooking it up. They would tell me I could not know who wrote the referee reports. The devil with the secrecy is that it can linger on for decades and that you cannot fight against it because one will argue that you cannot have any proof in principle. From what I have learned, I would just tell in a talk in big conference what happened, name the person, step down into the room and point at him in the complete audience: Here he is. If he continued to bother you, you might then get some support, if there are honest guys in the community, because everybody would know about the conflict,. In my case the group had a very strong grasp on the whole community and it just went on and on.

    I got ill from the years of continuous mobbing described above. I had to leave my lab, to leave my world-class instrument and to leave my research subjects.
    I had to start anew. I gradually grew into something new, that was heavily mathematical. I worked for ten years on it and it became a book of 300 pages with original research. I sent it in. The book was rejected as follows.
    An anonymous referee just picked one equation out of the whole book and made a hype about it that it would be obscene. He really used the word obscene. The equation is exact, it can be found in dozens of books. Two of these books are written by Nobel prize winners and a few other ones are written by really famous people. Nevertheless, this referee managed to get my book rejected, just with that hype about this equation and protesting to the editor did not help. Once again, this person is anonymous and the editor does not care.

    I think anonymous peer review is devilish and a crime against mankind. Editors know it but they act like the Germans after the war: “Wir haben es nicht gewusst”. They just deny it. I have been complaining about it for all these years and nobody listened. Now they are starting to see that it is broken. But I think most scientists just deserved only that. If you google “sham peer review” you will find what anonymous peer review does in health care in the US. An estimated 300 good physicians per year are committing suicide in the US due to sham peer review.

    I cannot give you very much advice, because all I tried to do to fight against dishonest people just did not work. If you are still young I can only strongly advice you to leave science and slam the door with a huge amount of noise. There are places where you can earn much better money, where you will have more time for your wife and your kids, etc…

  1. Who knows reviewers’ identities? » Chemistry Blog

    […] couple of posts ago I shared a pretty unpleasant experience I had after peer-reviewing a grant application. In short my anonymity appeared to have been […]

  2. 3 important steps to getting more credit for your peer reviews - Impactstory blog

    […] would prefer for criticisms of their work to remain private. Turns out, their assumptions weren’t backed up by […]

  3. Impact Challenge Day 19: Establish your expertise with Open Peer Review - Impactstory blog

    […] would prefer for criticisms of their work to remain private. Turns out, their assumptions weren’t backed up by […]

  4. The top blog posts of 2014 : The Sceptical Chymist

    […] The anonymity available to bloggers and commenters can be a blessing or a curse, allowing vulnerable people to speak the truth safely or offering malicious individuals a shield to hide behind. In January, Michael Eisen offered a thoughtful dissection of the differences between anonymity and pseudonymity and the ethical questions raised by social media and peer review. Questions of anonymity are not limited to social media, and Mark Lorch shared a story highlighting the importance and limitations of anonymity in peer review. […]

  5. Day 15: Establish your expertise with Open Peer Review – The FEDUA Research Impact Blog

    […] would prefer for criticisms of their work to remain private. Turns out, their assumptions weren’t backed up by […]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>