science policy

‘Anonymous’ peer-review – a cautionary tale

Some time ago I received a grant proposal to review. Its was from an eminent professor.

I was extremely disappointed, on numerous accounts, with the quality of the proposal. In short I thought it relied very heavily on the authors standing in the community and only paid lip service to the science. And so (naively) safe in the knowledge that my review would remain anonymous, I gave a frank assessment of it.  I advised the panel to reject the proposal.

Satisfied that I had done my job I gave it no more thought until, a few months later, I received an email from the proposal’s author. It started …

I have had a review – see below – for a pending application with [research council/charity]. It is of the “unhelpful” type the community gets, and looks as if it from an unsympathetic, uninformed, prejudiced non-expert – the other reviews were all very positive, but I now have to answer the points.

What followed was a spurious query about some detail in one of my papers, which he/she apparently needed to help address the comments I’d given as an ‘anonymous’ referee.

It seemed to me that the real reason for the email was to tell me off.  But how could this happen? How could the author have found out that I had refereed the proposal? So I sort advice from senior colleagues who had sat on research council/charity committees. And I was shocked by what they told me. It seemed that it wasn’t unusual for the committees to know who the referees are, but even worse committee members were often happy to share this information with eminent grant applicants.

Since this episode I’ve been in a quandary.  What should I do when asked to review a proposal from a high profile scientist? Should I be truthful about what I think, risk being found out and so make enemies? Should I lie and write glowing reviews based on who the applicant is and not what is proposed? Or should I refuse to review applications from the higher echelons of community? To be honest I’m still at a loss.

There’s more. The email went on …

Also, I am getting personal comments from others in the field [lists of other eminent names] to ensure that the [research council/charity name] panel does understand the importance of the approach, to help all of us doing [technique], and not create a negative and highly damaging impression of [technique] at their panel from this kind of review – it playing right into the hands of the [another technique] community which is well represented on this panel.

So it seems that, if you have the connections, its acceptable to seek one’s own referees and lobby the panel with them.

The whole episode has left me with a very bitter taste and a complete lack of confidence in the validity of the  peer-review system.

Finally, this all happened sometime ago, and since then I’ve thought long and hard about whether I should make the incident public. After all, pointing out these goings on is hardly going to make me any friends. But I’m convinced that giving an honest assessment of an application in a peer review has already damaged my chances of getting grants. Therefore I wanted to share this cautionary tale with early career scientists who may still believe in the system.

P.S. The proposal was funded.

By •March 28, 2014 •24 comments •opinion, science policy

The Geeks are Coming!

The geeks are coming. Mark Henderson is calling them to arms and politicians would do well to take notice.

Henderson is head of communications at medical research charity the Wellcome Trust and a former science editor for The Times. His latest book is a disturbing catalogue of the abuse of science by politicians to support their policies. He describes a political system (in both the UK and USA) that is rife with evidence abuse, including politicians cherry-picking evidence to support their policies or sacking advisors who don’t agree with policy, and confusion as departments take different approaches to the same evidence. A prime example is the UK’s environment and health departments’ approaches to homeopathy.  Defra’s (the department responsible for environment and food) advice is that homeopathy is unsuitable for animals, but the National Health Service funds the very same treatments for humans.

The worst of it is that politicians are fully aware of the abuse; some unashamedly admit to it. David Blunkett (a former senior member of the UK government)  recently appeared on BBC Radio to discuss science and politics, where it was put to him that politicians often shop around for experts to support their policies. “Yeh, of course, we’ve all done it,” he laughed. Why is this attitude acceptable? Henderson argues that it is because politicians do not understand the nature of science.

Science is much more than a body of knowledge; it is largely a way of thinking. It’s about rigorously testing ideas and so should be applied to every avenue of the decision-making. Why can’t education policy, say, undergo the same sort of randomly controlled trials as medical procedures? It’s really quite simple. If you want to judge the value of a new policy, just roll it out in a series of randomly selected schools then compare it to another set of schools that haven’t received an intervention.

Henderson argues that the lack of evidence collection and the evidence abuse is not really politicians’ fault. After all, only one of the the UK’s members of parliament has had a career in research science (and its no better in the USA). So one can hardly expect them to understand the scientific method. And our political systems do not support evidence-based decision-making. In science, if an experiment produces results that are contrary to one’s belief the scientists (should) change their mind. But in politics this sort of U-turn is seen as unacceptable; indeed, changing one’s mind in light of an evidence-based study may just result in the politician producing a stick to beat himself with. John Kerry is a prime example, he was renowned for changing his attitude to issues once new evidence emerged. His opponents called it flip-flopping, and it became a factor that may well have cost him the presidency.

This is where the scientist come in. Henderson is calling on them to make evidence-based policies a vote-winning issue, support politicians who change their mind in the face of evidence, and engage with their Members of Parliament, Senators and representatives to help them understand the scientific method. Above all, he wants geeks to form a voting block that appears on politicians’ radar.

Geeks have already taken their new manifesto to heart. In the UK hundreds have pledged to buy a copy of the book for their MPs. Right now, copies are arriving on desks throughout the Houses of Parliament. Other politicians around the world could probably do with reading it as well.

The Geek Manisfesto is available from Amazon UK in hardback and the USA and UK as an ebook.

This post is adapted from an article that originally appeared in Civil Service World.

By •June 26, 2012 •1 comment •science policy

Science for the future

A campaign group, calling itself ‘Science for the future’ is, today, delivering a coffin to Number 10 Downing street in London as a protest against what they claim to be the death of British science.

Their concern is that the UK state funded research councils,  (particularly the Engineering and physical sciences research council, (EPRSC) that funds most of the chemistry research in the UK) are giving priority to research that will deliver good economic outcomes over blue sky research.

This group isn’t just a rag tag bunch of disgruntled scientist who are peeved that their projects don’t get funded. Its backed by a heady list of Nobel Laureates and heavily honoured scientists who published their views on the matter in a letter to todays Daily Telegraph.

Between the press coverage in the Telegraph, the BBC etc. and twitter (#science4thefuture) things are being pretty well covered. So I thought I’d focus on one aspect of the group’s issues with the EPSRC.

Any grant proposal submitted to the EPSRC has to include a section addressing the ‘National Importance‘ of the research. The guidelines on how to construct this section of the proposal states:

National Importance is the extent, over the long term, for example 10-50 years, to which the research proposed;

  • contributes to, or helps maintain the health of other research disciplines contributes to addressing key UK societal challenges, contributes to current or future UK economic success and/or enables future development of key emerging industry(s)

 

Predicting the impact of our work 50 years into the future seems pretty incredible. There are plenty  that think completing this section of the grant proposal requires the employment of a soothsayer.

 

I too wonder about at it usefulness. Under this regime would lasers have been funded given that they were perceived to be little more than physicists’ toys?

But lets suppose for a moment that in 1958 Townes and  Schawlow had 20:20 foresight when they wrote a ‘National Importance’ statement for an EPSRC grant proposal. It may have gone something like this.

Over the next 50 years we envisage that lasers will have a extraordinary impact on research, technology and the everyday lives of people worldwide. Lasers will prove to be useful in every conceivable scientific discipline. For example, we expect them to be used as a viable means of inducing nuclear fission, eye surgens will correct abnormalities in the eyes by cutting into the cornea with lasers and so saving people the bother of wearing glasses, and we expect lasers to be used to manipulate single molecules with a technique that may become known as optical tweezers.

However lasers will not be restricted to the laboratory or trained medics. They  will become ubiquitous, every home will have a device that uses lasers to play their music, so replacing vinyl disks. In fact we expect lasers to become so cheap and readily available that they will even replace the pointy stick used to highlight the important parts of slide presentations.

Would anyone really have believed that?

 

By •May 15, 2012 •2 comments •opinion, science news, science policy

Gaussian’s Banhammer

In my last post, I briefly covered the ‘share or not to share’ debate involving non-commercial software. In this post I’ll delve deeper into the issue by discussing how commercially available research software further complicates the situation. I’ll focus on perhaps one of the most controversial conflicts in the chemistry software: Gaussian Inc. vs Banned by Gaussian.

In the 1950s and 60s Prof. John Pople (1998 Nobel Prize winner) and his research group at Carnegie-Mellon University were focused on the development of ab initio quantum calculation methods.  The group incorporated Gaussian orbitals – rather than Slater-type orbitals, which were more computationally intensive – into a computational chemistry program for molecular electronic structure calculations. The program, Gaussian 70, was released as open source software through the Quantum Chemistry Program Exchange (QCPE) in 1970.

In 1987, Carnegie Mellon University was issued a software license for the program and, ever since, it’s been developed and sold by Gaussian, Inc. Prices (pdf) for the Gaussian software package range from $2,500 for a single computer to $35,000 for an institution-wide license.

Gaussian was initially used only by theoreticians. However, as I mentioned in my last post, the continuously increasing power of personal computers as well as the addition of a user-friendly interfaces have made the software so accessible that even a computationally inept synthetic chemist (like myself) can perform high level ab initio calculations with a half dozen mouse clicks.

Gaussian is an important tool for many chemists, but it’s has also been a center of controversy. Since its commercial release a number of individuals and institutions have been Banned by Gaussian (BBG), which means they are prohibited by Gaussian Inc. from purchasing or using any version of Gaussian software.

Read more ›

By •May 9, 2012 •7 comments •science policy
1 2 3 4 … 6