ACS - Day 4: Peer-Review Reviewed
by mitch on Mar 25 2010 (14725 Views)
Wendy Warr gave a bleak and blistering critique on the current state of chemical peer-review at the recent ACS National Meeting in San Francisco. Her points are even more poignant as she is an associate editor for ACS. The salient features of her critique are listed below.
Problems with Peer-Review:
- It can delay publications for months.
- An editor can make or break a paper by sending it to the author's friends or competitors.
- Historically biased against women, single authors, etc...
- It costs reviewers' time (she gave a statistic that 41% of reviewers would like to be paid).
- Reviewers tend to favor conservative science and not far-out new ideas.
- Difficult finding qualified reviewers for multidisciplinary work.
- Basing the quality of a paper on 2 reviewers, basically just 2-data points, is statistically insignificant.
- As more papers are being submitted the burden for reviewers is increasing.
Warr did not give many solutions to these problems. However, she did point to resources addressing peer-review.
- Peer-to-Peer - Nature's blog specifically focusing on peer-review.
- Naboj - A website where you can comment on arxiv papers and pubmed papers.
- Faculty of 1000 - A website that tracks what authoritative people in the field think are the good current papers.
Unfortunately, Naboj and Faculty of 1000 do not really address the problems of peer-review. The former is just a comments hub, and the latter is just an aggregator. At the end of her talk George Purvis asked, why the government simply doesn't setup a system for peer-review like ebay. Where submitters can have a trust scale associated with them, and people could thumb through the history of a reviewer. The idea of a peer-review ebay is seductive, but I doubt it would be greatly used; I could live without seeing "A+++++ paper will read again!!!!"
Bluntly speaking, you can not expect to have a vibrant peer-review community without a vibrant post-review community, and the chemical community seems decidedly averse to putting their name on-the-line.
Mitch
Post Script: Before Dr. Warr began she made explicit that she was not speaking on behalf of the ACS-Pubs machine but as an independent scientist.
First!!!!
@Mitch, this was an interesting talk. The entire symposium had a lot to chew on.
BTW, post-review of the chemical literature can take many forms and I suspect there are a number of ways to do it that haven't been explored entirely or even considered in chemistry. As Wendy pointed out, most of the research on the topic of peer review has been done outside of chemistry.
Those who are interested in the subject might check out Chempedia Lab for a unique, chemistry-specific approach to the post-review problem:
http://lab.chempedia.com
Science Reddit? A scientific up-down again? Chemmunity was supposed to help offer a solution to this problem, but it totally failed. Also, the recent open-access journals that we have been talking about are also trying to think of ways to deal with the "reviewer issue". I don't know what the answer is, but I hope people like Wendy Warr continue to make it known that the current state of affairs is not OK.
Also, I hope she doesn't catch too much flak for this. If you remember you're post from a couple of years ago: http://www.chemistry-blog.com/2007/10/11/acs-open-access-and-chemmunity/
where you talk about the politics behind ACS publishing and salaries.
One important point back then: ACS hired Dezenhall Resources to undermine the open access journal movement. Dezenhall Resources did so by campaigning that open access journals mean "no more peer-review", which is a lie. Now, Ms. Warr is an ACS editor talking about how peer-review is broken.
So why'd they fight open access to begin with? The ACS managers receive bonuses based on how much money the publishing division generates. Hurt the publishing revenue; you hurt their bonuses.
Stay classy ACS.
@Maz Thank God you showed me that post. It was using the old styling information from http://www.chemicalforums.com and it made it look like most of those words were my own. In Dr. Warr's defense, she explicitly stated at the beginning of her talk that she was not speaking on behalf of ACS but that was not in my write-up until now. I think we all understand why ACS fought open-access, because it is competition not because it is inferior.
@Rich Definitely, the talks I saw from the symposium were quite good and will probably be the basis for more posts.
There is post publication review in the chemistry lit - it is just ad hoc.
I heard long ago at a talk, "I wait when a new reaction is published to see if there are further communications/pubs that use it, once I see a few, then I know it works..."
We all know this. If it is important and good, it is repeated and referred to again and again. The challenge is to publish more of this and less of the stuff that just collects dust.
The hard question that would drive us all to better science is, how do we know, before it is published or even before it is funded, whether it will be one of those ideas/experiments that gets repeated many times. That is a hard question to answer. READ Berson's "Chemical Creativity" for some thoughts on it.
I think that the comments above would indicate that often reviewers do not know the answer to that question.
How many years did the Bergman rearrangement sit before ene-diyne chemistry became vogue?
@Observer. The chemical literature is a wonderful self-correcting post-review forum. Most of us in the internet generation just wish things were faster. I'll make sure to checkout that book.
Mitch,
The guy who talked about ebay was George Purvis.
Your comments are pretty accurate but I did point out that at least 84% of scientists are in favor of peer review - and so am I!
Wendy
Thanks by the article Mitch. I didn't know that she gave a talk about this.
I think that the above list of problems with peer-review lacks the important item that peer-review has showed to be completely useless at detecting research fraud. In the work [1] I give a notorious case of scientific fraud in the top journals: the Schön scandal.
Let me do comments on the other items in the list:
1) "It can delay publications for months". That is in the optimal case, when the paper is accepted. If it is rejected, authors can easily go from rejection to rejection during years before they got it published anywhere. In one occasion, a scientific paper appeared in 1957 in the Journal of the American Chemical Society, 25 years after it was initially submitted [2].
Rosalyn Yalow, described how her Nobel-prize-winning paper was received by the journals in the next terms [1]:
2) "Reviewers tend to favor conservative science and not far-out new ideas". Most scientists seem to believe that new ideas are easily published and that, only in a pair of cases, peer-review was unfavorable. Objective data invalidates their belief. In [1] I give a list of thirty four Nobel laureates, whose awarded work was rejected by peer-review. This is an example:
I have also given some instances of works, not Nobel awarded but still important, that were also rejected. Mitchell Jay Feigenbaum, described the reception that his revolutionary papers on chaos received in the next terms:
3) "Basing the quality of a paper on 2 reviewers, basically just 2-data points, is statistically insignificant." My experience is that some physics journals use only one review to accept/reject papers!
The scope of the report [1] is broader and includes other problems of modern science as funding, "salami science", hidden financial issues of Open Access journals, and others. It contains several suggestions from Harry Morrow Brown, Lee Smolin, Linda Cooper, and myself for solving the problems.
Regarding peer-review some of our solutions are:
(i) reviewers would not be anonymous. This eliminates what the Nobel laureate Schwinger dubbed "anonymous censorship". Anonymous reviews are exceptionally accepted, but only if the author considers that the review is fair.
(ii) reviewers would be paid. We offer reviewers a premium account as paid for their services.
(iii) the number of reviewers may be enlarged to something as four-six. This eliminates statistical fluctuations associated to the use of a single reviewer. A larger number of reviewers also mean a better treatment of multidisciplinary works.
(iv) A system of post-review, where accepted papers are revised and updated/corrected. This is the basis for a living journal concept. Post-review can be seen as an extra guarantee for correcting errors that were not detected before. Living journal eliminates the need for Errata sections as in typical journals.
Of course there is other points of view. My colleague the electrochemist Alexander Shagaev has other ideas regarding peer-review [2].
REFERENCES:
[1] Science in the 21st century: social, political, and economic issues
[2] Russian-Spain Project sciencereform. See for instance,
http://sciencereform.narod.ru/YourProposalsEngl.html
http://sciencereform.narod.ru/LetterEngl.html
http://sciencereform.narod.ru/YourNotesEngl1.html
Wendy Warr said:
Science is not based over a popularity contest. Theories/hypothesis/ideas are not objectively right/wrong in base to percents of scientists believing on them or rejecting them. I have given many examples where a majority of scientists were wrong and a minority (even a single individual) completely right. There is many more examples out here. For instance, the reference [3] reports the started by Ahmed Zewail, who won the Nobel Prize in chemistry 1999. Nina Hall writes:
In a similar way, the current model of peer-review cannot be objectively right/wrong in base to percents. According to recent systematic investigation of the peer review system, its assumptions about fairness and objectivity are rarely tested [4].
REFERENCES:
[3] Chemistry and biology in the new age 2002: Chem. Commun. (Camb), 19, 2185-2187. Hall, Nina.
[4] Little evidence for effectiveness of scientific peer review 2003: BMJ 326, 241. White, Caroline.
Hi,
I agree absolutely with what you say. But don't you think sharing of ideas and skills is very significant in Science.
It is only when discoveries and inventions are shared that we take a step towards more evolution!!
Besides peer review can be proactive rather than reactive. As adults we could follow some protocols of peer review and see the difference it makes for the betterment.
Nadera Baig
Science Teacher