Too often I see calls for reforming the academic peer review process, or, another favorite, for the eradication of academic tenure, and you read the article and you realize the person has no idea what they are talking about.
There's another such article over at the New York Times, by Patricia Cohen, Wikipedia Age Challenges Scholars’ Sacred Peer Review. There are so many things wrong with the article, it's almost a challenge to know where to start. Let's start at the beginning, then, since I already gave you the summary.
Peer review is not "sacred", as in the title. It's not clear what that means. Peer review is a process that evolved over a period of time to serve a need that scholars have, which is, figuring out which articles are any good in a way which works.
The first sentence continues the misguided commentary. "For professors, publishing in elite journals is an unavoidable part of university life." As a writer, you like your first sentence to be accurate, among other things. Publishing in elite journals is not at all "unavoidable". It is quite avoidable: don't submit papers to elite journals. Trying to get published in elite journals is usually required, but that is very different from publishing in them. In fact, if the journals are elite, they won't publish papers by most people. And if you're at a teaching college, publications are not as important as they are at research universities.
The second sentence continues the comedy of errors: "The grueling process of subjecting work to the up-or-down judgment..." No. Subjecting the work to review is easy, you send it in. The grueling part is doing the research, and then writing it up perfectly. This means you have to have a great research question, great data, great methods, and interesting findings. And it's not an up-or-down judgement at all. For all of the journals I have reviewed for, it has never been up or down. I've done reviews for five different journals, and two conferences (and I have reviewed every year for the conferences, and more than once for some of the five journals). It's always a range of options for journals, and you give comments. Usually, the options are like the following:
- Accept as is.
- Accept with minor revisions.
- Resubmit with minor revisions.
- Resubmit with major revisions.
- Reject.
As you can see, that's not "up-or-down." Resubmits can be rejected, or sent back a second time as another resubmission, where they can be accepted or rejected (usually at that point they don't let you resubmit it again).
Cohen refers to peer review as a "monopoly". It's a process. She continues,
Instead of relying on a few experts selected by leading publications, they advocate using the Internet to expose scholarly thinking to the swift collective judgment of a much broader interested audience.Relying on experts for their opinion on the article is the entire point! These experts are not "selected by leading publications," reviewers are selected by the editors at the journal where the paper has been submitted. Use of the word "swift" implies that peer review is slow. Well, not like we get paid for it. And not like it counts towards tenure. We do it to help make journals better, which makes the field better as a whole. Sometimes you might get 10% off books by the publisher which publishes the journal. If you do a good job, the editor might remember it, but that also means more reviewing for you in the future. And relying on "a much broader interested audience" is a terrible idea, papers should not be judged by an interested audience, they should be judged by an expert audience.
As a writer, you don't want too many reviewers, because at some point they will disagree with each other and then there is no way to change the paper to make them all happy (which is the only way to get the paper published). Note that this is not pointless reviewer-pleasing, if they approve of the paper that means the paper makes a positive contribution to the field. I co-authored a paper for a conference recently and there were ten reviewers, which makes our job as authors very difficult because many of the reviewers disagreed with each other (but they have no idea that they are disagreeing with each other, this is not publishing by committee).
Regarding one of the crowd sourced peer review projects, Cohen writes, "In the end 41 people made more than 350 comments." As an author I don't want 350 comments. I want maybe five sets of comments at most, from qualified people.
There are a lot of other errors in the article, too many to detail at length. Here's one:
The traditional method, in which independent experts evaluate a submission, often under a veil of anonymity, can take months, even years.
This is not true. For the two most recent reviews I have done, I had a one-month deadline. It is true that for one of my co-authored papers it took over a year to finally get it published, but that's because the journal changed editors and they lost the paper in the switchover, and because all three of the authors moved. The editors were also slow, but if you're going to be slow about "traditional" reviews (which are all done by email these days), you're going to be slow about web-sourced reviews. All of the things by which you are judged and upon which your job depends (teaching, research, conferences, publishing, academic service) don't just vanish because you switched to web-based reviews.
And the "veil of anonymity" that Cohen claims "often" exists is not real. Reviewers are anonymous to the authors, but the editor knows who you are, and the editor or editors will read your comments. If you're rude or out-of-line, they may send your comments back, and you will have tarnished your reputation. But the reviewers doesn't know who wrote the paper (although at times you may have suspicions, or if you think you know you tell the editor and recuse yourself), and that's important. The reviewer judges the paper on its own merits. The reviewer and the author are anonymous to each other, but the editor knows who both are, and as such fills an important gatekeeping function. By lacking anonymity with the editor, both reviewers and authors want to do a good and accurate job (which may mean rejecting the paper as a reviewer, as I have done several times). But by simultaneously having anonymity with each other, the authors and reviewers can focus on the words of the paper and the reviews. As a reviewer, I can crush some poor sod for writing a terrible paper and know they won't hate me, as a writer I know the comments are solely about the paper and not about me (and I won't hold a grudge against the reviewer for not seeing the genius in the paper). This is a very important part of the process.
Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants. Open review more closely resembles Wikipedia behind the scenes, where anyone with an interest can post a comment. This open-door policy has made Wikipedia, on balance, a crucial reference resource.
What a horribly wrong paragraph. "Clubby"? Reviewers don't edit. Fraud? Perhaps Ms. Cohen has never heard of the Internet, speaking of fraud? As I just mentioned, reviewers are not anonymous to the editors, and are fully accountable to them. Exclusiveness does indeed help comment quality, which is the point. Limiting the participants is the whole idea. Otherwise, I'll just go get all my friends to say how awesome my paper is and I'll do the same for them. It is not clear what she means when she says Wikipedia is a "crucial reference resource," Wikipedia is, generally speaking, a giant mess of articles that seem correct enough to those who speak the loudest. Academic databases are crucial references resources, Google Scholar isn't bad, but Wikipedia is a disaster.
The beginning of the second page is a little better, and is actually accurate, which is a surprise based on the first page, however it degenerates into an error-filled comedy once again. For example,
Advocates of more open reviewing like Mr. Cohen at George Mason, argue that other important scholarly values besides quality control — for example, generating discussion, improving works in progress and sharing information rapidly — are given short shrift under the current system.Well, no. Discussion is what happens with your colleagues, on mailing lists, discussion boards, blogs, in the hallways, at the coffeeshops, and at conferences. Improving and sharing work is what conferences, email, and posting items online do. They have nothing to do with peer review.
Oh and we see the New York Times following its annoying idea about how people with PhDs aren't called Dr., as we deserve to be called. I could write, "Mr. [sic] Cohen...." but I think I've pointed out more than enough errors.
