Friday, 17 February 2012

How to be a good referee (hopefully!)

This post arises from a very recent referee response that particularly got my goat - see link here (Ref1, second review). Bearing in mind that my state of annoyance may not make what I say next in any ways objective or unbiased, it seems to me that journal editors are increasingly side-stepping their responsibilities - or they are increasingly busy would be a kinder way to phrase it. I have found editors passing the buck on publication decisions to referees more and more often. This effectively entails a two-stage review process as revised submissions are sent out for review again, hopefully to the same original referees but sometimes not. Or maybe I'm just writing trash that actually needs two-step oversight .....

Anyway, in light of the recent comment I received, it occurred to me that academics don't actually receive much training about how to referee other people's work - see some suggestions here and here. Like teaching, it is often assumed that we will pick it up as we go along. However, considering the sometimes horrifying attitude and actions of referees that most of us experience (or, at least, I hope other people experience them too!), I think this seems unlikely

I therefore thought it might be helpful to outline how to referee other people's work. I am not simply coming at this from the perspective of an author, I am also an associate editor on the journal Science as Culture where I have been working for nearly 5 years now and get to see numerous referee reports on a range of different quality submissions. I have also had the 'opportunity' - for want of a better term - to referee submissions for a range of journals, some more vaunted than others.

As for my own practice, I now routinely reveal my name to the authors I am reviewing. I do this for two reasons (which I outline in the reviews I write): First, transparency (e.g. authors can judge my knowledge and can therefore judge the validity of my comments); and second, for good practice (e.g. it forces me to be positive in my reviews to help the authors improve their papers rather than write negative or vague comments).

So, my suggestions are as follows .....

0. When to referee

There is always an issue about how many referee requests you should accept. As you become more well known you are generally invited more often and some people can get overwhelmed. Also, because some people work in more 'current' topics, they may also end up in more demand. It is also noticeable that a slew of submissions on similar topics come in around about the same time - for whatever reason. 

As an editorial I read as while ago suggested (and unfortunately I can't now remember where), we effectively owe our refereeing services to the journals that publish our work. The editorial suggested we should referee 3 papers per article we have accepted by a journal (or some similar amount). So, the idea is that we owe our time to compensate for the time others gave to review our work.

Personally, I now have a policy of doing one review for (almost) any journal that asks me, but then rejecting any others that I don't find interesting. I will review three papers for any journal that has published my work (or more if they've published me more than once).

1. Be on time

There is nothing worse than waiting ages - as either author or editor - for referees comments. This is especially so if you are an editor and the referees have assured you that they will do the review in the next month. If you can't do it in time then don't do it, but do make suggestions about other possible referees - as an editor, that is always helpful.

2. Be polite

Think about the authors! There really is nothing worse than receiving a review containing insulting personal statements: e.g. "I am not clear that the author [me] either has the depth of knowledge or the analytical abilities (from either the evidence in his cv or the proposal) to develop the ambitious project that is laid out here". It's not pleasant and it's totally unnecessary.

As a general rule, never make a personal comment about the author - whether or not you think they are spouting nonsense. Everyone is tempted on their bad days to let it rip, but it can have pretty demoralizing effects - I have heard of people's work being unfavourably compared to undergraduate essays and it then taking ages for them to get the confidence to resubmit elsewhere.

This kind of behaviour seems to be the reserve of particular kinds of academics; it might result from the context in which they have learned their craft, as it were. Having heard stories about what happens at some of the 'elite' universities during the PhD process, I have come to the conclusion that impoliteness is very possibly the result of too many bitter arguments around seminar tables.

3. Avoid vague comments

Don't make any claim without backing it up with an example: e.g. I received this comment without any examples, "There are too many convenient conflations and stereotypes in the MS that slide across the complexity of the issue of marketization". It is almost impossible to argue against this kind of statement since it contains no rationale or justification. It is also useless as a comment since the author can do nothing with it.

So, be precise when you make a comment, preferably by referring to a specific page or concept or empirical case. 

4. Avoid being too critical, try to be constructive

This may sounds strange, but often referees are overly critical without purpose; i.e. "this is bad ....". It is very easy to pan someone else's arguments, especially when you can do it anonymously. However, unless a paper is really unpublishable (i.e. poorly structured, incoherent, etc.), it is perfectly publishable somewhere, if not in that specific journal. Whether it should be published in a particular journal is not a decision referees should be making; that should be decided by editors who determine the editorial policy and direction of a journal. If a paper is not relevant or current enough then editors should make that decision.

Instead, it is important to make constructive comments; i.e. outline what the author needs to do to make the paper publishable. How could they restructure the paper? What ambiguities do they need to tease out? What literature would be helpful? That sort of thing. Again, be precise.

5. Don't demand something else

Often referees seem to think they know what an author should be writing about, and usually it relates to what the referee thinks is important. While there may be a case for suggesting that an author has missed something (e.g. literature, argument, theory etc.), it is never useful to tell an author to come back once they've rewritten their paper to take account of something completely different from what the authors set out to do.

Judge what the author has written, not what you want them to have written or what you want to read. This often involves putting aside your pet theories or pet approaches and considering the worth of the paper in its own light. I have seen numerous referee complaints about papers because they don't comply with the referees own perspective; usually because of an underlying conflict between qualitative and quantitative emphases or similar.

So, don't demand that authors do as you do. Judge on their own merits and if you feel you can't make that judgement then pass the responsibility to the editors, who have far more experience in these matters and should be pushed every now and again to make decisions.

6. Don't do the editors work for them

Editors have a responsibility to read submissions and make judgements on them rather than rely on referees to tell them what to think; otherwise, why have editors in the first place? In my opinion, this means not agreeing to referee a paper after it has been revised, or being very careful if you do re-review a paper that you only judge what the authors have done in response to the editorial / referee comment letter.

One example suggested by Professor Brian Martin is not to make a publication recommendation; instead he includes the statement:

"Scott Armstrong (Wharton School, University of Pennsylvania) for many years has studied and reviewed research on peer review. In order to encourage innovation, he recommends that referees do not make a recommendation about acceptance or rejection, but only comment on papers and how they might be improved, leaving decisions to editors. (See, for example, J.S. Armstrong, 'Peer review for journals: evidence on quality control, fairness, and innovation,' Science and Engineering Ethics, Vol. 3, 1997, pp. 63-84.) I am persuaded by his arguments and hence make no formal recommendation about this paper."

Leave the decision about what to do with the submission to the editors.