By Tom Pepinsky
The peculiar thing about peer review is that it is central to our professional lives as political scientists, yet we tend to talk about refereeing only in the most general and anonymous terms. I have never shown an anonymous referee report on my own submitted manuscripts to anyone, ever. The only other times I have seen other referee reports is when a journal gives me access to other reports for a manuscript that I have refereed. There is very little scope for dialogue on the value of particular referee reports; indeed, as an institution, peer review precludes dialogue between authors and referees unless, of course, the manuscript has already made it past the first stage! That means there is almost certainly a great diversity of viewpoints within the profession about what makes a good referee report, or about what exactly a referee report is supposed to do. Miller et al (2013), discuss the components of a good referee report, but leave unstated the function of the review itself, aside from to note that it is important for the scholarly process.
In my own view, a referee report has three functions. It is first a recommendation to an editor. This is the literal function of peer review, to communicate to the journal about whether or not the manuscript should be published, and under what conditions. The second function of a referee report is to provide comments to the author. The act of evaluating a manuscript entails explaining what its strengths and weaknesses are, and those evaluations are not just for the editors, they are comments for the author as well. Some referees also provide more than just comments on what is working and what is not, making suggestions about how the manuscript might be improved.
The third function of a referee report is an indirect one: shaping the discipline. Because editors use manuscript evaluations to decide whether or not to publish submissions, it follows that referee reports affect what gets published. Because publication is—appropriately—so central to the discipline, and because top journal slots are scarce, referee reports inevitably determine the direction of political science research.
These three functions of peer review differ from one another, and it follows that every referee report is necessarily doing many things at the same time. I suspect that the institution of peer review is probably not ideally suited for doing any of them. For example, the anonymity of peer review enables a kind of forthrightness that might not otherwise be possible with a personal interaction, but the impossibility of dialogue (unless the manuscript already has a good chance of being published) makes little sense if the goal is to provide meaningful feedback to authors. Peer review does effectively shape the discipline, but it may do so in inefficient and even counterproductive ways by discouraging novel, critical, or outside-the-box thinking.
Among political scientists, there are surely very different ideas about which of these three peer review functions are the important ones, and it is here where the anonymity of peer review as an institution makes it difficult to establish common expectations about what makes a referee report good or bad, or useful or not. My own tastes may be peculiar, but I attempt to focus exclusively on recommendations to the editor and comments to the author in my own referee reports. That is, I strive to be indifferent to concerns of the type if this manuscript is published, then people will work on this topic or adopt this methodology, even if I think it is boring or misleading? Instead, I try to focus on questions like is this manuscript accomplishing what it sets out to accomplish? and are there ways to my comments can make it better? My goal is to judge the manuscript on its own terms.
Why? Because as scientists, we must be radically skeptical that there is a single model for the discipline, or that we can identify ex ante what makes a contribution valuable. That is a case for the author to make. If we accept this, then true purpose of the referee report is simply to evaluate whether or not that case has been made.
Readers will note that focusing on feedback to the author and recommendations to the editor discourages certain kinds of commentary when evaluating a manuscript. The comment that this journal should not publish this kind of work, for example, is not relevant to my evaluation. I consider this to be a judgment for the editors, and I assume (perhaps naively) that any manuscript that I have been asked to referee has not been ruled out on grounds of its theoretical or methodological approach. I do not know how to evaluate this kind of research is also not a useful comment for either the editor or the author.
Most importantly, the comment that studies of this type inherently suffer from a fatal flaw only makes sense as a way to provide feedback about how a manuscript might be improved, not as a summary judgment against the manuscript. Imagine, for example, a referee who is implacably opposed to all survey experiments. My views about the function of peer review would hold that his or her referee report should provide concrete feedback about how the manuscript could be improved, with specific attention to whatever proposed flaw stems from the use of survey experiments. This view, in other words, does not insist that referees accept methodologies or frameworks that they find to be problematic. It does demand that referees be able to articulate directions for improvement; in this example, ways to complement the inferences that can be drawn from survey experiments with other forms of data, or suggestions about how the author might acknowledge limitations.
My views about what referee reports are for have evolved over time. Many younger scholars probably believe, as I once did, that the function of the referee is to act the part of the disciplinary police officer, to “protect” the community from “bad” research. What I now believe is that it is much harder to identity objectively “bad” research than we think, and the best way to orient my referee reports is around the question identified above: does this manuscript accomplish what the author sets out to do? These changing tastes on my part are probably a result of my experiences as an author, because I hate reading referee reports that conclude that my manuscripts should be about something else. And so I try to referee manuscripts with this concern in mind, and am especially mindful of this for manuscripts that do not match my own tastes.
There are two additional benefit of approaching peer review this way. First, it encourages the referee to try to articulate what the goal of the manuscript is. This should be obvious from the submission itself, and if it is not, then that is one big strike against the manuscript. (One useful exercise when writing a referee report is simply to summarize the manuscript.) Second, it should work against the incentives for disciplinary narrowness that are inherent to peer review. Some manuscripts are narrow and precise. Some manuscripts strive to be provocative. Some manuscripts seek to integrate disparate theories or literatures. Working from the assumption that each of these types of submissions can be valuable, judging a manuscript with an eye towards the author’s aims should encourage more creative research without punishing research in the normal science vein.
These points notwithstanding, it is surely true that referee tastes affect their evaluations, and to pretend otherwise is counterproductive. I suspect that editors have massive influence over the fate of individual manuscripts in choosing referees, and also in weighing different referees’ evaluations when their evaluations diverge.
Editorial judgments are probably most important when considering cross-disciplinary work, for the functions of peer review almost certainly vary across disciplines. I have refereed manuscripts from anthropology, religious studies, and Asian studies, among others. Before agreeing to referee these manuscripts, I always make clear to editors that they are going to get a referee report that follows my understanding of the conventions of political science. My logic is “if you go to a barber, you’ll get a haircut,” so I want to make sure that editors know that I am a barber, and so they should not expect a sandwich.
These presumed differences aside, no editor has ever thought that my training as a political scientist disqualifies me; conditional on inviting me to referee a manuscript, editors uniformly pledge accept my disciplinary biases. Even so, referees have a particular responsibility to authors of manuscripts outside of their disciplinary traditions, not because our evaluations are necessarily negative (I have indeed recommended in favor of publication in some cases), but because they probably draw on an entirely different understanding of what makes research valuable, and how authors can demonstrate that their work has accomplished the goals that they set out for themselves. Without such a shared understanding, it is hard to see how we are “peers” in any meaningful sense.
This point brings us back to my initial observation, that as a discipline we talk about referee reports only in the most general and anonymous terms, without consensus about what peer review is for. It helps, when receiving a painful negative review, to recognize that referees may have very different assumptions about what their reports are supposed to do. A more constructive focus on editorial recommendations and author comments will never soothe the sting of a rejection (or a “decline”; see Isaac 2015), but it could ensure that the dialogue remains focused on what the manuscript hopes to accomplish. And that—not policing the discipline—is what peer review is for.
References
Isaac, Jeffrey C. 2015. “Beyond ‘Rejection.’” Duck of Minerva, December 2, http://duckofminerva.com/2015/12/beyond-rejection.html.
Miller, Beth, Jon Pevehouse, Ron Rogowski, Dustin Tingley, and Rick Wilson. 2013. “How To Be a Peer Reviewer: A Guide for Recent and Soon-to-be PhDs.” PS: Political Science & Politics 46(1), pp. 120-123.