The Pleasure and (Mostly) Pain of Peer Review

The chairman of the editorial board of our sister publication, the Texas National Security Review, introduces Vol. 2, Iss. 2 of the journal and discusses the joys and pains of the review process, giving some advice for both reviewers and those submitting their work for review.

“I am sorry to report that I do not like this manuscript much.” As academics, we have all received such a note (and if you haven’t yet, you will at some point). And the negative reactions to our papers and books don’t ever go away, no matter how long you are in the profession or how distinguished your title. This was the first line of a referee report I received just last month for my latest book manuscript.

There is probably no harder part of scholarly life than sending our work out for anonymous peer review. We pour our heart and soul into our work, nurturing, digging, re-shaping our articles and books until we believe they are perfect. We carry around the arguments we make and the evidence we’ve collected in our heads throughout the day, evaluating them at all hours, from when our heads hit the pillow at night to when we get into the shower the next morning. As scholars, our arguments and research both reflect who we are and how we see the world. While we are intellectuals and pretend we are completely objective, we (understandably) become deeply and personally attached to our work. Our articles and books are like our children — we love them fiercely and, at times, irrationally, often blind to the flaws that others see. This is what makes receiving negative reviews so unsettling, especially for the young scholar.

Stay in the business long enough, and you collect stories. Years ago, I submitted an article, “The Myth of Flexible Response,” to a prestigious journal. One “anonymous” review was by someone who said he was involved in the Kennedy administration policy process I was writing on and suggested that I had no idea what I was talking about. The review was handwritten and the journal forgot to remove the initials — “CK” — at the end of the document. Since I had based much of my argument on the papers of Carl Kaysen and was quite familiar with his handwriting, the review was not especially anonymous. The journal passed on the piece. Another review, which I later learned was written by our most accomplished Cold War historian (note: No matter how hard we try not to, we all attempt to figure out the identity of our reviewers), asked, “Are we sure the author does not suffer from dyslexia”? I have countless more stories like this, but simply recalling them is generating a cold sweat!

At the Texas National Security Review, we have and will continue to think long and hard about how to encourage best practices in the review process. We have incorporated a number of measures, including paying our reviewers on a sliding scale depending upon how comprehensive the review is and how quickly it is returned. This has made these reviews even more rigorous than the typical review. Believe me, I know. You may have noticed that I did not write an introduction for the last issue. You may have also noticed that an article of mine appeared in our pages. This generated a whole lot of discussion and concern in our journal — how would it look for a new journal to publish an article by the chair of the editorial board? Did we have a process in place to treat my article in as demanding, ethical, and thorough a way as anyone else’s?

To my (not always pleasant) surprise, we did. I can honestly say that my article received the most rigorous review process I ever went through in over two decades of academic life. There were four anonymous reviews, each well over two pages long, as well as intense internal review. All of the reviews were sharp and penetrating, with a raft of (not always welcome) suggestions, but one in particular was especially harsh. I confess I may not have handled the criticism in the most mature way. I pouted and suggested I might pull the piece and send it somewhere else, arguing that I had gotten to a point in my career where I shouldn’t have to deal with this. Who was the “obvious” idiot TNSR had found to stand in judgment of my “obvious” expertise? It was not my best moment, to say the least. Ryan Evans — who in addition to being the publisher is also one of my best friends — never blinked. My piece, he stated calmly, would have to address all the criticisms if TNSR was to publish it. And no, I would never find out who had reviewed my piece. After putting the article aside for a few weeks, I swallowed my pride and went through the critiques line by line and made the changes. Ryan and his outstanding team of editors worked with me closely to improve the effort. The piece was much better for it. And I still have no idea who the reviewers were, though I am grateful for their extraordinary willingness to offer me honest criticism and helpful recommendations (although reviewer number four would be an unlikely addition to my holiday card list, as petty as that may be!).

I tell this story for a few reasons. First, the process of being evaluated and assessed never ends, no matter how long you stay in the academy. It is important for young people entering the academy to know this. Criticism hurts deeply and often feels unfair. To have your best work dismissed by an unknown, anonymous voice can be devastating. The key is to remember that, no matter how unpleasant, the reviewer took the time to read your manuscript and take it seriously. No matter how painful and even wrong-headed, almost every review I’ve received has been useful, if only to help me better understand how my arguments and evidence are received. Bear in mind, too, that just because an article is rejected does not mean it is wrong or the scholarship poor. As I think about my own work, it is often the pieces that were rejected once or twice (and in one case, four times before being published) that ended up being much better and more influential works than the pieces that sailed through review on a first pass. While you should take criticism seriously, if you believe in your ideas, arguments, and evidence, and have pursued your quest with rigor, honesty, and integrity, then never give up. Keep plodding, never let the critics get you down, and keep improving and pressing. Do this and you will get published and your work will — eventually — be recognized.

The second reason I have related this story is because TNSR is new and is working to establish its reputation as the best venue for rigorous, innovative scholarship on the most pressing questions of strategy, statecraft, and international relations. We want to be accessible and engage the world beyond academics, to include policymakers, without sacrificing the highest standards of scholarship. It is very easy for a new journal to be seen as an outlet for insiders or close friends. The great journal International Security is often — and to my mind, quite unfairly — seen as an outlet reserved largely for scholars within a certain self-contained network in security studies. It was very important to us that we implement the most demanding standards for review so that we could establish the highest scholarly credentials. My painful, if ultimately enormously helpful, experience with the TNSR review process convinced me we are doing well on this score.

The third reason I mention this is that we all wonder about the efficacy and fairness of the current system. Is the double-blind, peer-reviewed process that has become the norm the best way to advance knowledge? Is the current system too easily gamed, or does it encourage scholars — especially young thinkers at the height of their intellectual powers — to be risk-averse, to play small-ball, to write papers and books with the goal of getting through review, rather than expanding our understanding of the world? We all know the reasons we have the current system, but I think it is fair to ask whether it can be improved. Isaac Newton and Albert Einstein transformed our understanding of the world without it. Is double-blind peer review the worst way of evaluating scholarship, except all the others? Or are there ways we can improve the system? At TNSR we don’t know the answer but are very interested in hearing from our audience — both readers and authors. We are willing to break current norms and practices if and when we find better ways of doing things. We encourage you to send us your thoughts.

The final reason is a plea to all future reviewers, for this journal and all others: Be comprehensive, be rigorous, but don’t be a jerk. The benefits of anonymity are obvious, but so, too, are the pitfalls. We live in an age where our social media culture often prizes snark over substance, the witty cut over the empathetic suggestion, the pithy phrase over deep reflection. When writing a review, try to imagine the scholar on the other side of the process, a person who has no doubt invested a good part of their heart, soul, and mind in the work you are reading. This is not a plea to go easy — quite the contrary. At TNSR, we want our reviewers to bring their sharpest, most constructive insights, criticisms, and recommendations. Many reviewers, however, often forget the constructive part. If you are recommending a rejection, ask yourself, was your decision made because the ideas, methods, and evidence are lacking, or because it doesn’t comport with your long-held views (or those of your discipline or field)? Early in my career, I learned from my mentor, Marc Trachtenberg, that evidence and arguments that challenge your deepest beliefs are “like gold in your hands” — they should be embraced, encouraged, and relished. That is how we become smarter. We aren’t in this business to reify our own opinions, but to gain better understanding of enormously complex, often consequential issues. Does the article or book you are reviewing give us more purchase on a new question, new insight, even if the answer does not comport with how you understand the world? Will publication lead to energetic debate and discussion, even if you are somewhat skeptical of the claims? And if the answers to these questions are “yes,” are your critiques and suggestions oriented toward strengthening the piece, to helping the author make their strongest argument in the best possible way? Does your review offer helpful advice, demonstrate empathy, and provide the author with guidance that can help them move their project forward? And if the answers to these questions are “no,” might it be time to ask yourself some hard questions about who you are and why you are in this business? Truth be told, following the adage, “don’t be a jerk,” involves no sacrifice of standards or smarts or rigor. Quite the contrary, in fact.

Rest assured that the excellent scholarly contributions in this volume went through such a process, and that we at TNSR are committed to embracing the highest standards of scholarly review. Enjoy!

Francis J. Gavin is the chairman of the editorial board of the Texas National Security Review. He is the Giovanni Agnelli Distinguished Professor and the inaugural director of the Henry A. Kissinger Center for Global Affairs at SAIS-Johns Hopkins University. His writings include Gold, Dollars, and Power: The Politics of International Monetary Relations, 1958–1971 (University of North Carolina Press, 2004) and Nuclear Statecraft: History and Strategy in America’s Atomic Age (Cornell University Press, 2012).

Image: Creative Commons