Sobel’s “Mourning edition” offers an authoritative peek at the mechanics of peer review in economics
‘Teach me your tricks,’ I say to him; ‘I want to be a clown like you.’ He puts his hand on my shoulder. ‘I can’t teach you,’ he says. ‘This is my act, and to make it really mine I’ve been practicing it for decades. Every decent clown has his act. If you are patient, by the time you are an old man you’ll perfect your own.’
—G. Konrad, quoted in “Mourning Edition”
Read an introduction to the Talk and the Top series here.
Joel Sobel, a game theorist on the economics faculty at UC San Diego, served as co-editor of the American Economic Review from 2009 to 2010 and editor of Econometrica from 2015 to 2019. In a meandering and rather sweet meditation written in 2020 upon his departure from the latter post, he shares a wealth of institutional knowledge about editorial expectations at the Top 5, compiled on the basis of talks given to third-year graduate students at UCSD and similar audiences. He writes:
I have (electronically) signed about 2,000 decision letters in these roles and have written another 1,000 as an anonymous referee or Associate Editor… More people have read—and in many cases read with great care—my decision letters than my research.
The memo is addressed to researchers in their capacities as writers, reviewers, and editors, adorned with Sobel’s own pensive and often wry commentary on dysfunction in peer review and the future of the academic publishing industry. While the entire manuscript is worth lingering over—especially for graduate students and juniors—we extract a few gems below for more harried readers.
On what to expect from the review process
“If you face time constraints, do research on the response time and acceptance probabilities at different journals before you decide where to submit. Time constraints are most likely to be an important consideration for people nearing a tenure review. If you are in this situation, get specific advice from senior colleagues in your department to approach the difficult problem of balancing departmental deadlines and standards with variable response times at journals.”On submitting too soon
“Submitting prematurely is costly. In the best case, you may receive a speculative request for a revision. This request will likely remove some of your ability to frame the paper. Instead a sympathetic editor or reviewer will impose a frame on you. The author is typically better informed and probably has a better notion of what the paper should be, so I see no value in giving extra discretion to reviewers.”On suggesting your own reviewers
“It is not customary for authors to suggest referees in economics. Authors rarely do so and I was skeptical of suggestions when they were offered. Naming a referee increases the probability that the nominee would be asked to write a report, but decreases the weight placed on the reviewer’s evaluation. [footnote: I did consider the possibility that a strategic author would nominate a referee in order to lower the probability that the person gets asked.] I see nothing wrong with authors identifying reviewers, but these recommendations will always create the suspicion that the author is putting forward the name of someone biased in favor of the paper.”On receiving an R&R
“Some requests to resubmit are speculative and virtually every invitation carries the risk that you will ultimately be rejected, but a request to resubmit is a signal that someone influential in the editorial process is on your side.”
On “reject and resubmit”
“Sometimes, usually due to bad writing, reviewers were unable to evaluate the contribution and I wanted to give the authors another chance to make the case for publication. One way to respond to papers in this category is to reject the paper but leave the door open for a resubmission. These “reject and resubmit” decisions distinguish between a definitive rejection and an unambiguously positive decision to encourage a revision. … I found that most ‘reject and resubmits’ led to negative decisions, unhappy authors, and unhappy reviewers. I did not offer any ‘reject and resubmit’ decisions during the final two years of my term. What I did instead is increase the number of speculative revise and resubmit decisions.”On undertaking multiple rounds of revision
“If you are asked to revise your paper a second time, you can be confident that you will be able to publish the paper. Exceptions arise. In my experience they are due to unusual initial conditions (the first review is actually the review of a paper that had been rejected; the second revision involved a new set of referees; the editor points out a specific, necessary condition and frames the invitation to revise as a demand). I rejected papers after inviting two revisions, but I view these as cases where I failed to do my job effectively.”On receiving a rejection
“After a rejection, you go through stages that may parallel Kübler-Ross’s stages of dying (denial, anger, bargaining, depression, and acceptance). The denial phase may involve the hope that the rejection letter was sent by mistake, followed by the realization that the referees’ comments were fatally flawed and that a sensible process would reverse the decision on appeal. Anger begins when you realize that the incompetent reviewers control the situation and you are unlikely to change the decision. Bargaining arises if you appeal the decision. You become depressed when you realize that the editorial decision is final and you wonder whether all of your work will be poorly received, unfairly reviewed, or (perhaps) is not as good as you once thought. The depression may turn into negative reflections on your status in the profession. When you get to the acceptance stage, you are able to look at the paper again and, in most cases, prepare it for submission to another journal. How long it takes to pass through these stages (and the pain associated with each one) depends on you, on the quality of the project, on the quality of the review, and on the broader context (it helps to have a support network and job security). Take some consolation [that] at the end of the five stages you, unlike Kübler-Ross’s subjects, will probably still be alive.”On calibrating your expectations for referee feedback
“Politeness norms suggest that the reactions you get from advisers, colleagues, and seminar audiences will be more positive than what you can expect from anonymous reviewers.” “Referees can be lazy, insensitive, and stupid. But so can any reader of your paper. Treat negative comments—even ones that are objectively incorrect—as suggestions that you are demanding reviewers to do too much. Sometimes if readers understand what you are trying to do, they will be sympathetic. Better writing can reduce the chance of misunderstanding.”
On authors’ propensity to incorrectly guess the identities of their referees
“Blame the editor for failing to discount inappropriate criticism (you have my permission to blame me retroactively), but do not adjust the way you interact with a colleague because you conjecture that the colleague has reviewed your paper.”
On maintaining the integrity of your contribution throughout the review process
“Do not let the revision process degrade your work and fight for aspects of the paper that you believe important. You should be prepared to lose some of these fights. … If you face no time pressure and possibly in general, I recommend that you try to publish what you want in a different journal rather than let the review process degrade your work.”On the job of the referee
“The referee should identify arguments that are not clear or convincing, suggest literature that is relevant and incompletely discussed, and express doubts about the power or generality of the message. At some point, comments in these categories stop being constructive. … It is valuable for referees to contribute their opinions, but the process should not assume that the author must include a discussion simply because a referee requests it.” “Some referees do not make explicit recommendations in referee reports to give editors more discretion. This is not necessary. Editors already have enough discretion.” “It is not a referee’s job to show off for the editor. … Save the brilliance for your own research.”On who will make a good editor
“It is easy to measure conscientiousness (number of reports completed and time needed to complete the report) than the quality of advice. Generally search committees for open editorial positions have some idea about the quality of advice offered by candidates for editorial positions, but information about speed and willingness to review is easy to access and interpret. I was asked to be an editor primarily because I was a conscientious referee. Conscientiousness may not be correlated with good judgment.”On the returns to editorial service
“Editors get paid. I never asked how much I would get paid as an editor [footnote: When the first check arrived, I dreamed that the payments came monthly. When a second check didn’t arrive the next month, I feared that the payments were annual. They were quarterly.] and the level of compensation did not influence my decision to become an editor. Editors have residual authority. Their decision is pivotal in a significant fraction of cases. [footnote: I would say between 5 and 10 percent, but it is not clear what pivotal means.] I imagine some people get intrinsic pleasure from having authority. More people value the opportunity to use their authority to move the profession in a direction they prefer.”On the value of peer review
“Even if you have faith that reviewers are wise and public spirited, it is not clear that revisions lead to better papers. The review process adds more value than it subtracts (especially taking into account that the existence of the review process changes how authors prepare papers), but I am skeptical about whether it is worth the cost (from authors and reviewers).”
On Top 5-itis
“Heckman and Moktan document the importance of top five publications. The number of articles published in these journals has not kept pace with the number of scholars competing for senior faculty positions in strong research departments. Not all of the journals are sympathetic to submissions in all subfields, leading to biases in the opportunities available to scholars working in different subfields. Journals may impose space limitations that limit the kind of articles that they publish. Two of the journals are strongly associated with specific departments and the other three are associated with professional societies. So it is hard to ignore Heckman and Moktan’s concern that the weight that the profession places on these publications concentrates power in the hands of a small number of journal editors and distorts the kinds of research that young scholars pursue.”On the future of academic publishers
“One could make an argument that there was a time when academic publishers played a vital role. They earned their rents coordinating the publication process and created a permanent record of scholarly work. …. But it appears that the old business model of journals will become less profitable and commercial, academic publishers will need to make dramatic changes in order to survive.”
Our mission at Standard Error is to serve the international research community by providing disciplinary and editorial expertise throughout the submission and review process for social science publications of record. We aim to position ourselves as a reference trusted by social scientists for our editorial rigor, sharp judgment, high standards, commitment to quality and field expertise while maintaining accessibility for researchers worldwide. Standard Error also aims to provide a platform for highlighting the public impact of the research generated by the scholars whose work we support.
