Verified by 
“Peer review” involves scientists reading other scientists’ work and deciding whether it is worthy of being published. I had a mentor who used to say that science wasn’t truly science until it was disseminated (published or at least presented at scientific conferences). Until then, all that tinkering in the lab is really nothing more than a hobby, because the efforts have not contributed to our shared body of scientific knowledge.
The peer review process is an essential component of that, because not just anything counts as science. Regrettably, there is a certain amount of “scientism,” among academics as well as critics of science, who dress up their language with science-sounding terms but are not really practicing science. It is only science if there is a systematic attempt to meaningfully contribute to scientific knowledge.
In practice, the way it is determined if this bar has been met is by sending scientific reports to independent reviewers for comment and critique. They will give suggestions about how to improve the presentation or clarify details about the scientific method. It is rare to see an article in a scientific journal that has not gone through this review and revision process.
Although this is an essential part of the scientific process, surprisingly it is one that scientists often receive relatively little training in. Many graduate students, junior scholars, or people working at low-resource institutions have not had the opportunity to get some good instruction about how to conduct a scientific review. As an editor of a scientific journal, I am often asked for such guidelines for reviewers and share some suggestions below.
The peer review process relies on thoughtful, rigorous reviews. The field of psychology is fortunate because most researchers are dedicated to provided balanced, constructive reviewers that appropriately identify the main strengths and weaknesses of a manuscript. Providing reviews to other scholars is one important avenue for influencing the field and maximizing the quality of violence research.
The result is a body of evidence that is probably stronger in many ways than it is sometimes portrayed in the media or by politicians. It is not easy to publish a scientific study, especially in a top journal.
General Standards for the Review Process
1) It is important to maintain a constructive tone and identify strengths as well as weaknesses. Regardless of whether the manuscript gets published at a particular journal, the ultimate goal of the peer review process is to improve the overall level of science in the field. Hopefully, even when manuscripts are rejected, authors will find the comments useful to their future work. Over the years, I have received many appreciative comments from authors, even to rejection letters, about helpful advice for continuing their research.
If you are reading a paper that seems to have numerous problems, remember that many of those might be early efforts by students or other junior scholars. We don’t want to scare them out of the field with needlessly hostile reviews, we want to mentor people to reach the highest scientific standards.
2) There are usually numerous acceptable methodological and statistical choices. One of the first pieces of advice that I give new reviewers or new editors is to realize that other scientists need not have made the choices you did. There are almost always multiple reasonable options for measures, statistics, citations to other work. A plurality of approaches benefits the field.
The Elements of a Review
Narrative comments. The main part of the review is detailed narrative comments. If your only experience with reviews is getting a paper back in a college or high school class, you would probably be surprised how detailed the comments are. Usually there are comments from 2 or 3 anonymous reviewers plus the editor. Altogether, it is common that these would add up to several typed pages of feedback. I’ve gotten reviews back for several of my own papers that add up to 9 or 10 single-spaced pages of feedback!
The typical individual review ranges from a couple of paragraphs to a few pages of narrative that includes specific feedback for the authors. It is common for reviews to be organized in the order of the manuscript. Some reviewers prefer to organize their reviews from what they consider to be the most to least important comments. Either organization is acceptable. Authors often find it helpful if the comments are numbered, so they are more easily referred to in their cover letter explaining how they have revised the manuscript.
It might seem nice to write a very short, completely positive review, but in the long run these are not only harmful to science, they do not help scientists improve either. You would not tell someone that they could win Wimbledon who can barely swing a tennis racket, and it is not nice to suggest that a study is great if it is significantly flawed.
Narrative comments are shared with authors.
Overall recommendation. The names for different publication recommendations vary from journal to journal, but usually the following categories are included in some form. These recommendations CANNOT be seen by authors, only by the editor:
Accept As Is: This is generally reserved for manuscripts that have been through one or more revisions. It is important that manuscripts be as strong as possible, both in terms of the science and the clarity of presentation. Virtually every manuscript has room for improvement. “Accept As Is” is an extremely rare initial decision.
Minor Revision: Minor, specific revision refers to less extensive edits, such as clarifying points or elaborating on implications.
Questions about the adequacy of the rationale, the suitability of the measures, or the soundness of the statistical approach are seldom minor.
“Minor” also usually implies that there are only a few remaining issues to be addressed. Your review will be compiled with others who are likely to raise different points, so if you have identified many points to address (certainly 10 or more), then it is unlikely to fall into the “minor revision” category.
Major Revision: This recommendation is appropriate for manuscripts that have potential to make a sound contribution, but require more substantive revisions in order to fully evaluate the science. Recommendations for to re-do statistical analyses usually fall into this category, as a re-analysis of the data might cast the results in a very different light. This is also appropriate for papers that provide incomplete descriptions of one or more measures, if the adequacy of the operationalization cannot be determined. None of these issues, however, are likely to dramatically change the results or interpretation of the data, and there is no indication of a major insurmountable problem (such as a problem with the procedure that cannot be “fixed”).
Reject: This option is for manuscripts that have limited potential to contribute to the literature. This could be due to insurmountable sampling or measurement problems (statistical problems are potentially more “fixable”). It can also be due to the limited scope of the study, especially if that seems to indicate a piecemeal approach to publication (for example, focusing on only 2 or 3 variables from a larger dataset that could potentially provide a more comprehensive analysis).
Although of course no author hopes to receive a “reject” decision, rejection does serve several important purposes. Principally, these decisions help to maintain high standards of scientific rigor. Hopefully, reject decisions will also help researchers design future studies or give them ideas about how to strengthen their submission for another outlet.
Preparing a Review
There are an almost infinite number of issues that might be raised in a review, but some of the most commonly mentioned concerns involve the following:
1) Empirical papers need to establish how the current manuscript meaningfully builds on existing work (including the most recent work) and especially how the current manuscript will make a novel contribution to the field. Beyond this, it is not necessary, nor is there room in a manuscript, to provide detailed literature reviews in a paper presenting new data.
2) The introduction needs to provide a specific rationale, ideally one based in theory, for including the main variables under study.
3) Literature reviews and theoretical papers should also clearly indicate how the synthesis will advance the field.
4) Regarding the methodology, one of the most important issues is to focus on the adequacy of the operationalization of the constructs. High quality measurement is essential. No amount of sophisticated statistical analyses can make up for problematic measures or flawed sampling procedures.
5) Statistics need to be clearly justified and explained, and appropriate for testing the hypotheses.
6) The purpose of discussions is not to simply re-cap results, but to put the findings in the context of prior literature, acknowledge limitations of the current study, and suggest specific implications for future research and applications to prevention, intervention, or policy.
Learning about peer review and how to do good reviews is great for becoming a better scientist and also a better consumer of scientific information.
Note: This is adapted from an earlier reviewer guide I prepared for Psychology of Violence reviewers.
