How to Review
a Scientific Paper

write one that gets good reviews,

and address reviewers' comments

Review workflow

First review

Rejected

Editor

Rejected

Accepted

n-th review

Accepted

Final version

Revised version

First version

Journals only

What to assess in a review?

• List 8 things you check in a review
   
• Discuss in pairs and eliminate duplicates
   
• Pin them to the board, trying to cluster them around some key concepts

What to assess in a review?

• Relevance
  • problem motivation
  • assumptions
  • applicability
• Significance
  • contribution
• Originality
  • same authors didn't publish already
  • originality of solution
  • departs from literature
• Citations
  • literature review
  • primary sources are cited
  • references

• Technical correctness
  • maths and logic
  • technical rigor
  • correctness
  • simulations = ideas
  • design of experiments
  • simulation can be reproduced
• Presentation
  • technical presentation
  • clarity
  • logical flow
  • descriptive title
  • structure
  • balanced sections
  • language
  • self-explanatory diagrams
  • correct language
  • writing
  • mathematical notation

What to assess in a review?

• Relevance
• Significance
• Originality
• Adequacy of Citations
• Technical Quality
• Presentation

Relevance vs. Significance

Relevance

Significance

Originality & Adequacy of Citations

Check Journal policies!

• IEEE: When using their own previously published or submitted material as a basis for a new submission authors are expected to "cite the previous work(s) and very clearly indicate how the new submission differs from the previously published work(s)."
• IEEE TAC: If a submitted manuscript has been published or has been accepted for publication in the Proceedings of a conference, it may be considered for publication if evidence is provided that it adds value relative to its conference version.
• Operational Research: The manuscript has not been published previously (partly or in full). A single study is not split up into several parts to increase the quantity of submissions.

Technical Quality vs. Presentation

q.e.d.

Who checks

the math?

• Is the reviewer supposed to check the math?
• Does the author/journal maintain responsibility?
• What if I am unable to check (time/competence)?

• Syntax & grammar
• Document structure
• Notation
• Legibility of figures
• Accessibility
• Spelling

Anatomy of a review

Comments to the Editor

Comments to the Authors

Authors cannot read it.

Could contain

• Comments that disclose your identity
• Comments on the review process
• Ethical concerns
• Concise and direct recommendation
• List of things you could not assess and reason for that

Confidential

Your review is anonymous.

A common rule of thumb: don't say things you wouldn't say in person to the authors.

Anatomy of a review

Relevance

• Topic
• Methods
• Journal-specific

Comments to the Editor

Comments to the Authors

Confidential

Anatomy of a review

• Expected impact
• Incremental?
• Educational?
• New model?
• New method?

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Anatomy of a review

• Authors' work
• Other's work
• Other communities

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Anatomy of a review

• Other's work
• Your work?
• Fair recognition

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Anatomy of a review

• Ask for clarifications
• Challenge the approach
• Clarify your requirements
• Suggest fixes?

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Anatomy of a review

• Paper structure
• Second language
• Figures
• Spell-check?
• Suggestions

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Presentation

Anatomy of a review

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Presentation

Summary of the contribution

• You read it
• You show the authors what you understood
• You avoid misunderstanding

Anatomy of a review

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Presentation

Final recommendation

Don't assume here that a specific decision will be taken.

Is it possible to address the negative points?

How?

Reject    /      Major revision
Minor revision      /      Accept as it is

Exercise

• Look at the samples in
  https://people.ee.ethz.ch/~bsaverio/reviews/
• Mark the different parts (anything missing?)
• Assume you are the associate editor
  • Is this review an "expert opinion"?
  • Is there enough info to make a decision?
  • What would you like to ask to the reviewer?
  • What is your decision and requests?
• Assume you are the author
  • What are the main points to address?
  • Do you think you will be able to do that?
  • What will you do?

Integrity of Peer Review

Read each of the following stories, and list all the ethical issues that you can identify, before reading the solutions in the following slide.

Case 1

Professor Farnsworth runs a productive research group with both students and postdocs. To help his trainees develop their communication skills, they give talks in group meeting, seminars in the department, etc.

Professor Farnsworth is asked to review a paper for a journal, and he asks one of his students to study it, present it in the group meeting, where the paper is discussed.

Based on the discussion, he prepares his review for the editor of the journal.

Case 1 (solution)

• The confidentiality of the review process
  • Editors may authorize external discussion
• Taking credit for the work of others
  • Review requests can be transferred
  • The work of reviewers is acknowledged
• Misrepresentation to journal

Case 2

Dr. Wernstrom accepts to review a paper on a topic on which he has recently submitted a paper himself. His paper has been accepted, but it is not publicly available.

The paper he has to review covers a special case of his work. Dr. Wernstrom posts a preprint on arXiv, and writes in the comments to the authors that they should have a look at that, stating that the paper is not acceptable because it lacks originality.

Case 2 (solution)

• Accepting review requests on topics on which you are actively working.
  • Sometimes we receive the manuscript together with the request
  • It happens often, as we are selected as experts.
• False conflict of interest: you are not using the confidential information for your interest.
• The confidentiality of the review process
  • ​The "Comments to the editor" are a better choice to discuss the originality issue.

Case 3

Prof. Farnsworth is often asked to review papers for many journal and conferences. Because of that, he is very late on the deadline for one of these reviews.

He asks Dr. Wernstrom if he can help.

Dr. Wernstrom exceptionally accepts (he almost always declines any review request).

Prof. Farnsworth then sends the manuscript to Dr. Wernstrom and asks the Editor to transfer the review.

Case 3 (solution)

• Delaying a review affects the work and career of other researchers.
  • It is better not to accept the review request.
• Participating to the peer-review system is expected by ETH, the community, and society.
• Manuscripts remain confidential even if you decline to review.
  • Let the Editor handle the transfer.

Case 4

Dr. Wernstrom is reviewing a journal paper by some researchers that he doesn't know personally.

The paper under review cite Dr. Wernstrom's last work, and acknowledge him for making some Matlab code available on his website.

Dr. Wernstrom will visit the lab of these authors next week, because he is applying for a position there, and he has been invited for an interview.

He finds the paper brilliant, and recommends it.

During his interview, he congratulates for the paper.

Case 4 (solution)

• The only issue is the disclosure of his identity as a reviewer of the paper
  • Journal policy of peer-review anonymity
  • Other reviews may not be as positive as his
  • His review could be mis-interpreted
  • Conflict of interest in his job application

ETH Guidelines for research integrity

Art. 17 Principles of peer review

1. ETH Zurich researchers are willing to serve as experts, in particular concerning [...] acceptance of publications (peer review) [...].

2. The expert person's anonymity warrants the highest degree of objectivity, impartiality and confidentiality. The expert person:

a. must treat all information in question as confidential as long as it has not been published by the authors;
b. may not consult other experts on the subject of the expert opinion without the consent of the competent body who has appointed him;
c. may not use the confidential information disclosed to him in the course of his activities as expert; and
d. must provide timely, unbiased, constructive and well-established expert opinions. He must refrain from making emotional, derogatory or offensive remarks.

Who checks the math?

I only hope that computer-aided proof checking saves mathematics before it collapses under the weight of decades of irresponsible publishing.

Of all disciplines, peer review in mathematics should serve to guarantee nearly absolute confidence in the validity of published results.

Many subjects have grown so complex that one can't reasonably expect new people coming to the field to take responsibility for the correctness of all the literature that they might need to quote.

Leslie Lamport, "How to Write a 21st Century Proof"
http://research.microsoft.com/en-us/um/people/lamport/pubs/proof.pdf

Newton's Principia Mathematica (1728)

We no longer write mathematical formulas in prose.

On the other hand, proofs are still in prose, and surprisingly similar to 18th century proofs!

• Is each sentence a new fact?
• Is it obviously true, or something about to be proved?
• If it follows from previous facts, from which ones?
• What does the = sign means?

Other issues with 18th-century-style proofs:

• The author decides the level of details
  • too deep, you miss the big picture
  • too shallow, it's not rigorous enough

Solution: STRUCTURE + NAMING

Structured proof

Machine-verifiable proof

Error-proof
naming

analytics by Ryan Beck from the Noun Project

review by Daniel Nochta from the Noun Project

badge by Pham Thi Dieu Linh from the Noun Project

How to Review a Scientific Paper by Saverio Bolognani is licensed under a Creative Commons Attribution 4.0 International License.