How to Review
a Scientific Paper

write one that gets good reviews,

and address reviewers' comments

Review workflow

First review

Rejected

Editor

Rejected

Accepted

n-th review

Accepted

Final version

Revised version

First version

Journals only

What to assess in a review?

List 8 things you check in a review
Discuss in pairs and eliminate duplicates
Pin them to the board, trying to cluster them around some key concepts

What to assess in a review?

Relevance
- problem motivation
- assumptions
- applicability
Significance
- contribution
Originality
- same authors didn't publish already
- originality of solution
- departs from literature
Citations
- literature review
- primary sources are cited
- references

Technical correctness
- maths and logic
- technical rigor
- correctness
- simulations = ideas
- design of experiments
- simulation can be reproduced
Presentation
- technical presentation
- clarity
- logical flow
- descriptive title
- structure
- balanced sections
- language
- self-explanatory diagrams
- correct language
- writing
- mathematical notation

What to assess in a review?

Relevance
Significance
Originality
Adequacy of Citations
Technical Quality
Presentation

Relevance vs. Significance

Relevance

Significance

Originality & Adequacy of Citations

Check Journal policies!

IEEE: When using their own previously published or submitted material as a basis for a new submission authors are expected to "cite the previous work(s) and very clearly indicate how the new submission differs from the previously published work(s)."
IEEE TAC: If a submitted manuscript has been published or has been accepted for publication in the Proceedings of a conference, it may be considered for publication if evidence is provided that it adds value relative to its conference version.
Operational Research: The manuscript has not been published previously (partly or in full). A single study is not split up into several parts to increase the quantity of submissions.

Technical Quality vs. Presentation

\frac{d}{dx}x^2=\frac{d}{dx}(x+x+\cdots+x) \qquad x\ \mathrm{times}

x^2=x+x+\cdots+x \qquad x\ \mathrm{times}

2x=1+1+\cdots+1=x

2=1

q.e.d.

Who checks

the math?

Is the reviewer supposed to check the math?
Does the author/journal maintain responsibility?
What if I am unable to check (time/competence)?

Syntax & grammar
Document structure
Notation
Legibility of figures
Accessibility
Spelling

Anatomy of a review

Comments to the Editor

Comments to the Authors

Authors cannot read it.

Could contain

Comments that disclose your identity
Comments on the review process
Ethical concerns
Concise and direct recommendation
List of things you could not assess and reason for that

Confidential

Your review is anonymous.

A common rule of thumb: don't say things you wouldn't say in person to the authors.

Anatomy of a review

Relevance

Topic
Methods
Journal-specific

Comments to the Editor

Comments to the Authors

Confidential

Anatomy of a review

Expected impact
Incremental?
Educational?
New model?
New method?

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Anatomy of a review

Authors' work
Other's work
Other communities

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Anatomy of a review

Other's work
Your work?
Fair recognition

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Anatomy of a review

Ask for clarifications
Challenge the approach
Clarify your requirements
Suggest fixes?

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Anatomy of a review

Paper structure
Second language
Figures
Spell-check?
Suggestions

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Presentation

Anatomy of a review

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Presentation

Summary of the contribution

You read it
You show the authors what you understood
You avoid misunderstanding

Anatomy of a review

Originality

Relevance

Comments to the Editor

Comments to the Authors

Confidential

Significance

Citations

Technical Q.

Presentation

Final recommendation

Don't assume here that a specific decision will be taken.

Is it possible to address the negative points?

How?

Reject / Major revision
Minor revision / Accept as it is

Exercise

Look at the samples in
https://people.ee.ethz.ch/~bsaverio/reviews/
Mark the different parts (anything missing?)
Assume you are the associate editor
- Is this review an "expert opinion"?
- Is there enough info to make a decision?
- What would you like to ask to the reviewer?
- What is your decision and requests?
Assume you are the author
- What are the main points to address?
- Do you think you will be able to do that?
- What will you do?

Integrity of Peer Review

Read each of the following stories, and list all the ethical issues that you can identify, before reading the solutions in the following slide.

Case 1

Professor Farnsworth runs a productive research group with both students and postdocs. To help his trainees develop their communication skills, they give talks in group meeting, seminars in the department, etc.

Professor Farnsworth is asked to review a paper for a journal, and he asks one of his students to study it, present it in the group meeting, where the paper is discussed.

Based on the discussion, he prepares his review for the editor of the journal.

Case 1 (solution)

The confidentiality of the review process
- Editors may authorize external discussion
Taking credit for the work of others
- Review requests can be transferred
- The work of reviewers is acknowledged
Misrepresentation to journal

Case 2

Dr. Wernstrom accepts to review a paper on a topic on which he has recently submitted a paper himself. His paper has been accepted, but it is not publicly available.

The paper he has to review covers a special case of his work. Dr. Wernstrom posts a preprint on arXiv, and writes in the comments to the authors that they should have a look at that, stating that the paper is not acceptable because it lacks originality.

Case 2 (solution)

Accepting review requests on topics on which you are actively working.
- Sometimes we receive the manuscript together with the request
- It happens often, as we are selected as experts.
False conflict of interest: you are not using the confidential information for your interest.
The confidentiality of the review process
- The "Comments to the editor" are a better choice to discuss the originality issue.

Case 3

Prof. Farnsworth is often asked to review papers for many journal and conferences. Because of that, he is very late on the deadline for one of these reviews.

He asks Dr. Wernstrom if he can help.

Dr. Wernstrom exceptionally accepts (he almost always declines any review request).

Prof. Farnsworth then sends the manuscript to Dr. Wernstrom and asks the Editor to transfer the review.

Case 3 (solution)

Delaying a review affects the work and career of other researchers.
- It is better not to accept the review request.
Participating to the peer-review system is expected by ETH, the community, and society.
Manuscripts remain confidential even if you decline to review.
- Let the Editor handle the transfer.

Case 4

Dr. Wernstrom is reviewing a journal paper by some researchers that he doesn't know personally.

The paper under review cite Dr. Wernstrom's last work, and acknowledge him for making some Matlab code available on his website.

Dr. Wernstrom will visit the lab of these authors next week, because he is applying for a position there, and he has been invited for an interview.

He finds the paper brilliant, and recommends it.

During his interview, he congratulates for the paper.

Case 4 (solution)

The only issue is the disclosure of his identity as a reviewer of the paper
- Journal policy of peer-review anonymity
- Other reviews may not be as positive as his
- His review could be mis-interpreted
- Conflict of interest in his job application

ETH Guidelines for research integrity

Art. 17 Principles of peer review

1. ETH Zurich researchers are willing to serve as experts, in particular concerning [...] acceptance of publications (peer review) [...].

2. The expert person's anonymity warrants the highest degree of objectivity, impartiality and confidentiality. The expert person:

a. must treat all information in question as confidential as long as it has not been published by the authors;
b. may not consult other experts on the subject of the expert opinion without the consent of the competent body who has appointed him;
c. may not use the confidential information disclosed to him in the course of his activities as expert; and
d. must provide timely, unbiased, constructive and well-established expert opinions. He must refrain from making emotional, derogatory or offensive remarks.

Who checks the math?

I only hope that computer-aided proof checking saves mathematics before it collapses under the weight of decades of irresponsible publishing.

Of all disciplines, peer review in mathematics should serve to guarantee nearly absolute confidence in the validity of published results.

Many subjects have grown so complex that one can't reasonably expect new people coming to the field to take responsibility for the correctness of all the literature that they might need to quote.

Leslie Lamport, "How to Write a 21st Century Proof"
http://research.microsoft.com/en-us/um/people/lamport/pubs/proof.pdf