Reviewing a Paper

Original Text by Bill Griswold

The questions you want to have answered by reading a paper are the following:

• What are motivations for this work? For a research paper, there is an expectation that a problem has been solved that no one else has published in the literature. This problem intrinsically has two parts. The first is often unstated. Think of this as the people problem. The people problem is the benefits that are desired in the world at large; for example some issue of quality of life, such as saved time or increased safety. The second part is the technical problem, which is why the people problem does not have a trivial solution; that is, why a new technological or engineering solution may be required. Implicitly there is implication that previous solutions to the problem are inadequate. Occasionally an author will fail to state either point, making your job much more difficult.
• What is the proposed solution? This is also called the hypothesis or idea. There should also be an argument about why the solution solves the problem better than previous solutions. There should also be a discussion about how the solution is achieved (designed and implemented) or is at least achievable.
• What is the evaluation of the proposed solution? An idea alone is usually not adequate for publication of a research paper. What argument and/or experiment is made to make a case for the value of the ideas? What benefits or problems are identified? Are they convincing?
• What are the contributions? The contributions in a paper may be many and varied. Ideas, software, experimental techniques, and area survey are a few key possibilities.
• What are future directions for this research? Not only what future directions do the authors identify, but what ideas did you come up with while reading the paper?

More to Add
Copied from Harvard Site

Many conferences ask you to judge the paper (usually on a scale of 1 to 5) in a number of categories. For
example:

• Relevance: How relevant is the paper to the conference?
• Presentation: How well-written is the paper? Is it totally incomprehensible or lucid and eloquent?
• Originality: How novel is the paper? Are the technical ideas presented new?
• Correctness: Is the paper technically correct? Are the experiments performed or the analysis presented valid?
• Confidence: How well-versed are you, the reviewer, in this area? Are you an expert in the field and confident your feedback is correct or are you unfamiliar with the field and unsure of your feedback?
• Overall: What is your overall rating for this paper? Do you enthusiastically support acceptance of this paper into the conference, or would you be embarrassed to be on a Program Committee thataccepted a paper like this?