How to Really Review Papers

1
How to Really Review Papers

• CS 8803 AIC

2
Paper Reviewing Algorithm

• Read the paper
• Think about it
• Take a look at related work
• Leave it alone for a few days
• Read it again
• Write the review

3
Thinking about it

• What is the paper trying to do
• Observation/Motivation
• Programs often do X
• In the future, Y wont work
• Implementation
• Heres how to exploit this observation
• Evaluation
• We get great perf/power/reliaility/ improvement,
  or
• It didnt work and heres why

4
Contributions

• Did you gain an important insight
• A new problem?
• A new way to look at a problem?
• Counter-intuitive or unexpected?
• Did it propose a good solution to a problem?
• Explanation why solution should work
• Explanation why its a better solution
• Did it provide a good quantitative evaluation
• Should work in real processors?
• Should it gain or lose relevance in the future?
• Should work on real applications?

5
Summary

• Like a paper abstract
• Short and self-contained (no undefined acronyms)
• Short high-level description of contribution
• But without the sales pitch
• Avoid quoting numbers
• What is really the contribution

6
Good points

• Successful contributions
• If you learned something, thats a good thing
• If could be useful for real systems, thats good
• List the most important things
• If you only list Well written, paper is really
  bad
• There are ALWAYS good points ?

7
Bad points

• Motivation problems
• Unclear whether there is a problem
• Unclear why its an important problem
• Implementation problems
• Unclear why solve the problem this way
• Unclear what tradeoffs are
• Evaluation problems
• We all know about these
• Organization problems
• Difficult to follow?
• List the most important bad points
• If difficult to read is the only problem, its
  a great paper

8
Questions to answer

• Important questions only
• If answer cant change your recommendationdont
  ask the question here (see next slide)
• Can it be answered in a few days?
• If lots of work needed to answer, dont ask here
• Unsure if you got something right?
• If decision depends on something, andyou are not
  sure if your understanding is correctDO ask it
  here

9
General Comments Section

• Help the authors improve the paper
• Always be constructive
• Even if paper is hopelessThe problem of cache
  conflicts in 1024-way set associative caches is
  not very important, and the authors should focus
  on lower-associativity caches
• All grammar/writing/organizationissues go here!

10
Confidential Comments

• Only put things that are importantand would
  disclose your identity
• My group did this already in 1999 andwe
  published it in ISCA that year

11
Scoring (numbers)

• Compare to other papers in the same
  journal/conference/class
• So, compare to other papers in this class
• Weak accept
• Similar in quality to lower half of accepted
  papers
• Strong accept
• Similar to upper half of accepted papers
• Neutral/Undecided
• Would be in bottom 10 of accepted papers
• Strong reject
• You would argue against accepting it

12
Always the big picture

• Dont focus on minor issues
• Very easy to find lots of these
• Focus on make-or-break issues
• If X wasnt there, it would be a much worse paper
• If X done right, it would be a much better paper
• Always consider the space
• Dont ask for it if you dont think something
  else should be removed or made shorter