How to read and critique a technical paper

1
How to read and critique a technical paper?
2
3 phases to reading

• Determine if there is anything interesting at all
  in the paper.
• Determine which portion of the paper contains the
  interesting stuff.
• Read the whole paper.

3
Is there anything interesting?

• Ideally the abstract should tell you this, but
  frequently it does not.
• Need to jump about
• Read conclusion
• Read introduction
• Look at the bibliography
• Glance at the TOC (if any)

4
Which portion contains interesting stuff?

• Typically, a paper outlines its organization at
  the end of the introduction. Use this to
  determine which portion contains the exciting
  stuff.

5
Read the whole paper

• Read with the following questions in mind
• How can I use this stuff?
• Does this really do what the author claims to do?
• What if the assumptions and choices that the
  author made are discarded (or made invalid)?

6
Context and problem statement

• What problems are the author trying to solve or
  trying to convince you of?
• Are they important problems?
• Why?
• Why not?
• What is the authors thesis?

7
Related work evaluation

• Does the author describe other work in the field?
• If so, how does this research differ from the
  other work?

8
New idea

• What new idea is the author proposing?
• Architecture
• Algorithm
• Mechanism
• Methodology
• Perspective
• Is the idea useful and practical?

9
What to evaluate?

• What need to be evaluated to confirm the
  worthiness of the new idea?
• Runtime
• Throughput
• Resource utilization
• Model validation

10
How to evaluate?

• How did the author go about conducting the
  evaluation?
• Formalize and prove theorems
• Run simulations
• Artifact design and construction
• Collect traces from existing systems

11
Was the evaluation correct and adequate?

• How was the data collection done?
• Do you agree with the analysis of data?
• Do you agree with the conclusions about the data?
• Do you have any new interpretation of the data?
• Can you suggest new ways to evaluate the data?

12
Assumptions, drawbacks and extensions

• Can you think of other aspects of the idea that
  need to be evaluated?
• Can you think of extensions or modifications to
  the idea to improve it?
• How would you evaluate your improvement?

13
Assumptions, drawbacks and extensions

• Can you apply the idea or method of evaluation to
  your own project?
• Do the authors make any assumptions that are not
  valid or realistic?
• Can you come up with a more general solution that
  does not rely on one or more of the assumptions?

14
Future work

• Does the author indicate how the work should be
  followed up on?
• Does the paper generate new ideas?
• Does the paper implicitly or explicitly provide a
  new way of doing other things or of thinking
  about problems?