How to Read and Understand Research Papers

Research papers are technically dense and contain so much new information and detail that it is often difficult for a person new to that field to focus on what's really important about the paper. To avoid getting lost in details, as you read the paper, focus on the following key points:

  • What is the study?
  • What are the findings?
  • Who cares?

Now let's expand on each of these points.

What Is the Study?

Your goal is to understand the purpose of the paper. What is the key message that the authors are getting across? Why did they write this paper? That question will be answered differently depending on the kind of paper* you are reading. Here are some examples:

  • A "new idea" paper
    • Such a paper typically presents a new concept in system design or a new algorithm, or a new analytical model, or a new way to build systems (an engineering paper) etc. So the goal of such a paper is to present a new way of doing things and demonstrate potential benefits.
      Such papers typically dedicate lots of space to a detailed description of the idea and have small evaluation sections with preliminary experiments, as opposed to a thorough experimental evaluation.
  • An "evaluation" paper
    • Such a paper takes an existing idea or system and evaluates its impact in the context of multiple usage scenarios. So the goal of such a paper is to evaluate applicability of existing ideas or systems to multiple environments. Such papers typically dedicate a lot of space to the evaluation section. They present lots of data, resulting from varying parameters of the experiment, comparing different workloads and system configurations. The evaluation need not be quantitative; some ideas require qualitative evaluation.
  • A "characterization" or "classification" paper
    • Such a paper characterizes and classifies existing phenomena according to some valuable criteria. The characterization is used to better understand the behavior of the phenomenon and its interaction with the rest of the world. For example, a popular theme in systems research is workload characterization. Workloads are analyzed with respect to important properties, i.e., how they stress the CPU or network. This develops understanding of how to build systems to suit the needs of a particular workload. The goal of a characterization paper is to evaluate characteristics of some phenomena according to a set of criteria and understand implications of these characteristics for interactions with those phenomena.
  • An "experience" paper
    • Such a paper tells the reader about an experience with using a system. The goal is to offer insight into system design, i.e., suggest how future versions of the system should be designed to correct for mistakes of the past and what system features should be retained because they worked well.
  • A "survey" paper
    • Such a paper surveys state of the art in the field. It can survey, for example, approaches to solving a particular problem. The goal is to understand how to direct future research in that discipline.


You should be able to answer the "What is the study?" question by the time you've read the abstract, and surely by the time you've read the introduction (provided that the paper is well written). It is a good idea to summarize the goal of the paper for yourself before moving on to the rest of the paper.

What Are the Findings?

The key concepts here are methodology and conclusions.

  • Methodology
    • How did the authors approach the objective of their study? What methods did they use? The important questions to ask yourself are:
      • What steps did the authors follow to answer the questions they posed? Are these steps reasonable? Would you follow the same steps? What would be alternative approaches? Do the authors discuss them?
      • Was their methodology sound? For example, if the authors set out to evaluate performance of a wide range of applications on CMT processors, but their benchmarks included only commercial server applications, you can say that their methodology was poorly suited to achieve their goal.

  • Methodology is usually overviewed in the introduction of the paper and described in a greater detail in the first half of the paper (assuming this is a well written paper).

  • Conclusions
    • What were the authors' conclusions? How did they arrive at those conclusions? The important questions to ask here are:
      • What are the study's findings? Did you learn something new? What has the study contributed to the scientific field?
      • Has the study achieved its objectives? This is the time when you go back to your "What is the study?" question and match goals with conclusions. If the authors have not achieved their goals, do they explain why? Some papers present negative results. These papers are extremely valuable, because they show us that our expectations about certain phenomena were incorrect. Even though those papers do not achieve their goals per se, they further understanding of the field and are thus extremely valuable.
      • Were you convinced by the findings? Did the authors present enough evidence to support their conclusions? Do you believe them? Do their conclusions apply to scenarios they claim, or do you need to see more data to be convinced?

  • Conclusions are usually described in detail at the end of the paper and are also previewed in the abstract and introduction of the paper. As you read the paper, it is useful to keep in your mind what the conclusions will be - this way you know where the paper is leading you.

Who Cares?

As much as it is fun to do research, we don't do research just for fun's sake. The goal is to do something useful and valuable. Here we are talking about the concept of motivation. Each study should be well motivated. As you read the paper, answer for yourself the following questions:

  • Is this study imporant? Who is it important for?
  • How does this study make the world better?
  • Is this study novel or has this been done before? The authors will discuss related work to convince you that the study is novel.

Motivation is very important for research papers. The authors should convince you that it is worth your time reading their paper. Therefore, motivational arguments are strongly expressed in the abstract and introduction and also pop up here and there in the rest of paper.

 


*The "kind of paper" concept has been invented by Margo Seltzer