Research papers are technically dense
and contain so much new information and detail that it is often difficult
for a person new to that field to focus on what's really important about
the paper. To avoid getting lost in details, as you read the paper, focus
on the following key points:
- What is the study?
- What are the findings?
- Who cares?
Now let's expand on each of these points.
What Is the Study?
Your goal is to understand the purpose
of the paper. What is the key message that the authors
are getting across? Why did they write this paper? That
question will be answered differently depending on the kind of
paper* you are reading. Here are some examples:
- A "new idea" paper
- Such a paper typically presents a new concept in
system design or a new algorithm, or a new analytical model, or
a new way to build systems (an engineering paper) etc. So the goal
of such a paper is to present a new way of doing things and
demonstrate potential benefits.
Such papers typically dedicate lots of space to a detailed
description of the idea and have small evaluation sections with
preliminary experiments, as opposed to a thorough experimental evaluation.
- An "evaluation" paper
- Such a paper takes an existing idea or system and
evaluates its impact in the context of multiple usage scenarios.
So the goal of such a paper is to evaluate applicability of
existing ideas or systems to multiple environments. Such
papers typically dedicate a lot of space to the evaluation section.
They present lots of data, resulting from varying parameters of
the experiment, comparing different workloads and system configurations.
The evaluation need not be quantitative; some ideas require qualitative
- A "characterization" or "classification"
- Such a paper characterizes and classifies existing
phenomena according to some valuable criteria. The characterization
is used to better understand the behavior of the phenomenon and
its interaction with the rest of the world. For example, a popular
theme in systems research is workload characterization. Workloads
are analyzed with respect to important properties, i.e., how they
stress the CPU or network. This develops understanding of how to
build systems to suit the needs of a particular workload. The goal
of a characterization paper is to evaluate characteristics
of some phenomena according to a set of criteria and understand
implications of these characteristics for interactions with those
- An "experience" paper
- Such a paper tells the reader about an experience
with using a system. The goal is to offer insight
into system design, i.e., suggest how future versions of
the system should be designed to correct for mistakes of the past
and what system features should be retained because they worked
- A "survey" paper
- Such a paper surveys state of the art in the field.
It can survey, for example, approaches to solving a particular problem.
The goal is to understand how to direct future
research in that discipline.
You should be able to answer the "What is the study?"
question by the time you've read the abstract, and surely by the time
you've read the introduction (provided that the paper is well written).
It is a good idea to summarize the goal of the paper for yourself before
moving on to the rest of the paper.
What Are the Findings?
The key concepts here are methodology
- How did the authors approach
the objective of their study? What methods did they
use? The important questions to ask yourself are:
- What steps did the
authors follow to answer the questions they posed? Are
these steps reasonable? Would you follow the same
steps? What would be alternative approaches? Do the authors
- Was their methodology
sound? For example, if the authors set
out to evaluate performance of a wide range of applications
on CMT processors, but their benchmarks included only commercial
server applications, you can say that their methodology was
poorly suited to achieve their goal.
Methodology is usually overviewed in the introduction of the paper
and described in a greater detail in the first half of the paper (assuming
this is a well written paper).
- What were the authors' conclusions?
How did they arrive at those conclusions? The important questions
to ask here are:
- What are the study's
findings? Did you learn something new? What has the
study contributed to the scientific field?
- Has the study achieved
its objectives? This is the time when you go back to
your "What is the study?" question and match goals
with conclusions. If the authors have not achieved their goals,
do they explain why? Some papers present negative results. These
papers are extremely valuable, because they show us that our
expectations about certain phenomena were incorrect. Even though
those papers do not achieve their goals per se, they
further understanding of the field and are thus extremely valuable.
- Were you convinced
by the findings? Did the authors present enough evidence
to support their conclusions? Do you believe them? Do their
conclusions apply to scenarios they claim, or do you need to
see more data to be convinced?
Conclusions are usually described in detail at the end of the paper
and are also previewed in the abstract and introduction of the paper.
As you read the paper, it is useful to keep in your mind what the
conclusions will be - this way you know where the paper is leading
As much as it is fun to do research, we don't do research
just for fun's sake. The goal is to do something useful and valuable.
Here we are talking about the concept of motivation.
Each study should be well motivated. As you read the paper, answer
for yourself the following questions:
- Is this study imporant? Who is it important
- How does this study make the world better?
- Is this study novel or has this been done before?
The authors will discuss related work to convince you
that the study is novel.
Motivation is very important for
research papers. The authors should convince you that it is worth your
time reading their paper. Therefore, motivational arguments are strongly
expressed in the abstract and introduction and also pop up here and
there in the rest of paper.
*The "kind of paper" concept has been invented
by Margo Seltzer