For the first time, this semester, I have found myself in the position where I have to systematically read research papers rather than textbooks to stay abreast of my coursework. As I build up towards working on my Master’s thesis and eventually a career in research, I know this is a skill I will need to become adept at very soon.
In order to do that, I made three changes that have made me a better paper-reader (Is that the term?)
I started using a reference manager (Mendeley) to keep track of the papers I read and the context in which I read them. When I eventually need to build my list of references, I’ll already have them in categories according to how important the paper was to my work and I can quickly export the reference list.
I started making notes for each of the papers I read, so that when I needed to refer to the experimental procedure or compare results with papers I’ve read before, I don’t need to comb through introductions and paragraphs of text to find what I’m looking for.
I started writing about the concepts I learnt during the process of reading these papers. This served three purposes:
It gave me further incentive to read the paper thoroughly as I knew I would be writing about it soon .
It served as a version of the Feynman Technique that let me drill down to the most important aspects of the paper, solidifying the concepts in my own head.
The catalogue of notes could serve as a independent reference in itself for me or anyone else in the future who needs it.
But just this wasn’t enough. My process was still taking me almost a week to read each paper, and that just wasn’t sustainable in a world where I need to be able to read, assimilate and absorb a paper in a matter of days, if not hours, simply due to the volume of research that I need to be aware of.
So I went looking for tips on how to read papers, and ironically, I found... a paper.
Written by Dr. Srinivasan Keshav, a computer scientist at the University of Cambridge, during his time as a professor at the University of Waterloo in 2016, the paper describes a three pass approach to the process of reading a paper that I found extremely useful.
So in keeping with my process of writing what I read about, I wrote this post summarizing the contents of the paper, sprinkled with my own perspectives and insights.
How to Read a Paper
In keeping with the spirit of the paper, I’ve organized my notes into the 2 minute version, the 5 minute version, and the 10 minute version.
The 2 Minute Version: What is the Three Pass Approach?
Pass 1: 5-10 Minutes Purpose: To get a general overview of the paper, decide if you want to go for further passes
Read the title, abstract and introduction, conclusions in detail
Look at the section headings and sub-sections, ignore content
Glance at references to look for ones you’ve read
Pass 2: 1 Hour Purpose: Be able to summarize the main thrust of the paper with supporting evidence
Ignore details like proofs but read the contents of the paper - Make notes in this pass
Look carefully at figures, graphs, diagrams to evaluate quality of the work
Mark unread references for further reading
Pass 3: Beginner - 4 to 5 Hours, Experienced - 1 Hour Purpose: Get to know the work inside out
Essentially recreate the paper by implementing the concepts yourself, thinking about how you would present it and comparing the results
Question all assumptions and choices and understand the reasons for them
Note ideas for future work
The 5 Minute Version: How does three passes help?
(Read the 2 min version first)
The three pass approach captures goals a researcher has in reading a paper and provides multiple points of decision making about whether to continue based on the purpose of reading the particular paper.
The first pass provides an overview of the paper and what it is about. This helps you evaluate if it is clearly written, contains accurate information and whether it’s contents are relevant to the research you are doing. This pass will save you a lot of time reading papers that may not be very useful to your study.
The second pass lets you understand the contents of the paper, such as the experimental procedure, the evaluation of results or the concepts used in the paper. This is where most people might stop if the paper isn’t crucial to their study but still relevant. It gives enough information to understand the work and the results without going into too much detail about the specifics, which aren’t always necessary.
The third pass is for papers that are absolutely crucial to the study you are performing. Re-creating the paper lets you fully understand the assumptions, the procedure and the choices made by the authors and inform you about the direction you should take in your own study. This pass might also expose you to new tools, methods or presentation techniques that you can use in your own work. Comparing results by reimplementing the work helps you understand the challenges that may not be fully represented in the paper and gives you a deeper understanding of the area.
The 10 Minute Version: More about why breaking it into three passes is important
(Read the 2 and 5 min versions first)
First Pass
Knowing how to do a first pass can also help with writing a paper when the time comes. Most reviewers only perform a single pass over a paper, and if they do not get the gist after the one pass, they may reject it. Similarly, if a reader of your paper does not understand it after the first pass, the paper may never be read or cited.
Second Pass
In many a case, you may not fully understand a paper after going through a second pass. At this point, you have the opportunity to decide whether to move forward and what to do depending on why you didn’t understand the paper. You may may not have understood it due to a few reasons:
1. The subject matter is unfamiliar to you, or it uses techniques that you don’t understand What to do:
If the paper is important to your study or career, go through the references and look for ones that will help you understand the subject matter and the background better.
If it isn’t particularly important, it may be worth spending your time elsewhere. You don’t have to understand everything you read.
2. The paper is poorly written or has incomplete information What to do:
If the contents of the paper are important, look for a different paper that describes similar contents and read that
If the contents aren’t particularly important, set the paper aside. It’s not worth spending time and effort on untangling something that won’t be ultimately helpful to you
3. It’s been a long day and you’re tired I loved that Dr. Keshav added this in, because all too often, we forget that researchers are humans and are subject to ebbs and flows of focus and mental agility. I have been particularly guilty of believing the scientific method shines through no matter how tired you are, and going step by step is enough to beat tiredness. It’s not. Learning needs creativity to make connections between abstract ideas and creativity needs a fresh mind. What to do:
Call it a day and come back to the paper later, maybe after going through some background material that will help ease the process.
4. Two passes wasn’t enough Sometimes the material is just so dense and complex that two passes aren’t enough. What to do:
Make sure it’s not one of the other reasons, which would make persevering a waste of time and energy
If it’s worth persevering, go through a few more second passes or move on to the third pass.
Third Pass
This is what I thought reading a paper was all about. Go through every single line, read, assimilate, absorb and repeat. But there’s a reason there’s two passes that should happen before it. This level of detail is only required for a handful of papers where knowing everything about the paper is important. At the end of this pass, you should be able to reconstruct the entire structure of the paper from memory, as well as be able to identify its strong and weak points. In particular, you should be able to pinpoint implicit assumptions, missing citations to relevant work, and potential issues with experimental or analytical techniques.
This is the pass that lets you see potential for future work, building on existing work, as you begin to see assumptions that can be challenged or gaps that can be filled by your own work.
Reimplementing the paper also lets you see the assumptions and choices made that may not be explicitly detailed in the paper, that affects the outcomes.
Literature Survey
The last thing Dr. Keshav talks about in his paper is doing a literature survey. His description is concise enough that I decided to just include it here verbatim:
Paper reading skills are put to the test in doing a literature survey. This will require you to read tens of papers, perhaps in an unfamiliar field. What papers should you read? Here is how you can use the three-pass approach to help.
First, use an academic search engine such as Google Scholar or CiteSeer and some well-chosen keywords to find three to five recent papers in the area. Do one pass on each paper to get a sense of the work, then read their related work sections. You will find a thumbnail summary of the recent work, and perhaps, if you are lucky, a pointer to a recent survey paper. If you can find such a survey, you are done.
Read the survey, congratulating yourself on your good luck.
Otherwise, in the second step, find shared citations and repeated author names in the bibliography. These are the key papers and researchers in that area. Download the key papers and set them aside. Then go to the websites of the key researchers and see where they’ve published recently.
That will help you identify the top conferences in that field because the best researchers usually publish in the top conferences.
The third step is to go to the website for these top conferences and look through their recent proceedings. A quick scan will usually identify recent high-quality related work. These papers, along with the ones you set aside earlier, constitute the first version of your survey. Make two passes through these papers. If they all cite a key paper that you did not find earlier, obtain and read it, iterating as necessary.