Publications Part 3: Reading a Scientific Paper

I’m going to write a series of posts on the black art of scientific publications.  I’m planning a journal based exercise for one of my classes next year and need to provide some information on what journals are and why they are.  I thought the best way to help me think about this was to write some blog posts in an attempt to get the key ideas together.  These are written from the perspective of an academic chemist, and with an educated lay audience in mind.  Comments and suggestions are welcome.

So you’ve figured out your communications from your reviews, and worked out the difference between an abstract and a conclusion but its still really difficult to figure out how to read a scientific paper.  Every researcher has a different way of doing this.

Firstly, work out why you want to read the paper.  Does it have an experimental procedure in it that you want to follow?  If so, go straight to it, and then look for relevant parts of the results and discussion section that gives you more information.  You can read the rest of the paper later, but get the information you want first.  Same goes for a paper where they make a molecule you want to make.

If you’re reading papers to learn more about an area of research for a project, then you probably need to learn the vocabulary that’s used in that particular field.  Its probably easiest to look for a review article aimed at a general audience, and look up any scientific terms you are unfamiliar with.  Then, when you get to full papers, you’ll know what the terms mean and why they are used.

Sooner or later, you’re going to have to read a full paper and understand it.  Lets assume you’ve found a paper that is relevant to your project, and you know most of the specialist terms that will be used.  You can break it down into a series of questions:

  1. What are the authors trying to do?  Look for a hypothesis or statement of aims of the research.
  2. Why are they trying to do this? Why is this research important?
  3. How are the authors trying to do it? What methods are the authors going to use.  You can also make a judgement here based on your knowledge of the field – are those methods suitable, or would you do something different?
  4. Did it work?  You’ll probably have to skip to the conclusion here.  I should note that some people would say that you should go to the results section now, look at the data and draw your own conclusions as to whether it worked or not.   If you feel up to that, do it, if not, head for the conclusions.
  5. What else is there?  You need to go through the results and discussion and look for anything unexpected or interesting that the authors found.  Perhaps a certain reaction always gave an unexpected result, or an analytical technique was found to be complicated and difficult to run.
  6. Get critical.  You’ve already thought about whether the methods are appropriate, but now its time to think hard about the paper.  Do the results support the conclusions?  Do you understand why certain conclusions have been made?  If you were to design a series of experiments to test the hypothesis of the paper (or work through the aims), what would they be?  Would they be the same as the authors?  What would you do differently, and why?  This part is quite difficult, but often the part of the process where ideas for different work come.
  7. Put the paper in context.  Look up the paper on Web of Science or some other database that tracks citations (i.e. the number of papers published after your one that reference it).  Firstly, have the authors themselves published more on the topic?  Have other researchers picked up this work and used it in their own experiments?  The number of citations a paper has from other researchers (which grows over time) is a good indication of how important or useful the work is.  Are those citations mainly from researchers working in a similar area, or are they from a wider range of research fields?

3 Replies to “Publications Part 3: Reading a Scientific Paper”

  1. I think that you have achieved your objectives of clearly setting out the black arts of scientific publication, at least as far as journal articles go. Congratulations!

    What interests me is how far the principles you identify apply to other fields of science. My field of work, evidence-based medicine, has no parallel in chemistry, physics, archaeology, etc. Indeed the the very idea seems bizarre to these scientists. Can you imagine “evidence-based chemistry”?

    I think the key difference in clinical science is that in any experiment (or as we prefer to call them, clinical trial) there are a large number of unknown and uncontrollable confounding factors and sources of bias. Furthermore, trials can never be exactly replicated. There is therefore a need for secondary research, assessment of risk of bias, and systems to remove opportunities for bias.

    Secondary research systematically searches, appraises, collates, and analyses results from primary research. We call such studies systematic reviews and meta-analysis.

    Critical appraisal assesses the risk of bias in study results — I call it bullshit detection. There are many courses, textbooks, and published checklists on critical appraisal — for example

    Systems to prevent (some) sources of bias include guidelines on how trials should be reported, and guidelines on publicly registering trial protocols before the trial is started. The major medical journals will no longer publish research reports that do not follow these guidelines.

  2. There are two other important difference between the hard sciences and the soft sciences that I forgot to mention, and which are important for critical appraisal.

    The signal to noise ratio is often very low in the soft sciences. Large studies are required for reliable signal detection, but they are expensive, so corners are (unconsciously) cut by doing smaller studies and looking for more signals ie outcome measures. With smaller studies the confidence intervals around measurements of outcomes are wider; and with more outcome measurements the chances of finding false positive results increase.

    There are many untrained (to put it politely) people doing and publishing “research”, because the entry barriers to doing so are low. Much of the research into complementary and alternative medicine is very poor quality (high risk of biased results), but is used to to provide the appearance of evidence when this is useful for marketing.

  3. Thanks for the comments. I learned a little (emphasis on little) regarding clinical trials during my postdoc, and a little on the idea of controls etc for in vitro drug efficacy studies. It was an eye opener because it was completely different to the kind of research I’m used to. Simple example – chemists gladly give the yield of a reaction in a publication having done the reaction once, these new-to-me studies required repetition and statistics. Designing a good study to test a seemingly simple effect was very complex.

    I think the public regards many of the claims made by complementary/alternative medicines to be similar to the claims made for beauty or skin care products – they can be justified on the basis of a study but the actual study is of poor or limited quality. At least for beauty care products they have to state the number of individuals in the study used to make the claim (i.e. we gave 200 women a free pot of our skin cream then asked them how it made them feel after 2 weeks of use, 86 % said it improved {insert desired attribute here} mainly because we’ve trained them to like these sessions and give them travel expenses, free tea and nice biscuits).

    OK, I’m going to take my cynicism and turn it back to my work now 🙂

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.