Skip to Main Content

Systematic Reviews

A guide to conducting systematic reviews.

Study Inclusion/Exclusion

The decision criteria and process must be decided before the review begins. The review of the search results is usually conducted in two phases:
  • Review title/abstract
  • Review full-text

At least two independent reviewers are needed, with a third person available to resolve conflicts.

Keep a log of excluded studies with reasons for exclusion. Simple Excel spreadsheets can be designed to track the process. 

Critical Appraisal Worksheets

Critical Appraisal Worksheets:

  • Worksheets from the Centre for Evidence Based Medicine in Oxford
  • Worksheets from the Centre for Evidence Based Medicine in Toronto

Critical Appraisal: CRAP or CRAAP Test

The CRAAP test is a method for evaluating research based on the following criteria

  • When was the information published or posted?
  • Has the information been revised or updated?
  • Is the information current or too out-of-date for my topic?
  • Are all the links functional or are there dead links?*

Developed by Sarah Blakeslee and her team of librarians at California State University, Chico (CSU Chico) to check the reliability of sources across academic disciplines .

  • Does the information relate to my topic or answer my question?
  • Who is the intended audience?
  • Is the information at an appropriate level (i.e. not too simple or advanced) for my needs?
  • Did I look at a variety of sources before deciding to use this one?
  • Would I be comfortable using this source for my college research paper?
  • Who is the author/publisher/source/sponsor?
  • Are the author's credentials or organizational affiliations given?
  • What are the author's credentials or organizational affiliations?
  • What are the author's qualifications to write on the topic?
  • Is there contact information, such as a publisher or e-mail address?
  • Does the URL reveal anything about the author or source? Examples: .com .edu .gov .org .net*
  • Where does the information come from?
  • Is the information supported by evidence?
  • Has the information been reviewed by anyone else?
  • Can I verify any of the information in another source or from personal knowledge?
  • Does the language or tone seem biased? Or is it free of emotion?
  • Are there spelling, grammar, typographical, or other errors?
  • What is the purpose of the information? to inform? teach? sell? entertain? persuade?
  • Do the authors/sponsors make their intentions or purpose clear?
  • Is the information fact? opinion? propaganda?
  • Does the point of view appear objective and impartial?
  • Are there political, ideological, cultural, religious, institutional, or personal biases?

Critical Appraisal: AACODS Framework

  • Can you identify who is responsible for the intellectual content?
  • Is the work associated with a reputable organization?
  • Is the organization reputable? (e.g. W.H.O.)?
  • Can you easily identify the producing source?
  • Is he organization an authority in the field?
  • Does the item have a detailed reference list or bibliography?
  • Does the item have a clearly stated aim? If so, is it met?
  • Does it have a stated methodology?
  • Has it been peer-reviewed?
  • Has it been edited by a reputable authority?
  • The item refers to a particular population group, or excludes certain types of publications.
  • Are any limits clearly stated?
  • Opinion, expert or otherwise, is still opinion: is the author’s standpoint clear?
  • Does the work seem to be balanced in presentation?
  • Is there a conflicts of interest?
  • Does the item have a clearly stated date related to the content? 
  • For the item to inform your research, it needs to have a date that confirms relevance
  • No easily visible date is a strong concern
  • Have key contemporary material been included in the bibliography?
  • Is the item meaningful?
  • Does it enrich or negate a current position?
  • Is the publication relevant?

Covidence Screening and Extracting Tool

Covidence is a web-based tool that helps you with the systematic review process by enabling researchers to work with a team to screen title/abstracts and full text for inclusion in your review, as well as extract data from included studies.

The library has a subscription that allows for unlimited reviews once researchers join institutional account.

Other Screening and Extraction Tools

Data Extraction & Quality Assessment

For studies selected for inclusion:
  • What data will be extracted?
  • What software will be used?
  • What are the procedures for extraction
    • Number of reviewers
    • Dispute resolution process
  • Will authors be contacted for further data?
  • What is the translation process?

What criteria will be used for the quality assessment of the included studies? What process will be followed for the quality assessment?

  • Number of reviewers
  • Dispute resolution

How will the quality assessment be used in analysis?

Data Analysis and Synthesis

Things to consider:
  • Meta-analysis and/or narrative analysis?
  • Measures of heterogeneity
  • Outcomes of interest
  • Effect measures to be used
  • Subgroup analysis plans
  • Publication bias analysis

For qualitative data:

  • ATLAS.ti to code and analyze blocks of text/audio files (fee based)