Systematic Reviews

What Is A Systematic Review?

Guide Objectives

This guide will help you

• Describe the process of conducting a systematic review
• Identify helpful tools/resources to use while planning and completing your systematic review
• Determine if a systematic review is the correct method to answer your research question

What is a Systematic Review?

According to Cochrane, a systematic review is "A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review."

A systematic review contains the following components:

• Identification of a research question
• Creation of a plan
• Literature search
• Eligibility criteria to screen studies for inclusion in the review
• Appraisal and collection of data about each included study
• Interpreting data
• Reported findings
• Updates of review over time

What makes a systematic review different from other reviews? 

• Literature Reviews (Health Sciences) - step-by-step process of conducting a literature review.
• Cook et. al. (1997) describes the differences between a narrative and systematic reviews.
• This table breaks out the features of four types of reviews. 
• What Review is Right for You? - Five questions to identify the type of review that will best answer your research question

Creating a Plan

A systematic review plan contains the following elements:

Team Members: A project leader to coordinate and write the final report; librarians to develop comprehensive search strategies and identify appropriate databases; reviewers to screen abstracts and read the full text of articles; a statistician who can assist with data analysis; and individuals with expertise in systematic review methods and the clinical area of interest.

A clearly defined question: Clarify the key question(s) of your systematic review and the rationale for each question. Use the PICO framework to identify key concepts of the question.  

Check to see if a high-quality systematic review addressing your question is in progress or has been completed recently.

Eligibility criteria: Determine criteria for including and excluding studies from your review. Typical criteria: population characteristics, intervention details, language, study type, publication type.

Register your systematic review protocol on Prospero. 

Create a timeline: typical timeline is 12-18 months (see example detailed timeline)

Searching the Literature

The search must exhaustively encompass multiple databases and other literature sources. You can consult the library's research guides for your discipline to find the key databases.

1. Identify key terms in the research question, listing all synonyms for each term.
2. Identify all relevant databases.
3. Construct a search for each database that combines all relevant terms and subject headings for each concept using Boolean logic. 
   1. Record search strategies, date searched, and results for each search (see spreadsheet below)
   2. Amend searches as needed, including terminology specific to database or discipline
   3. Save final searches in each database
   4. Re-run the searches towards the end of the review to retrieve the most up-to-date results
4. Include a search of grey literature available on your topic.
5. Handsearching to find articles that may have been excluded in the electronic searching process.
6. Merge search results and remove duplicates using citation management tools.

Note that your library liaison can provide assistance with identifying databases and creating searches.

• Systematic Review Documentation Spreadsheet -
  Use this spreadsheet to keep track of your keywords, search strategies, inclusion/exclusion criteria, date searched, and results for each search.

Screening & Application of Eligibility Criteria

Use the following steps to screen your studies based on predefined eligibility criteria.  It is recommended that two or more people review each study during screening. Record the number of results at each step.

1. Screen studies based on titles and abstracts and remove irrelevant studies.
2. Screen full-text of studies for compliance with eligibility criteria. Remove irrelevant studies.
3. Correspond with investigators to make final decisions on study inclusion and proceed to data collection.

Based on Cochrane Handbook, Chapter 7.2.3

The PRISMA Flow Diagram depicts the number of records identified, included and excluded, and the reasons for exclusions during the screening process (PRISMA Flow Diagram Generator).

Consider using software to help organize and streamline the screening process.

• VCU Libraries currently has a three-year pilot subscription to Covidence systematic review management software.

Data Collection

Using a data collection form, extract all relevant data from each of the included studies. The specific elements collected will depend on both the question asked by the review and by the types of studies included.

Common fields included on a data extraction form include:

• study design/method
• setting and population information
• intervention(s)
• outcomes and measures
• study results
• author and review conclusions

Pilot your data collection form to make sure that all of the important data is collected and that all fields on the form are clear.

Cochrane checklist of items to consider in data collection (Cochrane Handbook 5.4, Part 2, Chapter 5)

Assessing Quality or Risk of Bias

Assessing the methodological quality of studies included in a systematic review is a critical step. This allows you to assess the risk of bias in each study. Whether you plan to asses randomized controlled trials, observational or mixed methods studies, or qualitative research, there are a variety of tools available to simplify this process. Some of the most common tools are included below.

• Cochrane Collaboration Risk of Bias assessment tool for randomized trials version 2 (RoB2)
• Cochrane Risk of Bias assessment tool for non-randomized studies (ROBINS-I)
• Newcastle - Ottawa Scale for assessing the quality of non-randomized studies (Deeks, 2003)
• SYRCLE Risk of Bias tool for animal studies
• Joanna Briggs Institute Critical Appraisal Tools
• Critical Appraisal Skills Programme (CASP)
• Medical Education Research Study Quality Instrument (MERSQI)

Reporting Systematic Reviews & Meta-Analyses

There are a number of reporting frameworks that ensure reproducibility, which can be found at the EQUATOR Network.

Some of the most common reporting guidelines are:

• PRISMA​ - Preferred Reporting Items for Systematic Reviews and Meta-Analyses
• MOOSE - Meta-analysis of Observational Studies in Epidemiological Studies
• RAMSES - Realist And Meta-narrative Evidence Syntheses: Evolving Standard
• JARS - APA Journal Article Reporting Standards for Meta-analysis