In this module, we will go over the different types of academic reviews and a quick overview of the steps involved in conducting them. If you are unsure about which review to choose for your own work this module will help you understand the difference between each review and help you make an informed decision.

If you know which review you will be conducting feel free to skip to relevant sections that will answer any questions you might have.

Both literature and systematic reviews are aimed at assembling, critically evaluating and reviewing existing research on a central topic or research question. Some differences between them include the method for determining what research to include or exclude, the extent or scope of the review, and the duration of time required to complete the process. To help you determine which review is most appropriate, please see Table 1.1 below for a detailed explanation of each as well as the differences between each type of review.

Adapted from Kysh, Lynn (2013): Difference between a systematic review and a literature review. Figshare (Poster), https://doi.org/10.6084/m9.figshare.766364.v1. Licensed under Creative Commons CC BY 4.0 /order and some words changed.

A literature review is an integrative summary of published research on a specific topic. The literature review seeks to synthesize what is already known about the topic, and sometimes, explicitly state what is not known, or not well understood.

The following are the key characteristics of a literature review:

• It contains a clear research question/problem that it is trying to answer.
• It is not a list describing or summarizing one piece of literature after another.
• It is organized into sections that present themes or identify trends, including relevant theory related to the research question.

Why Are Literature Reviews Important?

Literature reviews help authors and readers build their knowledge about a specific topic by synthesizing research on the subject. Literature reviews use published research to provide context to the topic and may expose debates within the field, gaps in the research, or provide a summary and analysis of research to date on the topic.

All reviews follow a familiar process:

1. Find and examine existing literature reviews.
2. Formulate a research question.
3. Search for sources.
4. Assess the quality of your results and select your sources.
5. Synthesize the important information from your sources.
6. Analyze what you’ve found.

Find and Examine Existing Literature Reviews

Prior to starting your own research, you will want to look at existing literature reviews – this is especially important so that you don’t duplicate existing work. It can also be helpful to look at the approaches taken for literature reviews similar to your own topic or discipline.  Below are some examples of locating existing reviews.

Formulate a Research Question

In general, your research question will tackle the problem you are trying to address by conducting the review. Since constructing a research question can be an in-depth process,  we go over it in more detail in Module 2: Formulating a Research Question and Searching for Sources.

Conduct Your Review Using the SALSA Framework

Once you have a research question, there are four stages you can follow when conducting your chosen review. These are known as the SALSA Framework: search, appraisal, synthesis and analysis.

What Are Systematic Reviews?

A systematic review is also known as evidence synthesis because it brings together information from a range of sources to answer a specific research question. It differs from a traditional literature review, in that it aims to synthesize and analyze the research in an unbiased, rigorous and systematic way so that it can be used to support evidence-based practice.

Characteristics  

• The scope of the review is established in advance (including the research question and pre-defined eligibility criteria).
• A systematic search is conducted in order to identify all studies/resources that would meet the eligibility criteria.
• The methodology used to search, assess, analyze and synthesize studies/resources is explicit and reproducible.
• The review assesses the validity of the studies/resources for a risk of bias .
• The review uses explicit methods for extracting and synthesizing study findings (qualitative or quantitative).

Types of Systematic Reviews

There are different types of reviews that involve evidence synthesis and a systematic review is the most well known version. Other examples include a rapid review or a scoping review.  We define the different types in more detail below.

Teams and Time Considerations

A review like a systematic review or a meta-analysis can take at least a year to complete and is usually conducted by a team.  If your review is for a class assignment, you can still conduct a systematic review without a team or a year to complete it.  If your aim is to eventually publish your review, keep in mind that one of the main goals of a systematic review is to try and eliminate potential bias, and working independently can be viewed negatively.  If you must work independently, you should identify this limitation when writing your review.

Systematic Reviews and Bias

Eliminating bias as much as possible is one of the key characteristics of systematic reviews. By bias, we mean that some type of systematic error has occurred during the review stage that leads to the acceptance of outcomes and conclusions of a study. This can result in the possibility of unfair or misleading information within the reviews. Bias is potentially introduced at any stage of the research process, from formulating your research question to choosing which sources to include.

In order to reduce bias in your review, you will need to undertake a quality assessment throughout the review process. Your protocol (see next section) and your screening process will  help you reduce your bias. Another way to reduce bias is to work in a team setting, and this is why some reviews require more than one person.  We will cover how to check for bias when screening sources in Module 3 Organizing, Managing and Screening Sources. 

Why Are Systematic Reviews Important?

A systematic review can generally give us the most dependable answer to a specific research question, and it can identify gaps in our knowledge that require further research. It also communicates the strength of the available evidence and the quality of included studies. This indicates how much confidence practitioners, service users, managers, policy makers, and the popular media should have in the results (Gough & Richardson, 2018).

What Type of Systematic Review is Right for You?

Booth et. al (2016) suggest that your choice of review methods should be determined by five main considerations captured by the acronym, TREAD

These and other factors may determine what kind of review is most appropriate to answer your research question.

The original version of this chapter contained H5P content. You may want to remove or replace this element.

This section is a quick summary of the main steps involved in conducting systematic reviews. By the end of this section you should have a better idea of the time and resources needed to conduct a successful review.

All reviews follow a familiar process as seen in Figure 1.1 below.

An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=39#h5p-4

Find Existing Systematic Reviews

Prior to starting your own research, you will want to look at existing systematic reviews – this is especially important so that you don’t duplicate existing work. It can also be helpful to look at the approaches taken for systematic reviews similar to your own topic or discipline.  You can find existing systematic reviews through a number of ways:

1. Search published journal articles. Systematic reviews can be published as journal articles. To identify them, add “systematic review” as an additional search term in databases, or look for publication type limits, if available. Here’s an example of a “systematic review” published as a journal article.
2. Search “Cochrane Database of Systematic Reviews” through your library. This database includes the full text of the regularly updated systematic reviews of the effects of healthcare prepared by The Cochrane Collaboration.
3. Search different protocol registries.  For example, PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome. PROSPERO aims to provide a comprehensive listing of systematic reviews registered at inception to help avoid duplication and reduce opportunity for reporting bias by enabling comparison of completed reviews with what was planned in the protocol.
4. Search the Campbell Collaboration. The Campbell Collaboration is an international network which publishes high quality systematic reviews of social and economic interventions around the world.

Assembling Your Research Team

If you are conducting a systematic review that requires a team these are the typical roles involved:

• Reviewers – You may need at least two reviewers working independently to screen abstracts, with a potential third as a tie-breaker
• Subject matter experts – Subject matter experts can clarify issues related to the topic,
• Statistician – A statistician can help with data analysis
• Project leader – A project leader can coordinate and write the final report
• Librarians – Librarian(s) can develop comprehensive search strategies and identify appropriate databases

Formulate Your Research Question

In general, your research question will tackle the problem you are trying to address by conducting the review. Since constructing a research question can be an in-depth process,  we go over it in more detail in Module 2: Formulating a Research Question and Searching for Sources.

Create a Review Protocol

Reviews like a systematic review require a protocol, which is essentially a planning document that indicates how your review will be carried out. Here is a sample protocol template from the Evidence Synthesis Coordinator at the Maritimes Strategy for Patient-Oriented Research (SPOR) Support Unit. This basic form includes all the relevant information needed for a simple protocol.

You may wish to register your protocol to avoid the duplication of work and to reduce the potential for bias by enabling a comparison between what was stated in the protocol to the completed review. It is also a way to share your current research interests with the research community at large, and help build your research profile.

Conducting Your Review Using the SALSA Framework

Once you have a research question, there are four stages you can follow when conducting your chosen review. These are known as the SALSA Framework: search, appraisal, synthesis and analysis.

Wait, What Happened to the “L” in SALSA?

Did you notice the missing L? We did too! The authors, Grant and Booth (2009) created  a simple analytical framework for conducting reviews: Search, Appraisal, Synthesis and Analysis. SASA, however, doesn’t make a memorable acronym, and Academics love a good acronym, so they derived the “L” from the last letter of appraisal:  Search, AppraisaL, Synthesis and Analysis (SALSA).

PRISMA: The Systematic Review Checklist

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA is the recognized standard for reporting evidence in systematic reviews and meta-analyses. The standards are endorsed by organizations and journals in the health sciences. It allows other researchers to assess strengths and weaknesses of the review and assists with future replication of the review methods. The 2020 PRISMA statement consists of a 27-item checklist and a 4-phase flow diagram.

Conducting a review requires good time management. Feel free to use the To Do List: Research Stage to keep track of your timeline and tasks for the research stage of your review.

For more information please see Workbook Instructions in the Introduction Module.

We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:

Literature Reviews

• A literature review is a qualitative integrative summary of published research on a specific topic.
  • It is not a list describing or summarizing one piece of literature after another.
  • It is organized into sections that present themes or identify trends, including relevant. theory related to the research question.
• One way to conduct reviews is with the SALSA Framework: search, appraisal, synthesis and analysis.

Systematic Reviews

• A systematic review brings together information from a range of sources to answer a specific research question. It differs from a traditional literature review, in that it aims to synthesize and analyze the research in an unbiased, rigorous and systematic way so that it can be used to support evidence-based practice.
• Some reviews need to be conducted in a team and can take up to 1 year to complete.
• Systematic reviews usually have a research protocol
• One way to conduct reviews is with the SALSA Framework: search, appraisal, synthesis and analysis.
• PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyse

Literature Reviews

Websites 

• Guide to Literature Reviews (McMaster University Library)
• Writing a Literature Review (OWL, Purdue University)

Articles and Books

Aveyard, H. (2019). Doing a literature review in health and social care: A practical guide (Fourth ed.) Open University Press, McGraw-Hill Education.

Jesson, J., Matheson, L., & Lacey, F. M. (2011). Doing your literature review: Traditional and systematic techniques. SAGE.

Ridley, D., Dr. (2008). The literature review: A step-by-step guide for students. SAGE.

Systematic Reviews

Helpful Websites 

• Systematic Reviews (Toronto Metropolitan University)
• Cochrane Database of Systematic Reviews
• Systematic Review Guide (Unity Health Toronto)

Registering Your Systematic Review

• PROSPERO
• Campbell Collaboration
• PRISMA 2020 Checklist

Articles and Books 

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. doi:10.1080/1364557032000119616

Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. Sage

SAGE.

Ganann, R., Ciliska, D., & Thomas, H. (2010). Expediting systematic reviews: Methods and implications of rapid reviews. Implementation Science, 5(1), 56-56. doi:10.1186/1748-5908-5-56

Glass, G. (1976). Primary, Secondary, and Meta-Analysis of Research. Educational Researcher, 5(10), 3-8. doi:10.2307/1174772

Gough, D., & Richardson, M. (2018). Systematic reviews. In Advanced research methods for applied psychology (pp. 63-75). Routledge.

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & Libraries Journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). (2021, February). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. www.training.cochrane.org/handbook.

In this module we go over the framework for searching the literature. Keep in mind that searching is a non-linear and potentially iterative process; as such you may need to review and revise earlier steps at any point.

The steps to searching the literature are:

1. Formulating a research question
2. Identify search terms (keywords)
3. Brainstorm more search terms
4. Construct a search strategy
5. Select and Search Databases
6. Manage your search strategies and results (Covered in Module 3: Organizing, Managing and Screening Sources)

We will show you how to use special techniques like forward and backward citation searching to find more sources. Finally, if you are conducting a systematic review you should check out our special section on how to document your search strategy and results.

Need a Refresher on Basic Searching Skills?

Check out these guides:

• Searching for Sources (Research Skills Tutorial). Toronto Metropolitan University Library
• Library Research Skills Tutorial. McMaster University Library

As noted in Module 1: Types of Reviews, conducting a “pre-search” is a crucial first step in devising the research question. A well-formulated research question informs the research process. It can focus your information needs (i.e. identify inclusion and exclusion criteria), help to identify key search concepts, and guide you in the direction of relevant resources.

Types of Research Questions

There are two general types of research questions: quantitative and qualitative.

Quantitative Research Questions

Types of quantitative questions can be categorized as explanatory (i.e., relationship-based), descriptive, or comparative.

• Explanatory questions aim to discover cause-and-effect relationships by comparing two or more variables, individuals or groups based on differing outcomes.
• Descriptive questions will often quantify a single variable but may include multiple variables  within a question. They typically ask for measurements, and can begin with: “how much”, “what percentage”, “how frequently”, or “how many”.
• Comparative questions are designed to identify the “difference between” a dependent variable and two or more groups. These questions tend to begin with “what is the difference” or “what are the differences”.

Qualitative Research Questions

Qualitative questions aim to discover meaning or gain an understanding of a phenomenon. They ask questions that cannot be measured with specific numbers and statistics. Qualitative research questions often contain words like "lived experience", “personal experience”, “understanding”, “meaning”, and “stories”.

A Comparison of Quantitative and Qualitative Questions

So how do quantitative and qualitative research questions differ when you are conducting a search? In Table 2.1 below, we provide some examples of research topics. Each topic can either be used for a quantitative or qualitative research question. For each question, the category of research question is clarified.

Formulas

Using a formula is another way to construct a research question and is recommended if you are conducting a systematic review. There are various formulas you can use to craft your question, see Table 2.2 below.

These frameworks aid in identifying the important parts (i.e., concepts) that can be used in formulating a research question. Below are two examples using the PS and PEO frameworks.

Is My Question Too Narrow or Too Broad?

When trying to settle on a research question, ensure you are choosing something of interest, but not so narrow that you are unlikely to uncover any information, but not too broad so that you are overwhelmed with content and don’t know where to start. Below are examples of too broad and too narrow questions and an explanation to help understand why.

A pre-search will assist you with this step. If you need a refresher on conducting a pre-search, see Module 1: Types of Reviews. Once you have a good understanding of the existing literature, you will be able to confidently formulate your research question.

Identifying keywords, (also known as search terms) is important for effective literature searching. ​Your search terms are terms that will appear somewhere within the resource (e.g. title, abstract, or author keywords).

You can identify search terms from your research question by highlighting, underlining or circling the main ideas that must appear in the article. Your search concepts are the most important words in your research question.

Keywords are easier to identify once the research question has been formulated using one of the stated frameworks (e.g. PICO or PS) listed in Table 2.2 of Formulating a Research Question.

Below is an example of how to use the PICO(T) framework to identify search concepts for a specific research question.

Your search strategy can contain your main keywords, similar or related keywords and controlled terms (also called subject headings).

Searching by a keyword will retrieve resources where the author(s) used that specific term. For this specific reason, you should also brainstorm similar or related keywords to incorporate into your search.

Methods for Identifying More Keywords

You can identify more keywords in multiple ways. Below are two methods.

Method 1: Use a Concept Model or Map

To use this method:

1. Write your research topic or question, along with any ideas and concepts associated with it on a blank sheet of paper.
2. Use themes to group your ideas, and connect related concepts using lines.
3. Remember to include the following:
   • Synonyms
   • Acronyms
   • Brand names and generic names
   • Variation in spelling (e.g. “paediatric” or “pediatric”)

Please see Figure 2.1 below for an example where your research question is “How effective is cognitive behavioral therapy in improving mild-to-moderate depression in adolescents?”

Method 2: Use Your Main Keywords in a Database

To use this method:

1. Locate an article on your topic.
2. Scan the title, abstract, and author keywords to identify more keywords (see Figure 2.2 below) to use in your search.

Controlled vocabularies are a standardized set of terms used to describe the content of a resource in a database. This is known as indexing. Using controlled vocabulary terms will usually generate fewer and more relevant results. However, you must know the exact term/vocabulary to use. An example is provided below.

When to Use Controlled Vocabularies

The following are situations in which you would use controlled vocabularies:

1. If your keyword searching is giving you too many or too few results, try using controlled vocabulary terms.
2. If you are doing a comprehensive search (e.g., a systematic review), you should combine both keywords and controlled vocabulary terms to increase the number of resources retrieved.

How to Use Controlled Vocabularies

Search strategies must be customized for each database for a number of reasons, one of which is that they use different controlled vocabularies. The tutorials in Table 2.3 below cover how to use controlled vocabularies in specific databases. Review these tutorials if you’d like to learn more.

How to Find More Controlled Vocabulary

Below are the steps to follow when finding more controlled vocabularies:

1. Search for each keyword in a database separately. This will allow you to identify additional controlled vocabulary. For example, the database PubMed allows you to search for MeSH headings one at a time.
   • The CADTH Search Filters Database can also help you look up the MeSH heading across databases such as Scopus, CINAHL, PubMed, PsycInfo, MEDLINE and Embase.
2. Find an article on your topic and look at the controlled vocabulary assigned to it. For example, in PubMed, look for the heading “MeSH” (the name for the controlled vocabulary in PubMed) under the heading of “Cited by.” See Figure 2.3 below for an example.
3. In other databases, such as Social Sciences Abstracts, find an article on your topic and look for the heading “Subjects” to help identify relevant controlled vocabularies. 

The standard method of constructing a search strategy is to use Boolean logic. Boolean relies on three essential operators, AND, OR and NOT. Using these operators allows you to combine words and phrases using the words AND, OR, NOT to limit, broaden, or define your search. A good researcher should know how to do a Boolean Search.

Boolean Operators

Boolean operators connect your search concepts and search terms (i.e., controlled vocabulary and keywords) together. The three basic operators are: OR, AND, and NOT. An example is shown below, followed by a deeper dive into each of these operators.

The OR Operator

The OR operator broadens your search results by retrieving records that contain either or both of your search terms or concepts. Usually, terms within a concept are combined using the Boolean OR operator.

In the Figure 2.4 below, the database will retrieve records that contain either or both of the terms Cognitive Behavioural Therapy OR CBT.

The AND Operator

The AND operator narrows the search results by retrieving records that contain both of your search terms or concepts. Usually, different concepts are combined using the Boolean AND operator.

In the Figure 2.5 below, the database will retrieve sources that contain Cognitive Behavioural Therapy AND Depression

NOT

The NOT operator narrows the search results by eliminating a specific search term. You should use the NOT operator with caution. This is because it can eliminate relevant results that happen to mention the term you are “noting” out. In Figure 2.6 below, any articles that mention both adult and adolescent will be excluded from the result set.

Using Truncation and Wildcards

Truncation and wildcards can be applied to a keyword search to broaden your results and allow you to look for variations of words.

Truncation

You can use the truncation symbol to avoid typing out all possible variations of a word. For example, surg* will retrieve surgery, surgeries, surgeon, or surgical. The truncation symbol should be used with caution to ensure relevant words are being retrieved.

The most common symbol used is an asterisk (*). The asterisk symbol is used in the following platforms and databases:

• EBSCO (e.g., CINAHL, Business Source Complete)
• Ovid (e.g., MEDLINE, Embase, PsycInfo)
• ProQuest
• ERI,
• PubMed
• Web of Science

Wildcard Symbols

Wildcard symbols can be used to substitute for one or more characters, or any single character. This is useful when dealing with variant spelling (e.g., pediatric and paediatric). One of the most common symbols used is the question mark (?). For example, in the Ovid database, p?ediatric* will retrieve pediatric OR paediatric OR pediatrics OR paediatrics.

Databases use different wildcard symbols, so check the database Help information or Search Tips for details about which symbol to use.

When conducting a review, it is important to search multiple resources. Just searching one subject database will not ensure you have found all the essential sources on your topic.

For systematic reviews, searching multiple databases is essential for reducing bias in your results. Only using one database like PubMed means you have ignored relevant studies found in other databases like Embase.

Subject Databases

Subject databases are accessible through your academic library and contain published material belonging or pertaining to a specific discipline or multiple disciplines, e.g.,health sciences, business, social work, or law. Many subject databases are hosted on the platforms EBSCO and Proquest for example. These search platforms allow users to search and retrieve content, and may look and work similarly to each other. The type of material in different subject databases depends on the subject, but most will include academic peer-reviewed articles, newspaper and magazine articles, and reports.

To find a subject specific database, go to your library’s website and look for a research guide for your discipline. This guide will include a list of article databases for your discipline, and the descriptions below them will help you select among them. Alternatively, look for a list of databases by subject on your library website, and choose an article database which covers your subject.

If you have questions about which database to choose, contact your librarian for help.

How to Search in Subject Databases

In general, all subject databases will have a basic search box and an advanced search box. We recommend you use the advanced search box, as it provides more options to optimize your search. The advanced search box will often allow you to choose where you ​want your keywords to appear within the resource (e.g., in the title or the abstract and body or anywhere). An example of an advanced search box is found in Figure 2.7 below.

Limiters are found in all subject databases and they allow you to limit your search by date, type of source, language, and in some databases by population and methodology (see Figure 2.8 below). This is particularly useful if you are conducting a systematic review, as the limiters will help with your exclusion criteria.

Once you have chosen your subject databases, take a note if they are owned by a particular platform (i.e., EBSCO, ProQuest, Ovid, etc). Each of these platforms will have tutorials on how to conduct an advanced search. Below are video tutorials from some of the major subject database platforms.

Google Scholar

Google Scholar searches articles, theses, books, abstracts and court opinions, from academic publishers, professional societies, online repositories, universities and other web sites.

The majority of sources will be behind pay-walls and you will not be able to access them. However, you can optimize Google Scholar to search your academic library’s collection, at which point you will be able to access any item your library owns. For a list of the pros and cons of using Google Scholar please see Table 2.6.

From McMaster University, Health Science Library, Google Scholar. Licensed under Creative Commons CC BY-NC-SA 4.0.

How to Search Google Scholar

The following video demonstrates how you can search Google Scholar.

UTS Library, Google Scholar: Advanced Searching [2:53]. Licensed under a Creative Commons CC BY 4.0. Closed Captions

Grey Literature and Other Sources

Grey literature refers to any information that is not commercially published, and can include a wide range of material types, such as:

• Theses and dissertations, e.g., databases such as Proquest Dissertations or Theses Canada
• Government documents and websites
• Clinical trials registries
• Conference proceedings, e.g.,  databases such as Scopus or Web of Science include citations to conference proceedings
• Research reports

How to Search Grey Literature

Knowing where to search for grey literature can be challenging. There are a variety of suggestions in the following library guide from Simon Fraser University: Grey literature: What it is & how to find it.

A good approach is to be strategic and consider where appropriate grey sources might be located. For example, if you know of any government agencies, non-profits, professional associations, research institutes, and other organisations that may be producing content relevant to your topic, you may wish to search their website for any publications and reports. At a very minimum, looking at graduate theses and dissertations and conference proceedings ( often sites of emerging research) should be a component of your grey literature search.

Searching for grey literature will likely involve the use of a search engine such as Google, so it will be important for you to critically evaluate sources before including them in your research.  For some tips on avoiding fake news and the spread of misinformation, review the following guide from Toronto Metropolitan University Library: How to Identify Fake News.

Replicating Your Search Strategy Across Different Databases

Your search strategy will need to be “translated” for use in multiple databases and you will have to look out for the following:

• Controlled vocabulary can vary across databases, e.g., MEDLINE uses MeSH while Embase uses Emtree
• Search options, limiting options, and availability of search fields will differ between databases

Besides your standard search there are additional options to identify relevant sources including searching key journals in your field, and tracking the work of relevant authors.

Another method used by researchers is to consult the citations within an article. By consulting the citations you can find relevant titles that you might have missed in your search. The advantage of this method is that you can find sources about a subject quickly and relatively easily. The disadvantage of this method is that you are searching backward, or retrospectively, so each source you find will be older than the previous one.

This is why you need to use a forward citation searching method in combination with your backward citation search. In the forward searching method, researchers use tools within databases like Google Scholar and Web of Science to see who has recently cited your source. This way you can find new and relevant sources.

Method 1: Backward Searching

In this method, review the references used in key articles. For example, search through the article for the “References” section as in Figure 2.9 below.

Method 2: Forward Searching

In this method, see what articles have cited your key articles. For example, search through the “Cited by” section of the article as in Figure 2.10 below.

Many databases will include live links to an article’s references, as well as live links to other articles that have cited that article. The following video demonstrates how to do citation searching in Google Scholar:

CSUDH Library. Citation Tracking. Licenced under Creative Commons CC BY 4.0.  Closed Captioned

If this is your first review, it’s normal to feel overwhelmed and unsure when to stop the searching process. Afterall, you don’t want to miss anything that’s possibly relevant, so the searching process never really seems to come to an end!

Unfortunately, we don’t have a firm easy answer like “if you find 10 articles you can stop.” There is always a chance that an article might slip past your extensive searching, but if you have done the steps below, then the chances of a really important article slipping past you is pretty slim.

Tips on When to Stop Searching

1. 1. Determine that your search strategy is working. You’ll know it is working when your search results include major studies/sources that have been recommended or, after a quick analysis of a few articles, you can tell they meet your criteria.
   2. You have searched all relevant databases.
   3. You used the citation searching techniques (See previous section). When using these methods you should see studies that you’ve already found in your database search and some new ones.
   4. You decided to add in some new search terms to a database search strategy and it yielded no new sources.
   5. You decided to remove a search term and the major studies you’ve found in previous searches have disappeared on the results page. 

Your search process must be documented in enough detail to ensure that it can be reported correctly in the systematic review and reproduced for verification. During your search process you will have to keep track of:

• Each database and platform (PsycINFO, PubMed, Medline etc) searched
• Date each search was conducted
• Subject headings and keywords used (including whether terms were exploded, truncated, etc)
• Number of results retrieved for each search within each database
• Total number of records over all
• Number of duplicates identified
• Numbers pre-screening and post-screening

Documentation Templates

In Module 1 we learned that PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. The PRISMA Group has developed an extension to PRISMA to assist researchers in documenting their literature searches for systematic reviews. The checklist identifies what to document in terms of information sources and methods, search strategies, peer review, and records management. The PRISMA flow diagram demonstrates how to document your final numbers.

Complete Activity 2.1: “Create Your Own Search Strategy” in your Research Roadmap Workbook

This exercise allows you to document  your search strategy.  Don’t worry if you are still trying to figure out some of the important details about your review as you can use this exercise to document what you have in mind.

For more information please see Workbook Instructions in the Introduction Module.

We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:

• Your research question can be qualitative or quantitative.
• A formula like PICO(T) can assist you in creating your question and identifying your key search terms.
• Your search terms can include related words and the controlled vocabulary of the subject database.
• Using Boolean operators will help you put your search terms together in a coherent search strategy.
• Subject databases have limiters that will help you filter your search results.
• Use search techniques like citation searching to make sure you have found relevant sources.
• If you have a working search strategy, have searched all your identified resources and used citation searching techniques, you can stop searching and start analyzing.
• You need to document your search process.
•  In some cases, more in-depth assistance is required and you can book a research consultation with your subject librarian to review your initial search.

Constructing a Question

Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting

research. SAGE.

Boland, A., Cherry, M. G., & Dickson, R. (2017). Chapter 3 in Doing a systematic review: A student’s guide (Second ed.). SAGE.

DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education. https://scientificinquiryinsocialwork.pressbooks.com/

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617

Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. (2021, February). Chapter 2: Determining the scope of the review and the questions it will address. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-02

How to Search

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies.In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04

Sayers, A. (2008). Tips and tricks in performing a systematic review. British Journal of General Practice, 58(547), 136-136. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2151802/

Controlled Vocabularies

The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube. https://www.youtube.com/watch?v=uyF8uQY9wys

UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube https://www.youtube.com/watch?v=6QQ0MW_jXfM

CADTH Search Filters Database: https://searchfilters.cadth.ca/

Documenting Your Search With PRISMA-S (Systematic Reviews)

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: An extension to the PRISMA Statement for reporting literature searches in systematic reviews. Systematic Reviews. 2021;10(1):39. https://dx.doi.org/10.1186/s13643-020-01542-z

After you have conducted your search, you will be confronted with the dilemma of what to do with your results. It is not unheard of for searches to produce hundreds if not thousands of results to shift through. Your next steps will be to download your results, save them somewhere you can manage them and finally screen them to pick the ones you will be keeping in your review.

At this point you should work out a plan for how you will organize your results. Staying organized and managing your results will require a new set of organizational skills, plus these skills will help protect you from accidental plagiarism or bias.

In any literature or systematic review project, exporting and saving results from databases is an important step prior to selecting your final sources.  Exporting and saving results allow you to permanently store the important bibliographic information or ‘metadata’ attached to your articles – this could include the author name(s), article title, journal title, etc., the article abstract,  and any associated keywords.

Exporting

The majority of the databases that you will likely be using for your review will have options to export your references. Some important tips when exporting large amounts of records:

• Export your records as an .ris file type, which will allow you to then upload these records to citation managers or systematic review software.
• Look for options to include the article abstracts when exporting, as this will be important for the first level of screening.
• For systematic reviews: When selecting your records for export, look for options to Select All – depending on the database, you may have to export your references in batches, if there are limits on the number that you can export at a time.

Create an Account to Save Results

Many database interfaces like EBSCO, OVID and ProQuest allow users to create a personal account. With an account you can save your searches including any keywords, MeSH terms or limiters you used. This means you can return to your account at any time to re-run your search or modify your search and run the search again. See Table 3.1 below on how to set up an account for the following major database vendors (always check with your library to see which databases are available to you as a patron).

After exporting your results, you will need to store and organize them. Since you will be dealing with a large amount of sources, it’s advisable to use some sort of citation management tool.

The original version of this chapter contained H5P content. You may want to remove or replace this element.

A citation tool is especially useful If you are conducting a systematic review. As part of your review, you need to document aspects of your search like the number of sources you retrieved per database, the number of duplicates and the final number of articles you choose to review. For more information, please see the PRISMA Checklist in Module 2.

Picking a Citation Management System

When picking a tool, we suggest trying out a few in order to compare their features. You might also want to keep the following questions in mind when trying them out:

• Is the tool within my budget?
• Does it work well with the databases I use frequently?
• Is it able to organize and filter a large number of citations?
• Can it produce a bibliography in the primary style used in my discipline?
• If I’m working with a research team, does the software allow me to collaborate with my colleagues smoothly?

Simple Solution: Using Excel/Google Sheets/Numbers

If you have a small and manageable number of sources, you can use a familiar spreadsheet program like Excel, Google Sheets or Numbers to manage your citation data and look for duplicates. You can create your own columns for important citation information like Author, Title, Year, Database, etc. and even a column to enter notes. You can enter your sources manually or use the exporting function found in most databases. For example, databases like PubMed and ProQuest allow you to export the bibliographic information attached to your sources as a .CSV file. This file can be opened as a spreadsheet in Excel, Google Sheets and Numbers. See Figure 3.1 below for an example of a spreadsheet used as a citation management tool.

Free Software

There are a number of free citation management tools available for you to use. The only requirement is that you sign up for an account and download the product to your device. The three most commonly used tools are Mendeley, Zotero and EndNote Basic. If you are having trouble picking between the three, Table 3.2 below offers a quick comparison chart. For a more in depth comparison you can use the following comparison chart from the University of Toronto Libraries to help you decide.

Paid Products

There are some important reasons for considering a paid product for your choice of citation management tool. They can offer technical support if anything goes wrong, larger storage space, collaboration, duplication and other special features that might meet your needs more than a free product. The following table offers a comparison of two popular paid products available on the market. Some libraries have institutional licenses to one or more of these products, which means as long as you are a student you will have access. Check with your library to see if you have access to these products. Table 3.3 below offers a quick comparison chart of two paid products, RefWorks and EndNote.

Products Specifically for Systematic Reviews

Some citation managers may not be robust enough to handle a large evidence synthesis review or remove duplicate citations. Instead, you might need to use a tool specifically designed for systematic reviews. These tools incorporate automated features that help organize and speed up the review process.

The Systematic Review Toolbox is a searchable collection of tools you could potentially use to develop your systematic review. Below is a sample of some free and paid tools designed for systematic reviews.

Complete Activity 3.1: “Selecting Your Citation Management Tool” in your Research Roadmap Workbook.

Now that you have been presented with an in-depth list of available tools. Take a moment to document at least 2-3 tools that you are interested in testing.

For more information please see Workbook Instructions in the Introduction Module.

Screening is the process of identifying suitable sources from your literature search to be “full-text” screened and eventually included in the review. As mentioned in Module 2, your inclusion and exclusion criteria can help inform which limiters to apply to exclude irrelevant results, e.g., publication year or language.

If you are conducting a systematic review, it is recommended to screen the results outside of the databases, as there will be duplication of records. A citation software tool or systematic review tool will aid in removing duplicates. Within the citation management or systematic review tool, you can screen the titles and abstracts to determine if articles are relevant, based on inclusion and exclusion criteria that could not be captured in your search strategy. Afterwards, the full-text must be retrieved and screened to determine inclusion.

Complete Activity 3.2: “Inclusion and Exclusion List” in your Research Roadmap Workbook.

This exercise allows you to document your criteria. Don’t worry if you are still trying to figure out some of the important details about your review as you can use this exercise to document what you have in mind.

For more information please see Workbook Instructions in the Introduction Module.

For systematic reviews, it is recommended that at least two reviewers screen the results independently. This is to help resolve disagreement by encouraging consensus and reduce the risk of biased decisions. When needed, a third reviewer can aid in resolving the deadlock.

Screening the Title and Abstract

For the first screening, each member of the team will independently scan the titles and abstracts of articles, and make a decision whether to include or exclude studies. Each reviewer should be asking themselves, “Is there enough information in the title and abstract to exclude this study at this stage?” To make sure both reviewers are on the same page, they will need to ensure they have a clear and pre-defined inclusion and exclusion criteria.

Screening the Full-Text

The next step is to read the full-text of the articles selected from the first round. Again, the reviewers will do this independently and use the same inclusion and exclusion criteria. This stage involves a more rigorous and in depth look at the articles. The reason for excluding a study must be recorded for reporting purposes. Reviewers can also look at the outcome of a study and determine if it is significant enough to include.

As noted in Module 2, you will be expected to document your search process and your selection process. The PRISMA flow diagram will help you map out the number of records identified, included and excluded, and the reasons for exclusions.

Addressing Bias in Selecting Sources for Systematic Reviews

Selection bias can happen consciously or unconsciously to a reviewer when selecting sources. Bias can appear in how and why we  make decisions on the potential eligibility or ineligibility of a study. This is why using a team approach is one method to reduce bias in selecting sources for systematic reviews. Another is to make sure each reviewer has a clear understanding of the key concepts, or inclusion and exclusion criteria. For example, has the team checked to ensure everyone has the same understanding of the key concepts in their research question and what to include and exclude? This way, everyone goes into selecting with the same understanding.

Complete Activity 3.3: “Check for Team Understanding” in your Research Roadmap Workbook.

Are there any search terms or items on your inclusion/exclusion list that might have multiple interpretations? Write them down with your own interpretation and have your team members do the same activity. Compare your interpretations to check for team understanding.

For more information please see Workbook Instructions in the Introduction Module.

Plagiarism happens when you claim the words, ideas, or data of another person as if it is your own, without appropriate referencing. Plagiarism can range from copying an entire paper to changing only a few words and not citing the original source. It’s important to familiarize yourself with what constitutes plagiarism. One way you can do this is to consult your institution’s academic integrity policy so you know how to avoid a charge of plagiarism.

Citation Styles

When it comes time to write your review, you will need to demonstrate that you understand when and how to cite your sources. Keeping them organized will help you with this task. By using a citation management tool, you will be able to access your sources in one place and pull the citation information needed to cite them correctly. Before sitting down to write your review, make sure you familiarize yourself with the rules of your citation style. While it is true that some citation management tools will offer you a plug-in for your word processor that seemingly does all the work, it can make mistakes. Getting to know the rules yourself will save you a lot of time and reduce the risk of plagiarism. Table 3.4 below provides a curated list of citation style resources to help you familiarize yourself with the rules.

We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:

• Get to know the exporting features of your chosen databases and sign up for an account with each database. This will not only save your results and search history, it will save you time if you have to re-do your search.
• Taking some time to pick a citation management tool that is right for you is an essential step in conducting a review. The right tool will help you maintain the integrity of your review and save you time.
• Screening your citations and abstracts against your inclusion and exclusion criteria is the first step in identifying suitable studies.
• If you are conducting a systematic review, it is important to reduce selection bias by having a team based approach to screening.
• Famarizlize yourself with the academic integrity policies of your institution to better understand how plagiarism is defined and how to avoid it.
• Commit to learning the rules or conventions of your chosen citation style.

Official Citation Manuels

APA

American Psychological Association (2020). Publication Manual of the American Psychological Association: The  official guide to APA style (Seventh ed.). American Psychological Association.

MLA

Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.

Turabian (Chicago)

Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.

IEEE

IEEE (n.d). IEEE Reference Guide. https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf

Vancouver or ICMJE

International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. http://www.icmje.org/icmje-recommendations.pdf

Screening

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04

(see See section 4.6 Selecting Studies)

Bias

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. (2021, February). Chapter 7: Considering bias and conflicts of interest among the included studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-07

Keenan, C. (2018, April 18). Assessing and addressing bias in systematic reviews. Meta-Evidence Blog, Campbell Collaboration, UK & Ireland. http://meta-evidence.co.uk/assessing-and-addressing-bias-in-systematic-reviews/

The purpose of this module is to teach you how to manage your reading workload efficiently, and how to identify the key points in the articles you are including in your review.

Generally speaking, there is a common flow to scholarly articles. While not a template per se, you can be assured that the following components will be present in most articles. Learning to identify each component is a key step in the strategic reading process, and will help you save time as you screen articles for relevance. Check out the interactive example below that describes each section.

Now that you know about the component parts of scholarly articles, you are prepared to scan your search results and make swift judgments about the relevance of an article to your own context. Note: this step in the process does not require you to read the entire article! You will begin by scanning the articles and follow these steps:

Now that you have scanned your search results and have amassed those that are relevant to your context and needs, you can settle in to do what we call critical reading. This is where you must plan to read the entire article thoroughly.

Critical Questions to Ask When Reading

Critical reading encourages the reader to think about how an article or argument is constructed, not just what it says or what the study found. Here are some questions to ask yourself when reading:

1. What are the central ideas or arguments in the paper?
2. Are there words, concepts or research methods used that you don’t understand? If so, ensure you follow up with quality resources to ensure comprehension and extend your learning.
3. What is the context for the argument? Have they made explicit connections to existing literature, and located their work within that context?
4. Is their literature review clearly outlined and presented in an inclusive manner? Are there any obvious exclusions? If so, are they accounted for?
5. Who is the author? Who are they writing for? Have they located themselves and their positionality to the topic at hand? Are there any declared (or perceived) conflicts of interest?
6. What evidence is given to support the conclusions?
7. Is the evidence credible, that is, does it come from reliable sources?
8. Is the logic of the argument sound? What are the steps in the argument that leads from the evidence to the conclusion?
9. Are any of their ideas problematic? Are there other lenses that could be used to frame the argument?

Tools for Critical Appraisal

Using a standardized tool for critical appraisal is a good practice to follow. It demonstrates consistency in evaluation of all articles, and when working on a research team, it ensures that each individual is asking the same questions.

There are numerous tools for appraisal to choose from. For the most part, there will be disciplinary norms where certain tools are chosen in order to best represent the specific context. Looking to other reviews in your field to see what tools are employed is a good step.

There are various tools such as checklists designed by research organizations that will help you determine if your source is appropriate. Table 4.1 below outlines a sample of free quality assessment tools available for you to use.

To find the tool right for your review check out the Quality Assessment and Risk of Bias Tool Repository from Duke University’s Medical Center Library & Archives. 

Complete Activity 4.1: “Assessing the Quality of Your Sources” in your Research Roadmap Workbook.

Using a tool for critical appraisal is a good practice to follow. It will help you evaluate your sources and if you are working as a research team, it ensures that each individual is asking the same questions.

Take a moment to review some of the tools and document which ones you might use in your review.

For more information please see Workbook Instructions in the Introduction Module.

Scanning an academic article for relevancy is an important component of research. In this activity you will scan the article and identify key components such as the research question, methodology used and the conclusion. For your own research you can use this method for the first read of your sources to decide if they are relevant to your review. If they are relevant, you would do a second more thorough reading of your source. We have provided two examples; a qualitative study and a quantitative study. You are free to do both or choose one.

Qualitative Study Example

Use this example study to identify the following components.Katajavuori, N., Vehkalahti, K., & Asikainen, H. (2021). Promoting university students’ well-being and studying with an acceptance and commitment therapy (ACT)-based intervention. Current Psychology, 1-13 https://link.springer.com/article/10.1007/s12144-021-01837-x#Sec15

Instructions: After taking a guess at the instructions on the front of each card, turn the card to reveal the response. How did you do?

The original version of this chapter contained H5P content. You may want to remove or replace this element.

Qualitative Study Example

Read this example study to identify the following components.Hickey, J., Powling, H., McKinney, P., Robbins, T., Carrier, N., & Nash, A. (2020). "It’s a change your life kind of program”: A healing focused camping weekend for urban Indigenous families living in Fredericton, New Brunswick. First Peoples Child & Family Review, 15(2), 23-44. https://fpcfr.com/index.php/FPCFR/article/view/45

Instructions: After taking a guess at the instructions on the front of each card, turn the card to reveal the response. How did you do?

The original version of this chapter contained H5P content. You may want to remove or replace this element.

The next step after reading and evaluating your sources is to organize them in a way that will help you start the writing process.

Review Matrix

One way to organize your literature is with a review matrix. The review matrix is a chart that sorts and categorizes the different arguments presented per topic or issue. Using a matrix enables you to quickly compare and contrast your sources in order to determine the scope of research across time. This will allow you  to spot similarities and differences between sources. It is particularly useful in the synthesis and analysis stages of a review (See Module 1 Conducting a Literature Review with the SALSA Framework).

Create a Review Matrix

Start with a charting tool you are most familiar with (for example MS Word, MS Excel, Google Sheets, Numbers etc).

1. Input your information.
   • Organize your sources from oldest to most recent. This way you can see how the research on your topic has changed over time.
2. Create your columns. The number of columns is up to you, but you will need the following columns to start:
   • First Column: citation (i.e., author, title, source, publication year)
   • Second Column: purpose or summary (1-2 sentences)
3. Depending on your review, you can now choose the remaining columns that will help you organize your sources. The columns should represent the specific content you are analyzing. For example:
   • methodology
   • population
   • geography
   • intervention
   • outcomes

We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:

• Learning to identify each component is a key step in the strategic reading process, and will help you save time as you screen articles for relevance.
• For your first read you can scan the outcome/results or conclusion section and make swift judgments about the relevance of an article to your own context.
• While selecting your sources, you will need to assess the quality of the research question, the methodology used, the validity of the results and whether the study is applicable to your own review. To help you assess the quality of your chosen sources, consult one of the free checklist tools suggested in Table 4.1.
• Finally a literature review matrix will help you organize your readings when it comes time to write your review.

Reading Strategies

Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246. https://doi.org/10.1136/bmj.315.7102.243

Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012, https://miriamsweeney.net/2012/06/20/readforgradschool/.

Quality Assessment

Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308. https://doi.org/10.1136/bmj.315.7103.305

Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743. https://doi.org/10.1136/bmj.315.7110.740

Literature Review Matrix

Evidence Synthesis Matrix Template, Jane Schmidt, Toronto Metropolitan University (Google Sheets)

The Matrix Method for Literature Reviews, Brandeis University, Writing Resources.

Literature Review Synthesis Matrix, Concordia University (MS Word)

Write a Literature Review: Synthesize. Johns Hopkins University, Sheridan Libraries

Helpful Websites

Systematic Review Guide: Critical Appraisal (St. Michael’s Hospital Health Science Library)

Systematic Reviews: Quality Assessment (University of North Carolina Health Science Library)

Congratulations on completing this short course on conducting an academic review! As you start to work through your review, we hope you notice that you are part of a long line of researchers adding their voice to the scholarly discourse of your topic. Your work is contributing to this discourse. As a researcher, you are an active member of creating new knowledge and not just a consumer of it.

We hope you have come away from this short course with an understanding that there is value in using your intellectual curiosity in learning new investigative methods. Finally, as a researcher, it is normal to encounter challenges and doubts when conducting research, but this can be overcome with persistence, adaptability, and flexibility.

How to Conduct a Literature Review

Aveyard, H. (2019). Doing a literature review in health and social care: A practical guide (Fourth ed.) Open University Press, McGraw-Hill Education.

Jesson, J., Matheson, L., & Lacey, F. M. (2011). Doing your literature review: Traditional and systematic techniques. SAGE.

Ridley, D., Dr. (2008). The literature review: A step-by-step guide for students. SAGE.

Websites

• Guide to Literature Reviews (McMaster University Library)

Constructing a Research Question

Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting research. SAGE.

DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education. https://scientificinquiryinsocialwork.pressbooks.com/

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617

How to Search

Videos

The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube. https://www.youtube.com/watch?v=uyF8uQY9wys

UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube https://www.youtube.com/watch?v=6QQ0MW_jXfM

Websites

• Searching for Sources (Research Skills Tutorial), Toronto Metropolitan University Library
• Library Research Skills Tutorial, McMaster University Library

Reading Strategies

Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246. https://doi.org/10.1136/bmj.315.7102.243

Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012, https://miriamsweeney.net/2012/06/20/readforgradschool/.

Quality Assessment

Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308. https://doi.org/10.1136/bmj.315.7103.305

Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743. https://doi.org/10.1136/bmj.315.7110.740

Checklists and Tools

• CASP Checklists Critical Appraisal Skills Programme (CASP) from the Public Health Resource Unit, NHS, England
• Critical Appraisal Tools The Centre for Evidence-Based Medicine (CEBM) from the University of Oxford
• Risk of Bias 2 (Rob 2) Tool  Cochrane Methods
• Critical Appraisal Tools Joanna Briggs Institute
• Study Quality Assessment Tools  National Heart, Lung and Blood Institute

Writing

Allen, J. (2019). The productive graduate student writer: How to manage your time, process, and energy to write your research proposal, thesis, and dissertation, and get published (First ed.). Stylus Publishing, LLC.

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617

Feak, C. B., & Swales, J. M. (2009). Telling a research story: Writing a literature review. University of Michigan Press.

Holland, K., & Watson, R. (2021). Writing for publication in nursing and healthcare: Getting it right (Second;2; ed.). John Wiley & Sons Ltd.

Walliman, N. (2006). Writing a literature review. In Social research methods (pp. 182-185). SAGE Publications, Ltd,

Website

• Writing a Literature Review (OWL, Purdue University)

Official Citation Manuels

APA

American Psychological Association (2020). Publication Manual of the American Psychological Association: The  official guide to APA style (Seventh ed.). American Psychological Association.

MLA

Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.

Turabian (Chicago)

Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.

IEEE

IEEE (n.d). IEEE Reference Guide. https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf

Vancouver or ICMJE

International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. http://www.icmje.org/icmje-recommendations.pdf

How to Conduct a Systematic Review

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. doi:10.1080/1364557032000119616

Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. SAGE.

Ganann, R., Ciliska, D., & Thomas, H. (2010). Expediting systematic reviews: Methods and implications of rapid reviews. Implementation Science, 5(1), 56-56. doi:10.1186/1748-5908-5-56

Glass, G. (1976). Primary, Secondary, and Meta-Analysis of Research. Educational Researcher, 5(10), 3-8. doi:10.2307/1174772

Gough, D., & Richardson, M. (2018). Systematic reviews. In Advanced research methods for applied psychology (pp. 63-75). Routledge.

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & Libraries Journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). (2021, February). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. www.training.cochrane.org/handbook.

Websites

• Systematic Reviews (Toronto Metropolitan University)
• Cochrane Database of Systematic Reviews (Cochrane)
• Systematic Review Guide (Unity Health Toronto)
• Knowledge syntheses: Systematic & Scoping Reviews, and other review types (University of Toronto Libraries, Gerstein Science Information Centre)

Online Courses on Systematic Reviews

The following guide provides a list of online self-directed learning on how to conduct a systematic review

Online Courses on Systematic Reviews (St. Michael’s Hospital Health Science Library)

Registering Your Systematic Review

• PROSPERO
• Campbell Collaboration
• Cochrane Collaboration

Creating Your Protocol

• PRISMA extension for Protocols
• MECIR (Methodological Expectations for Cochrane Intervention Reviews) Manual
• Sample Protocol Template from the Evidence Synthesis Coordinator at the Maritimes Strategy for Patient-Oriented Research (SPOR) Support Unit.

Documenting Your Review with PRISMA

• PRISMA 2020

Constructing a Research Question

Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting research. SAGE.

Boland, A., Cherry, M. G., & Dickson, R. (2017). Chapter 3 in Doing a systematic review: A student’s guide (Second ed.). SAGE.

DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education. https://scientificinquiryinsocialwork.pressbooks.com/

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617

Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. (2021, February). Chapter 2: Determining the scope of the review and the questions it will address. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-02

How to Search

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies.In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04

Sayers, A. (2008). Tips and tricks in performing a systematic review. British Journal of General Practice, 58(547), 136-136. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2151802/

CADTH Search Filters Database: https://searchfilters.cadth.ca/

Videos

• The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube. https://www.youtube.com/watch?v=uyF8uQY9wys
• UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube https://www.youtube.com/watch?v=6QQ0MW_jXfM

Websites

• Searching for Sources (Research Skills Tutorial), Toronto Metropolitan University Library
• Library Research Skills Tutorial, McMaster University Library

Screening

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04

(see See section 4.6 Selecting Studies)

Reading Strategies

Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246. https://doi.org/10.1136/bmj.315.7102.243

Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012, https://miriamsweeney.net/2012/06/20/readforgradschool/.

Bias and Quality Assessment

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. (2021, February). Chapter 7: Considering bias and conflicts of interest among the included studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-07 

Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308. https://doi.org/10.1136/bmj.315.7103.305

Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743. https://doi.org/10.1136/bmj.315.7110.740

Keenan, C. (2018, April 18). Assessing and addressing bias in systematic reviews. Meta-Evidence Blog, Campbell Collaboration, UK & Ireland. http://meta-evidence.co.uk/assessing-and-addressing-bias-in-systematic-reviews/

Checklists and Tools for Quality Assessment

• CASP Checklists Critical Appraisal Skills Programme (CASP) from the Public Health Resource Unit, NHS, England
• Critical Appraisal Tools The Centre for Evidence-Based Medicine (CEBM) from the University of Oxford
• Risk of Bias 2 (Rob 2) Tool  Cochrane Methods
• Critical Appraisal Tools Joanna Briggs Institute
• Study Quality Assessment Tools  National Heart, Lung and Blood Institute

Websites for Quality Assessment

• Systematic Review Guide: Critical Appraisal (St. Michael’s Hospital Health Science Library)
• Systematic Reviews: Quality Assessment (University of North Carolina Health Science Library)

Tools for Assessing Bias

• ROBIS – A tool for assessing the risk of bias in systematic reviews
• AMSTAR2 – AMSTAR stands for A MeaSurement Tool to Assess Systematic Reviews.

Analyzing Data

Card, N. A., & Little, T. D. (2012). Applied meta-analysis for social science research. Guilford Press.

Cheung, M. W. -., & Vijayakumar, R. (2016). A guide to conducting a meta-analysis. Neuropsychology Review, 26(2), 121-128. https://doi.org/10.1007/s11065-016-9319-z

Li T, Higgins JPT, Deeks JJ (editors). Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. https://training.cochrane.org/handbook/current/chapter-05

Writing

Allen, J. (2019). The productive graduate student writer: How to manage your time, process, and energy to write your research proposal, thesis, and dissertation, and get published (First ed.). Stylus Publishing, LLC.

Holland, K., & Watson, R. (2021). Writing for publication in nursing and healthcare: Getting it right (Second;2; ed.). John Wiley & Sons Ltd.

Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. SAGE.

Official Citation Manuels

APA

American Psychological Association (2020). Publication Manual of the American Psychological Association: The  official guide to APA style (Seventh ed.). American Psychological Association.

MLA

Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.

Turabian (Chicago)

Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.

IEEE

IEEE (n.d). IEEE Reference Guide. https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf

Vancouver or ICMJE

International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. http://www.icmje.org/icmje-recommendations.pdf