Toronto Metropolitan University Library
Advanced Research Skills: Conducting Literature and Systematic Reviews (2nd Edition) by Kelly Dermody; Cecile Farnum; Daniel Jakubek; Jo-Anne Petropoulos; Jane Schmidt; and Reece Steinberg is licensed under a Creative Commons Attribution 4.0 International License, except where otherwise noted.
This work is licensed under an Ontario Commons License (Version 1.0)
1
This Second Edition of Advanced Research Skills: Conducting Literature and Systematic Reviews has been revised and expanded to include new search resources and reading strategies. The accessibility features have also been enhanced.
2
This project is made possible with funding by the Government of Ontario and through eCampusOntario’s support of the Virtual Learning Strategy. To learn more about the Virtual Learning Strategy visit: https://vls.ecampusontario.ca.
3
Advanced Research Skills: Conducting Literature and Systematic Reviews was audited for accessibility issues using Sa11y.
Sa11y is an accessibility quality assurance tool that visually highlights common accessibility and usability issues.
The web version of Advanced Research Skills: Conducting Literature and Systematic Reviews includes the following features:
In addition to the web version, this book is available in a number of file formats including Digital PDF and EPUB (for eReaders).
There are no known accessibility issues at this time.
If accessibility issues are stopping you from accessing the information in this short course, please contact us to let us know and we will get it fixed. If you discover any other issues, please let us know of those as well.
Please include the following information:
You can contact us at kdermody@torontomu.ca
This statement was last updated on February 28, 2023
I
1
This short course is on the research component of conducting a literature or systematic review.
The purpose is to spark your intellectual curiosity and develop your research skills. Research is an important part of your review and your work will add to the scholarly discourse of your topic. This self-guided course will help you outline your goals for your review and demonstrate how adopting a mind-set of persistence, adaptability, and flexibility will help you achieve them.
Throughout the course, we provide examples and activities designed to help you develop your essential research skills. In Module 1, we provide a quick overview of the different types of academic reviews and the steps involved in conducting them. In Module 2, we demonstrate how to formulate your research question and how to search for sources. In Module 3, you will learn how to select and organize your sources. Finally, in Module 4, we outline techniques for reading and assessing the quality of your sources.
We have designed this short course to be done at your own pace and at your own point of need. Therefore, you can navigate to the parts you find relevant to your current situation.
This version of the course is hosted in Pressbooks (a WordPress-based online platform). Pressbooks is used to host open textbooks and, in this case, a short course. Please see the video below on how to navigate Pressbooks.
Iowa State University Digital Press. Navigating Your Course Pressbook. Licenced under Creative Commons CC BY-NC 4.0. (Closed Captioned)
The workbook is designed to help you reflect and document your research process. Throughout this short course, you will have the opportunity to use the workbook to document aspects of your review. You can refer back to this workbook during the research phase of your review.
The link below will create a personal copy of the workbook. This workbook is a Google Document and by clicking on the link, a copy of the workbook will save to your Google Drive. This is your personal copy of the Workbook for your own records that you can return to whenever you are prompted to do a workbook activity throughout this short course.
Advanced Research Skills: Conducting Literature and Systematic Reviews Workbook
Below are the learning outcomes for the overall course. Each module will include a list of objectives/outcomes specific to that module to guide you through your learning.
By the end of this course, you should be able to:
This course concentrates on the research component of conducting a review. We do not cover how to write a review. Also, we will not cover how to extract and analyze data.
In our last section, Resources to Keep you Going, we provide resources that you can consult if you need information on these topics.
2
Complete Activity 0: “Review Your Research Skills” in your Research Roadmap Workbook.
This survey will help you reflect on which research skills you have and what you need to work on. You can return to this document as you continue through your review to reflect on the progress you have made.
For more information please see Workbook Instructions in the Introduction Module.
II
In this module, we will go over the different types of academic reviews and a quick overview of the steps involved in conducting them. If you are unsure about which review to choose for your own work this module will help you understand the difference between each review and help you make an informed decision.
If you know which review you will be conducting feel free to skip to relevant sections that will answer any questions you might have.
By the end of this module, you will be able to:
Complete Activity 1.1: “My Review” in your Research Roadmap Workbook.
Before you start a review, it’s always a good idea to make sure you take some time to reflect on what your objectives are. This survey will help you articulate what time/resources you have available and what you tentatively plan to undertake. Keep your answers in mind as you read through the module to see if you are on the right track or if you need to modify your objectives.
For more information please see Workbook Instructions in the Introduction Module.
Both literature and systematic reviews are aimed at assembling, critically evaluating and reviewing existing research on a central topic or research question. Some differences between them include the method for determining what research to include or exclude, the extent or scope of the review, and the duration of time required to complete the process. To help you determine which review is most appropriate, please see Table 1.1 below for a detailed explanation of each as well as the differences between each type of review.
Component | Literature Review | Systematic reviews |
---|---|---|
Definition | A literature review is a qualitative integrative summary of published research on a specific topic. The literature review seeks to synthesize what is already known about the topic, and sometimes, explicitly state what is not known, or not well understood. | Systematic reviews bring together information from a range of sources to answer a specific research question. They differ from a traditional literature review or narrative review, in that they aim to synthesize and analyze the research in an unbiased, rigorous and systematic way so that it can be used to support evidence-based practice. |
Goals |
|
|
Question |
|
|
Components |
OR
|
|
Number of Authors |
|
|
Timeline |
|
|
Requirement |
|
|
Value |
|
|
Adapted from Kysh, Lynn (2013): Difference between a systematic review and a literature review. Figshare (Poster), https://doi.org/10.6084/m9.figshare.766364.v1. Licensed under Creative Commons CC BY 4.0 /order and some words changed.
There are two ways to present a literature review: it can be one section in an original study, or it can be a standalone full review. More examples are given in the section on Literature Reviews.
A literature review is an integrative summary of published research on a specific topic. The literature review seeks to synthesize what is already known about the topic, and sometimes, explicitly state what is not known, or not well understood.
The following are the key characteristics of a literature review:
A literature review may serve as a stand-alone piece or article such as this article by Francisco Javier Pérez Latre et al., on Social NetworksCitation: Pérez-Latre, F. J., Portilla, I., & Sánchez-Blanco, C. (2011). Social networks, media and audiences: A literature review. Communication & Society, (24)1, 63-74.https://revistas.unav.edu/index.php/communication-and-society/article/view/36221 *Communication & Society is licensed under Creative Commons Attribution-Noncommercial-NoDerivatives 4.0.
However, more often a literature review is part of a larger research publication such as this example from Nia Contini et al., on Boating-related fatalities in the Northwest TerritoriesCitation Nia Contini, Audrey R. Giles, Gordon Giesbrecht & Tyrone Raddi (2021) The adaptation of the beyond cold water bootcamp course for Inuvialuit communities in Northwest Territories, Canada, International Journal of Circumpolar Health, 80:1, DOI: 10.1080/22423982.2021.1969744
Literature reviews help authors and readers build their knowledge about a specific topic by synthesizing research on the subject. Literature reviews use published research to provide context to the topic and may expose debates within the field, gaps in the research, or provide a summary and analysis of research to date on the topic.
Literature reviews are not a summary of a couple of sources you found on your topic. Your review should present themes or identify trends, including relevant theory found in the literature you reviewed.
All reviews follow a familiar process:
Prior to starting your own research, you will want to look at existing literature reviews – this is especially important so that you don’t duplicate existing work. It can also be helpful to look at the approaches taken for literature reviews similar to your own topic or discipline. Below are some examples of locating existing reviews.
In general, your research question will tackle the problem you are trying to address by conducting the review. Since constructing a research question can be an in-depth process, we go over it in more detail in Module 2: Formulating a Research Question and Searching for Sources.
Once you have a research question, there are four stages you can follow when conducting your chosen review. These are known as the SALSA Framework: search, appraisal, synthesis and analysis.
Here is a quick summary of the SALSA steps.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=34#h5p-1
Did you notice the missing L? We did too! The authors, Grant and Booth (2009) created a simple analytical framework for conducting reviews: Search, Appraisal, Synthesis and Analysis. SASA, however, doesn’t make a memorable acronym, and Academics love a good acronym, so they derived the “L” from the last letter of appraisal: Search, AppraisaL, Synthesis and Analysis (SALSA).
A systematic review is also known as evidence synthesis because it brings together information from a range of sources to answer a specific research question. It differs from a traditional literature review, in that it aims to synthesize and analyze the research in an unbiased, rigorous and systematic way so that it can be used to support evidence-based practice.
There are different types of reviews that involve evidence synthesis and a systematic review is the most well known version. Other examples include a rapid review or a scoping review. We define the different types in more detail below.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=36#h5p-2
Determine what type of review would be the best fit for each of the following research questions.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=36#h5p-3
A review like a systematic review or a meta-analysis can take at least a year to complete and is usually conducted by a team. If your review is for a class assignment, you can still conduct a systematic review without a team or a year to complete it. If your aim is to eventually publish your review, keep in mind that one of the main goals of a systematic review is to try and eliminate potential bias, and working independently can be viewed negatively. If you must work independently, you should identify this limitation when writing your review.
Eliminating bias as much as possible is one of the key characteristics of systematic reviews. By bias, we mean that some type of systematic error has occurred during the review stage that leads to the acceptance of outcomes and conclusions of a study. This can result in the possibility of unfair or misleading information within the reviews. Bias is potentially introduced at any stage of the research process, from formulating your research question to choosing which sources to include.
In order to reduce bias in your review, you will need to undertake a quality assessment throughout the review process. Your protocol (see next section) and your screening process will help you reduce your bias. Another way to reduce bias is to work in a team setting, and this is why some reviews require more than one person. We will cover how to check for bias when screening sources in Module 3 Organizing, Managing and Screening Sources.
A systematic review can generally give us the most dependable answer to a specific research question, and it can identify gaps in our knowledge that require further research. It also communicates the strength of the available evidence and the quality of included studies. This indicates how much confidence practitioners, service users, managers, policy makers, and the popular media should have in the results (Gough & Richardson, 2018).
Booth et. al (2016) suggest that your choice of review methods should be determined by five main considerations captured by the acronym, TREAD
These and other factors may determine what kind of review is most appropriate to answer your research question.
The original version of this chapter contained H5P content. You may want to remove or replace this element.
Decision Tools
The following tools can help you pick the right systematic review:
It is important to understand and meet the specific requirements of your chosen review, especially if you plan to publish your review.
3
This section is a quick summary of the main steps involved in conducting systematic reviews. By the end of this section you should have a better idea of the time and resources needed to conduct a successful review.
All reviews follow a familiar process as seen in Figure 1.1 below.
Prior to starting your own research, you will want to look at existing systematic reviews – this is especially important so that you don’t duplicate existing work. It can also be helpful to look at the approaches taken for systematic reviews similar to your own topic or discipline. You can find existing systematic reviews through a number of ways:
Looking at published reviews and protocols can give you an idea of what has already been done and will help you ensure that your own research is original.
If you are conducting a systematic review that requires a team these are the typical roles involved:
In general, your research question will tackle the problem you are trying to address by conducting the review. Since constructing a research question can be an in-depth process, we go over it in more detail in Module 2: Formulating a Research Question and Searching for Sources.
Reviews like a systematic review require a protocol, which is essentially a planning document that indicates how your review will be carried out. Here is a sample protocol template from the Evidence Synthesis Coordinator at the Maritimes Strategy for Patient-Oriented Research (SPOR) Support Unit. This basic form includes all the relevant information needed for a simple protocol.
You may wish to register your protocol to avoid the duplication of work and to reduce the potential for bias by enabling a comparison between what was stated in the protocol to the completed review. It is also a way to share your current research interests with the research community at large, and help build your research profile.
How to Register your Protocol:
Please see this guide by the National Institute of Health (an agency of the United States government): Systematic Reviews Protocol and Protocol Registries.
By creating a protocol you are creating a document that will guide you through the systematic review process. Always refer to it throughout the process to ensure you are on track.
Once you have a research question, there are four stages you can follow when conducting your chosen review. These are known as the SALSA Framework: search, appraisal, synthesis and analysis.
Here is a quick summary of the SALSA steps.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=39#h5p-5
Did you notice the missing L? We did too! The authors, Grant and Booth (2009) created a simple analytical framework for conducting reviews: Search, Appraisal, Synthesis and Analysis. SASA, however, doesn’t make a memorable acronym, and Academics love a good acronym, so they derived the “L” from the last letter of appraisal: Search, AppraisaL, Synthesis and Analysis (SALSA).
Applying the SALSA Framework to Your Specific Review
We’ve provided a quick summary of the framework, and once you have chosen your specific type of review you should consult the following chart by Grant and Booth (2009) for a deep dive into each stage of the SALSA framework for your specific review.
A typology of reviews: an analysis of 14 review types and associated methodologies. Citation: Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & Libraries Journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA is the recognized standard for reporting evidence in systematic reviews and meta-analyses. The standards are endorsed by organizations and journals in the health sciences. It allows other researchers to assess strengths and weaknesses of the review and assists with future replication of the review methods. The 2020 PRISMA statement consists of a 27-item checklist and a 4-phase flow diagram.
For more information, consult the PRISMA Explanation and Elaboration document.
If you are conducting a scoping review, see The PRISMA-ScR (PRISMA Extension for Scoping Reviews)
A large amount of time and resources go into conducting a systematic review. To make sure you are ready to carry out a review, use the Knowledge Synthesis Readiness Checklist from Unity Health Toronto.
4
Conducting a review requires good time management. Feel free to use the To Do List: Research Stage to keep track of your timeline and tasks for the research stage of your review.
For more information please see Workbook Instructions in the Introduction Module.
5
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=43#h5p-6
We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:
Aveyard, H. (2019). Doing a literature review in health and social care: A practical guide (Fourth ed.) Open University Press, McGraw-Hill Education.
Jesson, J., Matheson, L., & Lacey, F. M. (2011). Doing your literature review: Traditional and systematic techniques. SAGE.
Ridley, D., Dr. (2008). The literature review: A step-by-step guide for students. SAGE.
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. doi:10.1080/1364557032000119616
Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. Sage
SAGE.
Ganann, R., Ciliska, D., & Thomas, H. (2010). Expediting systematic reviews: Methods and implications of rapid reviews. Implementation Science, 5(1), 56-56. doi:10.1186/1748-5908-5-56
Glass, G. (1976). Primary, Secondary, and Meta-Analysis of Research. Educational Researcher, 5(10), 3-8. doi:10.2307/1174772
Gough, D., & Richardson, M. (2018). Systematic reviews. In Advanced research methods for applied psychology (pp. 63-75). Routledge.
Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & Libraries Journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). (2021, February). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. www.training.cochrane.org/handbook.
III
6
In this module we go over the framework for searching the literature. Keep in mind that searching is a non-linear and potentially iterative process; as such you may need to review and revise earlier steps at any point.
The steps to searching the literature are:
We will show you how to use special techniques like forward and backward citation searching to find more sources. Finally, if you are conducting a systematic review you should check out our special section on how to document your search strategy and results.
By the end of this module, you will be able to:
7
Test your knowledge of Boolean, truncation and other searching techniques.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=52#h5p-7
Check out these guides:
8
As noted in Module 1: Types of Reviews, conducting a “pre-search” is a crucial first step in devising the research question. A well-formulated research question informs the research process. It can focus your information needs (i.e. identify inclusion and exclusion criteria), help to identify key search concepts, and guide you in the direction of relevant resources.
There are two general types of research questions: quantitative and qualitative.
Types of quantitative questions can be categorized as explanatory (i.e., relationship-based), descriptive, or comparative.
Qualitative questions aim to discover meaning or gain an understanding of a phenomenon. They ask questions that cannot be measured with specific numbers and statistics. Qualitative research questions often contain words like "lived experience", “personal experience”, “understanding”, “meaning”, and “stories”.
So how do quantitative and qualitative research questions differ when you are conducting a search? In Table 2.1 below, we provide some examples of research topics. Each topic can either be used for a quantitative or qualitative research question. For each question, the category of research question is clarified.
Topic | Quantitative Research Questions | Qualitative Research Questions |
---|---|---|
Housing insecurity for children | Explanatory:
How does experiencing housing-insecurity in childhood impact a child’s relationships in adulthood? |
Lived experience:
What are the lived experiences of children experiencing housing-insecurity while attending elementary School? |
Nursing and workplace violence | Descriptive:
What percentage of nurses experience violence in the workplace on night shifts? |
Personal understanding:
How do nurses conceptualize their role in mitigating workplace violence? |
Mental health of Canadian immigrants | Comparative:
What are the differences in perceptions towards accessing mental health services between immigrants and their Canadian-born children? |
Personal experience:
How do newly arrived Canadians experience seeking help for mental health concerns? |
Regardless of the type of question being asked, a good research question cannot be answered with a simple yes or no (as demonstrated by the sample questions in Table 2.1).
9
Using a formula is another way to construct a research question and is recommended if you are conducting a systematic review. There are various formulas you can use to craft your question, see Table 2.2 below.
Formula | Components Within the Formula | Example |
---|---|---|
PICO(T) |
Note: the time period and type of study are optional |
P: Patients who have undergone knee surgery
I: Post-operative infection C: Patients without post-operative infection O: Duration of recovery |
PS |
|
P: New immigrants
S: Accessing mental Health Care |
PIE |
|
P: Children experiencing homelessness
I: Breakfast program at school E: Academic performance |
PEO |
|
P: Health professionals
E: Caring for patients with Dementia O: Attitudes |
PCC |
|
P: Pediatric patients with sleep disturbance
C: Quality of life C: After tonsillectomy surgery |
SPICE |
|
S: Pediatric hospital rooms
P: Pediatric patients I: Therapy dog visits C: No therapy dog visits E: Reduced anxiety |
SPIDER |
|
S: Young parents
P of I: Early Literacy programs D: Survey E: Experiences R: Qualitative |
These frameworks aid in identifying the important parts (i.e., concepts) that can be used in formulating a research question. Below are two examples using the PS and PEO frameworks.
Can you think of a research question using these identified PS concepts?
Population/Problem: Family members’ of dying loved ones
Situation: Placing them in palliative care
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=56#h5p-8
Can you think of a research question using these identified PEO concepts?
Population/Problem: Caregivers of family members in palliative care
Exposure: Psycho-educational group intervention
Outcomes/Themes: Improved quality of life
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=56#h5p-9
When trying to settle on a research question, ensure you are choosing something of interest, but not so narrow that you are unlikely to uncover any information, but not too broad so that you are overwhelmed with content and don’t know where to start. Below are examples of too broad and too narrow questions and an explanation to help understand why.
Why is this example too broad a question?
“A systematic review of the literature on palliative care interventions.”
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=56#h5p-10
Why is this example too narrow a question?
“Effectiveness of early reading intervention programs for children aged 5-7 in the private school system in East Toronto.”
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=56#h5p-11
A pre-search will assist you with this step. If you need a refresher on conducting a pre-search, see Module 1: Types of Reviews. Once you have a good understanding of the existing literature, you will be able to confidently formulate your research question.
Keep in mind that your question may change over time as you delve deeper into the literature. This is a normal course of events. You are still in the outline stage! Part of the process is refining and leading to sharper focus as you learn. For guidance on developing the research question see the further reading section.
10
Identifying keywords, (also known as search terms) is important for effective literature searching. Your search terms are terms that will appear somewhere within the resource (e.g. title, abstract, or author keywords).
You can identify search terms from your research question by highlighting, underlining or circling the main ideas that must appear in the article. Your search concepts are the most important words in your research question.
What are the keywords you identify within this research question?
“What is the relationship between flexible work schedules and staff retention?”
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=58#h5p-12
Keywords are easier to identify once the research question has been formulated using one of the stated frameworks (e.g. PICO or PS) listed in Table 2.2 of Formulating a Research Question.
Below is an example of how to use the PICO(T) framework to identify search concepts for a specific research question.
Say you have the following research question: How effective is cognitive behavioural therapy in improving mild-to-moderate depression in adolescents? Let’s break down the formula components by concept.
Formula component | Concept |
---|---|
Patient, Population or Problem (P) | Adolescents with mild-to-moderate depression |
Intervention (I) | Cognitive behavioural therapy |
Comparison, Control Intervention (C) | There is no concept here |
Outcome (O) |
Symptom reduction |
Identify Search Concepts
Drag and drop the correct search terms using PICO (Patient/Population/Problem, Intervention, Comparison/Control, Outcome) for the following research question:
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=58#h5p-13
12
Controlled vocabularies are a standardized set of terms used to describe the content of a resource in a database. This is known as indexing. Using controlled vocabulary terms will usually generate fewer and more relevant results. However, you must know the exact term/vocabulary to use. An example is provided below.
Articles on cancer research can include terms like: tumour, cancer, or neoplasm. In Ovid MEDLINE, the controlled vocabulary for these terms is Neoplasms.
By searching for Neoplasms you should retrieve all the articles where the author(s) used the terms tumour, cancer, or neoplasm (as long as they have been properly indexed).
The following are situations in which you would use controlled vocabularies:
Search strategies must be customized for each database for a number of reasons, one of which is that they use different controlled vocabularies. The tutorials in Table 2.3 below cover how to use controlled vocabularies in specific databases. Review these tutorials if you’d like to learn more.
Database | Controlled Vocabulary | Tutorial |
---|---|---|
Ovid MEDLINE and PubMed | MeSH | MeSH tutorial |
APA PsycInfo | APA Thesaurus | APA PsycInfo Tutorial |
EBSCO CINAHL | CINAHL Subject Headings | CINAHL Tutorial |
Ovid Embase | Emtree | Embase Tutorial |
Databases like GEOBASE or Social Sciences Abstracts do not automatically identify relevant controlled vocabularies. Instead, you need to search the database’s thesaurus or browse their list of controlled terms. Consult the database’s help page, or contact your library for assistance.
Below are the steps to follow when finding more controlled vocabularies:
Not all search terms will map to an appropriate subject heading. If this is the case, search for it as a keyword. This option is available after the list of subject headings in most databases.
13
The standard method of constructing a search strategy is to use Boolean logic. Boolean relies on three essential operators, AND, OR and NOT. Using these operators allows you to combine words and phrases using the words AND, OR, NOT to limit, broaden, or define your search. A good researcher should know how to do a Boolean Search.
Boolean operators connect your search concepts and search terms (i.e., controlled vocabulary and keywords) together. The three basic operators are: OR, AND, and NOT. An example is shown below, followed by a deeper dive into each of these operators.
(Cognitive Behavioral Therapy OR CBT) AND Depression
The OR operator broadens your search results by retrieving records that contain either or both of your search terms or concepts. Usually, terms within a concept are combined using the Boolean OR operator.
In the Figure 2.4 below, the database will retrieve records that contain either or both of the terms Cognitive Behavioural Therapy OR CBT.
The AND operator narrows the search results by retrieving records that contain both of your search terms or concepts. Usually, different concepts are combined using the Boolean AND operator.
In the Figure 2.5 below, the database will retrieve sources that contain Cognitive Behavioural Therapy AND Depression
The NOT operator narrows the search results by eliminating a specific search term. You should use the NOT operator with caution. This is because it can eliminate relevant results that happen to mention the term you are “noting” out. In Figure 2.6 below, any articles that mention both adult and adolescent will be excluded from the result set.
Depending on the search engine (e.g., Google Scholar) or database (e.g., PubMed), the operator(s) may be required to be entered in upper case letters. Consult the search engine’s or database’s Help information or Search Tips for details about how to enter the Boolean Operators.
Test your Boolean Knowledge
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=66#h5p-16
Truncation and wildcards can be applied to a keyword search to broaden your results and allow you to look for variations of words.
You can use the truncation symbol to avoid typing out all possible variations of a word. For example, surg* will retrieve surgery, surgeries, surgeon, or surgical. The truncation symbol should be used with caution to ensure relevant words are being retrieved.
The most common symbol used is an asterisk (*). The asterisk symbol is used in the following platforms and databases:
Wildcard symbols can be used to substitute for one or more characters, or any single character. This is useful when dealing with variant spelling (e.g., pediatric and paediatric). One of the most common symbols used is the question mark (?). For example, in the Ovid database, p?ediatric* will retrieve pediatric OR paediatric OR pediatrics OR paediatrics.
Databases use different wildcard symbols, so check the database Help information or Search Tips for details about which symbol to use.
14
When conducting a review, it is important to search multiple resources. Just searching one subject database will not ensure you have found all the essential sources on your topic.
For systematic reviews, searching multiple databases is essential for reducing bias in your results. Only using one database like PubMed means you have ignored relevant studies found in other databases like Embase.
Subject databases are accessible through your academic library and contain published material belonging or pertaining to a specific discipline or multiple disciplines, e.g.,health sciences, business, social work, or law. Many subject databases are hosted on the platforms EBSCO and Proquest for example. These search platforms allow users to search and retrieve content, and may look and work similarly to each other. The type of material in different subject databases depends on the subject, but most will include academic peer-reviewed articles, newspaper and magazine articles, and reports.
To find a subject specific database, go to your library’s website and look for a research guide for your discipline. This guide will include a list of article databases for your discipline, and the descriptions below them will help you select among them. Alternatively, look for a list of databases by subject on your library website, and choose an article database which covers your subject.
If you have questions about which database to choose, contact your librarian for help.
There are also free options if you are not affiliated with an academic library. Public libraries have some databases you may search with a library card. Also, there are some free databases online such as PubMed (health sciences subjects), DOAJ (general subjects), Dimensions (general subjects) and Google Scholar (general subjects). Further free (Open Access) resources are listed on this Open Access Educational Resources Guide.
In general, all subject databases will have a basic search box and an advanced search box. We recommend you use the advanced search box, as it provides more options to optimize your search. The advanced search box will often allow you to choose where you want your keywords to appear within the resource (e.g., in the title or the abstract and body or anywhere). An example of an advanced search box is found in Figure 2.7 below.
Limiters are found in all subject databases and they allow you to limit your search by date, type of source, language, and in some databases by population and methodology (see Figure 2.8 below). This is particularly useful if you are conducting a systematic review, as the limiters will help with your exclusion criteria.
Once you have chosen your subject databases, take a note if they are owned by a particular platform (i.e., EBSCO, ProQuest, Ovid, etc). Each of these platforms will have tutorials on how to conduct an advanced search. Below are video tutorials from some of the major subject database platforms.
Advanced Searching Tutorial Videos:
Google Scholar searches articles, theses, books, abstracts and court opinions, from academic publishers, professional societies, online repositories, universities and other web sites.
The majority of sources will be behind pay-walls and you will not be able to access them. However, you can optimize Google Scholar to search your academic library’s collection, at which point you will be able to access any item your library owns. For a list of the pros and cons of using Google Scholar please see Table 2.6.
Pros | Cons |
---|---|
|
|
From McMaster University, Health Science Library, Google Scholar. Licensed under Creative Commons CC BY-NC-SA 4.0.
The following video demonstrates how you can search Google Scholar.
UTS Library, Google Scholar: Advanced Searching [2:53]. Licensed under a Creative Commons CC BY 4.0. Closed Captions
Grey literature refers to any information that is not commercially published, and can include a wide range of material types, such as:
Knowing where to search for grey literature can be challenging. There are a variety of suggestions in the following library guide from Simon Fraser University: Grey literature: What it is & how to find it.
A good approach is to be strategic and consider where appropriate grey sources might be located. For example, if you know of any government agencies, non-profits, professional associations, research institutes, and other organisations that may be producing content relevant to your topic, you may wish to search their website for any publications and reports. At a very minimum, looking at graduate theses and dissertations and conference proceedings ( often sites of emerging research) should be a component of your grey literature search.
Searching for grey literature will likely involve the use of a search engine such as Google, so it will be important for you to critically evaluate sources before including them in your research. For some tips on avoiding fake news and the spread of misinformation, review the following guide from Toronto Metropolitan University Library: How to Identify Fake News.
Your search strategy will need to be “translated” for use in multiple databases and you will have to look out for the following:
Systematic reviews require you to use the same search strategy across different databases. However, as mentioned above, different subject databases might not allow you to use the exact same search.
Consider the steps below on how to translate your search strategy from one database to anotherFrom McMaster University Library, How to Search the Literature (Advanced). Translate a Search Strategy.:
15
Besides your standard search there are additional options to identify relevant sources including searching key journals in your field, and tracking the work of relevant authors.
Another method used by researchers is to consult the citations within an article. By consulting the citations you can find relevant titles that you might have missed in your search. The advantage of this method is that you can find sources about a subject quickly and relatively easily. The disadvantage of this method is that you are searching backward, or retrospectively, so each source you find will be older than the previous one.
This is why you need to use a forward citation searching method in combination with your backward citation search. In the forward searching method, researchers use tools within databases like Google Scholar and Web of Science to see who has recently cited your source. This way you can find new and relevant sources.
In this method, review the references used in key articles. For example, search through the article for the “References” section as in Figure 2.9 below.
In this method, see what articles have cited your key articles. For example, search through the “Cited by” section of the article as in Figure 2.10 below.
Many databases will include live links to an article’s references, as well as live links to other articles that have cited that article. The following video demonstrates how to do citation searching in Google Scholar:
CSUDH Library. Citation Tracking. Licenced under Creative Commons CC BY 4.0. Closed Captioned
16
If this is your first review, it’s normal to feel overwhelmed and unsure when to stop the searching process. Afterall, you don’t want to miss anything that’s possibly relevant, so the searching process never really seems to come to an end!
Unfortunately, we don’t have a firm easy answer like “if you find 10 articles you can stop.” There is always a chance that an article might slip past your extensive searching, but if you have done the steps below, then the chances of a really important article slipping past you is pretty slim.
17
Your search process must be documented in enough detail to ensure that it can be reported correctly in the systematic review and reproduced for verification. During your search process you will have to keep track of:
In Module 1 we learned that PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. The PRISMA Group has developed an extension to PRISMA to assist researchers in documenting their literature searches for systematic reviews. The checklist identifies what to document in terms of information sources and methods, search strategies, peer review, and records management. The PRISMA flow diagram demonstrates how to document your final numbers.
Take a moment to review the checklist and the flowchart and save a copy for your own review.
18
Complete Activity 2.1: “Create Your Own Search Strategy” in your Research Roadmap Workbook
This exercise allows you to document your search strategy. Don’t worry if you are still trying to figure out some of the important details about your review as you can use this exercise to document what you have in mind.
For more information please see Workbook Instructions in the Introduction Module.
19
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=81#h5p-17
20
We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:
21
Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting
research. SAGE.
Boland, A., Cherry, M. G., & Dickson, R. (2017). Chapter 3 in Doing a systematic review: A student’s guide (Second ed.). SAGE.
DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education. https://scientificinquiryinsocialwork.pressbooks.com/
Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617
Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. (2021, February). Chapter 2: Determining the scope of the review and the questions it will address. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-02
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies.In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04
Sayers, A. (2008). Tips and tricks in performing a systematic review. British Journal of General Practice, 58(547), 136-136. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2151802/
The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube. https://www.youtube.com/watch?v=uyF8uQY9wys
UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube https://www.youtube.com/watch?v=6QQ0MW_jXfM
CADTH Search Filters Database: https://searchfilters.cadth.ca/
Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: An extension to the PRISMA Statement for reporting literature searches in systematic reviews. Systematic Reviews. 2021;10(1):39. https://dx.doi.org/10.1186/s13643-020-01542-z
IV
22
After you have conducted your search, you will be confronted with the dilemma of what to do with your results. It is not unheard of for searches to produce hundreds if not thousands of results to shift through. Your next steps will be to download your results, save them somewhere you can manage them and finally screen them to pick the ones you will be keeping in your review.
At this point you should work out a plan for how you will organize your results. Staying organized and managing your results will require a new set of organizational skills, plus these skills will help protect you from accidental plagiarism or bias.
By the end of this module, you will be able to:
23
In any literature or systematic review project, exporting and saving results from databases is an important step prior to selecting your final sources. Exporting and saving results allow you to permanently store the important bibliographic information or ‘metadata’ attached to your articles – this could include the author name(s), article title, journal title, etc., the article abstract, and any associated keywords.
The majority of the databases that you will likely be using for your review will have options to export your references. Some important tips when exporting large amounts of records:
The following video is an example of how to save and export results in EBSCO CINAHL.
Exporting from EBSCO CINAHL. University of East Anglia Library. Licensed under a Creative Commons CC BY 4.0. *Note this video does not have closed captions, a transcript has been provided below.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=90#h5p-18
Complete the following steps to practice exporting search results:
Many database interfaces like EBSCO, OVID and ProQuest allow users to create a personal account. With an account you can save your searches including any keywords, MeSH terms or limiters you used. This means you can return to your account at any time to re-run your search or modify your search and run the search again. See Table 3.1 below on how to set up an account for the following major database vendors (always check with your library to see which databases are available to you as a patron).
Database Vendor | Instructions on Setting Up an Account |
---|---|
EBSCO | How to Create & Manage my EBSCOhost account |
ProQuest | My Research Account |
OVID | Creating a OVID account |
Web of Science | Register for an an email |
Get to know the exporting features of your chosen databases and sign up for an account with each database. This will not only save your results and search history, it will save you time if you have to re-do your search.
24
After exporting your results, you will need to store and organize them. Since you will be dealing with a large amount of sources, it’s advisable to use some sort of citation management tool.
The original version of this chapter contained H5P content. You may want to remove or replace this element.
A citation tool is especially useful If you are conducting a systematic review. As part of your review, you need to document aspects of your search like the number of sources you retrieved per database, the number of duplicates and the final number of articles you choose to review. For more information, please see the PRISMA Checklist in Module 2.
The following video provides a quick overview of how a reference management tool works.
Reference Management Software: How it Works and What it Does. University of York Library. Licensed under a Creative Commons CC BY 4.0. Closed Captions
25
When picking a tool, we suggest trying out a few in order to compare their features. You might also want to keep the following questions in mind when trying them out:
If you are interested in a more comprehensive list of citation management tools available worldwide, please see the following chart from Wikipedia: Comparison of Reference Management Software.
If you have a small and manageable number of sources, you can use a familiar spreadsheet program like Excel, Google Sheets or Numbers to manage your citation data and look for duplicates. You can create your own columns for important citation information like Author, Title, Year, Database, etc. and even a column to enter notes. You can enter your sources manually or use the exporting function found in most databases. For example, databases like PubMed and ProQuest allow you to export the bibliographic information attached to your sources as a .CSV file. This file can be opened as a spreadsheet in Excel, Google Sheets and Numbers. See Figure 3.1 below for an example of a spreadsheet used as a citation management tool.
There are a number of free citation management tools available for you to use. The only requirement is that you sign up for an account and download the product to your device. The three most commonly used tools are Mendeley, Zotero and EndNote Basic. If you are having trouble picking between the three, Table 3.2 below offers a quick comparison chart. For a more in depth comparison you can use the following comparison chart from the University of Toronto Libraries to help you decide.
Product | Mendeley | Zotero | EndNote Basic |
---|---|---|---|
Price | Free | Free and open source |
*Note: this is the free basic version of EndNote 2.0 which is a paid product. |
Availability | Desktop and online | Desktop and online | Online only |
Word processor | MS Word Plugin | MS Word and Google Doc Plugin | Download “Cite While you Write” for both Windows and Mac. |
Features |
|
|
|
Important links |
There are some important reasons for considering a paid product for your choice of citation management tool. They can offer technical support if anything goes wrong, larger storage space, collaboration, duplication and other special features that might meet your needs more than a free product. The following table offers a comparison of two popular paid products available on the market. Some libraries have institutional licenses to one or more of these products, which means as long as you are a student you will have access. Check with your library to see if you have access to these products. Table 3.3 below offers a quick comparison chart of two paid products, RefWorks and EndNote.
Product | RefWorks | EndNote |
Company | Product of Ex Libris, a ProQuest company | Product of Clarivate (owns Web of Science) |
Availability | Web based | Desktop interface for Windows and Mac |
Word Processor | MS Word and Google Docs plugin | “Cite While You Write” download for Windows and Mac |
Features |
|
|
Important Links | RefWorks Account | EndNote Account |
Some citation managers may not be robust enough to handle a large evidence synthesis review or remove duplicate citations. Instead, you might need to use a tool specifically designed for systematic reviews. These tools incorporate automated features that help organize and speed up the review process.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=95#h5p-19
The Systematic Review Toolbox is a searchable collection of tools you could potentially use to develop your systematic review. Below is a sample of some free and paid tools designed for systematic reviews.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=95#h5p-20
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=95#h5p-21
Complete the following steps to locate the tools available to you:
Taking some time to pick a citation management tool that is right for you is an essential step in conducting a review. The right tool will help you maintain the integrity of your review and save you time.
26
Complete Activity 3.1: “Selecting Your Citation Management Tool” in your Research Roadmap Workbook.
Now that you have been presented with an in-depth list of available tools. Take a moment to document at least 2-3 tools that you are interested in testing.
For more information please see Workbook Instructions in the Introduction Module.
27
Screening is the process of identifying suitable sources from your literature search to be “full-text” screened and eventually included in the review. As mentioned in Module 2, your inclusion and exclusion criteria can help inform which limiters to apply to exclude irrelevant results, e.g., publication year or language.
If you are conducting a systematic review, it is recommended to screen the results outside of the databases, as there will be duplication of records. A citation software tool or systematic review tool will aid in removing duplicates. Within the citation management or systematic review tool, you can screen the titles and abstracts to determine if articles are relevant, based on inclusion and exclusion criteria that could not be captured in your search strategy. Afterwards, the full-text must be retrieved and screened to determine inclusion.
Screening Practice: In this activity, you are doing a review on the effectiveness of cognitive behavioural therapy in improving mild to moderate depression in teenagers. Your inclusion criteria includes:
Instructions:
First, screen the abstract to see if it fits your inclusion criteria. * Don’t click on the plus symbols yet.*
Then, when you are ready, click on the plus symbols to find out if this article fits with your inclusion criteria.
The original version of this chapter contained H5P content. You may want to remove or replace this element.
Abstract of Kroll, L., Harrington, R., Jayson, D., Fraser, J., & Gowers, S. (1996). Pilot study of continuation cognitive-behavioral therapy for major depression in adolescent psychiatric patients. Journal of the American Academy of Child and Adolescent Psychiatry, 35(9), 1156–1161. https://doi.org/10.1097/00004583-199609000-00013 This screen grab from PubMed , National Library of Medicine (NLM) is included on the basis of fair dealing.
28
Complete Activity 3.2: “Inclusion and Exclusion List” in your Research Roadmap Workbook.
This exercise allows you to document your criteria. Don’t worry if you are still trying to figure out some of the important details about your review as you can use this exercise to document what you have in mind.
For more information please see Workbook Instructions in the Introduction Module.
29
For systematic reviews, it is recommended that at least two reviewers screen the results independently. This is to help resolve disagreement by encouraging consensus and reduce the risk of biased decisions. When needed, a third reviewer can aid in resolving the deadlock.
For the first screening, each member of the team will independently scan the titles and abstracts of articles, and make a decision whether to include or exclude studies. Each reviewer should be asking themselves, “Is there enough information in the title and abstract to exclude this study at this stage?” To make sure both reviewers are on the same page, they will need to ensure they have a clear and pre-defined inclusion and exclusion criteria.
The next step is to read the full-text of the articles selected from the first round. Again, the reviewers will do this independently and use the same inclusion and exclusion criteria. This stage involves a more rigorous and in depth look at the articles. The reason for excluding a study must be recorded for reporting purposes. Reviewers can also look at the outcome of a study and determine if it is significant enough to include.
30
As noted in Module 2, you will be expected to document your search process and your selection process. The PRISMA flow diagram will help you map out the number of records identified, included and excluded, and the reasons for exclusions.
Save a copy of the PRISMA Flow Diagram for your own review.
Selection bias can happen consciously or unconsciously to a reviewer when selecting sources. Bias can appear in how and why we make decisions on the potential eligibility or ineligibility of a study. This is why using a team approach is one method to reduce bias in selecting sources for systematic reviews. Another is to make sure each reviewer has a clear understanding of the key concepts, or inclusion and exclusion criteria. For example, has the team checked to ensure everyone has the same understanding of the key concepts in their research question and what to include and exclude? This way, everyone goes into selecting with the same understanding.
Your group is planning on using the following criteria. What bias or misinterpretation issues might be present within each one?
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=105#h5p-22
Screening your citations and abstracts against your inclusion and exclusion criteria is the first step in identifying suitable studies. If you are conducting a systematic review, it is important to reduce selection bias by having a blind team based approach to screening.
31
Complete Activity 3.3: “Check for Team Understanding” in your Research Roadmap Workbook.
Are there any search terms or items on your inclusion/exclusion list that might have multiple interpretations? Write them down with your own interpretation and have your team members do the same activity. Compare your interpretations to check for team understanding.
For more information please see Workbook Instructions in the Introduction Module.
32
Plagiarism happens when you claim the words, ideas, or data of another person as if it is your own, without appropriate referencing. Plagiarism can range from copying an entire paper to changing only a few words and not citing the original source. It’s important to familiarize yourself with what constitutes plagiarism. One way you can do this is to consult your institution’s academic integrity policy so you know how to avoid a charge of plagiarism.
When it comes time to write your review, you will need to demonstrate that you understand when and how to cite your sources. Keeping them organized will help you with this task. By using a citation management tool, you will be able to access your sources in one place and pull the citation information needed to cite them correctly. Before sitting down to write your review, make sure you familiarize yourself with the rules of your citation style. While it is true that some citation management tools will offer you a plug-in for your word processor that seemingly does all the work, it can make mistakes. Getting to know the rules yourself will save you a lot of time and reduce the risk of plagiarism. Table 3.4 below provides a curated list of citation style resources to help you familiarize yourself with the rules.
Style | Resource |
---|---|
APA |
|
MLA |
|
Turabian (Chicago) |
|
Vancouver (also known as ICMJE) |
|
IEEE |
|
How much do you know about plagiarism? Take this quiz to find out.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=109#h5p-23
This quiz is adapted from the University of Guelph, Academic Integrity Plagiarism Quiz. Licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License.
Familiarize yourself with the academic integrity policies of your institution to better understand how plagiarism is defined and how to avoid it. Commit to learning the rules or conventions of your chosen citation style.
33
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=111#h5p-24
34
We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:
35
American Psychological Association (2020). Publication Manual of the American Psychological Association: The official guide to APA style (Seventh ed.). American Psychological Association.
Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.
Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.
IEEE (n.d). IEEE Reference Guide. https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf
International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. http://www.icmje.org/icmje-recommendations.pdf
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04
(see See section 4.6 Selecting Studies)
Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. (2021, February). Chapter 7: Considering bias and conflicts of interest among the included studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-07
Keenan, C. (2018, April 18). Assessing and addressing bias in systematic reviews. Meta-Evidence Blog, Campbell Collaboration, UK & Ireland. http://meta-evidence.co.uk/assessing-and-addressing-bias-in-systematic-reviews/
V
36
The purpose of this module is to teach you how to manage your reading workload efficiently, and how to identify the key points in the articles you are including in your review.
By the end of this module, you will be able to:
37
Generally speaking, there is a common flow to scholarly articles. While not a template per se, you can be assured that the following components will be present in most articles. Learning to identify each component is a key step in the strategic reading process, and will help you save time as you screen articles for relevance. Check out the interactive example below that describes each section.
Click on the purple question marks to learn more about each component of an academic article.
The original version of this chapter contained H5P content. You may want to remove or replace this element.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=120#h5p-25
Structure of an Academic Article by Emma Seston. Licenced under Creative Commons CC BY-NC 4.0.
Learning to identify each component is a key step in the strategic reading process, and will help you save time as you screen articles for relevance.
38
Now that you know about the component parts of scholarly articles, you are prepared to scan your search results and make swift judgments about the relevance of an article to your own context. Note: this step in the process does not require you to read the entire article! You will begin by scanning the articles and follow these steps:
Click on the purple question marks to learn more about each step in the process of scanning an academic article for relevance.
The original version of this chapter contained H5P content. You may want to remove or replace this element.
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=122#h5p-26
Initial Scan for Relevance by Emma Seston. Licenced under Creative Commons CC BY-NC 4.0.
For your first read you can scan the outcome/results or conclusion section and make swift judgments about the relevance of an article to your own context.
39
Now that you have scanned your search results and have amassed those that are relevant to your context and needs, you can settle in to do what we call critical reading. This is where you must plan to read the entire article thoroughly.
Critical reading encourages the reader to think about how an article or argument is constructed, not just what it says or what the study found. Here are some questions to ask yourself when reading:
Using a standardized tool for critical appraisal is a good practice to follow. It demonstrates consistency in evaluation of all articles, and when working on a research team, it ensures that each individual is asking the same questions.
There are numerous tools for appraisal to choose from. For the most part, there will be disciplinary norms where certain tools are chosen in order to best represent the specific context. Looking to other reviews in your field to see what tools are employed is a good step.
There are various tools such as checklists designed by research organizations that will help you determine if your source is appropriate. Table 4.1 below outlines a sample of free quality assessment tools available for you to use.
Tool | Organization | Description |
---|---|---|
CASP Checklists | Critical Appraisal Skills Programme (CASP) from the Public Health Resource Unit, NHS, England | Checklists for assessing the reliability, importance, and applicability of studies. Studies include qualitative studies, systematic reviews, randomized control trials and others. |
Critical Appraisal Tools | The Centre for Evidence-Based Medicine (CEBM) from the University of Oxford | A checklist with questions for assessing the reliability, importance, and applicability of studies based on the research question and inclusion criteria. |
Risk of Bias 2 (Rob 2) Tool | Cochrane Methods | A manual from the Cochrane Method Network that includes standards for assessing risk of bias in your included studies. |
Critical Appraisal Tools | Joanna Briggs Institute | Checklists assessing the trustworthiness, relevance and results of different types of studies. |
Study Quality Assessment Tools | National Heart, Lung and Blood Institute | Quality assessment tools for assessing potential flaws in study methods or implementation of different types of studies. |
To find the tool right for your review check out the Quality Assessment and Risk of Bias Tool Repository from Duke University’s Medical Center Library & Archives.
While selecting your sources, you will need to assess the quality of the research question, the methodology used, the validity of the results and whether the study is applicable to your own review. To help you assess the quality of your chosen sources, consult one of the free checklist tools suggested in Table 4.1.
40
Complete Activity 4.1: “Assessing the Quality of Your Sources” in your Research Roadmap Workbook.
Using a tool for critical appraisal is a good practice to follow. It will help you evaluate your sources and if you are working as a research team, it ensures that each individual is asking the same questions.
Take a moment to review some of the tools and document which ones you might use in your review.
For more information please see Workbook Instructions in the Introduction Module.
41
Scanning an academic article for relevancy is an important component of research. In this activity you will scan the article and identify key components such as the research question, methodology used and the conclusion. For your own research you can use this method for the first read of your sources to decide if they are relevant to your review. If they are relevant, you would do a second more thorough reading of your source. We have provided two examples; a qualitative study and a quantitative study. You are free to do both or choose one.
Use this example study to identify the following components.Katajavuori, N., Vehkalahti, K., & Asikainen, H. (2021). Promoting university students’ well-being and studying with an acceptance and commitment therapy (ACT)-based intervention. Current Psychology, 1-13 https://link.springer.com/article/10.1007/s12144-021-01837-x#Sec15
Instructions: After taking a guess at the instructions on the front of each card, turn the card to reveal the response. How did you do?
The original version of this chapter contained H5P content. You may want to remove or replace this element.
Read this example study to identify the following components.Hickey, J., Powling, H., McKinney, P., Robbins, T., Carrier, N., & Nash, A. (2020). "It’s a change your life kind of program”: A healing focused camping weekend for urban Indigenous families living in Fredericton, New Brunswick. First Peoples Child & Family Review, 15(2), 23-44. https://fpcfr.com/index.php/FPCFR/article/view/45
Instructions: After taking a guess at the instructions on the front of each card, turn the card to reveal the response. How did you do?
The original version of this chapter contained H5P content. You may want to remove or replace this element.
42
The next step after reading and evaluating your sources is to organize them in a way that will help you start the writing process.
One way to organize your literature is with a review matrix. The review matrix is a chart that sorts and categorizes the different arguments presented per topic or issue. Using a matrix enables you to quickly compare and contrast your sources in order to determine the scope of research across time. This will allow you to spot similarities and differences between sources. It is particularly useful in the synthesis and analysis stages of a review (See Module 1 Conducting a Literature Review with the SALSA Framework).
Example of a Review Matrix
My research question:
How can we use machine learning to analyze social media data related to HIV?
Sources | Methods | Concept 1 | Concept 2 | Concept 3 | Gaps, Problems, Questions, Notes |
Source 1:
Signorini, A., Segre, A. M., & Polgreen, P. M. (2011). The use of Twitter to track levels of disease activity and public concern in the U.S. during the influenza A H1N1 pandemic. PloS one, 6(5), e19467. |
Collected and stored a large sample of public tweets that matched a set of pre-specified search terms and geocoded. Estimated rate of disease and public sentiment toward swine flu | Able to make predictions about swine flu using social media data. This data is vital given that “an influenza surveillance program does not exist” (p. 3) | “When and where tweets are less frequent (or where only a subset of tweets contain geographic information), the performance of our model may suffer.” | ||
Source 2:
Chiu, C. J., Menacho, L., Fisher, C., & Young, S. D. (2015). Ethics issues in social media–based HIV prevention in low-and middle-income countries. Cambridge Quarterly of Healthcare Ethics, 24(3), 303-310. |
Quantitative survey assessing participants’ perspectives on educational intervention | Increasing social media use in low- and middle-income countries. Participant took part in an HIV education program on Facebook | Most participants felt like they benefited positively from the program and learned about HIV prevention. All participants were MSM | Note: Helpful article for including diverse perspectives | |
Source 3:
Bollen, J., Mao, H., & Zeng, X. (2011). Twitter mood predicts the stock market. Journal of computational science, 2(1), 1-8 |
Collected public tweets and analyzed mood | Gathered data from Twitter posts that explicitly states moods (e.g. “I’m feeling…”). Found that positive/negative sentiment on Twitter is 87.6% accurate for predicting stock market average | Used a “Self-Organizing Fuzzy Neural Network” to predict Dow Jones Industrial Average (p. 1) |
Writing a Literature Review Modified from The WI+RE Team, UCLA. Creative Commons CC-BY-NA-SA
Start with a charting tool you are most familiar with (for example MS Word, MS Excel, Google Sheets, Numbers etc).
Key Takeaways
Here are some examples of different review matrices and templates:
43
An interactive H5P element has been excluded from this version of the text. You can view it online here:
https://ecampusontario.pressbooks.pub/tmuh5ptest/?p=132#h5p-27
44
We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:
45
Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246. https://doi.org/10.1136/bmj.315.7102.243
Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012, https://miriamsweeney.net/2012/06/20/readforgradschool/.
Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308. https://doi.org/10.1136/bmj.315.7103.305
Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743. https://doi.org/10.1136/bmj.315.7110.740
Evidence Synthesis Matrix Template, Jane Schmidt, Toronto Metropolitan University (Google Sheets)
The Matrix Method for Literature Reviews, Brandeis University, Writing Resources.
Literature Review Synthesis Matrix, Concordia University (MS Word)
Write a Literature Review: Synthesize. Johns Hopkins University, Sheridan Libraries
Systematic Review Guide: Critical Appraisal (St. Michael’s Hospital Health Science Library)
Systematic Reviews: Quality Assessment (University of North Carolina Health Science Library)
VI
46
Congratulations on completing this short course on conducting an academic review! As you start to work through your review, we hope you notice that you are part of a long line of researchers adding their voice to the scholarly discourse of your topic. Your work is contributing to this discourse. As a researcher, you are an active member of creating new knowledge and not just a consumer of it.
We hope you have come away from this short course with an understanding that there is value in using your intellectual curiosity in learning new investigative methods. Finally, as a researcher, it is normal to encounter challenges and doubts when conducting research, but this can be overcome with persistence, adaptability, and flexibility.
Below are the learning outcomes for this course that we shared with you in the Introduction section. Please take some time to review the list and see how many outcomes you achieved.
By completing this short course, you should now be able to:
47
Aveyard, H. (2019). Doing a literature review in health and social care: A practical guide (Fourth ed.) Open University Press, McGraw-Hill Education.
Jesson, J., Matheson, L., & Lacey, F. M. (2011). Doing your literature review: Traditional and systematic techniques. SAGE.
Ridley, D., Dr. (2008). The literature review: A step-by-step guide for students. SAGE.
Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting research. SAGE.
DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education. https://scientificinquiryinsocialwork.pressbooks.com/
Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617
The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube. https://www.youtube.com/watch?v=uyF8uQY9wys
UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube https://www.youtube.com/watch?v=6QQ0MW_jXfM
Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246. https://doi.org/10.1136/bmj.315.7102.243
Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012, https://miriamsweeney.net/2012/06/20/readforgradschool/.
Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308. https://doi.org/10.1136/bmj.315.7103.305
Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743. https://doi.org/10.1136/bmj.315.7110.740
Allen, J. (2019). The productive graduate student writer: How to manage your time, process, and energy to write your research proposal, thesis, and dissertation, and get published (First ed.). Stylus Publishing, LLC.
Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617
Feak, C. B., & Swales, J. M. (2009). Telling a research story: Writing a literature review. University of Michigan Press.
Holland, K., & Watson, R. (2021). Writing for publication in nursing and healthcare: Getting it right (Second;2; ed.). John Wiley & Sons Ltd.
Walliman, N. (2006). Writing a literature review. In Social research methods (pp. 182-185). SAGE Publications, Ltd,
American Psychological Association (2020). Publication Manual of the American Psychological Association: The official guide to APA style (Seventh ed.). American Psychological Association.
Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.
Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.
IEEE (n.d). IEEE Reference Guide. https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf
International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. http://www.icmje.org/icmje-recommendations.pdf
48
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. doi:10.1080/1364557032000119616
Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. SAGE.
Ganann, R., Ciliska, D., & Thomas, H. (2010). Expediting systematic reviews: Methods and implications of rapid reviews. Implementation Science, 5(1), 56-56. doi:10.1186/1748-5908-5-56
Glass, G. (1976). Primary, Secondary, and Meta-Analysis of Research. Educational Researcher, 5(10), 3-8. doi:10.2307/1174772
Gough, D., & Richardson, M. (2018). Systematic reviews. In Advanced research methods for applied psychology (pp. 63-75). Routledge.
Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & Libraries Journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). (2021, February). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. www.training.cochrane.org/handbook.
The following guide provides a list of online self-directed learning on how to conduct a systematic review
Online Courses on Systematic Reviews (St. Michael’s Hospital Health Science Library)
Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting research. SAGE.
Boland, A., Cherry, M. G., & Dickson, R. (2017). Chapter 3 in Doing a systematic review: A student’s guide (Second ed.). SAGE.
DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education. https://scientificinquiryinsocialwork.pressbooks.com/
Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234. https://doi.org/10.1080/10511253.2012.730617
Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. (2021, February). Chapter 2: Determining the scope of the review and the questions it will address. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-02
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies.In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04
Sayers, A. (2008). Tips and tricks in performing a systematic review. British Journal of General Practice, 58(547), 136-136. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2151802/
CADTH Search Filters Database: https://searchfilters.cadth.ca/
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-04
(see See section 4.6 Selecting Studies)
Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246. https://doi.org/10.1136/bmj.315.7102.243
Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012, https://miriamsweeney.net/2012/06/20/readforgradschool/.
Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. (2021, February). Chapter 7: Considering bias and conflicts of interest among the included studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook/current/chapter-07
Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308. https://doi.org/10.1136/bmj.315.7103.305
Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743. https://doi.org/10.1136/bmj.315.7110.740
Keenan, C. (2018, April 18). Assessing and addressing bias in systematic reviews. Meta-Evidence Blog, Campbell Collaboration, UK & Ireland. http://meta-evidence.co.uk/assessing-and-addressing-bias-in-systematic-reviews/
Card, N. A., & Little, T. D. (2012). Applied meta-analysis for social science research. Guilford Press.
Cheung, M. W. -., & Vijayakumar, R. (2016). A guide to conducting a meta-analysis. Neuropsychology Review, 26(2), 121-128. https://doi.org/10.1007/s11065-016-9319-z
Li T, Higgins JPT, Deeks JJ (editors). Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. https://training.cochrane.org/handbook/current/chapter-05
Allen, J. (2019). The productive graduate student writer: How to manage your time, process, and energy to write your research proposal, thesis, and dissertation, and get published (First ed.). Stylus Publishing, LLC.
Holland, K., & Watson, R. (2021). Writing for publication in nursing and healthcare: Getting it right (Second;2; ed.). John Wiley & Sons Ltd.
Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. SAGE.
American Psychological Association (2020). Publication Manual of the American Psychological Association: The official guide to APA style (Seventh ed.). American Psychological Association.
Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.
Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.
IEEE (n.d). IEEE Reference Guide. https://ieeeauthorcenter.ieee.org/wp-content/uploads/IEEE-Reference-Guide.pdf
International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. http://www.icmje.org/icmje-recommendations.pdf