Complete Activity 0: “Review Your Research Skills” in your Research Roadmap Workbook.

This survey will help you reflect on which research skills you have and what you need to work on. You can return to this document as you continue through your review to reflect on the progress you have made.

For more information please see Workbook Instructions in the Introduction Module.

This section is a quick summary of the main steps involved in conducting systematic reviews. By the end of this section you should have a better idea of the time and resources needed to conduct a successful review.

All reviews follow a familiar process as seen in Figure 1.1 below.

Figure 1.1 What Authors Do: Systematic Reviews. Designed by Jessica Kaufman, Cochrane Consumers & Communication Review Group, Centre for Health Communication & Participation, La Trobe University, 2011. Licensed under Creative Commons CC BY 4.0.

Find Existing Systematic Reviews

Prior to starting your own research, you will want to look at existing systematic reviews – this is especially important so that you don’t duplicate existing work. It can also be helpful to look at the approaches taken for systematic reviews similar to your own topic or discipline.  You can find existing systematic reviews through a number of ways:

1. Search published journal articles. Systematic reviews can be published as journal articles. To identify them, add “systematic review” as an additional search term in databases, or look for publication type limits, if available. Here’s an example of a “systematic review” published as a journal article.
2. Search “Cochrane Database of Systematic Reviews” through your library. This database includes the full text of the regularly updated systematic reviews of the effects of healthcare prepared by The Cochrane Collaboration.
3. Search different protocol registries.  For example, PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome. PROSPERO aims to provide a comprehensive listing of systematic reviews registered at inception to help avoid duplication and reduce opportunity for reporting bias by enabling comparison of completed reviews with what was planned in the protocol.
4. Search the Campbell Collaboration. The Campbell Collaboration is an international network which publishes high quality systematic reviews of social and economic interventions around the world.

Assembling Your Research Team

If you are conducting a systematic review that requires a team these are the typical roles involved:

• Reviewers – You may need at least two reviewers working independently to screen abstracts, with a potential third as a tie-breaker
• Subject matter experts – Subject matter experts can clarify issues related to the topic,
• Statistician – A statistician can help with data analysis
• Project leader – A project leader can coordinate and write the final report
• Librarians – Librarian(s) can develop comprehensive search strategies and identify appropriate databases

Formulate Your Research Question

In general, your research question will tackle the problem you are trying to address by conducting the review. Since constructing a research question can be an in-depth process,  we go over it in more detail in Module 2: Formulating a Research Question and Searching for Sources.

Create a Review Protocol

Reviews like a systematic review require a protocol, which is essentially a planning document that indicates how your review will be carried out. Here is a sample protocol template from the Evidence Synthesis Coordinator at the Maritimes Strategy for Patient-Oriented Research (SPOR) Support Unit. This basic form includes all the relevant information needed for a simple protocol.

You may wish to register your protocol to avoid the duplication of work and to reduce the potential for bias by enabling a comparison between what was stated in the protocol to the completed review. It is also a way to share your current research interests with the research community at large, and help build your research profile.

Conducting Your Review Using the SALSA Framework

Once you have a research question, there are four stages you can follow when conducting your chosen review. These are known as the SALSA Framework: search, appraisal, synthesis and analysis.

Wait, What Happened to the “L” in SALSA?

Did you notice the missing L? We did too! The authors, Grant and Booth (2009) created a simple analytical framework for conducting reviews: Search, Appraisal, Synthesis and Analysis. SASA, however, doesn’t make a memorable acronym, and Academics love a good acronym, so they derived the “L” from the last letter of appraisal:  Search, AppraisaL, Synthesis and Analysis (SALSA).

PRISMA: The Systematic Review Checklist

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA is the recognized standard for reporting evidence in systematic reviews and meta-analyses. The standards are endorsed by organizations and journals in the health sciences. It allows other researchers to assess strengths and weaknesses of the review and assists with future replication of the review methods. The 2020 PRISMA statement consists of a 27-item checklist and a 4-phase flow diagram.

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & Libraries Journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Conducting a review requires good time management. Feel free to use the To Do List: Research Stage to keep track of your timeline and tasks for the research stage of your review.

For more information please see Workbook Instructions in the Introduction Module.

SFU Continuing Studies students at work in the classroom. By Flickr user Simon Fraser University  – Communications & Marketing under Creative Commons CC BY-ND 2.0

In this module we go over the framework for searching the literature. Keep in mind that searching is a non-linear and potentially iterative process; as such you may need to review and revise earlier steps at any point.

The steps to searching the literature are:

1. Formulating a research question
2. Identify search terms (keywords)
3. Brainstorm more search terms
4. Construct a search strategy
5. Select and Search Databases
6. Manage your search strategies and results (Covered in Module 3: Organizing, Managing and Screening Sources)

We will show you how to use special techniques like forward and backward citation searching to find more sources. Finally, if you are conducting a systematic review you should check out our special section on how to document your search strategy and results.

Need a Refresher on Basic Searching Skills?

Check out these guides:

• Searching for Sources (Research Skills Tutorial). Toronto Metropolitan University Library
• Library Research Skills Tutorial. McMaster University Library

As noted in Module 1: Types of Reviews, conducting a “pre-search” is a crucial first step in devising the research question. A well-formulated research question informs the research process. It can focus your information needs (i.e. identify inclusion and exclusion criteria), help to identify key search concepts, and guide you in the direction of relevant resources.

Types of Research Questions

There are two general types of research questions: quantitative and qualitative.

Quantitative Research Questions

Types of quantitative questions can be categorized as explanatory (i.e., relationship-based), descriptive, or comparative.

• Explanatory questions aim to discover cause-and-effect relationships by comparing two or more variables, individuals or groups based on differing outcomes.
• Descriptive questions will often quantify a single variable but may include multiple variables  within a question. They typically ask for measurements, and can begin with: “how much”, “what percentage”, “how frequently”, or “how many”.
• Comparative questions are designed to identify the “difference between” a dependent variable and two or more groups. These questions tend to begin with “what is the difference” or “what are the differences”.

Qualitative Research Questions

Qualitative questions aim to discover meaning or gain an understanding of a phenomenon. They ask questions that cannot be measured with specific numbers and statistics. Qualitative research questions often contain words like "lived experience", “personal experience”, “understanding”, “meaning”, and “stories”.

A Comparison of Quantitative and Qualitative Questions

So how do quantitative and qualitative research questions differ when you are conducting a search? In Table 2.1 below, we provide some examples of research topics. Each topic can either be used for a quantitative or qualitative research question. For each question, the category of research question is clarified.

Formulas

Using a formula is another way to construct a research question and is recommended if you are conducting a systematic review. There are various formulas you can use to craft your question, see Table 2.2 below.

These frameworks aid in identifying the important parts (i.e., concepts) that can be used in formulating a research question. Below are two examples using the PS and PEO frameworks.

Is My Question Too Narrow or Too Broad?

When trying to settle on a research question, ensure you are choosing something of interest, but not so narrow that you are unlikely to uncover any information, but not too broad so that you are overwhelmed with content and don’t know where to start. Below are examples of too broad and too narrow questions and an explanation to help understand why.

A pre-search will assist you with this step. If you need a refresher on conducting a pre-search, see Module 1: Types of Reviews. Once you have a good understanding of the existing literature, you will be able to confidently formulate your research question.

Identifying keywords, (also known as search terms) is important for effective literature searching. ​Your search terms are terms that will appear somewhere within the resource (e.g. title, abstract, or author keywords).

You can identify search terms from your research question by highlighting, underlining or circling the main ideas that must appear in the article. Your search concepts are the most important words in your research question.

Keywords are easier to identify once the research question has been formulated using one of the stated frameworks (e.g. PICO or PS) listed in Table 2.2 of Formulating a Research Question.

Below is an example of how to use the PICO(T) framework to identify search concepts for a specific research question.

Frandsen, T. F., Nielsen, M. F. B., Lindhardt, C. L., & Eriksen, M. B. (2020). Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements. Journal of Clinical Epidemiology, 127, 69-75.

Your search strategy can contain your main keywords, similar or related keywords and controlled terms (also called subject headings).

Searching by a keyword will retrieve resources where the author(s) used that specific term. For this specific reason, you should also brainstorm similar or related keywords to incorporate into your search.

Methods for Identifying More Keywords

You can identify more keywords in multiple ways. Below are two methods.

Method 1: Use a Concept Model or Map

To use this method:

1. Write your research topic or question, along with any ideas and concepts associated with it on a blank sheet of paper.
2. Use themes to group your ideas, and connect related concepts using lines.
3. Remember to include the following:
   • Synonyms
   • Acronyms
   • Brand names and generic names
   • Variation in spelling (e.g. “paediatric” or “pediatric”)

Please see Figure 2.1 below for an example where your research question is “How effective is cognitive behavioral therapy in improving mild-to-moderate depression in adolescents?”

Figure 2.1. Example of a “Concept Map” to generate more keywords. Source: Jo-Ann Petropoules. Licensed under Creative Commons CC BY 4.0

Method 2: Use Your Main Keywords in a Database

To use this method:

1. Locate an article on your topic.
2. Scan the title, abstract, and author keywords to identify more keywords (see Figure 2.2 below) to use in your search.

Figure 2.2. Example of how to find keywords in a journal article. From Stanzani-Maserati, M., De Matteis, M., Bosco, L., Baccari, F., Zenesini, C., Mitolo, M., La Morgia, C., Gallassi, R., & Capellari, S. (2024). “Build Your Village”-Conducting the Village Test on Cognitively Impaired Patients: A First Journey into Alzheimerland. Brain sciences, 14(6), 523. https://doi.org/10.3390/brainsci14060523. Licensed under a Creative Commons CC BY 4.0.

Controlled vocabularies are a standardized set of terms used to describe the content of a resource in a database. This is known as indexing. Using controlled vocabulary terms will usually generate fewer and more relevant results. However, you must know the exact term/vocabulary to use. An example is provided below.

When to Use Controlled Vocabularies

The following are situations in which you would use controlled vocabularies:

1. If your keyword searching is giving you too many or too few results, try using controlled vocabulary terms.
2. If you are doing a comprehensive search (e.g., a systematic review), you should combine both keywords and controlled vocabulary terms to increase the number of resources retrieved.

How to Use Controlled Vocabularies

Search strategies must be customized for each database for a number of reasons, one of which is that they use different controlled vocabularies. The tutorials in Table 2.3 below cover how to use controlled vocabularies in specific databases. Review these tutorials if you’d like to learn more.

How to Find More Controlled Vocabulary

Below are the steps to follow when finding more controlled vocabularies:

1. Search for each keyword in a database separately. This will allow you to identify additional controlled vocabulary. For example, the database PubMed allows you to search for MeSH headings one at a time.
   • The CADTH Search Filters Database can also help you look up the MeSH heading across databases such as Scopus, CINAHL, PubMed, PsycInfo, MEDLINE and Embase.
2. Find an article on your topic and look at the controlled vocabulary assigned to it. For example, in PubMed, look for the heading “MeSH” (the name for the controlled vocabulary in PubMed) under the heading of “Cited by.” See Figure 2.3 below for an example.
3. In other databases, such as Social Sciences Abstracts, find an article on your topic and look for the heading “Subjects” to help identify relevant controlled vocabularies. 

Figure 2.3. MeSH terms in an article in PubMed. From Vittorini, M. G., Sahin, A., Trojan, A., Yusifli, S., Alashvili, T., Bonifácio, G. V., Paposhvili, K., Tischler, V., Lampl, C., Sacco, S., & School of Advanced Studies of the European Headache Federation (EHF-SAS) (2024). The glymphatic system in migraine and other headaches. The journal of headache and pain, 25(1), 34.

The standard method of constructing a search strategy is to use Boolean logic. Boolean relies on three essential operators, AND, OR and NOT. Using these operators allows you to combine words and phrases using the words AND, OR, NOT to limit, broaden, or define your search. A good researcher should know how to do a Boolean Search.

Boolean Operators

Boolean operators connect your search concepts and search terms (i.e., controlled vocabulary and keywords) together. The three basic operators are: OR, AND, and NOT. An example is shown below, followed by a deeper dive into each of these operators.

The OR Operator

The OR operator broadens your search results by retrieving records that contain either or both of your search terms or concepts. Usually, terms within a concept are combined using the Boolean OR operator.

In the Figure 2.4 below, the database will retrieve records that contain either or both of the terms Cognitive Behavioural Therapy OR CBT.

Figure 2.4. Example of the “OR” operator in Boolean.

The AND Operator

The AND operator narrows the search results by retrieving records that contain both of your search terms or concepts. Usually, different concepts are combined using the Boolean AND operator.

In the Figure 2.5 below, the database will retrieve sources that contain Cognitive Behavioural Therapy AND Depression

Figure 2.5. Example of the “AND” operator in Boolean.

NOT

The NOT operator narrows the search results by eliminating a specific search term. You should use the NOT operator with caution. This is because it can eliminate relevant results that happen to mention the term you are “noting” out. In Figure 2.6 below, any articles that mention both adult and adolescent will be excluded from the result set.

Figure 2.6. Example of the “NOT” operator in Boolean.

Using Truncation and Wildcards

Truncation and wildcards can be applied to a keyword search to broaden your results and allow you to look for variations of words.

Truncation

You can use the truncation symbol to avoid typing out all possible variations of a word. For example, surg* will retrieve surgery, surgeries, surgeon, or surgical. The truncation symbol should be used with caution to ensure relevant words are being retrieved.

The most common symbol used is an asterisk (*). The asterisk symbol is used in the following platforms and databases:

• EBSCO (e.g., CINAHL, Business Source Complete)
• Ovid (e.g., MEDLINE, Embase, PsycInfo)
• ProQuest
• ERI,
• PubMed
• Web of Science

Wildcard Symbols

Wildcard symbols can be used to substitute for one or more characters, or any single character. This is useful when dealing with variant spelling (e.g., pediatric and paediatric). One of the most common symbols used is the question mark (?). For example, in the Ovid database, p?ediatric* will retrieve pediatric OR paediatric OR pediatrics OR paediatrics.

Databases use different wildcard symbols, so check the database Help information or Search Tips for details about which symbol to use.

When conducting a review, it is important to search multiple resources. Just searching one subject database will not ensure you have found all the essential sources on your topic.

For systematic reviews, searching multiple databases is essential for reducing bias in your results. Only using one database like PubMed means you have ignored relevant studies found in other databases like Embase.

Subject Databases

Subject databases are accessible through your academic library and contain published material belonging or pertaining to a specific discipline or multiple disciplines, e.g.,health sciences, business, social work, or law. Many subject databases are hosted on the platforms EBSCO and Proquest for example. These search platforms allow users to search and retrieve content, and may look and work similarly to each other. The type of material in different subject databases depends on the subject, but most will include academic peer-reviewed articles, newspaper and magazine articles, and reports.

To find a subject specific database, go to your library’s website and look for a research guide for your discipline. This guide will include a list of article databases for your discipline, and the descriptions below them will help you select among them. Alternatively, look for a list of databases by subject on your library website, and choose an article database which covers your subject.

If you have questions about which database to choose, contact your librarian for help.

How to Search in Subject Databases

In general, all subject databases will have a basic search box and an advanced search box. We recommend you use the advanced search box, as it provides more options to optimize your search. The advanced search box will often allow you to choose where you ​want your keywords to appear within the resource (e.g., in the title or the abstract and body or anywhere). An example of an advanced search box is found in Figure 2.7 below.

Figure 2.7. Example of an advanced search box in PubMed. This screen grab, from PubMed National Library of Medicine (NLM) is included on the basis of fair dealing.

Limiters are found in all subject databases and they allow you to limit your search by date, type of source, language, and in some databases by population and methodology (see Figure 2.8 below). This is particularly useful if you are conducting a systematic review, as the limiters will help with your exclusion criteria.

Figure 2.8. Example of date range limits in PubMed. This screen grab, from PubMed National Library of Medicine (NLM) is included on the basis of fair dealing. 

Once you have chosen your subject databases, take a note if they are owned by a particular platform (i.e., EBSCO, ProQuest, Ovid, etc). Each of these platforms will have tutorials on how to conduct an advanced search. Below are video tutorials from some of the major subject database platforms.

Google Scholar

Google Scholar searches articles, theses, books, abstracts and court opinions, from academic publishers, professional societies, online repositories, universities and other web sites.

The majority of sources will be behind pay-walls and you will not be able to access them. However, you can optimize Google Scholar to search your academic library’s collection, at which point you will be able to access any item your library owns. For a list of the pros and cons of using Google Scholar please see Table 2.6.

From McMaster University, Health Science Library, Google Scholar. Licensed under Creative Commons CC BY-NC-SA 4.0.

How to Search Google Scholar

The following video demonstrates how you can search Google Scholar.

UTS Library, Google Scholar: Advanced Searching [2:53]. Licensed under a Creative Commons CC BY 4.0. Closed Captions

Grey Literature and Other Sources

Grey literature refers to any information that is not commercially published, and can include a wide range of material types, such as:

• Theses and dissertations, e.g., databases such as Proquest Dissertations or Theses Canada
• Government documents and websites
• Clinical trials registries
• Conference proceedings, e.g.,  databases such as Scopus or Web of Science include citations to conference proceedings
• Research reports

How to Search Grey Literature

Knowing where to search for grey literature can be challenging. There are a variety of suggestions in the following library guide from Simon Fraser University: Grey literature: What it is & how to find it.

A good approach is to be strategic and consider where appropriate grey sources might be located. For example, if you know of any government agencies, non-profits, professional associations, research institutes, and other organisations that may be producing content relevant to your topic, you may wish to search their website for any publications and reports. At a very minimum, looking at graduate theses and dissertations and conference proceedings ( often sites of emerging research) should be a component of your grey literature search.

Searching for grey literature will likely involve the use of a search engine such as Google, so it will be important for you to critically evaluate sources before including them in your research.  For some tips on avoiding fake news and the spread of misinformation, review the following guide from Toronto Metropolitan University Library: How to Identify Fake News.

Replicating Your Search Strategy Across Different Databases

Your search strategy will need to be “translated” for use in multiple databases and you will have to look out for the following:

• Controlled vocabulary can vary across databases, e.g., MEDLINE uses MeSH while Embase uses Emtree
• Search options, limiting options, and availability of search fields will differ between databases

McMaster University Library. (2024, June 6). Translate a Search Strategy. How to Search the Literature (Advanced).

Besides your standard search there are additional options to identify relevant sources including searching key journals in your field, and tracking the work of relevant authors.

Another method used by researchers is to consult the citations within an article. By consulting the citations you can find relevant titles that you might have missed in your search. The advantage of this method is that you can find sources about a subject quickly and relatively easily. The disadvantage of this method is that you are searching backward, or retrospectively, so each source you find will be older than the previous one.

This is why you need to use a forward citation searching method in combination with your backward citation search. In the forward searching method, researchers use tools within databases like Google Scholar and Web of Science to see who has recently cited your source. This way you can find new and relevant sources.

Method 1: Backward Searching

In this method, review the references used in key articles. For example, search through the article for the “References” section as in Figure 2.9 below.

Figure 2.9. Example of backward citation searching in PubMed by looking at the “References” section of an article. This screen grab from PubMed, National Library of Medicine (NLM) is included on the basis of fair dealing.

Method 2: Forward Searching

In this method, see what articles have cited your key articles. For example, search through the “Cited by” section of the article as in Figure 2.10 below.

Figure 2.10. Example of forward citation searching in PubMed. This screen grab from PubMed , National Library of Medicine (NLM) is included on the basis of fair dealing.

Many databases will include live links to an article’s references, as well as live links to other articles that have cited that article. The following video demonstrates how to do citation searching in Google Scholar:

CSUDH Library. Citation Tracking. Licenced under Creative Commons CC BY 4.0.  Closed Captioned

Citation Mapping Tools

If you’ve ever clicked on a link titled something like “suggested for you” or “you may also like” on your favourite online store, you were taking advantage of an algorithm that uses your interests to identify related items. Algorithms search for metadata to collate similar items. This is similar to clicking a hashtag on social media sites like Instagram.

Citation mapping tools are an academic version of this process. Not only do these tools look for items that have similar metadata, they also leverage the world of bibliometric data to create a visual representation of the world that surrounds a single article. These tools require you to enter a citation (often referred to as a “seed paper”) you are interested in further exploring. The results will vary depending on the tool you select, but in general, you will be led to further works related to the seed paper either by the similarity in their bibliographies or adjacency to similarly cited works.

When should you use a citation mapping tool?

It may be very tempting to want to rely on a tool that works quickly, and uncovers potentially new articles. However, be cautious when deciding how to make use of a citation mapping tool. The majority of these tools are using algorithms that are not transparent in their composition and operation. This is problematic if you are engaged in a systematic review where replicability of the search is essential to the methodology. Moreover, over-reliance on AI tools when engaging in an academic exercise is a slippery slope. One must always keep in mind that the entire purpose of a literature review is to gain a deeper understanding of the existing literature and thought in your field. AI tools can be helpful to enhance productivity; they are not a replacement for critical or in-depth thinking about information.

Secondly, there are existing tools and methods that are thoroughly documented and have been used for decades. The novelty of an AI tool should not replace the use of existing approaches to forward and backward citation searching (also known as citation chasing) that can be replicated, and are widely acknowledged as standard in evidence synthesis. The old adage of “if it ain’t broke, don’t fix it” is applicable here. Using traditional citation indexes such as Scopus or Web of Science should be sufficient for reliable citation chasing.

Finally, it can be very tempting to use a tool that you feel will save you time. However, research should not feel like a race, and skimming your way through a literature review is a mistake. The devil, they say, is in the details. Poor reading habits such as over-reliance on abstracts or machine-generated summaries can lead to misreadings of studies in your review. Using an abstract or summary to determine relevance is appropriate; relying solely on them for interpretation of a studies’ purpose, methods, findings and conclusions is a recipe for misinterpretation. This can snowball and lead to embarrassing errors, and worse, charges of dishonesty.

Image source: França, T. F., & Monserrat, J. M. (2019). To read more papers, or to read papers better? A crucial point for the reproducibility crisis. BioEssays, 41(1), 1800206.

If, however, you are not engaged in a proper systematic review or meta-synthesis, and instead are working in the area of qualitative evidence synthesis (QES), or are engaged in a wide-ranging scoping search, you may want to consider using a citation mapping tool to expand the pool of sources as a supplementary search method. According to the Cochrane Handbook for Systematic Reviews of Interventions, “poor indexing of qualitative studies makes citation searching (forward and backward) and the Related Articles features of electronic databases particularly useful”(Harris et al., 2018). Furthermore, methodological studies have found that supplementary search methods (i.e. those methods undertaken outside of database searching), including citation chasing, is beneficial in identifying unique studies, especially in the social sciences (Cooper et al., 2017).

In short, using a citation mapping tool for discovery when supplementary search strategies are called for can be immensely useful. As a tool to enhance productivity, however, it should be given careful thought and not be solely relied upon, particularly given the mysterious nature of its functionality.

How to select a citation mapping tool?

It is challenging to recommend any one tool. Much will depend on your discipline, the goal of your research, or even just personal preference. Many tools are open access, others require users to create a user account, and still others are subscription based. Aaron Tay, a librarian from Singapore Management University, maintains a comprehensive list of currently available citation mapping tools on his blog. The best way to determine which tool is right for you is to go ahead and experiment with them! Select an article with which you are very familiar and use it as your “seed paper” in the tools you are keen to try.

Cooper, C., Booth, A., Britten, N., & Garside, R. (2017). A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: A methodological review. Systematic Reviews, 6(1), 234.

Harris, J. L., Booth, A., Cargo, M., Hannes, K., Harden, A., Flemming, K., Garside, R., Pantoja, T., Thomas, J., & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: Methods for question formulation, searching, and protocol development for qualitative evidence synthesis. Journal of Clinical Epidemiology, 97, 39–48.

If this is your first review, it’s normal to feel overwhelmed and unsure when to stop the searching process. Afterall, you don’t want to miss anything that’s possibly relevant, so the searching process never really seems to come to an end!

Unfortunately, we don’t have a firm easy answer like “if you find 10 articles you can stop.” There is always a chance that an article might slip past your extensive searching, but if you have done the steps below, then the chances of a really important article slipping past you is pretty slim.

Tips on When to Stop Searching

1. 1. Determine that your search strategy is working. You’ll know it is working when your search results include major studies/sources that have been recommended or, after a quick analysis of a few articles, you can tell they meet your criteria.
   2. You have searched all relevant databases.
   3. You used the citation searching techniques (See previous section). When using these methods you should see studies that you’ve already found in your database search and some new ones.
   4. You decided to add in some new search terms to a database search strategy and it yielded no new sources.
   5. You decided to remove a search term and the major studies you’ve found in previous searches have disappeared on the results page. 

Your search process must be documented in enough detail to ensure that it can be reported correctly in the systematic review and reproduced for verification. During your search process you will have to keep track of:

• Each database and platform (PsycINFO, PubMed, Medline etc) searched
• Date each search was conducted
• Subject headings and keywords used (including whether terms were exploded, truncated, etc)
• Number of results retrieved for each search within each database
• Total number of records over all
• Number of duplicates identified
• Numbers pre-screening and post-screening

Documentation Templates

In Module 1 we learned that PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. The PRISMA Group has developed an extension to PRISMA to assist researchers in documenting their literature searches for systematic reviews. The checklist identifies what to document in terms of information sources and methods, search strategies, peer review, and records management. The PRISMA flow diagram demonstrates how to document your final numbers.

Librarians are important resources during the review process, and their contributions can take many forms. It is recommended that you explore what options are available at your institutional library prior to starting your review. Libraries may have a formal service for supporting systematic review projects, which could include a more standardized level of support.  Regardless of the level of support available, the advice and support of a librarian, particularly at the outset of a review project, is highly recommended, and supported by best practices.

There are a number of service models in existence, but the most common approaches are the Consultant Model or the Co-Authorship Model.

In the Consultant Model, the Library provides advice and support to a review project through an in-depth consultation, or a series of consultations. This model provides a time-limited level of support and advice, which could include:

• Recommendations on relevant databases and grey literature sources
• Advice on developing search strategies
• Guidance on the use of citation management tools or systematic review software

Where extensive support has been provided, it is common to ask the librarian how they would like their contributions to be acknowledged. This would typically include a written acknowledgement of their contribution in the published version. See an example:

Inglis, S. C., Clark, R. A., McAlister, F. A., Stewart, S., & Cleland, J. G. (2011). Which components of heart failure programmes are effective? A systematic review and meta-analysis of the outcomes of structured telephone support or telemonitoring as the primary component of chronic heart failure management in 8323 patients: Abridged Cochrane Review. European journal of heart failure, 13(9), 1028–1040.

In the Co-Authorship Model, a librarian joins your review project team as a co-author. This model provides a more substantial level of support to the project, and could include:

• Selection of databases and grey literature resources
• Developing the search strategy
• Translating the search strategy across all databases
• Exporting results to a citation management tool or systematic review software
• Writing the methods section specific to searching

See an example:

Stotz, S. A., McNealy, K., Begay, R. L., DeSanto, K., Manson, S. M., & Moore, K. R. (2021). Multi-level Diabetes Prevention and Treatment Interventions for Native People in the USA and Canada: a Scoping Review. Current diabetes reports, 21(11), 46.

Complete Activity 2.1: “Create Your Own Search Strategy” in your Research Roadmap Workbook

This exercise allows you to document  your search strategy.  Don’t worry if you are still trying to figure out some of the important details about your review as you can use this exercise to document what you have in mind.

For more information please see Workbook Instructions in the Introduction Module.

We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:

• Your research question can be qualitative or quantitative.
• A formula like PICO(T) can assist you in creating your question and identifying your key search terms.
• Your search terms can include related words and the controlled vocabulary of the subject database.
• Using Boolean operators will help you put your search terms together in a coherent search strategy.
• Subject databases have limiters that will help you filter your search results.
• Use search techniques like citation searching to make sure you have found relevant sources.
• If you have a working search strategy, have searched all your identified resources and used citation searching techniques, you can stop searching and start analyzing.
• You need to document your search process.
•  In some cases, more in-depth assistance is required and you can book a research consultation with your subject librarian to review your initial search.

Constructing a Question

Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting

research. SAGE.

Boland, A., Cherry, M. G., & Dickson, R. (2017). Chapter 3 in Doing a systematic review: A student’s guide (Second ed.). SAGE.

DeCarlo, M. (2018). Chapter 8: Creating and refining a research question. In Scientific Inquiry in Social Work. Open Social Work Education.

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234.

Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. (2021, February). Chapter 2: Determining the scope of the review and the questions it will address. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

How to Search

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

Sayers, A. (2008). Tips and tricks in performing a systematic review. British Journal of General Practice, 58(547), 136-136.

Controlled Vocabularies

The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube.

UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube.

CADTH Search Filters Database

Documenting Your Search With PRISMA-S (Systematic Reviews)

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: An extension to the PRISMA Statement for reporting literature searches in systematic reviews. Systematic Reviews. 2021;10(1):39.

The Role of Librarians

Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from Synthesis Manual.

Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015). Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of Clinical Epidemiology, 68(6), 617–626.

Spencer, A. J., & Eldredge, J. D. (2018). Roles for librarians in systematic reviews: a scoping review. Journal of the Medical Library Association : JMLA, 106(1), 46–56.

Group members studying together. From Flicker user Archivists Round Table. Licensed under Creative Commons CC BY-ND 2.0

After you have conducted your search, you will be confronted with the dilemma of what to do with your results. It is not unheard of for searches to produce hundreds if not thousands of results to shift through. Your next steps will be to download your results, save them somewhere you can manage them and finally screen them to pick the ones you will be keeping in your review.

At this point you should work out a plan for how you will organize your results. Staying organized and managing your results will require a new set of organizational skills, plus these skills will help protect you from accidental plagiarism or bias.

In any literature or systematic review project, exporting and saving results from databases is an important step prior to selecting your final sources.  Exporting and saving results allow you to permanently store the important bibliographic information or ‘metadata’ attached to your articles – this could include the author name(s), article title, journal title, etc., the article abstract,  and any associated keywords.

Exporting

The majority of the databases that you will likely be using for your review will have options to export your references. Some important tips when exporting large amounts of records:

• Export your records as an .ris file type, which will allow you to then upload these records to citation managers or systematic review software.
• Look for options to include the article abstracts when exporting, as this will be important for the first level of screening.
• For systematic reviews: When selecting your records for export, look for options to Select All – depending on the database, you may have to export your references in batches, if there are limits on the number that you can export at a time.

Create an Account to Save Results

Many database interfaces like EBSCO, OVID and ProQuest allow users to create a personal account. With an account you can save your searches including any keywords, MeSH terms or limiters you used. This means you can return to your account at any time to re-run your search or modify your search and run the search again. See Table 3.1 below on how to set up an account for the following major database vendors (always check with your library to see which databases are available to you as a patron).

After exporting your results, you will need to store and organize them. Since you will be dealing with a large amount of sources, it’s advisable to use some sort of citation management tool.

A citation tool is especially useful If you are conducting a systematic review. As part of your review, you need to document aspects of your search like the number of sources you retrieved per database, the number of duplicates and the final number of articles you choose to review. For more information, please see the PRISMA Checklist in Module 2.

Picking a Citation Management System

When picking a tool, we suggest trying out a few in order to compare their features. You might also want to keep the following questions in mind when trying them out:

• Is the tool within my budget?
• Does it work well with the databases I use frequently?
• Is it able to organize and filter a large number of citations?
• Can it produce a bibliography in the primary style used in my discipline?
• If I’m working with a research team, does the software allow me to collaborate with my colleagues smoothly?

Simple Solution: Using Excel/Google Sheets/Numbers

If you have a small and manageable number of sources, you can use a familiar spreadsheet program like Excel, Google Sheets or Numbers to manage your citation data and look for duplicates. You can create your own columns for important citation information like Author, Title, Year, Database, etc. and even a column to enter notes. You can enter your sources manually or use the exporting function found in most databases. For example, databases like PubMed and ProQuest allow you to export the bibliographic information attached to your sources as a .CSV file. This file can be opened as a spreadsheet in Excel, Google Sheets and Numbers. See Figure 3.1 below for an example of a spreadsheet used as a citation management tool.

Figure 3.1. Example of using a spreadsheet for citation management. Source: Kelly Dermody, licensed under a Creative Commons CC BY 4.0

Free Software

There are a number of free citation management tools available for you to use. The only requirement is that you sign up for an account and download the product to your device. The three most commonly used tools are Mendeley, Zotero and EndNote Basic. If you are having trouble picking between the three, Table 3.2 below offers a quick comparison chart. For a more in depth comparison you can use the following comparison chart from the University of Toronto Libraries to help you decide.

Paid Products

There are some important reasons for considering a paid product for your choice of citation management tool. They can offer technical support if anything goes wrong, larger storage space, collaboration, duplication and other special features that might meet your needs more than a free product. The following table offers a comparison of two popular paid products available on the market. Some libraries have institutional licenses to one or more of these products, which means as long as you are a student you will have access. Check with your library to see if you have access to these products. Table 3.3 below offers a quick comparison chart of two paid products, RefWorks and EndNote.

Products Specifically for Systematic Reviews

Some citation managers may not be robust enough to handle a large evidence synthesis review or remove duplicate citations. Instead, you might need to use a tool specifically designed for systematic reviews. These tools incorporate automated features that help organize and speed up the review process.

The Systematic Review Toolbox is a searchable collection of tools you could potentially use to develop your systematic review. Below is a sample of some free and paid tools designed for systematic reviews.

Complete Activity 3.1: “Selecting Your Citation Management Tool” in your Research Roadmap Workbook.

Now that you have been presented with an in-depth list of available tools. Take a moment to document at least 2-3 tools that you are interested in testing.

For more information please see Workbook Instructions in the Introduction Module.

Screening is the process of identifying suitable sources from your literature search to be “full-text” screened and eventually included in the review. As mentioned in Module 2, your inclusion and exclusion criteria can help inform which limiters to apply to exclude irrelevant results, e.g., publication year or language.

If you are conducting a systematic review, it is recommended to screen the results outside of the databases, as there will be duplication of records. A citation software tool or systematic review tool will aid in removing duplicates. Within the citation management or systematic review tool, you can screen the titles and abstracts to determine if articles are relevant, based on inclusion and exclusion criteria that could not be captured in your search strategy. Afterwards, the full-text must be retrieved and screened to determine inclusion.

Complete Activity 3.2: “Inclusion and Exclusion List” in your Research Roadmap Workbook.

This exercise allows you to document your criteria. Don’t worry if you are still trying to figure out some of the important details about your review as you can use this exercise to document what you have in mind.

For more information please see Workbook Instructions in the Introduction Module.

For systematic reviews, it is recommended that at least two reviewers screen the results independently. This is to help resolve disagreement by encouraging consensus and reduce the risk of biased decisions. When needed, a third reviewer can aid in resolving the deadlock.

Screening the Title and Abstract

For the first screening, each member of the team will independently scan the titles and abstracts of articles, and make a decision whether to include or exclude studies. Each reviewer should be asking themselves, “Is there enough information in the title and abstract to exclude this study at this stage?” To make sure both reviewers are on the same page, they will need to ensure they have a clear and pre-defined inclusion and exclusion criteria.

Screening the Full-Text

The next step is to read the full-text of the articles selected from the first round. Again, the reviewers will do this independently and use the same inclusion and exclusion criteria. This stage involves a more rigorous and in depth look at the articles. The reason for excluding a study must be recorded for reporting purposes. Reviewers can also look at the outcome of a study and determine if it is significant enough to include.

As noted in Module 2, you will be expected to document your search process and your selection process. The PRISMA flow diagram will help you map out the number of records identified, included and excluded, and the reasons for exclusions.

Addressing Bias in Selecting Sources for Systematic Reviews

Selection bias can happen consciously or unconsciously to a reviewer when selecting sources. Bias can appear in how and why we  make decisions on the potential eligibility or ineligibility of a study. This is why using a team approach is one method to reduce bias in selecting sources for systematic reviews. Another is to make sure each reviewer has a clear understanding of the key concepts, or inclusion and exclusion criteria. For example, has the team checked to ensure everyone has the same understanding of the key concepts in their research question and what to include and exclude? This way, everyone goes into selecting with the same understanding.

Complete Activity 3.3: “Check for Team Understanding” in your Research Roadmap Workbook.

Are there any search terms or items on your inclusion/exclusion list that might have multiple interpretations? Write them down with your own interpretation and have your team members do the same activity. Compare your interpretations to check for team understanding.

For more information please see Workbook Instructions in the Introduction Module.

Plagiarism happens when you claim the words, ideas, or data of another person as if it is your own, without appropriate referencing. Plagiarism can range from copying an entire paper to changing only a few words and not citing the original source. It’s important to familiarize yourself with what constitutes plagiarism. One way you can do this is to consult your institution’s academic integrity policy so you know how to avoid a charge of plagiarism.

Citation Styles

When it comes time to write your review, you will need to demonstrate that you understand when and how to cite your sources. Keeping them organized will help you with this task. By using a citation management tool, you will be able to access your sources in one place and pull the citation information needed to cite them correctly. Before sitting down to write your review, make sure you familiarize yourself with the rules of your citation style. While it is true that some citation management tools will offer you a plug-in for your word processor that seemingly does all the work, it can make mistakes. Getting to know the rules yourself will save you a lot of time and reduce the risk of plagiarism. Table 3.4 below provides a curated list of citation style resources to help you familiarize yourself with the rules.

The topic of generative AI (GAI) is fairly new and is evolving quickly. Although artificial intelligence more broadly has been around for decades, discussions about generative AI became widespread with the introduction of ChatGPT in 2022. Since then, there has been a dramatic increase in the creation of AI-assisted tools, which can be used to expedite tasks during the review process.

These tools can be used (with varying quality) to: summarize literature, extract information like themes, find sources on a topic, show relationships between works, and ‘synthesize’ findings (or so they claim). However, there are many issues with these tools and they should never be used without human intervention. When considering AI tools, ensure that you have permission (some journals do not allow AI authorship, although this is subject to change), double check the accuracy of the output, and always cite the tool and how it has been used.

Issues with Generative AI

As with any tool, generative AI is not perfect. There are many issues and limitations, including:

• Inaccuracy and errors
• Bias and discrimination
• Risks to privacy

Inaccuracy and errors

The content that is generated by Large Language Models (LLMs) can be inaccurate or false. LLMs have been known to ‘hallucinate’, giving responses that are false or imaginative (IBM, n.d.). LLMs have also been trained with data up to a certain point in time and cannot account for new information past that cut-off date. For instance, ChatGPT 3.5 does not currently have knowledge of events that occurred after 2021 and can, therefore, produce outdated results (OpenAI, 2024). It’s important to always verify any information that is generated by AI.

Bias and discrimination

LLMs have been created and trained by humans and are therefore not free from bias. Bias can appear in multiple stages, including data collection, data labeling, model training, and deployment (Chapman University, n.d.). LLMs can perpetuate stereotypes and cause discrimination; reinforce exclusion by perpetuating social norms; use harmful language; and can perform better for certain languages and social groups over others (Weidinger et. al., 2021).

Risks to privacy

Training data can include personal and private information which can be revealed by an LLM as a response to a prompt, creating a privacy leak as with Scatterlab’s chatbot Lee-Luda (Weidinger et. al., 2021). As LLMs can also include the information you have provided as part of their training data, that information is also at risk. It’s important to treat all conversations with generative AI models as public, since you cannot control how the information is used once it has been added to a model’s input. Never share sensitive information.

These are just three concerns with Generative AI, but there are many others, some of which have been mentioned in the Generative AI Guide.

AI in Literature & Systematic Reviews

The use of Generative AI in academic reviews is a new area of research that is generating interest. Below are some early benefits, cautions and recommendations from van Dijk et al., (2023) unless otherwise stated.

A benefit of using an AI tool is that it can save you time compared to other tools and methods.

However, we must also consider that:

• GAI tools miss some relevant articles for systematic reviews
• Deduplication of articles is required, as GAI deduplication is not always accurate

Recommendations:

• Be transparent about your use of GAI in reviews: note it prominently in your methodology or equivalent section, and cite the tool(s) you use (See Citations for more information)
• Further research and guidelines are needed to ensure quality standards are met using GAI (Cacciamani et al., 2023).

Tools and recommendations are rapidly changing;  for current information, please see this guide which includes GAI tools for academic reviews, current research on AI screening of academic reviews and more; updated regularly by authors of this book.

Citations

As mentioned in the section above, transparency is vital if you are using GAI tools in your research, screening and/or writing. Citation is part of this. GAI use in research is so new that style guides may not have current guidance, but many styles such as APA, MLA and Chicago Style have updated recommendations on their websites. For the most current recommendations, please see Citing Artificial Intelligence, updated regularly by authors of this book.

AI Prompts

To interact with a generative AI tool, you will need to give it a prompt. Prompts are written in natural language, which can feel counterintuitive to keyword searching in databases. Typically, the more information you provide, the better.

Providing generative AI tools with an effective prompt is an important part of getting the desired response. The following prompt guidance has been adapted from Harvard University Information Technology (2023).

Be clear

Tell a GAI tool exactly what you would like it to do, as well as what you would like it not to do. Use the words ‘do’ and ‘don’t’ to clarify your criteria. It also helps to be clear about how you would like to receive the output. Maybe you want a list or a couple of paragraphs or in the format of a letter. You can also use examples to provide further clarification, but be careful not to use copyrighted works as an example.

Be specific

Asking a GAI tool to do something generic like ‘Write a speech’ will produce an equally generic result. To improve this prompt, provide context and background information on your request. You can specify the tone of the response and the audience it is intended for. Typically, the more specific your request the better your result, with the caveat that sometimes being too specific may cause a hallucination as the AI model tries to fill in the blanks of what it does not know.

Have a Conversation

Part of the appeal of generative AI is that you can speak to it as you would another person and build on your request. If you do not get the results you want or would like the results to be modified, you can provide this feedback to the model. For instance, you could ask a chatbot to: summarize its answer in one paragraph, speak more about a specific aspect of its answer, or write in a more formal language. If you’re stuck on a prompt or would like to improve a prompt, you can also ask the chatbot what it needs from you to fulfill this request. This is part of a process of refining your prompts through trial and error.

Definitions

Deep learning is what makes generative AI and LLMs possible. It “uses neural networks with multiple layers to model and solve complex problems.” (University of Manitoba Libraries, 2024)

Generative artificial intelligence is a broad term that encompasses AI systems that generate content. These systems are trained on large amounts of data to produce a response to a user’s prompt. They continually learn and improve on themselves. E.g., Text generators like ChatGPT, image generators like Midjourney, audio generators and video generators like Canva AI.

Large Language Model (LLM) is a language model that uses deep learning and large training datasets to recognize, classify, create, predict, and summarize textual content. E.g.. Open AI’s GPT-4

Neural Networks process and analyze data for AI, using algorithms to identify patterns and relationships. They are intended to imitate the operation of a human brain (Chen, 2024).

Prompts are the information entered into a GAI tool in order to receive an output. GAI analyzes the prompt, and generates a response based on relationships identified by its neural networks (Harvard University Information Technology, 2023).

Cacciamani, G. E., Chu, T. N., Sanford, D. I., Abreu, A., Duddalwar, V., Oberai, A., Kuo, C.-C. J., Liu, X., Denniston, A. K., Vasey, B., McCulloch, P., Wolff, R. F., Mallett, S., Mongan, J., Kahn, C. E., Jr, Sounderajah, V., Darzi, A., Dahm, P., Moons, K. G. M., … Hung, A. J. (2023). PRISMA AI reporting guidelines for systematic reviews and meta-analyses on AI in healthcare. Nature Medicine, 29(1), 14–15.

Chapman University. (n.d.). Bias in AI.

Chen, J. (2024, Feb. 7). What Is a Neural Network? Investopedia.

Harvard University Information Technology (2023). Getting started with prompts for text-based Generative AI tools.

IBM. (n.d.). What are AI Hallucinations? Think.

OpenAI. (2024). What is ChatGPT?

University of Manitoba Libraries. (2024). Definitions. Using Generative AI for Library Research.

van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023). Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open, 13(7), e072254–e072254.

Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., … Gabriel, I. (2021). Ethical and social risks of harm from Language Models (arXiv:2112.04359). arXiv.

We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:

• Get to know the exporting features of your chosen databases and sign up for an account with each database. This will not only save your results and search history, it will save you time if you have to re-do your search.
• Taking some time to pick a citation management tool that is right for you is an essential step in conducting a review. The right tool will help you maintain the integrity of your review and save you time.
• Screening your citations and abstracts against your inclusion and exclusion criteria is the first step in identifying suitable studies.
• If you are conducting a systematic review, it is important to reduce selection bias by having a team based approach to screening.
• Famarizlize yourself with the academic integrity policies of your institution to better understand how plagiarism is defined and how to avoid it.
• Commit to learning the rules or conventions of your chosen citation style.

Official Citation Manuels

APA

American Psychological Association (2020). Publication Manual of the American Psychological Association: The  official guide to APA style (Seventh ed.). American Psychological Association.

MLA

Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.

Turabian (Chicago)

Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.

IEEE

IEEE (n.d). IEEE Reference Guide.

Vancouver or ICMJE

International Committee of Medical Journal Editors. (2021, December).Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals (PDF).

Screening

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

(see See section 4.6 Selecting Studies)

Bias

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. (2021, February). Chapter 7: Considering bias and conflicts of interest among the included studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

Keenan, C. (2018, April 18).Assessing and addressing bias in systematic reviews. Meta-Evidence Blog, Campbell Collaboration, UK & Ireland.

Citation Mapping

Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: Consideration of a range of search techniques. Health Information and Libraries Journal, 27(2), 114–122.

Tay, A. (n.d.). Citation based literature mapping tools—An update—Tools offering premium accounts, the effects of the loss of MAG and use for evidence synthesis? Aaron Tay’s Musings about Librarianship.

Students looking at a laptop. From Flicker user Sac State. Licensed under Creative Commons CC BY-ND 2.0

The purpose of this module is to teach you how to manage your reading workload efficiently, and how to identify the key points in the articles you are including in your review.

Generally speaking, there is a common flow to scholarly articles. While not a template per se, you can be assured that the following components will be present in most articles. Learning to identify each component is a key step in the strategic reading process, and will help you save time as you screen articles for relevance. Check out the interactive example below that describes each section.

Now that you know about the component parts of scholarly articles, you are prepared to scan your search results and make swift judgments about the relevance of an article to your own context. Note: this step in the process does not require you to read the entire article! You will begin by scanning the articles and follow these steps:

Now that you have scanned your search results and have amassed those that are relevant to your context and needs, you can settle in to do what we call critical reading. This is where you must plan to read the entire article thoroughly.

Critical Questions to Ask When Reading

Critical reading encourages the reader to think about how an article or argument is constructed, not just what it says or what the study found. Here are some questions to ask yourself when reading:

1. What are the central ideas or arguments in the paper?
2. Are there words, concepts or research methods used that you don’t understand? If so, ensure you follow up with quality resources to ensure comprehension and extend your learning.
3. What is the context for the argument? Have they made explicit connections to existing literature, and located their work within that context?
4. Is their literature review clearly outlined and presented in an inclusive manner? Are there any obvious exclusions? If so, are they accounted for?
5. Who is the author? Who are they writing for? Have they located themselves and their positionality to the topic at hand? Are there any declared (or perceived) conflicts of interest?
6. What evidence is given to support the conclusions?
7. Is the evidence credible, that is, does it come from reliable sources?
8. Is the logic of the argument sound? What are the steps in the argument that leads from the evidence to the conclusion?
9. Are any of their ideas problematic? Are there other lenses that could be used to frame the argument?

Tools for Critical Appraisal

Using a standardized tool for critical appraisal is a good practice to follow. It demonstrates consistency in evaluation of all articles, and when working on a research team, it ensures that each individual is asking the same questions.

There are numerous tools for appraisal to choose from. For the most part, there will be disciplinary norms where certain tools are chosen in order to best represent the specific context. Looking to other reviews in your field to see what tools are employed is a good step.

There are various tools such as checklists designed by research organizations that will help you determine if your source is appropriate. Table 4.1 below outlines a sample of free quality assessment tools available for you to use.

To find the tool right for your review check out the Quality Assessment and Risk of Bias Tool Repository from Duke University’s Medical Center Library & Archives. 

Complete Activity 4.1: “Assessing the Quality of Your Sources” in your Research Roadmap Workbook.

Using a tool for critical appraisal is a good practice to follow. It will help you evaluate your sources and if you are working as a research team, it ensures that each individual is asking the same questions.

Take a moment to review some of the tools and document which ones you might use in your review.

For more information please see Workbook Instructions in the Introduction Module.

Scanning an academic article for relevancy is an important component of research. In this activity you will scan the article and identify key components such as the research question, methodology used and the conclusion. For your own research you can use this method for the first read of your sources to decide if they are relevant to your review. If they are relevant, you would do a second more thorough reading of your source. We have provided two examples; a qualitative study and a quantitative study. You are free to do both or choose one.

Quantitative Study Example

Use this Quantitative example study to identify the following components (Katajavuori et al., 2021).

Instructions: After taking a guess at the instructions on the front of each card, turn the card to reveal the response. How did you do?

Qualitative Study Example

Read this Qualitative example study to identify the following components (Hickey, et al., 2020).

Instructions: After taking a guess at the instructions on the front of each card, turn the card to reveal the response. How did you do?

Hickey, J., Powling, H., McKinney, P., Robbins, T., Carrier, N., & Nash, A. (2020). “It’s a change your life kind of program”: A healing focused camping weekend for urban Indigenous families living in Fredericton, New Brunswick. First Peoples Child & Family Review, 15(2), 23-44.

Katajavuori, N., Vehkalahti, K., & Asikainen, H. (2021). Promoting university students’ well-being and studying with an acceptance and commitment therapy (ACT)-based intervention. Current Psychology, 1-13.

The next step after reading and evaluating your sources is to organize them in a way that will help you start the writing process.

Review Matrix

One way to organize your literature is with a review matrix. The review matrix is a chart that sorts and categorizes the different arguments presented per topic or issue. Using a matrix enables you to quickly compare and contrast your sources in order to determine the scope of research across time. This will allow you  to spot similarities and differences between sources. It is particularly useful in the synthesis and analysis stages of a review (See Module 1 Conducting a Literature Review with the SALSA Framework).

Create a Review Matrix

Start with a charting tool you are most familiar with (for example MS Word, MS Excel, Google Sheets, Numbers etc).

1. Input your information.
   • Organize your sources from oldest to most recent. This way you can see how the research on your topic has changed over time.
2. Create your columns. The number of columns is up to you, but you will need the following columns to start:
   • First Column: citation (i.e., author, title, source, publication year)
   • Second Column: purpose or summary (1-2 sentences)
3. Depending on your review, you can now choose the remaining columns that will help you organize your sources. The columns should represent the specific content you are analyzing. For example:
   • methodology
   • population
   • geography
   • intervention
   • outcomes

Synthesis is an incredibly challenging concept to define in simple terms. Entire volumes have been written about the process, the concept and the various methods that can be used to engage in the activity. It is not the intention of this chapter to provide an in-depth analysis of synthesis as an intellectual activity; however, this is a component of the literature review process that consistently challenges students and researchers. It also happens to be the step in the review process that produces the most value – that is, the synthesis is where the findings from all of the searching, reading and analysis results come together. Therefore, one must not gloss over the approach to synthesis, and understanding the various methods that can be used is an important consideration. In this chapter, the concept of synthesis will be defined in straightforward terms, and various methods will be introduced, with examples offered, and further reading suggested.

Definition:

Combining parts into a newly conceived whole is crucial to synthesis. Observations made from distinct pieces, and then assembled into an entirely new perspective is how synthesis is generated.

Imagine a researcher who is studying play based learning in kindergarten. They have amassed a collection of articles on the topic, and have done their reading. Based on the conclusions from these studies, they have summarized the findings:

• One study found that teachers in Ontario believed that play-based learning (PBL) led to increased social inclusion of children with neurodevelopmental delay (NDD).
• One study found that a play-based training programme for parents in Hong Kong increased the social skills of their children
• One study found that an outdoor play-based learning curriculum in Wales led to increased involvement in tasks and deeper learning when children perceived their lessons as play
• One study looked to determine where children would choose to focus their play when given choices of stations; results showed that stations that promoted physical activity were given preference, indicating that when available, children will choose more active play than stationary.
• One study found that when play stations encouraged children to write, authentic opportunities for creative expression resulted, but only when intentional teaching took place concurrently

The above list is not a synthesis. It is simply a summary of the findings from distinct studies that have very loose connections. It is now the job of the researcher to try to make connections that may form a cohesive new interpretation of what these other researchers have concluded. This is the process of integration. What are some potential observations that could be made about the above studies?

• Play-based learning increased inclusion in the classroom
• Participation of adults (both teachers and parents) have measurable effects on outcomes
• Play-based learning is effective for learning
• Play-based learning is a global practice

None of the example studies drew these conclusions, nor was it the intent of those studies to draw those conclusions. Rather, by analyzing the findings of studies done in very specific circumstances, the intellectual exercise of synthesis allows the researcher to make higher level observations based on the similarities (i.e. they all focus on play based learning) and differences (i.e. they each focus on different participants in different circumstances) in a critical mass of studies that share many similar characteristics, but are not – strictly speaking – specifically about the same thing.

Approaches to Synthesis

There is a great deal of flexibility and individual choice around approaches to synthesis.  As stated earlier, this section is not meant to be an exhaustive exploration of the varieties of methods and techniques available; rather, some general guidelines for consideration.

Quite broadly, as described by Booth (2012), synthesis relies on pattern recognition in the data.  Key considerations would include the comparability of the study settings and the interventions – “in short…whether…it makes sense to treat all included studies as a group (lumping) or whether they make more sense as a series of separate and distinct groups (splitting)” (p. 179).

A common approach to synthesis includes narrative approaches which summarize the characteristics and findings of a body of research, primarily using words and text (Booth, 2012).  Tabulation can also be used in synthesis; providing a chart of the relevant data points – “eyeballing patterns across rows or down columns.” (p.185).

Examples:

Thematic analysis is a popular choice, which uses ‘codes’ or ‘themes’ that may be pre-specified or emergent from the data (Gough, 2017). The data is then mapped to these themes and codes in a consistent manner. This approach is common when the synthesis is focussed on qualitative studies.

See it in use here:

Sartain, S. A., Stressing, S., & Prieto, J. (2015). Patients’ views on the effectiveness of patient-held records: A systematic review and thematic synthesis of qualitative studies. Health Expectations, 18(6), 2666–2677. https://doi.org/10.1111/hex.12240

An alternative approach to a narrative or thematic approach is a meta-analysis, which uses a set of statistical procedures to integrate, summarize or organize a set of reported statistical findings of studies that investigate the same research question using the same methods of measurement.” (Booth, 2012)  This methodology is appropriate when there are homogenous amounts of quantitative data under consideration.

See it in use here:

Ettehad, D., Emdin, C. A., Kiran, A., Anderson, S. G., Callender, T., Emberson, J., Chalmers, J., Rodgers, A., & Rahimi, K. (2016). Blood pressure lowering for prevention of cardiovascular disease and death: a systematic review and meta-analysis. Lancet (London, England), 387(10022), 957–967. https://doi.org/10.1016/S0140-6736(15)01225-8

Evidence Synthesis Chart

Creating an evidence synthesis chart will assist you by compiling and presenting the relevant data points to assist with observing the patterns in the data.

Take an article that you’re already familiar with, and practice extracting data into the evidence synthesis chart provided.

Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis of qualitative research: A critical review. BMC Medical Research Methodology, 9(1), 59.

Booth, A., Papaioannou, D., & Sutton, A. (2012). Systematic approaches to a successful literature review. Sage.

Gough, D., Oliver, S, & Thomas, J. (2017). An Introduction to Systematic Reviews. Sage.

Strike K & Posner G. (1983). Types of synthesis and their criteria. In Ward, S & L. Reed (Eds.), Knowledge structure and use: implications for synthesis and interpretation (pp. 343-362). Temple University Press.

We’ve come to the end of this module. Below are the key points that we hope you will take away from the topics covered:

• Learning to identify each component is a key step in the strategic reading process, and will help you save time as you screen articles for relevance.
• For your first read you can scan the outcome/results or conclusion section and make swift judgments about the relevance of an article to your own context.
• While selecting your sources, you will need to assess the quality of the research question, the methodology used, the validity of the results and whether the study is applicable to your own review. To help you assess the quality of your chosen sources, consult one of the free checklist tools suggested in Table 4.1.
• Finally a literature review matrix will help you organize your readings when it comes time to write your review.

Reading Strategies

Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246.

Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012.

Quality Assessment

Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308.

Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743.

Literature Review Matrix

Evidence Synthesis Matrix Template, Jane Schmidt, Toronto Metropolitan University (Google Sheets)

The Matrix Method for Literature Reviews, Brandeis University, Writing Resources.

Literature Review Synthesis Matrix (MS Word), Concordia University

Write a Literature Review: Synthesize. Johns Hopkins University, Sheridan Libraries

Helpful Websites

Systematic Review Guide: Critical Appraisal (St. Michael’s Hospital Health Science Library)

Systematic Reviews: Quality Assessment (University of North Carolina Health Science Library)

Congratulations on completing this short course on conducting an academic review! As you start to work through your review, we hope you notice that you are part of a long line of researchers adding their voice to the scholarly discourse of your topic. Your work is contributing to this discourse. As a researcher, you are an active member of creating new knowledge and not just a consumer of it.

We hope you have come away from this short course with an understanding that there is value in using your intellectual curiosity in learning new investigative methods. Finally, as a researcher, it is normal to encounter challenges and doubts when conducting research, but this can be overcome with persistence, adaptability, and flexibility.

How to Conduct a Literature Review

Aveyard, H. (2019). Doing a literature review in health and social care: A practical guide (Fourth ed.) Open University Press, McGraw-Hill Education.

Jesson, J., Matheson, L., & Lacey, F. M. (2011). Doing your literature review: Traditional and systematic techniques. SAGE.

Ridley, D., Dr. (2008). The literature review: A step-by-step guide for students. SAGE.

Websites

• Guide to Literature Reviews (McMaster University Library)

Constructing a Research Question

Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting research. SAGE.

DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education.

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234.

How to Search

Videos

The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube.

UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube.

Websites

• Searching for Sources (Research Skills Tutorial), Toronto Metropolitan University Library
• Library Research Skills Tutorial, McMaster University Library

Reading Strategies

Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246.

Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012.

Quality Assessment

Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308.

Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743.

Checklists and Tools

• CASP Checklists Critical Appraisal Skills Programme (CASP) from the Public Health Resource Unit, NHS, England
• Critical Appraisal Tools The Centre for Evidence-Based Medicine (CEBM) from the University of Oxford
• Risk of Bias 2 (Rob 2) Tool  Cochrane Methods
• Critical Appraisal Tools Joanna Briggs Institute
• Study Quality Assessment Tools  National Heart, Lung and Blood Institute

Writing

Allen, J. (2019). The productive graduate student writer: How to manage your time, process, and energy to write your research proposal, thesis, and dissertation, and get published (First ed.). Stylus Publishing, LLC.

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234.

Feak, C. B., & Swales, J. M. (2009). Telling a research story: Writing a literature review. University of Michigan Press.

Holland, K., & Watson, R. (2021). Writing for publication in nursing and healthcare: Getting it right (Second;2; ed.). John Wiley & Sons Ltd.

Walliman, N. (2006). Writing a literature review. In Social research methods (pp. 182-185). SAGE Publications, Ltd.

Website

• Writing a Literature Review (OWL, Purdue University)

Official Citation Manuals

APA

American Psychological Association (2020). Publication Manual of the American Psychological Association: The  official guide to APA style (Seventh ed.). American Psychological Association.

MLA

Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.

Turabian (Chicago)

Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.

IEEE

IEEE (n.d). IEEE Reference Guide (PDF)

Vancouver or ICMJE

International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals (PDF).

How to Conduct a Systematic Review

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32.

Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. SAGE.

Ganann, R., Ciliska, D., & Thomas, H. (2010). Expediting systematic reviews: Methods and implications of rapid reviews. Implementation Science, 5(1), 56-56.

Glass, G. (1976). Primary, Secondary, and Meta-Analysis of Research. Educational Researcher, 5(10), 3-8. doi:10.2307/1174772

Gough, D., & Richardson, M. (2018). Systematic reviews. In Advanced research methods for applied psychology (pp. 63-75). Routledge.

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & Libraries Journal, 26(2), 91-108.

Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). (2021, February). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

Websites

• Systematic Reviews (Toronto Metropolitan University)
• Cochrane Database of Systematic Reviews (Cochrane)
• Systematic Review Guide (Unity Health Toronto)
• Knowledge syntheses: Systematic & Scoping Reviews, and other review types (University of Toronto Libraries, Gerstein Science Information Centre)

Online Courses on Systematic Reviews

The following guide provides a list of online self-directed learning on how to conduct a systematic review

Online Courses on Systematic Reviews (St. Michael’s Hospital Health Science Library)

Registering Your Systematic Review

• PROSPERO
• Campbell Collaboration
• Cochrane Collaboration

Creating Your Protocol

• PRISMA extension for Protocols
• MECIR (Methodological Expectations for Cochrane Intervention Reviews) Manual
• Sample Protocol Template from the Evidence Synthesis Coordinator at the Maritimes Strategy for Patient-Oriented Research (SPOR) Support Unit.

Documenting Your Review with PRISMA

• PRISMA 2020

Constructing a Research Question

Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting research. SAGE.

Boland, A., Cherry, M. G., & Dickson, R. (2017). Chapter 3 in Doing a systematic review: A student’s guide (Second ed.). SAGE.

DeCarlo, M (2018) Chapter 8: Creating and refining a research question in Scientific Inquiry in Social Work. Open Social Work Education.

Denney, A. S., & Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218–234.

Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. (2021, February). Chapter 2: Determining the scope of the review and the questions it will address. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

How to Search

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

Sayers, A. (2008). Tips and tricks in performing a systematic review. British Journal of General Practice, 58(547), 136-136.

CADTH Search Filters Database

Videos

• The National Library of Medicine (2013, February 14). Use MeSH to Build a Better PubMed Query [Video]. YouTube.
• UTS Library (2021, February 23). Medline Ovid: Advanced Searching [Video]. YouTube.

Websites

• Searching for Sources (Research Skills Tutorial), Toronto Metropolitan University Library
• Library Research Skills Tutorial, McMaster University Library

Screening

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. (2021, February). Chapter 4: Searching for and selecting studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

(see See section 4.6 Selecting Studies)

Reading Strategies

Greenhalgh T. (1997). How to read a paper. Getting your bearings (deciding what the paper is about). BMJ (Clinical research ed.), 315(7102), 243–246.

Sweeney, M. “How to Read for Grad School.” Miriam E. Sweeney, 20 June 2012.

Bias and Quality Assessment

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. (2021, February). Chapter 7: Considering bias and conflicts of interest among the included studies. In Higgins J.P.T, Thomas J, Chandler J, Cumpston M, Li T, Page M.J, Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane.

Greenhalgh T. (1997). Assessing the methodological quality of published papers. BMJ (Clinical research ed.), 315(7103), 305–308.

Greenhalgh, T., & Taylor, R. (1997). Papers that go beyond numbers (qualitative research). BMJ (Clinical research ed.), 315(7110), 740–743.

Keenan, C. (2018, April 18). Assessing and addressing bias in systematic reviews. Meta-Evidence Blog, Campbell Collaboration, UK & Ireland.

Checklists and Tools for Quality Assessment

• CASP Checklists Critical Appraisal Skills Programme (CASP) from the Public Health Resource Unit, NHS, England
• Critical Appraisal Tools The Centre for Evidence-Based Medicine (CEBM) from the University of Oxford
• Risk of Bias 2 (Rob 2) Tool  Cochrane Methods
• Critical Appraisal Tools Joanna Briggs Institute
• Study Quality Assessment Tools  National Heart, Lung and Blood Institute

Websites for Quality Assessment

• Systematic Review Guide: Critical Appraisal (St. Michael’s Hospital Health Science Library)
• Systematic Reviews: Quality Assessment (University of North Carolina Health Science Library)

Tools for Assessing Bias

• ROBIS – A tool for assessing the risk of bias in systematic reviews
• AMSTAR2 – AMSTAR stands for A MeaSurement Tool to Assess Systematic Reviews.

Analyzing Data

Card, N. A., & Little, T. D. (2012). Applied meta-analysis for social science research. Guilford Press.

Cheung, M. W. -., & Vijayakumar, R. (2016). A guide to conducting a meta-analysis. Neuropsychology Review, 26(2), 121-128.

Li T, Higgins JPT, Deeks JJ (editors). Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021.

Writing

Allen, J. (2019). The productive graduate student writer: How to manage your time, process, and energy to write your research proposal, thesis, and dissertation, and get published (First ed.). Stylus Publishing, LLC.

Holland, K., & Watson, R. (2021). Writing for publication in nursing and healthcare: Getting it right (Second ed.). John Wiley & Sons Ltd.

Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: A student’s guide. SAGE.

Official Citation Manuels

APA

American Psychological Association (2020). Publication Manual of the American Psychological Association: The  official guide to APA style (Seventh ed.). American Psychological Association.

MLA

Modern Language Association of America. (2021). MLA handbook (Ninth ed.). Modern Language Association of America.

Turabian (Chicago)

Turabian, Booth, W. C., Colomb, G. G., & Williams, J. M. (2018). Manual for Writers of Research Papers, Theses, and Dissertations (9th edition.). University of Chicago Press.

IEEE

IEEE (n.d). IEEE Reference Guide (PDF).

Vancouver or ICMJE

International Committee of Medical Journal Editors. (2021, December). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals (PDF).