Skip to Main Content

Systematic Reviews: Grey Literature & Handsearching

A guide to conducting systematic reviews.

What is Grey Literature

Grey literature refers to both published and unpublished research material that is not available commercially. In general, grey literature publications are non-conventional and sometimes ephemeral publications that are not indexed in databases such as PubMed and Embase.

Grey literature includes:

  • Clinical trials
  • Dissertations and theses
  • Conference proceedings
  • Government reports and documents
  • Maps
  • Newsletters
  • Pamphlets
  • Reports
  • Social media: Twitter, Facebook, LinkedIn, etc.
  • Statistics and data
  • Technical specifications and standards
  • Technical and commercial documentation
Why search grey literature:
  • Including unpublished results reduces publication bias, which includes the phenomenon that studies reporting a positive result are more likely to be published than those finding a negative result.
  • Incorporating unpublished trial data can change statistical results.
  • Global literature should be included.
  • Grey literature is often more current.
  • Current IOM systematic review standards call for grey literature inclusion.

Referencing and Citing Grey Literature

Consider using a citation management program to organize your search results.

  • Data to include in your references:
    • Author or (organizational author)
    • Year of publication
    • Title
    • Publisher information (for printed documents)
    • Retrieved from and URL (for online material)

Read More about Grey Literature

Where to Find Grey Literature

Advanced Search

Handsearching

IOM Standard 3.2.4 states: that researcher should "Handsearch selected journals and conference abstracts." Handsearching can be done of either paper or electronic journals and involves a page-by-page search. Reasons for handsearching include:

  • Indexing inconsistencies in bibliographic databases
  • Selective inclusion of articles
  • Difficulty of retrieving non-English articles in a database search

 

Critical Appraisal: AACODS Framework

  • Can you identify who is responsible for the intellectual content?
  • Is the work associated with a reputable organization?
  • Is the organization reputable? (e.g. W.H.O.)?
  • Can you easily identify the producing source?
  • Is he organization an authority in the field?
  • Does the item have a detailed reference list or bibliography?

 

  • Does the item have a clearly stated aim? If so, is it met?
  • Does it have a stated methodology?
  • Has it been peer-reviewed?
  • Has it been edited by a reputable authority?
  • The item refers to a particular population group, or excludes certain types of publications.
  • Are any limits clearly stated?
  • Opinion, expert or otherwise, is still opinion: is the author’s standpoint clear?
  • Does the work seem to be balanced in presentation?
  • Is there a conflicts of interest?
  • Does the item have a clearly stated date related to the content? 
  • For the item to inform your research, it needs to have a date that confirms relevance
  • No easily visible date is a strong concern
  • Have key contemporary material been included in the bibliography?
  • Is the item meaningful?
  • Does it enrich or negate a current position?
  • Is the publication relevant?

Critical Appraisal: CRAP or CRAAP Test

The CRAAP test is a method for evaluating research based on the following criteria

  • When was the information published or posted?
  • Has the information been revised or updated?
  • Is the information current or too out-of-date for my topic?
  • Are all the links functional or are there dead links?*

Developed by Sarah Blakeslee and her team of librarians at California State University, Chico (CSU Chico) to check the reliability of sources across academic disciplines .

 

  • Does the information relate to my topic or answer my question?
  • Who is the intended audience?
  • Is the information at an appropriate level (i.e. not too simple or advanced) for my needs?
  • Did I look at a variety of sources before deciding to use this one?
  • Would I be comfortable using this source for my college research paper?
  • Who is the author/publisher/source/sponsor?
  • Are the author's credentials or organizational affiliations given?
  • What are the author's credentials or organizational affiliations?
  • What are the author's qualifications to write on the topic?
  • Is there contact information, such as a publisher or e-mail address?
  • Does the URL reveal anything about the author or source? Examples: .com .edu .gov .org .net*
  • Where does the information come from?
  • Is the information supported by evidence?
  • Has the information been reviewed by anyone else?
  • Can I verify any of the information in another source or from personal knowledge?
  • Does the language or tone seem biased? Or is it free of emotion?
  • Are there spelling, grammar, typographical, or other errors?
  • What is the purpose of the information? to inform? teach? sell? entertain? persuade?
  • Do the authors/sponsors make their intentions or purpose clear?
  • Is the information fact? opinion? propaganda?
  • Does the point of view appear objective and impartial?
  • Are there political, ideological, cultural, religious, institutional, or personal biases?

Record Keeping

  • When grey literature is searched, a detailed account of the approach should be documented
  • It is important that your protocol documents the types of grey literature you will use and where you will look
  • When searching, you should record exactly what you do from the minute you start
  • This is important to ensure that you can reproduce your search later if necessary
  • Print/save the complete record and content of the results
Specific tools
•Archive.is. https://archive.is/

Managing and Reporting Results

MS Office

  • MS Word
  • Excel

Google Docs

List any trials and other research registries, web search engines, specific web sites, conference proceedings, or other resources searched, including their dates of coverage

  • List all websites searched, search methods (advanced or basic)
  • File a print copy or save a screen shot of items found on the internet, rather than simply ‘book-marking’ the site, in case the record of the trial is removed or altered at a later stage (beware of "link rot."
  • Which search engines, search terms used?
  • It’s important to keep a record of the date the web site was accessed, dates of coverage and Limitations, for citation purposes.                         - - Cochrane handbook guidelines

Writing Up Search Methods

BE TRANSPARENT

  • Search approach
    • Focused or broader
  • Inclusion/exclusion criteria
  • Sources search
    • Databases, structured internet searching
  • Document types
    • Unpublished clinical trials, dissertations
  • Search terms
  • Number of results
  • Limitations

Avoid Predatory Publishers

Is your inbox inundated with invitations from unfamiliar publishers to submit your manuscript to a journal you've never heard of? The number of journals in the biomedical sciences has increased exponentially in the past few years. How can you tell if a journal is legit?

Here are a few questions you can ask yourself to help guide your decision about publishing in a particular journal:

  1. Does our library have a subscription?
  2. Is the journal indexed in PubMed?
  3. Can you find the journal’s website?
    1. Is the explanation of fees clear?
    2. Who is on the editorial board?
  4. Is it listed in Directory of Open Access Journals (DOAJ)?
  5. Does it belong to the Committee on Publication Ethics (COPE)?
  6. Does it belong to the Open Access Scholarly Publishers’ Association (OASPA)?