Elsevier

Environment International

Volumes 92–93, July–August 2016, Pages 611-616
Environment International

GRADE: Assessing the quality of evidence in environmental and occupational health

https://doi.org/10.1016/j.envint.2016.01.004Get rights and content

Highlights

  • A structured framework is needed for decision-making in environmental health.

  • GRADE has been applied in many disciplines and holds great promise for the field.

  • Methods development and assessment is needed to address environmental health data.

  • Methods assessment priorities are evaluation and integration of diverse evidence streams.

  • GRADE evidence-to-decision framework informs risk and other management decisions.

Abstract

There is high demand in environmental health for adoption of a structured process that evaluates and integrates evidence while making decisions and recommendations transparent. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework holds promise to address this demand. For over a decade, GRADE has been applied successfully to areas of clinical medicine, public health, and health policy, but experience with GRADE in environmental and occupational health is just beginning. Environmental and occupational health questions focus on understanding whether an exposure is a potential health hazard or risk, assessing the exposure to understand the extent and magnitude of risk, and exploring interventions to mitigate exposure or risk. Although GRADE offers many advantages, including its flexibility and methodological rigor, there are features of the different sources of evidence used in environmental and occupational health that will require further consideration to assess the need for method refinement. An issue that requires particular attention is the evaluation and integration of evidence from human, animal, in vitro, and in silico (computer modeling) studies when determining whether an environmental factor represents a potential health hazard or risk. Assessment of the hazard of exposures can produce analyses for use in the GRADE evidence-to-decision (EtD) framework to inform risk-management decisions about removing harmful exposures or mitigating risks. The EtD framework allows for grading the strength of the recommendations based on judgments of the certainty in the evidence (also known as quality of the evidence), as well as other factors that inform recommendations such as social values and preferences, resource implications, and benefits. GRADE represents an untapped opportunity for environmental and occupational health to make evidence-based recommendations in a systematic and transparent manner. The objectives of this article are to provide an overview of GRADE, discuss GRADE's applicability to environmental health, and identify priority areas for method assessment and development.

Introduction

There is high demand in environmental and occupational health for using systematic review methodology and structured frameworks to evaluate and integrate evidence to support evidence-based and transparent decisions and recommendations (Agency for Toxic Substances and Disease Registry (ATSDR), 2012; Bruce et al., 2014, EFSA, 2010, Johnson et al., 2014, Koustas et al., 2014, Lam et al., 2014, Mandrioli and Silbergeld, 2015, Mandrioli et al., 2014, Murray and Thayer, 2014, NRC, 2007, NRC, 2014a, NRC, 2014b, Silbergeld and Scherer, 2013, Whaley et al., 2016, Woodruff and Sutton, 2011, Woodruff and Sutton, 2014). Environmental health, which includes occupational health, is a broad field in which data address all the physical, chemical, and biological factors external to a person, and all the related factors impacting behaviors (WHO, 2015). Environmental health questions focus on understanding whether an exposure is a potential health hazard or risk using exposure assessments to recognize the extent and magnitude of exposure, and interventions to prevent or mitigate exposure or risk.

The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach has the potential to improve transparency in addressing these questions in environmental health assessments. GRADE represents a rigorous, structured, and transparent process to inform decision-making beginning with well-defined questions, followed by an assessment of the certainty in the evidence (also called confidence in the effect or other estimates, or quality of the evidence) (Guyatt et al., 2011d, Schünemann et al., 2003), and leading to development of recommendations and decisions.

GRADE is widely used internationally to address topics related to clinical medicine, public health, and health policy (Atkins et al., 2004, Guyatt et al., 2011d, Guyatt et al., 2008, Schünemann et al., 2008), including by programs within the U.S. Centers for Disease Control and Prevention (CDC), World Health Organization (WHO), the U.S. Agency for Healthcare Research and Quality (AHRQ), and National Institute for Health and Clinical Excellence (NICE) in the United Kingdom and the National Health and Medical Research Council in Australia (Ahmed et al., 2011, National Health and Medical Research Council, 2011, Thornton et al., 2013, Viswanathan et al., 2012, WHO, 2014b). The Cochrane Collaboration, which prepares, maintains, and promotes the accessibility of systematic reviews, uses the GRADE system for reporting on the quality of evidence for outcomes in systematic reviews (Higgins et al., 2011, Schünemann et al., 2011b). Formed in 2000, the GRADE Working Group now includes over 500 active members from 40 countries and serves as a think tank for advancing evidence-based decision-making in multiple disciplines (Schünemann et al., 2003)(see also http://www.gradeworkinggroup.org/).

Advantages of using the GRADE approach have already been recognized by some within the environmental health field. The Navigation Guide proposed adapting GRADE for an environmental health context (Woodruff and Sutton, 2011) and followed-up with a series of case studies to demonstrate the feasibility of applying GRADE to epidemiological and animal studies (Johnson et al., 2014, Koustas et al., 2014, Lam et al., 2014, Vesterinen et al., 2014). In 2013, the National Toxicology Program's (NTP) Office of Health Assessment and Translation (OHAT) at the National Institute of Environmental Health Sciences announced plans to use GRADE in its evaluations to assess the evidence for associations between environmental exposures and non-cancer health effects (NTP, 2013, NTP, 2015, Rooney et al., 2014). The SYstematic Review Center for Laboratory animal Experimentation (SYRCLE), is currently applying the GRADE approach to assess the quality of evidence from preclinical animal intervention studies (Hooijmans et al., 2014). GRADE has also been used in recent systematic reviews of epidemiological studies of shift work and breast cancer risk (Ijaz et al., 2013), shift work and cardiovascular disease (Vyas et al., 2012), and adverse effects related to reduced indoor air quality related to household fuel use (Bruce et al., 2013, WHO, 2014a). GRADE, including its adoption by NTP/OHAT and the Navigation Guide, was specifically identified in the National Academy of Sciences' National Research Council (NRC) review of the U.S. Environmental Protection Agency's (EPA) Integrated Risk Information System as an approach that would increase the transparency of evaluating evidence (NRC, 2014a). Use of GRADE in environmental health is likely to grow as systematic reviews become more common in the field and the limitations of expert-based narrative review methods are increasingly recognized (Aiassa et al., 2015, EFSA, 2010, EPA, 2013, Mandrioli and Silbergeld, 2015, NRC, 2014b, Woodruff and Sutton, 2014).

An additional advantage of GRADE is the GRADE Working Group's commitment to ongoing methods development and assessment of applicability to different areas of research. This is critical because experience with GRADE in the environmental health context is limited. Work to-date from the Navigation Guide, NTP, and WHO show the GRADE framework is sufficiently flexible to support use now (Johnson et al., 2013, Johnson et al., 2014, Koustas et al., 2014, Lam et al., 2014, NTP, 2015, WHO, 2014a); however, areas for further method assessment have been identified. In this respect, the GRADE Working Group serves as a vehicle to leverage transdisciplinary skills, knowledge, and resources to bridge the fields of clinical and environmental health. The objectives of this article are to provide an overview of the GRADE framework, discuss applicability of GRADE to environmental and occupational health, and identify priority areas for method development.

Section snippets

Formulating the research question

GRADE requires that decision-makers specify key-elements to formulate a relevant and focused question for decision-making (e.g., to inform clinical and public health guidelines, formulate scientific consensus statements, etc.) (Aiassa et al., 2015, Guyatt et al., 2011b). The key elements are the components of the question that identify what information must be provided in a primary study to evaluate the intervention under assessment and hence answer the question (Aiassa et al., 2015). For

Formulating the research question

The GRADE approach has been utilized predominantly to answer questions on interventions in health care, like “what is the impact of an intervention (including diagnostic tests and strategies) compared with an alternative on patient or population important outcomes?” or “should intervention A or B be used for X?” In the context of decision-making in environmental health, the term intervention has somewhat different connotations. First, an intervention can be thought of as a specific

Future directions

This paper provides an overview of important aspects of adapting GRADE to decision-making in environmental health. In 2014, several project groups were formed within the GRADE Working Group to focus on methods assessment needs that are directly applicable to environmental and occupational health, including project groups for environmental health, observational studies, public health, application of GRADE to laboratory animal research, and non-randomized study risk of bias integration. Priority

Conclusions

This paper examines several key components of GRADE as they can be assessed and expanded as a standardized methodology for research and decision-making in environmental and occupational health. Over 90 organizations from 18 countries worldwide have adopted the GRADE framework to assess evidence and inform decision-making. With a focus on rigorous and transparent methods, the GRADE approach has been applied successfully to clinical medicine, public health, diagnostic decision-making, questions

Conflict of interest

The authors declare they have no financial interests with respect to this manuscript, or its content, or subject matter.

Acknowledgments

This research was supported by the Intramural Research Program of the National Institute of Environmental Health Sciences and the MacGRADE center at the McMaster University. The contribution of AH was supported by the Natural Sciences and Engineering Research Council of Canada. The contribution of DG was supported by Australia’s National Health and Medical Research Council. The contribution of UCSF Program on Reproductive Health and the Environment co-authors (TW and PS) to this research was

References (73)

  • P. Whaley et al.

    Implementing Systematic Review Techniques in Chemical Risk Assessment: Challenges, Opportunities and Recommendations

    Environ. Int.

    (2016)
  • E. Aiassa et al.

    Applicability and feasibility of systematic review for performing evidence-based risk assessment in food and feed safety

    Crit. Rev. Food Sci. Nutr.

    (2015)
  • D. Atkins et al.

    Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group

    BMC Health Serv. Res.

    (2004)
  • ATSDR

    The Future of Science at ATSDR: A symposium

    (2012)
  • G.S. Bilotta et al.

    Quality assessment tools for evidence from environmental science

    Environ. Evid.

    (2014)
  • N. Bruce et al.

    Tackling the Health Burden from Household air Pollution: Development and Implementation of new WHO Guidelines

    (2013)
  • N. Bruce et al.

    WHO indoor air quality guidelines on household fuel combustion: Strategy implications of new evidence on interventions and exposure–risk functions

    Atmos. Environ.

    (2014)
  • Guidelines for Systematic Review and Evidence Synthesis in Environmental Management. Version 4.2. Environmental Evidence

    (2013)
  • B. Djulbegovic et al.

    Quality of Evidence is a key Determinant for Making a Strong GRADE Guidelines Recommendation

    (2015)
  • EFSA

    Application of systematic review methodology to food and feed safety assessments to support decision making

    EFSA J.

    (2010)
  • EPA

    EPA Announces Actions to Address Chemicals of Concern, Including Phthalates

    (2009)
  • EPA

    Design for the Environment Alternatives Assessments

    (2011)
  • EPA

    Hazard Identification

    (2012)
  • EPA

    Applying systematic review to assessments of health effects of chemical exposures

    (2013)
  • GRADE Working Group

    Criteria for Applying or Using GRADE

    (2010)
  • G.H. Guyatt et al.

    GRADE: an emerging consensus on rating quality of evidence and strength of recommendations

    BMJ

    (2008)
  • J.P. Higgins et al.

    The Cochrane collaboration's tool for assessing risk of bias in randomised trials

    BMJ

    (2011)
  • C.R. Hooijmans et al.

    SYRCLE's risk of bias tool for animal studies

    BMC Med. Res. Methodol.

    (2014)
  • IARC

    IARC: Diesel Engine Exhaust Carcinogenic. Press release

    (2012)
  • S. Ijaz et al.

    Night-shift work and breast cancer—a systematic review and meta-analysis

    Scand. J. Work Environ. Health

    (2013)
  • A. Iorio et al.

    Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients

    BMJ

    (2015)
  • P.I. Johnson et al.

    Applying the Navigation Guide: Case Study #1: The impact of developmental exposure to perfluorooctanoic acid (PFOA) on fetal growth (Final protocol)

    (2013)
  • P.I. Johnson et al.

    The Navigation Guide—evidence-based medicine meets environmental health: systematic review of human evidence for PFOA effects on fetal growth

    Environ. Health Perspect.

    (2014)
  • R. Judson et al.

    The toxicity data landscape for environmental chemicals

    Environ. Health Perspect.

    (2009)
  • R. Kavlock et al.

    Update on EPA's ToxCast program: providing high throughput decision support tools for chemical risk management

    Chem. Res. Toxicol.

    (2012)
  • E. Koustas et al.

    The Navigation Guide—evidence-based medicine meets environmental health: systematic review of nonhuman evidence for PFOA effects on fetal growth

    Environ. Health Perspect.

    (2014)
  • Cited by (206)

    View all citing articles on Scopus
    View full text