GRADE: Assessing the quality of evidence in environmental and occupational health
Introduction
There is high demand in environmental and occupational health for using systematic review methodology and structured frameworks to evaluate and integrate evidence to support evidence-based and transparent decisions and recommendations (Agency for Toxic Substances and Disease Registry (ATSDR), 2012; Bruce et al., 2014, EFSA, 2010, Johnson et al., 2014, Koustas et al., 2014, Lam et al., 2014, Mandrioli and Silbergeld, 2015, Mandrioli et al., 2014, Murray and Thayer, 2014, NRC, 2007, NRC, 2014a, NRC, 2014b, Silbergeld and Scherer, 2013, Whaley et al., 2016, Woodruff and Sutton, 2011, Woodruff and Sutton, 2014). Environmental health, which includes occupational health, is a broad field in which data address all the physical, chemical, and biological factors external to a person, and all the related factors impacting behaviors (WHO, 2015). Environmental health questions focus on understanding whether an exposure is a potential health hazard or risk using exposure assessments to recognize the extent and magnitude of exposure, and interventions to prevent or mitigate exposure or risk.
The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach has the potential to improve transparency in addressing these questions in environmental health assessments. GRADE represents a rigorous, structured, and transparent process to inform decision-making beginning with well-defined questions, followed by an assessment of the certainty in the evidence (also called confidence in the effect or other estimates, or quality of the evidence) (Guyatt et al., 2011d, Schünemann et al., 2003), and leading to development of recommendations and decisions.
GRADE is widely used internationally to address topics related to clinical medicine, public health, and health policy (Atkins et al., 2004, Guyatt et al., 2011d, Guyatt et al., 2008, Schünemann et al., 2008), including by programs within the U.S. Centers for Disease Control and Prevention (CDC), World Health Organization (WHO), the U.S. Agency for Healthcare Research and Quality (AHRQ), and National Institute for Health and Clinical Excellence (NICE) in the United Kingdom and the National Health and Medical Research Council in Australia (Ahmed et al., 2011, National Health and Medical Research Council, 2011, Thornton et al., 2013, Viswanathan et al., 2012, WHO, 2014b). The Cochrane Collaboration, which prepares, maintains, and promotes the accessibility of systematic reviews, uses the GRADE system for reporting on the quality of evidence for outcomes in systematic reviews (Higgins et al., 2011, Schünemann et al., 2011b). Formed in 2000, the GRADE Working Group now includes over 500 active members from 40 countries and serves as a think tank for advancing evidence-based decision-making in multiple disciplines (Schünemann et al., 2003)(see also http://www.gradeworkinggroup.org/).
Advantages of using the GRADE approach have already been recognized by some within the environmental health field. The Navigation Guide proposed adapting GRADE for an environmental health context (Woodruff and Sutton, 2011) and followed-up with a series of case studies to demonstrate the feasibility of applying GRADE to epidemiological and animal studies (Johnson et al., 2014, Koustas et al., 2014, Lam et al., 2014, Vesterinen et al., 2014). In 2013, the National Toxicology Program's (NTP) Office of Health Assessment and Translation (OHAT) at the National Institute of Environmental Health Sciences announced plans to use GRADE in its evaluations to assess the evidence for associations between environmental exposures and non-cancer health effects (NTP, 2013, NTP, 2015, Rooney et al., 2014). The SYstematic Review Center for Laboratory animal Experimentation (SYRCLE), is currently applying the GRADE approach to assess the quality of evidence from preclinical animal intervention studies (Hooijmans et al., 2014). GRADE has also been used in recent systematic reviews of epidemiological studies of shift work and breast cancer risk (Ijaz et al., 2013), shift work and cardiovascular disease (Vyas et al., 2012), and adverse effects related to reduced indoor air quality related to household fuel use (Bruce et al., 2013, WHO, 2014a). GRADE, including its adoption by NTP/OHAT and the Navigation Guide, was specifically identified in the National Academy of Sciences' National Research Council (NRC) review of the U.S. Environmental Protection Agency's (EPA) Integrated Risk Information System as an approach that would increase the transparency of evaluating evidence (NRC, 2014a). Use of GRADE in environmental health is likely to grow as systematic reviews become more common in the field and the limitations of expert-based narrative review methods are increasingly recognized (Aiassa et al., 2015, EFSA, 2010, EPA, 2013, Mandrioli and Silbergeld, 2015, NRC, 2014b, Woodruff and Sutton, 2014).
An additional advantage of GRADE is the GRADE Working Group's commitment to ongoing methods development and assessment of applicability to different areas of research. This is critical because experience with GRADE in the environmental health context is limited. Work to-date from the Navigation Guide, NTP, and WHO show the GRADE framework is sufficiently flexible to support use now (Johnson et al., 2013, Johnson et al., 2014, Koustas et al., 2014, Lam et al., 2014, NTP, 2015, WHO, 2014a); however, areas for further method assessment have been identified. In this respect, the GRADE Working Group serves as a vehicle to leverage transdisciplinary skills, knowledge, and resources to bridge the fields of clinical and environmental health. The objectives of this article are to provide an overview of the GRADE framework, discuss applicability of GRADE to environmental and occupational health, and identify priority areas for method development.
Section snippets
Formulating the research question
GRADE requires that decision-makers specify key-elements to formulate a relevant and focused question for decision-making (e.g., to inform clinical and public health guidelines, formulate scientific consensus statements, etc.) (Aiassa et al., 2015, Guyatt et al., 2011b). The key elements are the components of the question that identify what information must be provided in a primary study to evaluate the intervention under assessment and hence answer the question (Aiassa et al., 2015). For
Formulating the research question
The GRADE approach has been utilized predominantly to answer questions on interventions in health care, like “what is the impact of an intervention (including diagnostic tests and strategies) compared with an alternative on patient or population important outcomes?” or “should intervention A or B be used for X?” In the context of decision-making in environmental health, the term intervention has somewhat different connotations. First, an intervention can be thought of as a specific
Future directions
This paper provides an overview of important aspects of adapting GRADE to decision-making in environmental health. In 2014, several project groups were formed within the GRADE Working Group to focus on methods assessment needs that are directly applicable to environmental and occupational health, including project groups for environmental health, observational studies, public health, application of GRADE to laboratory animal research, and non-randomized study risk of bias integration. Priority
Conclusions
This paper examines several key components of GRADE as they can be assessed and expanded as a standardized methodology for research and decision-making in environmental and occupational health. Over 90 organizations from 18 countries worldwide have adopted the GRADE framework to assess evidence and inform decision-making. With a focus on rigorous and transparent methods, the GRADE approach has been applied successfully to clinical medicine, public health, diagnostic decision-making, questions
Conflict of interest
The authors declare they have no financial interests with respect to this manuscript, or its content, or subject matter.
Acknowledgments
This research was supported by the Intramural Research Program of the National Institute of Environmental Health Sciences and the MacGRADE center at the McMaster University. The contribution of AH was supported by the Natural Sciences and Engineering Research Council of Canada. The contribution of DG was supported by Australia’s National Health and Medical Research Council. The contribution of UCSF Program on Reproductive Health and the Environment co-authors (TW and PS) to this research was
References (73)
- et al.
Methods for developing evidence-based recommendations by the Advisory Committee on Immunization Practices (ACIP) of the US Centers for Disease Control and Prevention (CDC)
Vaccine
(2011) - et al.
GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations
J. Clin. Epidemiol.
(2013) - et al.
GRADE guidelines: 3. Rating the quality of evidence
J. Clin. Epidemiol.
(2011) - et al.
GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables
J. Clin. Epidemiol.
(2011) - et al.
GRADE guidelines: 2. Framing the question and deciding on important outcomes
J. Clin. Epidemiol.
(2011) - et al.
GRADE guidelines: 8. Rating the quality of evidence–indirectness
J. Clin. Epidemiol.
(2011) - et al.
GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology
J. Clin. Epidemiol.
(2011) - et al.
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
J. Clin. Epidemiol.
(2011) Methodological idiosyncracies, frameworks and challenges of non-pharmaceutical and non-technical treatment interventions
Z. Evid. Fortbild. Qual. Gesundhwes.
(2013)- et al.
Introducing GRADE across the NICE clinical guideline program
J. Clin. Epidemiol.
(2013)
Implementing Systematic Review Techniques in Chemical Risk Assessment: Challenges, Opportunities and Recommendations
Environ. Int.
Applicability and feasibility of systematic review for performing evidence-based risk assessment in food and feed safety
Crit. Rev. Food Sci. Nutr.
Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group
BMC Health Serv. Res.
The Future of Science at ATSDR: A symposium
Quality assessment tools for evidence from environmental science
Environ. Evid.
Tackling the Health Burden from Household air Pollution: Development and Implementation of new WHO Guidelines
WHO indoor air quality guidelines on household fuel combustion: Strategy implications of new evidence on interventions and exposure–risk functions
Atmos. Environ.
Guidelines for Systematic Review and Evidence Synthesis in Environmental Management. Version 4.2. Environmental Evidence
Quality of Evidence is a key Determinant for Making a Strong GRADE Guidelines Recommendation
Application of systematic review methodology to food and feed safety assessments to support decision making
EFSA J.
EPA Announces Actions to Address Chemicals of Concern, Including Phthalates
Design for the Environment Alternatives Assessments
Hazard Identification
Applying systematic review to assessments of health effects of chemical exposures
Criteria for Applying or Using GRADE
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
BMJ
The Cochrane collaboration's tool for assessing risk of bias in randomised trials
BMJ
SYRCLE's risk of bias tool for animal studies
BMC Med. Res. Methodol.
IARC: Diesel Engine Exhaust Carcinogenic. Press release
Night-shift work and breast cancer—a systematic review and meta-analysis
Scand. J. Work Environ. Health
Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients
BMJ
Applying the Navigation Guide: Case Study #1: The impact of developmental exposure to perfluorooctanoic acid (PFOA) on fetal growth (Final protocol)
The Navigation Guide—evidence-based medicine meets environmental health: systematic review of human evidence for PFOA effects on fetal growth
Environ. Health Perspect.
The toxicity data landscape for environmental chemicals
Environ. Health Perspect.
Update on EPA's ToxCast program: providing high throughput decision support tools for chemical risk management
Chem. Res. Toxicol.
The Navigation Guide—evidence-based medicine meets environmental health: systematic review of nonhuman evidence for PFOA effects on fetal growth
Environ. Health Perspect.
Cited by (206)
Maternal exposure to ambient air temperature and adverse birth outcomes: An umbrella review of systematic reviews and meta-analyses
2024, Science of the Total EnvironmentThe association between prenatal exposure to bisphenol A and offspring obesity: A systematic review
2024, Environmental PollutionDietary mycotoxin exposure and human health risks: A protocol for a systematic review
2024, Environment InternationalGlobal association between air pollution and COVID-19 mortality: A systematic review and meta-analysis
2024, Science of the Total EnvironmentTyre granulate on the loose; How much escapes the turf? A systematic literature review
2023, Science of the Total Environment