Reports from predatory journals in systematic reviews

The presence of reports from potential predatory or questionable journals in systematic reviews is a major methodological concern because these data may bias results. Deciding whether a journal is legitimate or potentially predatory is a complex decision and often subjective, especially considering the grey zone created by legitimate, although comparatively low-quality, content-specific regional journals.8 It might be argued that papers published in predatory journals may be of sufficient quality. For example, this might hold value in case of hijacked journals.9 However, publishing a paper in predatory journals reflects lack of knowledge and careful consideration of target journal by the authors. Experiments performed previously have shown that predatory journals are likely to accept anything irrespective of the quality of the paper.10 It should also be noted that some authors deliberately publish in predatory journals making it rather a symbiotic relationship.11 Subsequently, discussion specifically on the quality of papers published in such journals is complex and beyond the scope of this editorial.

There are several approaches that may aid identifying these predatory or questionable journals, including, yet not limited to, Think-Check-Submit (https://thinkchecksubmit.org) Initiative and Open Access Journal Quality Indicators (https://www.gvsu.edu/library/sc/open-access-journal-quality-indicators-5.htm).8 12 13 Once identified, the ethical quandary of how to handle these reports is similarly surrounded by layers of complexity. For instance, some authors (1) might exclude records from journals in the grey zone (poor-quality regional journals) whereby potential records might be missed, or (2) might include records from a journal they consider legitimate rather than predatory and ultimately end up including such studies. Notably, forward and backward citation tracking is recommended when conducting a systematic review, yet one common method of using Google Scholar increases the potential of finding publications from predatory journals given the breadth of indexing.14 For instance, one recent review reported that reports from predatory journals are often cited in systematic reviews in health sciences.15 Given that existing systematic review guidelines, such as Cochrane Collaboration, Campbell Collaboration and the Joanna Briggs Institute, have no clear guidance on how to deal with reports from potentially predatory journals, we suggest authors establish the following prior to conducting a systematic review:

1. Criteria to define a journal as ‘potentially predatory’, which can be achieved in light of many resources describing their characteristics.8 13 16

2. Whether reports identified as ‘potentially predatory’ will be included in primary syntheses.

3. Whether sensitivity analyses will be employed to evaluate the impact of including ‘potentially predatory’ reports.

Similar to how search strategies and intended methods of synthesising effect estimates are registered a priori, we contend this should also apply to methods for identifying and handling ‘potentially predatory’ reports.

Duplicate publications in systematic reviews

The presence of multiple reports from one study impacts evidence synthesis and effect estimates.17 This often happens when resulting reports from the same study do not cite prior reports and/or clinical trial registry identifiers.18 19 The Cochrane Handbook provides guidance on this in the section 5.2.1 and suggests ways in which authors can identify when multiple reports emanate from one study, such as trial registration numbers, authors’ names, numbers of participants and baseline data.7 A related problem is the inclusion of participants from the same intervention (or control) group based on data from multiple reports of one study. Potential solutions to address this issue are available in the Cochrane Handbook in the section 23.3.4 and include combining groups to create a single pairwise comparison and selecting one pair of interventions and excluding the others.7 Once again, we contend considering these methods a priori and ensuring these are integrated into the synthesis component of any systematic review.

Reporting recommendations for systematic reviews

Journals suggest authors to use Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) checklist,6 which ultimately improves the structure and transparency of reporting. For example, reviews should be prospectively registered (eg, PROSPERO) and should report the risk of bias assessment.20 However, many systematic reviews do not adhere to these reporting guidelines, despite commonly including a completed checklist as a supplement accompanying the final publication and/or review process.20 21

Specifically from our experience in the sports and exercise medicine field, common examples of PRISMA not being followed include, but are not restricted to: (a) providing list of excluded studies at full text with reasons; (b) providing access to the statistical code, raw tabulated data and extraction sheet such as on a data-sharing platform (eg, osf.io); (c) transparently reporting deviations to the a priori study protocol; (d) reporting the search strategy in full22 and (e) structuring the abstract in accordance with the relevant extensions of PRISMA statement (http://prisma-statement.org/Extensions/Abstracts).6 Including these elements not only facilitates the peer-review process insofar that reviewers are able to provide higher-level feedback and editors are able to reach a decision in a more timely manner, yet also conveys the methods and results of the systematic review in a more transparent and interpretable format to expected end users (eg, researchers, clinicians and policymakers). Therefore, we recommend authors: (a) familiarise themselves with not only the PRISMA checklist, but also the supporting documentation regarding its implementation,23 (b) design and draft the systematic review through the lens of the PRISMA checklist, rather than applying it post hoc, and (c) engage with the relevant extensions, such as that relevant to abstracts (http://prisma-statement.org/Extensions/Abstracts).6

Grey literature

The term ‘grey literature’ is used to describe reports published outside of traditional commercial publishing, such as dissertations, preprints, conference abstracts and reports.7 This is an important step as part of secondary searches which helps reduce the risk of publication bias. This step is crucial in cases where literature is expected to be small (eg, qualitative studies). However, these studies may be unrepresentative sample of all unpublished studies. The Cochrane Handbook provides guidance on this in the sections 3.4, 4.3.5 and 21.7 and provides guidance on how to deal with grey literature.7

In conclusion, authors of systematic reviews in the field of sports and exercise medicine are encouraged to check the methodological guideline for conducting systematic reviews from resources such as Cochrane Collaboration, Campbell Collaboration and the Joanna Briggs Institute Reports. Authors of systematic reviews should also consider issues related to predatory journals in systematic reviews, duplicate publications in systematic reviews and reporting recommendations for systematic reviews in light of this editorial.