Article Text
PDF
Abstract
Background Research evidence is commonly compiled into expert-agreed consensus statements or guidelines, with an increasing trend towards their publication in peer-reviewed journals. Prominent among these has been the publication of several International Olympic Committee (IOC) tatements to help inform sport and exercise medicine (SEM) practice. This study aimed to assess the citation impact and reach of the IOC statements published between 2003 and 2020.
Method Bibliometric analysis focused on identifying core publications (original statement and linked publications) and quantifying their academic citations (number and Field-Weighted Citation Impact (FWCI)) in journal articles up to February 2022. The analysis includes descriptive data on the country of IOC statement authorship affiliations, where they were published and by whom. The extent to which the IOC statements have been cited in the peer-reviewed literature is presented, together with information about the country of authorship of the citing papers as a measure of international academic reach.
Results 29 IOC statements were composed of 61 core publications. The IOC statements have had 9535 citations from 7863 citing publications. Individual FWCI ranged from 1.2 to 24.3 for core publications. The IOC statements were coauthored by multiple authors, mostly affiliated to countries with well-resourced SEM Authors of citing publications reflected the same geographical regions (ie, the USA, Canada, Australia, UK and western Europe.)
Conclusion Disseminating the IOC statements as open access papers in peer-reviewed journals has resulted in strong citation impact. However, this impact is centred on well-resourced academic circles that may not represent the diversity of SEM. Further research is required to identify if, and to what extent, the IOC statements have impacted SEM practice worldwide.
- consensus statement
- knowledge translation
- position statement
- sports & exercise medicine
- sports medicine
Data availability statement
All data relevant to the study are included in the article or uploaded as online supplemental information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Through its medical and scientific commission, the International Olympic Committee (IOC) has supported the development and publication of consensus statements on various sports medicine topics. The reach and impact of the IOC statements in peer-reviewed publications has not been evaluated.
WHAT THIS STUDY ADDS
Between 2003 and 2020, 29 IOC statements were published. Publishing the IOC statements as peer-reviewed open access publications has been a successful strategy for increasing citation numbers among academic audiences. However, there is a limited group of authors within well-resourced academic circles. There was a noticeable lack of IOC statement authors and citing authors from large parts of the world, including Asia, Africa (excluding South Africa), the Middle East and Oceania.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
When developing new IOC statements, the audience and purpose of the document should be made clear. Consideration should be given to how a broader representation of authors can be included in the authorship and wider promotion of key findings.
Introduction
International consensus statements are a popular format for summarising and sharing complex information, particularly in areas of importance or controversy. In several specialist medical fields, including sport and exercise medicine,1 consensus statements are popular as a quick way to access best practice knowledge. While practitioners value a consensus statement for its evidence summary, such statements can also present an expert-agreed understanding for a particular clinical scenario in the absence of strong empirical evidence. Consensus statements are generally developed through a structured process of identifying an issue to be addressed, bringing together acknowledged topic experts, compiling evidence through a series of (systematic) reviews, a consensus reaching process among the experts, and drafting and agreement on a final document.2 3
As a peak representative for sport and exercise medicine, the International Olympic Committee (IOC) Sports Medical and Scientific Commission has supported the development and dissemination of consensus statements (hereafter referred to as the IOC statements). The IOC statements are one strategy that the IOC Medical and Scientific Commission has used to provide consistent, evidence-based guidance to promote athlete health and well-being across the IOC community. Many of the IOC statements have been published online or linked to the IOC Medical and Scientific Commission website (https://olympics.com/ioc/medical-and-scientific-commission).
Many statements, particularly those most recent, have also been copublished in peer-reviewed medical and scientific journals. The implicit assumption behind this dual publishing approach is that sport and exercise medicine practitioners access and read these journals in their pursuit to provide evidence-based medicine to athletes.
To date, the reach (ie, dissemination, access) and impact (ie, implementation adoption, actions arising) of the IOC statements has not been evaluated. This means that it is currently unknown if there has been global uptake of the IOC statements by their intended users (ie, the IOC’s clinical stakeholders including the medical committees and staff of International Sporting Federations and National Olympic Committees) or if, indeed, the overall goal of the IOC statements—to improve athlete health and well-being—has been achieved. Before further investing in the ongoing development or revision of the IOC statements, it is important to consider their value to stakeholders and users of the information.
This paper reports the first stage of a formal assessment of the citation impact and reach of the IOC statements published between 2003 and 2020. This assessment involved a bibliometric analysis of the published scientific literature based on citation counts and associated metadata as a quantitative method for measuring citation impact.4 5 This work in cataloguing the IOC statements was also designed specifically to inform subsequent stages of the broader assessment, which considers policy and practice impacts through qualitative case study and quantitative survey methods.
Methods
This study comprised a bibliometric analysis of published IOC statements and associated documents, as available up to 31 December 2021.
Documents relating to each IOC statement were identified from several sources, including, where available:
The meeting title where an IOC statement was discussed, as obtained from the IOC website or via direct communication with the IOC Medical Commission.
Material available on the IOC website, associated with the meeting topic (eg, a meeting report).
Copies or links to published journal articles available from the IOC website.
Other published journal articles that were identified as being associated with the meeting topic but were not listed on the IOC website. These IOC statement papers were identified from the Scopus citation database (https://www.elsevier.com/en-au/solutions/scopus).
There were 29 IOC statements identified on the IOC website. One statement (female reproductive system in sport) was excluded from further analysis as it was a ‘statement’ with no associated meeting, rather than a ‘consensus statement’ resulting from expert group involvement.
The peer-reviewed journal articles associated with each statement (#3 and #4 in the above list) were identified to form a ‘core’ set of published documents for each topic. Other material on the IOC website associated with the Statement (#2 in the above list) was not included in further analysis as it is not found in citation databases such as Scopus. Some statements had more than one associated publication as they were republished in multiple journals so as to extend the audience—in these cases all publications and citations associated with the Statement are grouped (table 1).
Details of the IOC consensus statements (2003–2020) and bibliometric measures of their impact and academic reach (as at December 2021)
For each statement, the set of publications that cite them was also extracted to form a ‘citing’ set of published documents for each topic.
Open access status was determined for individual ‘core’ publications from the Unpaywall dataset (Unpaywall, last accessed 15 March 2022 https://unpaywall.org/). A manual search was used for the three ‘core’ publications that did not have a Digital Object Identifier.
Grouped publication data for each IOC statement were analysed within Elsevier’s SciVal tool for research benchmarking, using the Scopus citation database. The following summary data for the ‘core’ and ‘citing’ publications were exported from SciVal on 30 January 2022 and directly from Scopus on 26 February 2022. Summary data for the publication groups based on open access status were exported from SciVal on 28 March 2022 and from Scopus data of 16 March 2022:
Citation count—the number of citations (including self-citations) to the IOC statement since its publication in the published literature up to 31 December 2021 (SciVal).
Field Weighted Citation Impact (FWCI)—the ratio of the actual number of citations relative to the expected world average number of citations for the subject field, publication type and publication year (SciVal).6 The FWCI is calculated from citations received in the year in which an item was published and the following 3 years. An FWCI >1 indicates an item has been cited more frequently than the world average of comparator publications from that year and area of research as determined by Scopus from their database.
Country of affiliation listed for authors of the ‘core’ publication set and the ‘citing’ set of the publications that cited them, recognising that multiple authors may have coauthored multiple publications (Scopus).
Other publication metadata, such as authors and sources (ie, the journal that the publication was published in) (Scopus).
All data processing was through the R language and environment for statistical computing (R Core Team, 2013 V.3.5.3 of R was used through the RStudio Cloud integrated development environment V.1.2.1335 L (RStudio Team, 2018)), specifically drawing on the bibliometrix library.7
Results
Core IOC statement publications
Of the 29 IOC statements, 21 were distributed as 61 peer-reviewed journal articles published (table 1). The number of peer-reviewed journal articles associated with any given IOC statement ranged from 1 to 17. Eight IOC statements had no associated publication in a peer-reviewed journal. Therefore, bibliometric analysis was only performed for the 21 statements with copublication in a peer-reviewed journal. The documents were published throughout the period of interest, with peak publication years being in 2009 (n=11 of 61), 2013 (n=10) and 2018 (n=11).
The 61 IOC statement ‘core’ documents were published in 18 different peer-reviewed journal sources, with some representing multisource publications of a single statement. Twenty-nine (48%) of the ‘core’ documents were available as published journal articles in the British Journal of Sports Medicine (BJSM); other common journal sources were the Clinical Journal of Sport Medicine (n=6, 10%), International Journal of Sport Nutrition and Exercise Metabolism (n=3, 5%), Orthopaedic Journal of Sports Medicine (n=3, 5%) and the Journal of Athletic Training (n=3, 5%). Overall, 69% of the ‘core’ publications were published in an open access form.
The IOC statements were coauthored by multiple authors, such that the 61 ‘core’ IOC statement publications were associated with 355 unique authors from 944 authorships. Each author authored between 1 and 32 ‘core’ IOC statement publications, with 169 (48%) of authors only involved in a single publication. The most common countries of authorship affiliation were the USA (25% of all named authors), Canada (18%), Switzerland (12%), Australia (12%), Norway (7%) and the UK (7%). Other western European and Scandinavian countries accounted for 12% of the remaining authorship affiliations and the African continent (largely South Africa) for 3%. There was <1% authorship representation from the Middle East, South America, Asia or Oceania. As the 61 ‘core’ IOC statement publications had many authors, a different way to look at the country of authorship is to simply count the number of publications with at least one author from a country. These results showed that 45 of the 61 (74%) ‘core’ publications had at least one author from the USA, 44 (72%) from Switzerland, 42 (69%) from Canada, 41 (67%) from Australia, 36 (59%) from Norway and 36 (59%) from the UK.
Impact
Table 1 summarises individual groups of publications associated with IOC statements. Raw citation counts ranged from 42 (Molecular basis of connective tissue and muscle injuries in sport) to 5198 (Concussion in sport). The highest FWCI (24.3) was for the IOC statement on Methods for recording and reporting of epidemiological data on injury and illness in sport. Other highly ranked IOC statements in terms of FWCI were Youth athletic development (20.2) Concussion in sport (16.9), Beyond the Female Athlete Triad—Relative Energy Deficiency in Sport (14.6), Health Consequences of a saturated Sports Calendar (12.7), Use of platelet-rich plasma in sports medicine (10.3) and Mental Health in Elite Athletes (10.4). The IOC statements with the lowest FWCI were Sports nutrition (1.2), Pain Management (1.4), Periodic Health Evaluation of Elite Athletes (1.6), Prevention and Management of Chronic Disease (1.6) and Molecular basis of connective tissue and muscle injuries in sport (1.7). Nonetheless, each had still been cited more often than the global average for journal publications in the same field (all FWCI >1).
The FWCI of the 42 open access IOC statement publications (‘core’-open access) was 14.2, and for the 19 non-open access IOC statement publications it was 2.4.
Assessment of academic reach
As a group, the 61 ‘core’ IOC statement publications were cited 9535 times in 7863 peer-review journal articles between 2005 and 2021, with some ‘citing’ publications citing multiple ‘core’ publications. As an indicator of international academic reach, the country of affiliation of the authors of all published journal articles that had formally cited the journal-published IOC statements was obtained. Of these ‘citing’ publications, the most common countries of affiliation of the citing authors were the USA (43% of 20 996 citing authors with country data), Canada (13%), Australia and UK (7% each) and Germany, Italy and Switzerland (2% each); Spain, Brazil, Sweden, Norway, France, South Africa, Netherlands, Ireland, New Zealand and Portugal were each associated with 1%–2% (more than 300) of citing authorships. Every other country accounted for fewer than 1%, if any, of all citing author affiliations. Both the ‘core’ IOC statement peer-review publication sources and their citing papers tended to be published in the same journals.
Discussion
There is a large volume of published research and strong clinical expertise in sport and exercise medicine practice globally. Much of this knowledge and expertise has been compiled into consensus statements and guidelines informed by context-specific expert groups, such as those coordinated by the IOC. This is the first study to assess the citation impact and reach of the IOC statements. The first main finding is that the IOC statements have been highly cited in the orthopaedic and sports medicine peer-reviewed literature, many at levels well above the field publication averages indicating that the statements can be regarded as having had a strong citation impact. The second main finding is that the academic reach of the IOC statements in terms of their citing literature is largely within the same countries of authorship as the original IOC statements.
Increasingly, the IOC has actively supported the publication of their consensus statements in peer-reviewed journals. This publication strategy has two main benefits. First, it provides additional assurance of quality, with the requirement for the statement to have undergone peer review processes. Second, it directly informs the sport and exercise medicine community, especially those who belong to the societies that have publishing agreements with the relevant journals. While such journals certainly have international reach, it is also the case that there are many parts of the world where ready access to such items is not possible due to a lack of resources and infrastructure. Publishing the IOC statements as open access can help to overcome this barrier to a degree; the IOC statements that were published open access were cited more (14 times higher than the world average of comparator publications from that year and area of research) than those not published open access (twice the world average). Open access also enables ease of distribution and sharing compared with the copyright and licensing restrictions imposed by the publisher for closed publications. For example, printed copies of open access publications can be freely distributed to those without reliable internet access, and not having to pay to obtain publications removes one barrier to access.
The choice of journal for publishing the IOC statements has been limited to a small number of journals, most notably the BJSM, which probably reflects the IOC’s formal commitment to supporting the publishing of targeted issues in that journal (see bjsm.bmj.com/pages/about). Increasingly, the IOC has also encouraged copublication across specialist journals as one way to further enhance their reach.
The FWCI data presented in this paper demonstrate that some of the IOC statements have achieved exceptionally high citation counts relative to other papers published in the same field and time of publication internationally. Even the IOC statements with the lowest FWCI values have citation counts above the expected world average of comparator publications from that year and area of research. The IOC statements with the greatest impact in the scientific literature, as measured by the FWCI, cover sports medicine topics of relevance beyond just the direct IOC stakeholders or competitive setting, but to sports medicine more broadly, such as load management and concussion. Overall, the IOC statements appear to have had a strong citation impact, with citations to them suggesting value for research. However, it is important to recognise that citation counts are only one tool for evaluation and are not without controversy and caution for their use as a measure of impact.8 9
The purpose of the IOC statements is to inform, support and guide sports medicine clinical practice. Social marketing theory and health promotion frameworks argue that before people will act, they first need to be aware of the advice, deem it relevant to them and need to be able to act on it.10–16 The IOC statements can only achieve this if their target audiences (sport and exercise medicine practitioners) are fully aware of them and deem them directly relevant to themselves and their real-world sports medicine clinical practice contexts. Our analysis has highlighted that there may be some challenges in this respect, particularly with authorship representation.
The authorship of the statements has been largely restricted to experts from well-resourced areas such as Northern America (the USA and Canada), Western Europe, Scandinavia, the UK and Australia. Moreover, several authors have contributed to multiple statements. There is a noticeable lack of contributors from many parts of the world, including Asia, Africa (excluding South Africa), the Middle East and Oceania. Our analysis also showed low citation of the statements outside of North America, Europe, UK and Australia. It is possible that the absence of perspectives, advice and considerations arising from expert contributors in these regions could suggest to sports medicine practitioners in those areas that the IOC statements are not relevant to them, or this finding might reflect publishing patterns more generally. The fact that the citing papers are being published in the same journals as the original IOC statements could suggest a similar audience for both, limiting their reach. It could also result from self-citation by the IOC statement authors or certain countries being better resourced for access to peer-reviewed sports medicine resources than others. It is recommended that for future statements, the IOC extends opportunities for contributions to the IOC statement to experts from outside of western Europe, North America, the UK and Scandinavia, to ensure that they truly do have international relevance. This inclusion will need to be balanced with expert views and might include contextualisation or translation of key concepts.
Not every sports medicine practitioner reads scientific journals, and even fewer write clinical or scientific papers in peer-reviewed journals that reference other papers they contain. The measure of citation impact presented in this paper was obtained from a bibliometric analysis of the scholarly peer-reviewed literature and so is based on reactions from a specific subset of the international sports medicine community only.
There were initially some challenges in identifying available documents for some IOC statements, even when there was a journal publication in a peer-reviewed journal. This was partly because of inconsistent terminology and branding of the statements, which reduced the value of search strings in search engines to identify them. Multiple sources of documents were explored to minimise the likelihood of overlooking any key documents. However, there is still the possibility that some of the relevant documents relating to a given IOC statement may have been missed. While this is a limitation of the research reported here, this issue reflects broader challenges that sports medicine practitioners may have when trying to source this information. There would be value in consistent terminology and branding being used by the IOC in the future to enable ready recognition of the IOC statements.
This study relied on a single publicly available citation database (ie, Scopus) to identify citations of the IOC statements only from peer-reviewed literature contained within the same citation database. While the information on the source and authorship country affiliation of the literature that had cited the ‘core’ IOC statements was summarised, it was beyond the scope of this study to explore details of citation patterns from that literature or to look further into details of gender, professional standing, or expertise of the authors. As noted in the introduction, the second goal of this work was the compilation of documents in readiness for evaluating practice and policy impacts through separately conducted studies.
Conclusion
This study has shown that publishing the IOC statements through the peer-review literature has been a successful strategy for increasing their citation impact in terms of citation numbers. While used for scoping the use of literature, citations do come with caveats and, often, controversy. Further research, using different study designs and approaches, is required to identify the extent to which the IOC statements have impacted clinical sport and exercise medicine practice and policy worldwide.17
Supplemental material
Data availability statement
All data relevant to the study are included in the article or uploaded as online supplemental information.
Ethics statements
Patient consent for publication
Ethics approval
No ethics approval was required as there was no involvement of human participants or their personal data in the study.
Acknowledgments
Early results were presented and discussed in two meetings linked to the IOC research centres network, with attendance from representatives of eleven research centres and the IOC Medical and Scientific Commission. Neither the IOC nor its Medical and Scientific Commission contributed to details or decisions about the study design, conduct or interpretation of results.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @bechandcock, @caemery@CarolynAEmery, @evertverhagen, @CarolineFinch
Contributors CFF and LVF designed the larger study that this research was aligned to, with input from EV, KP, CE, MS and WD. RNH developed the analysis plan and conducted the analyses. CFF and LVF drafted the manuscript with critical input from all coauthors. All authors approved the final version for submission. LVF is study guarantor.
Funding The research was supported by an internationally competitive research grant from the International Olympic Committee Medical and Scientific Commission awarded to CFF, LVF, WD, CE, KP, MS and EV. All authors are affiliated with an IOC supported Research Centre for the Prevention of Injury and Illness (see www.olympic.org/medical-and-scientific-commission). CFF, LVF and RNH are associated with the Australian Centre for Research into Injury in Sport and its Prevention (ACRISP). EV is associated with the Amsterdam Collaboration on Health and Safety in Sports (ACHSS). CAE and KP are associated with the Sport Injury Prevention Research Centre (SIPRC). CAE holds a Canada Research Chair (Tier 1) in Concussion. WD and MS are associated with the IOC Research Centre of South Africa.
Competing interests Several authors have been contributors to the published IOC statements reviewed in this paper. CE, CFF, EV, MS and WD all coauthored the injury and illness surveillance statement. CE coauthored the following statements: concussion and the developing youth athlete. WD coauthored the following statements: pain management in the elite athlete. WD and MS both coauthored the following statements: molecular basis of connective tissue, muscle injuries in sport, prevention, and management of chronic disease. MS coauthored the following statements: periodic health evaluation of elite athletes, use of platelet-rich plasma in sports medicine, and health consequences of a saturated sports calendar (part 1 and part 2). Author EV is editor in chief of BMJSEM.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.