You are here

Article Text

Article menu
Original research
Assessing Open Science practices in physical activity behaviour change intervention evaluations
  1. Emma Norris1,
  2. Isra Sulevani1,
  3. Ailbhe N Finnerty2,
  4. Oscar Castro2,3
  1. 1Department of Health Sciences, Brunel University London, Uxbridge, Middlesex, UK
  2. 2Centre for Behaviour Change, University College London, London, UK
  3. 3Future Health Technologies, Singapore-ETH Centre, Campus for Research Excellence And Technological Enterprise, Singapore
  1. Correspondence to Dr Emma Norris; Emma.Norris{at}brunel.ac.uk

Abstract

Objectives Concerns on the lack of reproducibility and transparency in science have led to a range of research practice reforms, broadly referred to as ‘Open Science’. The extent that physical activity interventions are embedding Open Science practices is currently unknown. In this study, we randomly sampled 100 reports of recent physical activity randomised controlled trial behaviour change interventions to estimate the prevalence of Open Science practices.

Methods One hundred reports of randomised controlled trial physical activity behaviour change interventions published between 2018 and 2021 were identified, as used within the Human Behaviour-Change Project. Open Science practices were coded in identified reports, including: study pre-registration, protocol sharing, data, materials and analysis scripts sharing, replication of a previous study, open access publication, funding sources and conflict of interest statements. Coding was performed by two independent researchers, with inter-rater reliability calculated using Krippendorff’s alpha.

Results 78 of the 100 reports provided details of study pre-registration and 41% provided evidence of a published protocol. 4% provided accessible open data, 8% provided open materials and 1% provided open analysis scripts. 73% of reports were published as open access and no studies were described as replication attempts. 93% of reports declared their sources of funding and 88% provided conflicts of interest statements. A Krippendorff’s alpha of 0.73 was obtained across all coding.

Conclusion Open data, materials, analysis and replication attempts are currently rare in physical activity behaviour change intervention reports, whereas funding source and conflict of interest declarations are common. Future physical activity research should increase the reproducibility of their methods and results by incorporating more Open Science practices.

  • open science
  • reproducibility
  • transparency
  • meta-research
  • physical activity
  • behaviour change

Data availability statement

Data are available in a public, open access repository. All data from this study are available here: https://osf.io/t5gw4/

https://creativecommons.org/licenses/by/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

https://doi.org/10.1136/bmjsem-2021-001282

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    WHAT IS ALREADY KNOWN ON THIS TOPIC

    • Open Science practices support research being more transparent and reproducible.

    • Assessment of Open Science practices in physical activity research is limited.

    WHAT THIS STUDY ADDS

    • We reviewed Open Science practices within 100 reports of physical activity behaviour change intervention randomised controlled trials.

    • Open data, materials, analysis and replication attempts are currently rare in physical activity behaviour change intervention research.

    HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE AND/OR POLICY

    • Our study draws attention to the practical solutions and resources that are most needed to promote transparency, reproducibility and accessibility in future physical activity research.

    Introduction

    Across scientific research, there is an increased awareness of highly prevalent problematic research practices, often referred to as questionable research practices,1 such as p-hacking: mining data for significant results2 3 and Hypothesising After the Results are Known (or ‘HARKing’).4 Open Science is an umbrella term of research behaviours intending to reduce these questionable research practices.5–8 Open Science research practices can be applied across the whole research process: from conception to publication.7–9 At research conception, pre-registrations provide time-stamped evidence of study hypotheses, methods and analysis plans,10 11 with these details made publicly available through online repositories such as Open Science Framework.12 In contrast to research protocols that specify research details but may be published before or after the study is in-progress or even completed,13 pre-registrations are completed and published prior to data collection to minimise biases.14 Open data, open materials (including questionnaires and intervention materials used) and open analysis scripts help make the processes and outputs of research more transparent, accessible and shareable.15 At publication, open access publishing makes reporting of research available to anyone at no cost to the reader.16

    Questionable research practices are likely rife in physical activity, sport and exercise medicine research.17 18 A recent study assessed the prevalence of questionable research practices within sport and exercise medicine research, including 129 studies published in leading sports medicine journals in 2019.19 Their analysis found that 82.2% of all reported hypotheses, and 70.8% of primary hypotheses, were supported by study results identified as implausibly high.19 Meta research has assessed Open Science practices in domains related to physical activity, behaviour change and life sciences.20 A recent study exploring 250 psychology studies of varying study designs published between 2014 and 2017 found that while open access publication was relatively common (65%), sharing of open materials (14%), data (2%) and analysis scripts (1%), as well as pre-specification of research plans via pre-registration (3%) and study protocols (0%) were low.21 In addition, transparency of reporting was inconsistent for funding statements (62%) and conflict of interest disclosure statements (39%).21 Meta-science studies have also assessed these Open Science practices within smoking cessation behaviour change research,22 social sciences,20 biomedicine23 and biostatistics.24

    However, to our knowledge, no study has evaluated the extent to which Open Science practices are used within physical activity research. Gaining a better understanding of these practices could inform future recommendations and policy development to promote open, transparent science within the physical activity field and to reduce the threat of questionable research practices. Therefore, the aim of this study was to assess Open Science practices within physical activity behaviour change intervention randomised controlled trial (RCT) reports assessing moderate-to-vigorous physical activity outcomes.

    Methods

    Study design

    This was a retrospective observational study with a cross-sectional design. Sampling units were individual physical activity behaviour change intervention reports. This study applied an established methodology used to assess Open Science practices in smoking cessation interventions,22 psychological sciences21 and social sciences.20 This study was pre-registered on the Open Science Framework.25 All deviations from this protocol are explicitly acknowledged in online supplemental file 1. Deviations included adding an additional item to specify whether a declared study pre-registration was registered ahead of data collection, or whether it was actually retrospectively registered after data collection had commenced, as well as adding ‘funded by a non-profit’ options within funding source and conflict of interest assessment items.

    Supplemental material

    Search strategy

    All papers included in this study were reports of physical activity behaviour change interventions, evaluated via RCTs. These reports were identified for inclusion within the Human Behaviour-Change Project (HBCP), which is developing an artificial intelligence system to extract information from published intervention studies and make recommendations for real-world practice and future research.26 27 The selection criteria for the HBCP are comparable to the one used for the present study (ie, both projects have a broad scope and aim to identify a subsample of reports describing RCTs of physical activity behaviour change interventions). Therefore, we used the same pool of articles remaining after the HBCP’s title and abstract screening (see figure 1 for a complete overview). Physical activity behaviour change intervention reports were identified in the HBCP using Microsoft Academic, one of the biggest, most comprehensive bibliographic databases of scientific literature.28 The search strategy was performed on 20 January 2021 and included the terms ‘MVPA or moderate-to-vigorous physical activity or MPA or VPA or moderate physical activity or vigorous physical activity or strenuous physical activity or hard physical activity’, with studies additionally filtered using the RCT classifier within Microsoft Academic. The terms were identified through a scoping search in which one of the study authors (OC) manually scanned the terms used in 20 physical activity behaviour change intervention reports.

    Figure 1
    Figure 1

    Flow diagram for the physical activity behaviour change intervention reports included in the analysis of Open Science practices. * Steps performed as part of the Human Behaviour-Change Project (HBCP) and involving two independent reviewers.

    Inclusion criteria were reports describing RCTs of physical activity behaviour change interventions and published between 2018 and 2021. The rationale for the recency of these included papers is to best represent current Open Science practices, given the relatively recent nature of Open Science practices.1 In addition, we focused on RCTs only due to their recognition as ‘gold-standard’ for studying intervention effectiveness.26 Exclusion criteria were trial protocols, conference submissions, abstract-only entries, qualitative research and economic or process evaluations. Full texts of identified papers within the HBCP were screened by one researcher (EN) to double-check relevance against inclusion and exclusion criteria, with piloting of the screening strategy by two authors (EN and OC). Of the 171 reports remaining after applying these criteria, 100 reports were selected due to time and resource constraints using the Calculator Soup Random Number Generator.29

    Measures

    Article characteristics recorded were: (i) author name, (ii) publication year and (iii) country of the corresponding author. Open Science practices were assessed by recording presence of the following in included reports: (i) Pre-registration: whether pre-registration was reported as carried out, where the pre-registration was hosted (eg, Open Science Framework, ClinicalTrials.gov), whether it could be accessed, what aspects of the study (hypotheses, methods and analysis plans) were pre-registered and whether the pre-registration was logged prospectively (prior to data collection commencing) or retrospectively (after data collection had commenced)30; (ii) Protocol sharing: whether a protocol was reported as published and what aspects of the study (hypotheses, methods and analysis plans) were included in the protocol; (iii) Data sharing: whether data were reported as available, where it was available (eg, online repository such as Open Science Framework, on request from authors, as a journal supplementary file), whether the data were downloadable and accessible, whether data files were clearly documented and the extent that data reported were sufficient to allow replication of study findings; (iv) Materials sharing: whether study materials were reported as available, where they were available (eg, online repository such as Open Science Framework, on request from authors, as a journal supplementary file) and whether the materials were downloadable and accessible; (v) Analysis script-sharing: whether analysis scripts were reported as available, where they were available (eg, online repository such as Open Science Framework, on request from authors, as a journal supplementary file) and whether the analysis scripts were downloadable and accessible; (vi) Replication of a previous study: whether the study was described as being a replication attempt of a previous study; (vii) Open access publication: whether the study was published as open access, assessed via the open access button website31 which harvests deposited publication from 1000s of academic institutions32; (viii) Funding sources: whether funding sources were declared and if research was funded by public organisations (such as research councils or charities), pharmaceutical, activity-related or other companies; and (ix) Conflicts of interest: whether conflicts of interest were declared and whether conflicts were with public organisations (such as research councils or charities), pharmaceutical, activity-related or other companies. The journal impact factor of identified papers was intentionally not assessed to evaluate papers, due to well-documented issues with manipulation and inflation of these figures.33 34 All measured variables are shown in table 1.

    View this table:
    Table 1

    Measured characteristics within identified physical activity behaviour change intervention reports

    Procedure

    Coding of identified intervention reports took place between July and September 2021, with all data extracted onto a Google Form.35 All reports were independently coded by two researchers (IS coded all 100 papers, EN and OC coded 50 each). Any discrepancies were resolved through discussion, with input from a third researcher who was not involved in the initial coding of that specific paper (EN or OC).

    Analysis

    Raw numbers and percentages were identified for each variable. Inter-rater reliability of the independent coding by the two researchers, prior to any changes after discrepancy discussions, was calculated using Krippendorff’s alpha36 using R package ‘irr’ V.0.84.1,37 as performed in other related research coding studies.22 38

    Results

    Sample characteristics

    Twenty-two out of the 100 physical activity behaviour change intervention reports were published in 2018, 33 in 2019, 37 in 2020 and 8 in 2021. The 100 reports evaluated studies conducted in 24 different countries, taking place most commonly in the USA (n=24), Australia (n=19), Canada (n=10) and the UK (n=7). A full summary of countries in included reports is presented in online supplemental file 2.

    Supplemental material

    Open Science practices in physical activity behaviour change intervention reports

    Final reconciled coding of Open Science practices for all 100 included physical activity behaviour change intervention reports can be found in online supplemental file 3.

    Supplemental material

    Article availability (open access)

    Seventy-three out of 100 physical activity behaviour change intervention reports were available via open access, with 27 of them only accessible behind a paywall (figure 2A).

    Figure 2
    Figure 2

    (A) Article availability, (B) pre-registration, (C) protocol availability, (D) data availability, (E) material availability, (F) analysis script availability, (G) replication study, (H) funding statement and (I) conflict of interest statement.

    Pre-registration

    Seventy-eight out of 100 physical activity behaviour change intervention reports included a statement indicating existence of a study pre-registration. Of those, 77 could be accessed. Forty-three of all accessible pre-registrations were recorded prospectively (ie, before data collection commenced) and 34 were recorded respectively (ie, after data collection commenced). Seventy-seven of all accessible pre-registrations declared specifications relating to study methods, 24 declared hypotheses and 5 declared analysis plans. Thirty-seven of all accessible pre-registrations were hosted on ClinicalTrials.gov (48.1%), 18 on the Australian and New Zealand Clinical Trials Registry (ANZCTR: 23.4%), 14 on the International Standard Randomised Clinical Trial Number registry (ISRCTN: 18.2%), 3 on Netherlands Trial Register (NTR: 3.9%) and 1 on Deutsches Register Klinischer Studien (DRKS), Iranian Registry of Clinical Trials (IRCT), Registro Brasileiro de Ensaios Clinicos (REBEC), UMIN Clinical Trials Registry (UMIN-CTR) and Chinese Clinical Trial Registry (ChCTR) (1.3% each). One included study was a Registered Report,39 logged with an International Registered Report Identifier (1.3%) (figure 2B).

    Protocol sharing

    Forty-one out of 100 physical activity behaviour change intervention reports included a statement about protocol availability. All 41 (100%) of these protocols specified study methods, 40 (97.6%) specified analysis plans and 14 (34.1%) specified hypotheses (figure 2C).

    Data sharing

    Thirty-two out of 100 physical activity behaviour change intervention reports included a data availability statement. Of those, 22 stated data were only available on request from the authors, 5 stated that data were available within the reports’ supplementary files, 1 stated that data were available via a personal or institutional website and 4 stated that data were not available. Only 4 out of these 32 reports included a data availability statement that data files that were actually accessible to download, with two of these providing clear documentation for the data files and two providing sufficient detail needed to reproduce findings (figure 2D).

    Material sharing

    Seventeen out of 100 physical activity behaviour change intervention reports included a materials availability statement. Of those, 10 reports stated that materials were available within the reports’ supplementary files and 7 stated that materials were only available on request from the authors. Eight out of the 10 studies which stated that materials were provided as supplementary files actually provided accessible and downloadable materials, such as full or sample intervention activities (figure 2E).

    Analysis script sharing

    One out of 100 physical activity behaviour change intervention reports included an analysis script availability statement,40 with this provided as a supplementary file (figure 2F).

    Replication study

    None of the 100 physical activity behaviour change intervention reports were described as replication studies (figure 2G).

    Funding

    Ninety-three out of the 100 physical activity behaviour change intervention reports included a statement about funding sources. Most of the reports disclosed public funding only, such as via government-funded research grants, charities or universities (n=85). One report disclosed both public funding and funding from private activity-related companies41 and one report disclosed funding from private activity-related companies only.42 Six reports reported receiving no funding (figure 2H).

    Conflicts of interest

    Eighty-eight out of the 100 articles provided a conflict of interest statement. Most of these reports stated that there were no conflicts of interest (n=77). Eleven reports stated that there was at least one conflict of interest, including from an activity company (n=5), a public organisation such as government or charities (n=2), a pharmaceutical company (n=1), a non-activity or pharmaceutical company (n=1), a combination of activity, pharmaceutical and other private companies (n=1), or that researchers were involved in the development and evaluation of the reported intervention (n=1) (figure 2I).43

    Inter-rater reliability assessment

    Inter-rater reliability of all coding across the 100 reports was assessed as good, a=0.73.

    Discussion

    This study aimed to assess Open Science practices within physical activity behaviour change intervention reports. It was found that Open Science practices varied among the assessed 100 physical activity behaviour change intervention reports. Most reports were open access and pre-registered, with reported funding sources and conflicts of interest. However, open materials, data and analysis scripts were not frequently provided and no replication studies were identified.

    Pre-registration of studies was found to be slightly more common for physical activity intervention RCTs (78%), than found in smoking cessation intervention RCTs (73%)22 and much more common than in wider psychological research of varying study designs (3%).21 In our study, similar amounts of studies were pre-registered prospectively (55.7%: prior to data collection commencing) or retrospectively (44.2%: after data collection had commenced),30 although this distinction between pre-registrations has not been assessed in comparable research. The common prevalence of retrospective pre-registration via clinical trials is arguably not true pre-registration, nor transparent from the study’s outset.12 13 One included study was noted as a Registered Report,39 where in-principle acceptance to journals is given based on study proposals at conception stage, rather than based on completed studies and their reported findings.44 45 No Registered Reports were identified in smoking cessation22 and psychology,21 perhaps reflecting a slow increase in Registered Report numbers over time.15 Protocols were available as separate papers or linked publications in 41% of included physical activity studies, which is higher than in smoking cessation studies (29%)22; and wider psychology research (0%).21 The increased prevalence of protocols within physical activity and smoking cessation likely reflects greater availability of health-related protocol publications,46 via specific journals such as JMIR Research Protocols and via protocols as specific types of publications within wider journals such as BMC Public Health and Trials. High prevalence of protocols in this study is also indicative of RCTs being both a common study design in health and intervention research47 and a study design typically accompanied by research protocols.48

    Open access reports were at similarly moderate levels in physical activity (73%) than in smoking cessation (71%)22; and psychology (65%),21 but greater than the 45% observed in the social sciences,20 the 45% across scientific literature published in 201516 and the 25% in biomedicine.23 This high rate of open access publishing in physical activity interventions may reflect increasing requirements by health funding bodies for open access publications,49 as well as increasing usage of preprint servers such as medRxiv for medical sciences and PsyArXiv for the psychological sciences.50

    Open materials were less commonly available in physical activity reports (8%) than in smoking cessation reports (13%),22 psychology (14%)21; and biomedicine (33%).23 Open data were also less common across physical activity reports (4%) than in smoking cessation reports (7%),22 but greater than the 2% of wider psychological research.21 Provision of raw data as supplementary files to published intervention reports or via trusted third-party repositories such as the Open Science Framework is important to facilitate evidence synthesis. Open analysis scripts were found to be as infrequently provided in physical activity studies than in smoking interventions and wider psychological research (all 1%).21 22 No replication attempts were identified in this sample of physical activity intervention reports, same as within smoking cessation reports22 but less than in the social sciences (1%)20; and in wider psychology studies (5%).21

    Declaration of funding sources were declared in physical activity reports (93%) similarly to smoking cessation reports (95%)22; more so than wider psychology (62%),21 social sciences (31%)20 and biomedical science reports (69%).23 Similarly, a conflict of interest statement was provided as commonly in physical activity reports than in smoking cessation reports (88% in both)22 and higher than in wider psychology (39%),21 social sciences (39%)20 and biomedical sciences reports (65%).23 Eight per cent of studies reported conflicts from private companies including activity, pharmaceutical and other companies, less than the 20% of studies reporting company funding in smoking cessation interventions.22

    Future steps to increase Open Science in physical activity interventions

    This research has demonstrated a need to address the low levels of Open Science engagement in physical activity research, particularly in the areas of open materials, data, analysis scripts and replication attempts. As with any complex behaviour change, this transformation requires systems change across bodies involved in the development, running and publication of physical activity research: researchers, research institutions, funding organisations, journals and beyond.1 9 In order to develop effective behaviour change interventions, it is important to use a systematic and comprehensive approach to intervention development, underpinned by a model of behaviour and theoretically predicted mechanisms of action.51–53 The Capability, Opportunity, Motivation, Behaviour model54 posits that changing behaviour involves changing one or more of the following: capability (psychological and physical capacity to engage in the behaviour), opportunity (external factors that make the execution of a particular behaviour possible or prompt it) and motivation (internal processes that energise and direct behaviour). We argue that understanding the capability, opportunity and motivation associated with Open Science practices9 and developing interventions to address these determinants of behaviour change,55 is key to increase engagement with Open Science.

    For example, low perceived capability towards Open Science practices in physical activity researchers can be addressed by providing researchers with training tailored to the context of activity intervention research (eg, online training on how to make anonymised activity monitor data openly available, how to use preprint servers most relevant to activity research or how to make their activity analysis reproducible). Opportunity to engage in Open Science practices can be facilitated within institutions, encouraging discussions around Open Science in the context of physical activity research19 and in science more broadly,21 23 56 as well as developing a research culture valuing and promoting the benefits of Open Science practices.16 23 Motivation for Open Science can be addressed by providing incentives, such as awarding funding to research-embedding open practices.57 Similarly, Open Science badges recognising open data, materials and pre-registration have been adopted by journals as a simple, low-cost scheme to reward these research behaviours.58 However, uptake of Open Science badges in physical activity journals is currently low and is rife for increased uptake in the field.

    Strengths and limitations

    A strength of this study is the implementation of a comprehensive and previously used approach to identify Open Science practices. Moreover, two researchers independently carried out the coding of Open Science practices, reducing the risk of human error and maximising reliability.59 A limitation is that the search and screening processes were conducted by a single author. However, unlike systematic reviews, we did not attempt to conduct a comprehensive search to identify all relevant research but to select a somewhat random subsample to analyse Open Science practices and inform specific recommendations for future research. In this regard, it is worth acknowledging that results are based on a relatively small sample of physical activity behaviour change reports, meaning findings may not be applicable to all physical activity research. Last, the assessment of Open Science practices was entirely dependent on what was described within evaluation reports. Direct requests to authors or additional wider searching of third-party registries such as Open Science Framework may have identified additional information.

    Conclusions

    Open Science practices in physical activity behaviour change intervention reports were varied. Open access publication and pre-registration of research plans were common, although pre-registration was often done retrospectively, that is, after data collection has started, hence not in the most transparent manner. Provision of open data, materials and analysis was rare and replication attempts were non-existent. Funding sources and conflicts of interest were usually declared. Urgent initiatives are needed to increase the uptake of all Open Science practices in physical activity, with a particular focus on open materials, data, analysis scripts and replication attempts.

    Data availability statement

    Data are available in a public, open access repository. All data from this study are available here: https://osf.io/t5gw4/

    Ethics statements

    Patient consent for publication

    Ethics approval

    Not applicable.

    Acknowledgments

    Thanks to the Human Behaviour-Change Project.

    References

    1. ↵
      1. Munafò MR,
      2. Nosek BA,
      3. Bishop DVM, et al
      . A manifesto for reproducible science. Nat Hum Behav 2017;1:1–9.doi:10.1038/s41562-016-0021
      OpenUrl
    2. ↵
      1. Head ML,
      2. Holman L,
      3. Lanfear R, et al
      . The extent and consequences of p-hacking in science. PLoS Biol 2015;13:e1002106.doi:10.1371/journal.pbio.1002106pmid:http://www.ncbi.nlm.nih.gov/pubmed/25768323
      OpenUrlCrossRefPubMed
    3. ↵
      1. Simmons JP,
      2. Nelson LD,
      3. Simonsohn U
      . False-Positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011;22:1359–66.doi:10.1177/0956797611417632pmid:http://www.ncbi.nlm.nih.gov/pubmed/22006061
      OpenUrlCrossRefPubMed
    4. ↵
      1. Murphy KR,
      2. Aguinis H
      . HARKing: how badly can Cherry-Picking and question Trolling produce bias in published results? J Bus Psychol 2019;34:1–17.doi:10.1007/s10869-017-9524-7
      OpenUrl
    5. ↵
      1. Munafò MR,
      2. Hollands GJ,
      3. Marteau TM
      . Open science prevents mindless science. BMJ 2018;363:k4309.doi:10.1136/bmj.k4309pmid:http://www.ncbi.nlm.nih.gov/pubmed/30322848
      OpenUrlFREE Full Text
    6. ↵
      1. Nosek BA,
      2. Spies JR,
      3. Motyl M
      . Scientific utopia: II. restructuring incentives and practices to promote truth over Publishability. Perspect Psychol Sci 2012;7:615–31.doi:10.1177/1745691612459058pmid:http://www.ncbi.nlm.nih.gov/pubmed/26168121
      OpenUrlCrossRefPubMed
    7. ↵
      1. Science FO
      . Open science definition | foster, 2021. Available: https://www.fosteropenscience.eu/foster-taxonomy/open-science-definition [Accessed 03 Nov 2021].
    8. ↵
      1. Kathawalla U-K,
      2. Silverstein P,
      3. Syed M
      . Easing into open science: a guide for graduate students and their advisors. Collabra Psychol 2021;7.doi:10.1525/collabra.18684
    9. ↵
      1. Norris E,
      2. O'Connor DB
      . Science as behaviour: using a behaviour change approach to increase uptake of open science. Psychol Health 2019;34:1397–406.doi:10.1080/08870446.2019.1679373pmid:http://www.ncbi.nlm.nih.gov/pubmed/31661325
      OpenUrlPubMed
    10. ↵
      1. Field SM,
      2. Wagenmakers E-J,
      3. Kiers HAL, et al
      . The effect of preregistration on trust in empirical research findings: results of a registered report. R Soc Open Sci 2020;7:181351.doi:10.1098/rsos.181351pmid:http://www.ncbi.nlm.nih.gov/pubmed/32431853
      OpenUrlPubMed
    11. ↵
      1. Yamada Y
      . How to crack PRE-REGISTRATION: toward transparent and open science. Front Psychol 2018;9:1831.doi:10.3389/fpsyg.2018.01831pmid:http://www.ncbi.nlm.nih.gov/pubmed/30319516
      OpenUrlPubMed
    12. ↵
      1. Sullivan I,
      2. DeHaven A,
      3. Mellor D
      . Open and reproducible research on open science framework. Curr Protoc Essent Lab Tech 2019;18:e32.doi:10.1002/cpet.32
    13. ↵
      1. Shamseer L,
      2. Moher D,
      3. Clarke M, et al
      . Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:p. 7647:g7647.doi:10.1136/bmj.g7647
      OpenUrl
    14. ↵
      1. Bakker M,
      2. Veldkamp CLS,
      3. van Assen MALM, et al
      . Ensuring the quality and specificity of preregistrations. PLoS Biol 2020;18:e3000937.doi:10.1371/journal.pbio.3000937pmid:http://www.ncbi.nlm.nih.gov/pubmed/33296358
      OpenUrlPubMed
    15. ↵
      1. Nosek BA,
      2. Alter G,
      3. Banks GC, et al
      . Promoting an open research culture. Science 2015;348:1422–5.doi:10.1126/science.aab2374
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Piwowar H,
      2. Priem J,
      3. Larivière V, et al
      . The state of oa: a large-scale analysis of the prevalence and impact of open access articles. PeerJ 2018;6:e4375.doi:10.7717/peerj.4375pmid:http://www.ncbi.nlm.nih.gov/pubmed/29456894
      OpenUrlCrossRefPubMed
    17. ↵
      1. Caldwell AR,
      2. Vigotsky AD,
      3. Tenan MS, et al
      . Moving sport and exercise science forward: a call for the adoption of more transparent research practices. Sports Med 2020;50:449–59.doi:10.1007/s40279-019-01227-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/32020542
      OpenUrlCrossRefPubMed
    18. ↵
      1. Halperin I,
      2. Vigotsky AD,
      3. Foster C, et al
      . Strengthening the practice of exercise and Sport-Science research. Int J Sports Physiol Perform 2018;13:127–34.doi:10.1123/ijspp.2017-0322pmid:http://www.ncbi.nlm.nih.gov/pubmed/28787228
      OpenUrlCrossRefPubMed
    19. ↵
      1. Büttner F,
      2. Toomey E,
      3. McClean S, et al
      . Are questionable research practices facilitating new discoveries in sport and exercise medicine? the proportion of supported hypotheses is implausibly high. Br J Sports Med 2020;54:1365–71.doi:10.1136/bjsports-2019-101863pmid:http://www.ncbi.nlm.nih.gov/pubmed/32699001
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Hardwicke TE,
      2. Wallach JD,
      3. Kidwell MC, et al
      . An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). R Soc Open Sci 2020;7:190806.doi:10.1098/rsos.190806pmid:http://www.ncbi.nlm.nih.gov/pubmed/32257301
      OpenUrlPubMed
    21. ↵
      1. Hardwicke TE,
      2. Thibault RT,
      3. Kosie JE
      . Estimating the prevalence of transparency and Reproducibility-Related research practices in psychology (2014–2017). Perspect Psychol Sci p 2021.doi:10.1177/1745691620979806
    22. ↵
      1. Norris E,
      2. He Y,
      3. Loh R
      . Assessing markers of reproducibility and transparency in smoking behaviour change intervention evaluations. J Smok Cessat 2021;2021:e6694386:12.doi:10.1155/2021/6694386pmid:34306236
      OpenUrlPubMed
    23. ↵
      1. Wallach JD,
      2. Boyack KW,
      3. Ioannidis JPA
      . Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. PLoS Biol 2018;16:e2006930.doi:10.1371/journal.pbio.2006930pmid:http://www.ncbi.nlm.nih.gov/pubmed/30457984
      OpenUrlCrossRefPubMed
    24. ↵
      1. Rowhani-Farid A,
      2. Barnett AG
      . Badges for sharing data and code at biostatistics: an observational study. F1000Res 2018;7:90.doi:10.12688/f1000research.13477.1pmid:http://www.ncbi.nlm.nih.gov/pubmed/29862016
      OpenUrlPubMed
    25. ↵
      1. Norris E,
      2. Castro O
      . Assessing open science practices in physical activity behaviour change intervention evaluations. medRxiv 2021.doi:10.1101/2021.12.01.21267126
    26. ↵
      1. Michie S,
      2. Thomas J,
      3. Johnston M, et al
      . The human Behaviour-Change project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation. Implement Sci 2017;12:121.doi:10.1186/s13012-017-0641-5pmid:http://www.ncbi.nlm.nih.gov/pubmed/29047393
      OpenUrlCrossRefPubMed
    27. ↵
      1. Michie S,
      2. Thomas J,
      3. Mac Aonghusa P, et al
      . The human Behaviour-Change project: an artificial intelligence system to answer questions about changing behaviour. Wellcome Open Res 2020;5:122.doi:10.12688/wellcomeopenres.15900.1pmid:http://www.ncbi.nlm.nih.gov/pubmed/32566761
      OpenUrlPubMed
    28. ↵
      1. Microsoft
      . Academic Microsoft. Available: https://www.microsoft.com/en-us/research/project/academic/ [Accessed 11 Apr 2022].
    29. ↵
      1. Soup C
      . Random number generator. Available: https://www.calculatorsoup.com/calculators/statistics/random-number-generator.php [Accessed 06 Jan 2021].
    30. ↵
      1. Loder E,
      2. Loder S,
      3. Cook S
      . Characteristics and publication fate of unregistered and retrospectively registered clinical trials submitted to The BMJ over 4 years. BMJ Open 2018;8:e020037.doi:10.1136/bmjopen-2017-020037pmid:http://www.ncbi.nlm.nih.gov/pubmed/29453302
      OpenUrlAbstract/FREE Full Text
    31. ↵
      1. Button OA
      . Open access button, 2021. Available: https://openaccessbutton.org/ [Accessed 06 Jan 2021].
    32. ↵
      1. Bowley C
      . Bringing open access into Interlibrary loan with the open access button. LIS Scholarship Archive, 2018.
    33. ↵
      1. Larivière V,
      2. Sugimoto CR
      . The Journal Impact Factor: A Brief History, Critique, and Discussion of Adverse Effects. In: Glänzel W, Moed HF, Schmoch U, et al., eds. Springer Handbook of science and technology indicators. Cham: Springer International Publishing, 2019: 3–24.doi:10.1007/978-3-030-02511-3_1
    34. ↵
      1. Chua SK,
      2. Qureshi AM,
      3. Krishnan V, et al
      . The impact factor of an open access Journal does not contribute to an article's citations. F1000Res 2017;6:208.doi:10.12688/f1000research.10892.1pmid:http://www.ncbi.nlm.nih.gov/pubmed/28649365
      OpenUrlPubMed
    35. ↵
      1. Norris E,
      2. Finnerty AN,
      3. Castro O
      . Google form for data extraction. Available: https://osf.io/5vsg6/ [Accessed 06 Apr 2022].
    36. ↵
      1. Hayes AF,
      2. Krippendorff K
      . Answering the call for a standard reliability measure for coding data. Commun Methods Meas 2007;1:77–89.doi:10.1080/19312450709336664
      OpenUrlCrossRef
    37. ↵
      1. Finnerty AN
      . Open science in physical activity Irr script. Available: https://osf.io/t5gw4/ [Accessed 06 Apr 2022].
    38. ↵
      1. Wright AJ, et al
      . Ontologies relevant to behaviour change interventions: a method for their development [version 3; peer review: 2 approved, 1 approved with reservations]. Wellc Open Res 2020;5.
    39. ↵
      1. Ek A,
      2. Alexandrou C,
      3. Söderström E, et al
      . Effectiveness of a 3-month mobile Phone-Based behavior change program on active transportation and physical activity in adults: randomized controlled trial. JMIR Mhealth Uhealth 2020;8:e18531.doi:10.2196/18531pmid:http://www.ncbi.nlm.nih.gov/pubmed/32510462
      OpenUrlPubMed
    40. ↵
      1. Tudor-Locke C,
      2. Schuna JM,
      3. Swift DL, et al
      . Evaluation of Step-Counting interventions differing on intensity messages. J Phys Act Health 2020;17:21–8.doi:10.1123/jpah.2018-0439pmid:http://www.ncbi.nlm.nih.gov/pubmed/31698336
      OpenUrlPubMed
    41. ↵
      1. Kayser JW,
      2. Cossette S,
      3. Côté J, et al
      . A web-based tailored nursing intervention (TAVIE en m@rche) aimed at increasing walking after an acute coronary syndrome: Multicentre randomized trial. J Adv Nurs 2019;75:2727–41.doi:10.1111/jan.14119pmid:http://www.ncbi.nlm.nih.gov/pubmed/31225667
      OpenUrlPubMed
    42. ↵
      1. Nooijen CFJ,
      2. Blom V,
      3. Ekblom Örjan, et al
      . The effectiveness of multi-component interventions targeting physical activity or sedentary behaviour amongst office workers: a three-arm cluster randomised controlled trial. BMC Public Health 2020;20:1329.doi:10.1186/s12889-020-09433-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/32873260
      OpenUrlPubMed
    43. ↵
      1. Poppe L,
      2. De Bourdeaudhuij I,
      3. Verloigne M, et al
      . Efficacy of a Self-Regulation-Based electronic and mobile health intervention targeting an active lifestyle in adults having type 2 diabetes and in adults aged 50 years or older: two randomized controlled trials. J Med Internet Res 2019;21:e13363.doi:10.2196/13363pmid:http://www.ncbi.nlm.nih.gov/pubmed/31376274
      OpenUrlPubMed
    44. ↵
      1. Chambers C
      . What's next for registered reports? Nature 2019;573:187–9.doi:10.1038/d41586-019-02674-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/31506624
      OpenUrlCrossRefPubMed
    45. ↵
      1. Scheel AM,
      2. Schijen MRMJ,
      3. Lakens D
      . An excess of positive results: comparing the standard psychology literature with registered reports. Adv Methods Pract Psychol Sci 2021;4:25152459211007468:251524592110074.doi:10.1177/25152459211007467
      OpenUrl
    46. ↵
      1. Basu AP,
      2. Pearse JE,
      3. Rapley T
      . Publishing protocols for trials of complex interventions before trial completion - potential pitfalls, solutions and the need for public debate. Trials 2017;18:5.doi:10.1186/s13063-016-1757-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/28069042
      OpenUrlPubMed
    47. ↵
      1. Deaton A,
      2. Cartwright N
      . Understanding and misunderstanding randomized controlled trials. Soc Sci Med 2018;210:2–21.doi:10.1016/j.socscimed.2017.12.005
      OpenUrlPubMed
    48. ↵
      1. Tetzlaff JM,
      2. Chan A-W,
      3. Kitchen J, et al
      . Guidelines for randomized clinical trial protocol content: a systematic review. Syst Rev 2012;1:43.doi:10.1186/2046-4053-1-43pmid:http://www.ncbi.nlm.nih.gov/pubmed/23006870
      OpenUrlCrossRefPubMed
    49. ↵
      1. Severin A,
      2. Egger M,
      3. Eve MP
      . Discipline-specific open access publishing practices and barriers to change: an evidence-based review. F1000Res 2020.
    50. ↵
      1. Flanagin A,
      2. Fontanarosa PB,
      3. Bauchner H
      . Preprints involving medical Research-Do the benefits outweigh the challenges? JAMA 2020;324:1840–3.doi:10.1001/jama.2020.20674pmid:http://www.ncbi.nlm.nih.gov/pubmed/33170226
      OpenUrlCrossRefPubMed
    51. ↵
      1. McVay MA,
      2. Conroy DE
      . Transparency and openness in behavioral medicine research. Transl Behav Med 2021;11:287–90.doi:10.1093/tbm/ibz154pmid:http://www.ncbi.nlm.nih.gov/pubmed/31675096
      OpenUrlPubMed
    52. ↵
      1. Michie S,
      2. Johnston M
      . Theories and techniques of behaviour change: developing a cumulative science of behaviour change. Health Psychol Rev 2012;6:1–6.doi:10.1080/17437199.2012.654964
      OpenUrlCrossRefWeb of Science
    53. ↵
      1. Willmott T,
      2. Rundle-Thiele S
      . Are we speaking the same language? call for action to improve theory application and reporting in behaviour change research. BMC Public Health 2021;21:479.doi:10.1186/s12889-021-10541-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/33691658
      OpenUrlPubMed
    54. ↵
      1. Michie S,
      2. van Stralen MM,
      3. West R
      . The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011;6:42.doi:10.1186/1748-5908-6-42pmid:http://www.ncbi.nlm.nih.gov/pubmed/21513547
      OpenUrlCrossRefPubMed
    55. ↵
      1. Osborne C,
      2. Norris E
      . Pre-registration as behaviour: developing an evidence-based intervention specification to increase pre-registration uptake by researchers using the behaviour change wheel. Cogent Psychol 2022;9:1.doi:10.1080/23311908.2022.2066304
      OpenUrl
    56. ↵
      1. Orben A
      . A Journal Club to fix science. Nature 2019;573:465–6.doi:10.1038/d41586-019-02842-8pmid:http://www.ncbi.nlm.nih.gov/pubmed/31551562
      OpenUrlPubMed
    57. ↵
      1. Munafò MR,
      2. Chambers CD,
      3. Collins AM, et al
      . Research culture and reproducibility. Trends Cogn Sci 2020;24:91–3.doi:10.1016/j.tics.2019.12.002pmid:http://www.ncbi.nlm.nih.gov/pubmed/31892459
      OpenUrlPubMed
    58. ↵
      1. Kidwell MC,
      2. Lazarević LB,
      3. Baranski E, et al
      . Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLoS Biol 2016;14:e1002456.doi:10.1371/journal.pbio.1002456pmid:http://www.ncbi.nlm.nih.gov/pubmed/27171007
      OpenUrlCrossRefPubMed
    59. ↵
      1. Gwet KL
      . Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. 4th Ed. Advanced Analytics LLC, 2014I.

    Supplementary materials

    Footnotes

    • Twitter @EJ_Norris

    • Contributors EN: Conceptualisation, data curation, formal analysis, investigation, methodology, project administration, resources, software, supervision, validation, visualisation, writing—original draft, writing—review and editing, guarantor. IS: Data curation, formal analysis, writing—review and editing. ANF: Formal analysis, software, writing—original draft, writing—review and editing. OC: Conceptualisation, data curation, formal analysis, investigation, methodology, writing—original draft, writing—review and editing.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.