You are here

Article Text

Article menu
Original article
The International Rugby Board (IRB) Pitch Side Concussion Assessment trial: a pilot test accuracy study
  1. Gordon Ward Fuller1,
  2. Simon P T Kemp2,
  3. Philippe Decq3
  1. 1Emergency Medicine Research in Sheffield Group, School of Health and Related Research, University of Sheffield, Sheffield, UK
  2. 2Rugby Football Union, Middlesex, UK
  3. 3French Federation of Rugby, Neurosurgical Department, Medical School René Diderot Paris 7, Assistance Publique-Hôpitaux de Paris, Hôpital Beaujon, Clichy, Paris, France
  1. Correspondence to Dr Gordon Ward Fuller, Emergency Medicine Research in Sheffield Group, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield S1 4DA, UK; g.fuller{at}sheffield.ac.uk

Abstract

Background ‘On the field and on the run’ assessments of head impact events in professional rugby have resulted in a high proportion of players subsequently diagnosed with confirmed concussion not leaving the field of play at the time of injury. The International Rugby Board (IRB) consequently developed a process to support team doctors in the recognition and management of players at risk of concussion, including development of a multimodal assessment instrument—the Pitch Side Concussion Assessment (PSCA) tool.

Methods This was a pilot cohort study designed to determine the feasibility of assessing the accuracy of the IRB PSCA tool in elite male rugby. The study population comprised consecutive players identified with a head impact event with the potential to result in concussion during eight international/national competitions. Players were assessed off field by match-day or team doctors, following a temporary substitution. The accuracy of the PSCA tool was measured against a reference standard of postmatch confirmed concussion, based on clinical judgement aided by an established concussion support instrument.

Results A total of 165 head injury events with the potential to cause concussion were included in the study. The PSCA tool demonstrated a sensitivity of 84.6% (95% CI 73.5% to 92.4%) and a specificity of 74% (95% CI 64.3% to 82.3%) to identify players subsequently diagnosed with confirmed concussion. The negative predictive value was 88.1% (95% CI 79.2% to 94.1%); the positive predictive value was 67.9% (95% CI 56.6% to 77.8%). There were no major barriers identified that would prevent the evaluation of the PSCA process or tool in a future large-scale study.

Conclusions This pilot study has provided the first preliminary estimates for the performance of the PSCA tool, suggesting a potentially favourable balance between positive and negative predictive values. The study has also offered a strong basis to conduct a further larger trial, providing information for sample size calculations and highlighting areas for methodological development.

  • Concussion
  • Head Injuries
  • Injury Prevention
  • Rugby

https://doi.org/10.1136/bjsports-2014-093498

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Background

    Concussion represents a subset of traumatic brain injury, pragmatically defined as “a traumatically induced transient disturbance of brain function caused by a complex pathophysiological process.”1

    The management of head impact events with the potential for concussion is a prominent and contentious issue in collision and full-contact sports.2 ,3 Players who immediately return to play following a concussive event may risk further injuries and exacerbation of concussive symptoms.2

    Although controversial, it has also been suggested that a catastrophic diffuse cerebral swelling can occur rarely following a successive ‘second impact’.4 In addition to these early sequelae, possible causal links between repetitive concussive episodes and increased risk of depression,5 chronic traumatic encephalopathy6 and other neurodegenerative diseases have been proposed.7

    Concerns about sport-related concussion were addressed in the recent consensus statement from the 4th International Conference on Concussion in Sport,8 and in a position statement from the American Medical Society for Sports Medicine,1 aiming to mitigate the risk from concussive events. The importance of a multimodal assessment, with subsequent removal from play for any athlete with suspected concussion, is emphasised in both publications. However, it was acknowledged that the accuracy and reliability of the current on-pitch and sideline assessment tests are ‘largely undefined’.

    Rugby is a highly physical, dynamic, full-contact sport with an estimated incidence of concussion between 4 and 9 injuries/1000 player-match hours in professional competitions.9–12 Despite this burden, the acute management of concussion in elite level rugby may be suboptimal, typically involving brief ‘on the field and on the run’ assessments. For example, the English Rugby Football Union reported that between 2002 and 2006, 58% of confirmed concussion cases were not removed from the field of play despite medical practitioner education.9 The challenges of assessing players with a potential head injury in rugby are well recognised. The laws of the game do not allow free player interchanges for off field appraisal of injuries and the in-game environment may limit the evaluation of important clinical domains or distract players during assessments. In addition, there is concern about the potential for rule manipulation, which was previously observed following the introduction of temporary substitutions for ‘blood injuries’.13

    Consequently, a Pitch Side Concussion Assessment (PSCA) working group was convened by the International Rugby Board (IRB) to improve the management of head injuries and concussion in elite rugby. At the initial meeting of this working group, it was recognised that concussion was a complex injury, difficult to diagnose and assess, and that no validated diagnostic test existed for an immediate conclusive diagnosis. Informed by recent consensus statements, a PSCA process was developed to support team doctors in the recognition of a player at risk of concussion, including a multimodal assessment instrument for the pitchside (rather than on-pitch) assessment of concussion—the PSCA tool.

    The aim of the pilot study reported here was to determine the feasibility of examining the validity of the PSCA tool in a definitive investigation. Specific objectives were to identify barriers for implementing a large-scale study; estimate the prevalence of concussion within the target population to guide future sample size calculations and obtain a preliminary estimate of the sensitivity and specificity of the PSCA tool. Implementation of the PSCA tool was also evaluated in secondary analyses.

    Methods

    Study design and setting

    This was a pilot cohort study designed to determine the feasibility of assessing the accuracy of the IRB PSCA tool in elite male rugby. To maximise internal validity, the study followed expert recommendations on the conduct and reporting of pilot and diagnostic accuracy studies.14 ,15

    Study population

    The source population consisted of elite rugby players participating in eight international/national competitions between October 2012 and July 2013, comprising: The Rugby Championship, Northern hemisphere autumn test matches, ITM Cup, Aviva Premiership, Currie Cup, Super Rugby, Top 14 and ProD2. Further details on participating competitions are provided in web appendix 1. The subsequent study sample included consecutive players identified during play by match officials, team doctors or match-day doctors as having sustained a head impact event, either a direct blow or indirect transmission of force, with the potential for a concussion. Players with immediate, clear clinical signs or symptoms of concussion (eg, confirmed loss of consciousness, tonic posturing, convulsions) were excluded.

    Index test

    The index test under consideration was the IRB PSCA tool (see web appendix 1), an assessment instrument designed to support clinicians in identifying suspected concussion and guide their clinical decisions on removal from play following a head impact event as part of the IRB PSCA process. This instrument includes assessment of four sequential domains, comprising the Maddocks’ questions,16 a tandem balance test,17 a symptom assessment and a brief evaluation of mental status. An affirmative finding in any one of these aspects is regarded as a positive test result indicating suspected concussion, which mandates removal of the player from the game. Negative test results were interpreted within the context of overall clinical judgement, and did not automatically mean a return to play.

    The PSCA tool had a central role within the wider PSCA process, summarised in figure 1. Testing with the PSCA tool could be requested by the match referee, team doctor or independent match-day doctor following a head impact event (either a direct blow or indirect contact with transmission of impulsive force to the head) with the potential for a player to sustain a concussion. Players with immediate, clear clinical signs or symptoms of concussion were directly removed from play and thus not subjected to assessment with the PSCA tool. Additionally, the PSCA process also allowed players to be permanently removed from the game if a player was clinically suspected of having a concussion, regardless of a negative PSCA test result.

    Figure 1
    Figure 1

    Summary of the Pitch Side Concussion Assessment (PSCA) process for evaluating head impact events with the potential for concussion.

    The team doctor had the primary responsibility for conducting tests with the PSCA tool but could delegate this role to the match-day doctor. The test was administered in the stadium's medical room or other agreed venue if this site was too distant from the field. The PSCA process had to be completed within 5 min (absolute time, not playing time), starting when a player left the field of play and finishing when the player presented to the fourth match official for permission to return to play. A temporary player interchange was allowed during the assessment period.

    The overall framework for assessing players with the potential for concussion, and the components of the PSCA tool, were formulated following a review of the scientific literature, consideration of position statements and consultation with sports medicine doctors, coaches, players and referees. The 5 min assessment period was considered sufficient time to safely complete all steps in the PSCA process and judge whether a player had a suspected concussion, with minimal disruption to play in the event of a negative test.

    Reference test

    The reference standard, against which the accuracy of the PSCA tool was tested, was the ultimate clinical diagnosis of concussion determined postmatch with the help of an established concussion support instrument. All players evaluated by the PSCA tool underwent confirmatory postgame testing with the reference standard by their team doctors. The choice of the postgame concussion support tool was at the discretion of individual team doctors and the timing of the application varied from immediately postmatch to 48 h postinjury. Reference standard assessors were not blinded to the results of the PSCA tool.

    Data collection

    Index test findings were recorded contemporaneously using standardised paper forms. Concussion confirmatory test results were also immediately recorded by the administrating clinician, with exact documentation varying according to the specific support instrument used. All data were subsequently collected using a central IRB web-based data portal by specific competition coordinators. All data were anonymised.

    Statistical analysis

    Primary analysis: accuracy of the PSCA tool

    The accuracy of the PSCA tool was assessed in the primary analysis. Sample characteristics were initially examined using descriptive statistics. Each case was then coded according to the index test and reference standard result, with a 2×2 contingency table constructed to determine true positives, false positives, true negatives and false negatives. Prevalence of concussion, sensitivity and specificity, with their 95% CIs, were subsequently calculated. The characteristics of false-negative cases were compared descriptively with true-positive cases to explore the determinants of test sensitivity. The influence of each subcomponent of the PSCA tool on overall accuracy was also examined. Sensitivity and specificity for identifying concussion were computed separately for each of the four individual domains of the assessment instrument.

    Secondary analyses: implementation of the PSCA tool

    Implementation of the index test was investigated by examining the concordance of observed decisions on removal from play with those recommended by the PSCA tool. Sensitivity and specificity were subsequently recalculated to estimate accuracy for reported removal from play decisions, rather than those suggested by PSCA tool results alone, and for the hypothetical contingency of allowing clinician discretion to overrule negative PSCA tool results only.

    Statistical analysis, ethics and funding

    In the absence of unique patient identifiers, it was assumed that each observation was independent with no player sustaining more than one head impact event. A conventional significance level (α) of 0.05 was used in all analyses. Statistical analyses were carried out in Stata V.12.1 (StataCorp, College Station, USA). Agreement with study processes, affirmation of informed consent from study participants and permission for the use of anonymised data were formally confirmed with the presiding medical officers of each participating competition. Research funding was provided by the IRB.

    Results

    Derivation and characteristics of study participants

    A total of 173 consecutive head injury events with the potential to cause concussion were reported during 727 matches (29 080 player hours) over the study period. All 173 players were assessed with the PSCA tool but consent for inclusion of data was not obtained in eight cases, leaving 165 players for inclusion in complete case analyses. There were no missing data on index test parameters or reference standard results. Figure 2 presents a flow chart describing the derivation of study participants.

    Figure 2
    Figure 2

    STARD flow chart showing derivation of study participants for assessment of accuracy of the Pitch Side Concussion Assessment (PSCA) tool.

    A wide range of mechanisms of injury were observed with head contact during tackles predominating (n=68, 41.2%, either being tackled or tackling). PSCA tests were most commonly requested by, and performed by, team doctors (request: n=145, 87.8%; performed: n=117, 70.9%). A range of support tools were used to confirm or refute the final diagnosis of concussion, of which the Sport Concussion Assessment Tool 2nd Edition (SCAT2) was the most common (n=87, 52.7%). Sample characteristics are shown in table 1.

    View this table:
    Table 1

    Characteristics of PSCA tool assessments

    Primary analyses: accuracy of the PSCA tool

    Sixty-five of the included players had a confirmed postmatch clinical diagnosis of concussion, giving a target disorder prevalence of 39% (95% CI 32.0% to 47.3%). Of these, 55 had a positive PSCA test (true positives) resulting in a sensitivity of 84.6% (95% CI 73.5% to 92.4%) for identifying concussion. One hundred players were reference standard negative with no confirmed concussion, of which 74 cases were classified as true negatives with negative PSCA test results. The specificity to correctly identify players without concussion in this study group was therefore 74% (95% CI 64.3% to 82.3%). The positive and negative predictive values of the PSCA tool were 67.9% (95% CI 56.6% to 77.8%) and 88.1% (79.2% to 94.1%), respectively. Table 2 summarises the performance of the PSCA tool in a 2×2 contingency table. Table 3 presents point estimates of metrics of test accuracy together with their precision. The characteristics of true-positive and false-negative cases are shown in web appendix 2. There were no obvious distinguishing features of false-negative cases.

    View this table:
    Table 2

    2×2 contingency table summarising the performance of the IRB PSCA tool

    View this table:
    Table 3

    Statistical metrics describing the accuracy of the Pitch Side Concussion Assessment (PSCA) tool

    The separate subcomponents of the PSCA tool notably varied in their sensitivities and specificities. No individual subscale demonstrated satisfactory sensitivity alone, with clinical symptoms demonstrating the highest sensitivity for identifying concussion (sensitivity 76.9%, 95% CI 64.8% to 86.5%). In contrast, clinical signs, Maddocks’ questions and the tandem balance test all manifested high specificity (specificity point estimates 93–95%). The sensitivity and specificity of individual components of the PSCA tool are detailed in web appendix 2.

    Secondary analyses: implementation of the PSCA tool

    Decisions on exclusion from play deviated from that indicated by the PSCA tool result in 22 (13.3%) cases. Conspicuously, four cases that were ultimately confirmed with concussion had positive findings on subcomponents of the PSCA tool (abnormal tandem balance test: 1; positive symptoms: 3) but returned to play despite their positive PSCA. Conversely, three cases had no documented abnormal finding on any of the PSCA subscales, with removal from the game presumably enforced due to clinical suspicion. There were no significant differences between match-day doctors and team physicians (Pearson's χ2 test, p=0.47) in return to play decisions for patients with positive PSCA test results.

    The sensitivity of actual removal from play decisions for identifying concussion was 83.1% (95% CI 71.7% to 91.2%), with a specificity of 77% (95% CI 67.5% to 84.8%). The positive and negative predictive values for the observed implementation of the PSCA tool were 70.1% (95% CI 58.6% to 80.0%) and 87.5% (95% CI 78.7% to 93.6%), respectively. In the contingency that removal from play had been enforced with a positive PSCA result, but clinical judgement could override a negative PSCA result, the sensitivity of the PSCA tool would have improved to 89.2% (95% CI 79.1% to 95.6%) with a corresponding reduction in specificity to 68% (95% CI 57.9% to 77.0%).

    Web appendix 2 provides further details on observed decisions on removal from play compared with the actual PSCA test results, the accuracy of PSCA test implementation and the performance of the index test under different assumptions for clinician compliance with the PSCA tool.

    Discussion

    Summary of results

    The PSCA tool demonstrated a sensitivity of 85% and a specificity of 74% for the ultimate presence of clinically diagnosed concussion in elite rugby players. Given the reported prevalence, team doctors could be between 79% and 94% sure that a player will not have concussion following a negative PCSA test at a 95% confidence level. False-negative cases appeared to be similar to true-positive patients with no notable differing characteristics. Individual components of the PCSA tool did not have sufficient sensitivity to be used individually for the identification of concussion. There were no major barriers identified to evaluate the PSCA tool further in a large-scale study.

    Interpretation of findings

    An important consideration in determining the utility of the PSCA tool is the risk tolerance for failing to remove a suspected concussion case from play versus the repercussions of excluding a non-injured player. As there are serious concerns regarding the consequences of mild head injuries, a sufficient negative predictive value will be required, necessitating a high test sensitivity given the large observed prevalence of confirmed concussion in players presenting with the potential for concussion. However, given the potential detriment to team performance from excluding non-concussed players, acceptable test specificity is also important to maintain stakeholder credibility in the PSCA tool.

    Although the PSCA tool exhibits potentially favourable test characteristics, instrument subcomponents were individually inadequate, confirming the need for a multimodal assessment of suspected concussion. The very low sensitivity manifested by the Maddocks’ questions alone challenges the common practice of using this test as a single screening assessment.2 Conversely, the relatively higher sensitivity of assessing detailed concussive symptoms may highlight the need for off-pitch assessments.

    Clinicians’ decisions on players’ removal from play deviated from that indicated by the PSCA tool in a substantial minority of cases. The observed lack of compliance with positive PSCA results may reflect lack of training or awareness of the study protocol. Alternatively, in the real world, practice clinicians may have had greater confidence in their own experience and personal clinical acumen than in an experimental assessment tool. Pressure to return players to competitive play did not appear to be a factor in decisions, with no significant difference in non-compliance with positive PSCA results observed between PSCA implemented by team and match-day doctors. Interestingly, there was the potential for improved negative predictive value if negative PSCA test results were overruled on clinical grounds (as intended in the PSCA process), but positive results were fully implemented.

    There are other important practical considerations relevant to the PSCA instrument, in addition to narrow attention to test accuracy. The ease of administration, influence on outcome of matches and acceptability to relevant stakeholders will be of great importance in any implementation. The IRB PSCA procedure is consistent with recommendations from the American Medical Society for Sports Medicine,1 and broadly follows the Zurich consensus statement guidance.8 However, after a head impact event, the Zurich approach advocates an undefined appraisal of symptoms and signs, followed by a 15 min break for players with suspected concussion, prior to a formal concussion assessment. In contrast, the PSCA process immediately and permanently removes players with obvious signs of concussion from play, and objectively screens other players for suspected concussion with the PSCA tool. The opportunity to return to play after the 5 min assessment period, if suspected concussion is excluded, combined with the PSCA tool performance reported herein, is likely to have increased credibility and applicability to competitive rugby.

    The optimal timing for the assessment of suspected concussion has not been fully defined and false-negative cases could arise if players who initially appear normal develop a delayed onset of symptoms.3 It has been reported that up to 16% of players with evidence of concussion at 3 h postinjury are originally asymptomatic, some of whom may be detected with later pitchside assessment.18 A period of rest before testing could also alleviate the potentially confounding influence of fatigue on a player's test performance.8 However, the natural history of concussive symptoms is uncertain, and other suspected concussion assessment instruments have demonstrated superior sensitivity with immediate testing,19 supporting the PSCA tool's early application.

    Comparison with related studies

    Recent systematic literature reviews have highlighted the paucity of studies evaluating the accuracy of pitchside tests for identifying concussion during adult sport.1 ,3 Identified studies reporting measures of test performance are summarised in table 4.16 ,19–21 Although these investigations have been limited to case–control studies, which are well known to strongly overestimate test performance,22 the PSCA test demonstrates comparable or superior sensitivity. The external validity of these previous studies is also questionable given the non-rugby study samples. Additionally, many established concussion assessment tools have not been validated in the pitchside setting, for example, the NFL Sideline Concussion Assessment Tool.23

    View this table:
    Table 4

    Performance of other pitchside tools for assessment of concussion

    There are no other comparable non-concussive pitchside tests with which to compare the PSCA tool's performance. Clinical decision rules used in other injuries, including the Ottawa ankle rule or Ottawa knee rule,24 ,25 are derived for use in hospital environments to determine imaging decisions, and are calibrated to 100% sensitivity. Such tests have demonstrated higher sensitivity than the PSCA test (97.6% and 95.5%, respectively), but at a much lower specificity level (31.5% and 48.6%), which is likely to be unacceptable in professional rugby.24 ,25

    Strengths and limitations

    There are several strengths to this study. The index tests and reference standard were independently applied with no potential for partial verification bias. Additionally, the broad source population should ensure that these results are robust to spectrum effects and are generalisable throughout elite rugby competitions in developed countries.

    However, there are limitations which may challenge internal validity. First, and most importantly, assessment of the reference standard was not blinded to the index test result, with both tests potentially performed by the same clinician. Diagnostic review bias may have resulted and, given the subjective nature of concussion assessment, knowledge of the PSCA result could have influenced the final clinical diagnosis.

    Second, selection bias may have arisen from incomplete enrolment or study dropouts. The average number of PSCA tool tests varied across the studied tournaments, which may suggest that not all players potentially eligible for the PSCA process were identified. Moreover, a small number of players were excluded due to unobtainable consent. Systematic differences between study participants and excluded players could have biased accuracy estimates. Any ensuing selection bias could be ameliorated through video surveillance of matches and improved consent procedures in a future definitive study.

    Third, the reference standard was not standardised for all cases, with variations in timing of the final assessment and in choice of the concussion support instrument used, resulting in a consequent risk of differential verification bias. If there is correlation between the performance of a particular reference standard and certain test results, measures of accuracy could be affected unpredictably. Additionally, even though the final diagnosis of concussion was based on clinical judgement, similarity between assessment domains of the PSCA tool and postmatch concussion support instruments could conceivably lead to incorporation bias.

    Fourth, as acknowledged in the Zurich consensus document, the diagnosis and definition of concussion is an evolving area of medicine.8 Misclassification of the reference standard by inaccurate clinical assessment could therefore lead to errors in the reported accuracy metrics. Players may also have deliberately concealed symptoms to avoid missing games through graduated return to play protocols.26 Furthermore, reference standard misclassification could also have arisen from temporal changes in players’ clinical condition, with some false-positive cases sustaining short-lived concussions that had resolved by the time of the reference standard assessment, although the clinical significance of such injuries is uncertain.

    Fifth, test reproducibility has not been assessed, and unreliability could adversely affect the accuracy and clinical utility of the PSCA instrument. Ideally, the intraobserver and interobserver reliability of administering the index test would be assessed in any future study (eg, using the κ statistic), although the practicalities of independent clinicians implementing the PSCA tool within a 5 min assessment window during a professional sports match may preclude this investigation.

    Sixth, this was a pilot study aiming to test the feasibility of evaluating the PSCA tool in a future conclusive study, rather than to provide precise estimates of sensitivity and specificity. The sample size is relatively small and the reported 95% CIs are consistent with either favourable or unfavourable performance metrics for the PSCA tool.

    Finally, it is possible that individual players sustained multiple head impact events, within or between games, with clustering of outcomes. Owing to the anonymised nature of the data collection, it was not possible to account for this. Although the intracluster correlation might be expected to be minimal, the reported CIs may not have the exactly correct coverage.27

    Conclusions

    This pilot study has provided the first preliminary estimates for the performance of the IRB PSCA tool, suggesting a potentially favourable balance between positive and negative predictive values. The reported results have also offered a strong basis to design and conduct a further larger trial, providing information for sample size calculations and highlighting areas for methodological development.

    The PSCA working group will now proceed to a definitive study to provide a precise estimate of the PSCA tool's accuracy and to evaluate the PSCA process in detail. Study design improvements will include compulsory PSCA training for match-day medical staff, standardised reference standard assessment postgame and at 48 h, and a video review system to identify all suspicious head injuries. Final decisions on the utility of the PSCA process will ultimately depend on a judgement of its observed performance against the perceived risks of missed concussions.

    What are the new findings?

    • Implementation of the International Rugby Board Pitch Side Concussion Assessment (IRB PSCA) tool in elite rugby appears feasible.

    • The PSCA tool demonstrated a sensitivity of 84.6% and a specificity of 74% in the identification of players subsequently diagnosed with confirmed concussion.

    • There were no major barriers identified that would prevent the evaluation of the PSCA process or tool in a future large-scale study.

    How might it impact on clinical practice in the future?

    • Accurate pitchside assessment of players sustaining head injury events will reduce the number of players returning to play with subsequently confirmed concussion.

    Acknowledgments

    The authors gratefully acknowledge the support and input of Dr Martin Raftery (IRB Chief Medical Officer) in the development of the PSCA Tool and Process. The authors also thank the following members of the IRB PSCA working group for their expert recommendations: Michael Makdissi (The Florey Institute of Neuroscience and Mental Health, Australia), Conor McCarthy (Irish RFU Medical Director), Deborah Robinson (New Zealand RU Medical Director), Rob Nichol (Chief executive International Rugby Players' Association), Vincenzo Ieracitano (Chairman Medical Commission of Italian FIR), Mike England (Director English RFU Injured Players Foundation), Paul Watson (Medical Director Rugby Canada – independent advisors); Mark Harrington (IRB Training and Medical Manager – IRB representative). Methodological advice from Matt Cross and Keith Stokes (University of Bath), and data management by Marc Douglas (IRB Research Coordinator), are also gratefully acknowledged.

    References

    1. ↵
      1. Harmon KG,
      2. Drezner J,
      3. Gammons M,
      4. et al
      . American Medical Society for Sports Medicine position statement: concussion in sport. Br J Sports Med 2013;47:15–26.
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Helmy A,
      2. Agarwal M,
      3. Hutchinson PJ
      . Concussion and sport. BMJ 2013;347: f5748.
      OpenUrlFREE Full Text
    3. ↵
      1. Putukian M,
      2. Raftery M,
      3. Guskiewicz K,
      4. et al
      . Onfield assessment of concussion in the adult athlete. Br J Sports Med 2013;47:285–8.
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. McCrory P,
      2. Davis G,
      3. Makdissi M
      . Second impact syndrome or cerebral swelling after sporting head injury. Curr Sports Med Rep 2012;11:21.
      OpenUrlCrossRefPubMed
    5. ↵
      1. Kerr ZY,
      2. Marshall SW,
      3. Harding HP Jr.,
      4. et al
      . Nine-year risk of depression diagnosis increases with increasing self-reported concussions in retired professional football players. Am J Sports Med 2012;40:2206–12.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Gardner A,
      2. Iverson GL,
      3. McCrory P
      . Chronic traumatic encephalopathy in sport: a systematic review. Br J Sports Med 2014;48:84–90.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Mortimer JA,
      2. van Duijn CM,
      3. Chandra V,
      4. et al
      . Group head trauma as a risk factor for Alzheimer's disease: a collaborative re-analysis of case-control studies. Int J Epidemiol 1991;20(Suppl 2):S28–35.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. McCrory P,
      2. Meeuwisse WH,
      3. Aubry M,
      4. et al
      . Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Br J Sports Med 2013;47:250–8.
      OpenUrlFREE Full Text
    9. ↵
      1. Kemp SPT,
      2. Hudson Z,
      3. Brooks JHM,
      4. et al
      . The epidemiology of head injuries in English professional rugby union. Clin J Sport Med 2008;18:227–34.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Fuller CW,
      2. Sheerin K,
      3. Targett S
      . Rugby World Cup 2011: International Rugby Board Injury Surveillance Study. Br J Sports Med 2012;47:1184–91.
      OpenUrlPubMedWeb of Science
    11. ↵
      1. Bathgate A,
      2. Best JP,
      3. Craig G,
      4. et al
      . A prospective study of injuries to elite Australian rugby union players. Br J Sports Med 2002;36:265–9.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Fuller CW,
      2. Taylor A,
      3. Molloy MG
      . Epidemiological study of injuries in international rugby sevens. Clin J Sport Med 2010;20:179–84.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Anderson L
      . Bloodgate: were the punishments fair? Br J Sports Med 2011;45:948–9.
      OpenUrlFREE Full Text
    14. ↵
      1. Bossuyt PM,
      2. Reitsma JB,
      3. Bruns DE,
      4. et al
      . The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 2003;138:W1–12.
      OpenUrlCrossRefPubMed
    15. ↵
      1. Thabane L,
      2. Ma J,
      3. Chu R,
      4. et al
      . A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol 2010;10:1.
      OpenUrlCrossRefPubMed
    16. ↵
      1. Maddocks DL,
      2. Dicker GD,
      3. Saling MM
      . The assessment of orientation following concussion in athletes. Clin J Sport Med 1995;5:32–5.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Spirduso WW,
      2. Francis KL,
      3. MacRae PG
      . Balance and posture: evaluating the multiple dimensions of balance. In: Spirduso WW, Francis KL, MacRae PG. eds Physical dimensions of aging. Champaign, IL: Human Kinetics, 2005:144.
    18. ↵
      1. Guskiewicz KM,
      2. McCrea M,
      3. Marshall SW,
      4. et al
      . Cumulative effects associated with recurrent concussion in collegiate football players: the NCAA Concussion Study. JAMA 2003;290:2549–55.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. McCrea M,
      2. Barr WB,
      3. Guskiewicz K,
      4. et al
      . Standard regression-based methods for measuring recovery after sport-related concussion. J Int Neuropsychol Soc 2005;11:58–69.
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Broglio SP,
      2. Macciocchi SN,
      3. Ferrara MS
      . Sensitivity of the concussion assessment battery. Neurosurgery 2007;60:1050–7.
      OpenUrlPubMedWeb of Science
    21. ↵
      1. Barr WB,
      2. McCrea M
      . Sensitivity and specificity of standardized neurocognitive testing immediately following sports concussion. J Int Neuropsychol Soc 2001;7:693–702.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Rutjes AW,
      2. Reitsma JB,
      3. Vandenbroucke JP,
      4. et al
      . Case-control and two-gate designs in diagnostic accuracy studies. Clin Chem 2005;51:1335–41.
      OpenUrlAbstract/FREE Full Text
    23. ↵
      NFL sideline concussion assessment tool (online). http://nflhealthandsafety.files.wordpress.com/2011/01/nfl-concussion-tool-post-injury.pdf (accessed 4 Nov 2013).
    24. ↵
      1. Bachmann LM,
      2. Kolb E,
      3. Koller MT,
      4. et al
      . Accuracy of Ottawa ankle rules to exclude fractures of the ankle and mid-foot: systematic review. BMJ 2003;326:417.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Bachmann LM,
      2. Haberzeth S,
      3. Steurer J,
      4. et al
      . The accuracy of the Ottawa knee rule to rule out knee fractures: a systematic review. Ann Intern Med 2004;140:121–4.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
    27. ↵
      1. Genders TS,
      2. Spronk S,
      3. Stijnen T,
      4. et al
      . Methods for calculating sensitivity and specificity of clustered data: a tutorial. Radiology 2012;265:910–16.
      OpenUrlCrossRefPubMedWeb of Science
    View Abstract

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Correction notice This paper has been amended since it was published Online First. The third author's affiliation has been changed and the wording ‘French Federation of Rugby’ has been added.

    • Contributors PD and SPTK conceived and designed the study in collaboration with the International Rugby Board PSCA working group. They also critically revised the manuscript for important intellectual content and gave final approval to the version to be published. GWF analysed and interpreted the data, and wrote and prepared the manuscript for publication.

    • Competing interests None.

    • Ethics approval The study protocol received ethical approval from the University of Bath for the Aviva Premiership.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Further details on unpublished data are available on request by contacting the corresponding author, subject to permission from the International Rugby Board.