You are here

Article Text

Article menu

Download PDFPDF

Original article
An observational method to code concussions in the National Hockey League (NHL): the heads-up checklist
  1. Michael G Hutchison1,2,
  2. Paul Comper1,3,
  3. Willem H Meeuwisse4,
  4. Ruben J Echemendia5,6
  1. 1David L. MacIntosh Sport Medicine Clinic, Faculty of Kinesiology and Physical Education, University of Toronto, Toronto, Ontario, Canada
  2. 2Injury Prevention Research Office, Keenan Research Centre, St. Michael's Hospital, Toronto, Ontario, Canada
  3. 3Faculty of Kinesiology and Physical Education & Graduate Department of Rehabilitation Science, University of Toronto, Toronto, Ontario, Canada
  4. 4Faculty of Kinesiology and Hotchkiss Brain Institute, Sport Injury Prevention Research Centre, University of Calgary, Calgary, Alberta, Canada
  5. 5Psychological and Neurobehavioral Associates, Inc, State College, Pennsylvania, USA
  6. 6University of Missouri—Kansas City, Kansas City, Missouri, USA
  1. Correspondence to Dr Michael G Hutchison David L. MacIntosh Sport Medicine Clinic, Faculty of Kinesiology and Physical Education, University of Toronto 55 Harbord Street, Toronto M5S 2W6, Ontario, Canada; michael.hutchison{at}utoronto.ca

Abstract

Background Development of effective strategies for preventing concussions is a priority in all sports, including ice hockey. Digital video records of sports events contain a rich source of valuable information, and are therefore a promising resource for analysing situational factors and injury mechanisms related to concussion.

Aim To determine whether independent raters reliably agreed on the antecedent events and mechanisms of injury when using a standardised observational tool known as the heads-up checklist (HUC) to code digital video records of concussions in the National Hockey League (NHL).

Methods The study occurred in two phases. In phase 1, four raters (2 naïve and 2 expert) independently viewed and completed HUCs for 25 video records of NHL concussions randomly chosen from the pool of concussion events from the 2006–2007 regular season. Following initial analysis, three additional factors were added to the HUC, resulting in a total of 17 factors of interest. Two expert raters then viewed the remaining concussion events from the 2006–2007 season, as well as all digital video records of concussion events up to 31 December 2009 (n=174).

Results For phase 1, the majority of the factors had a κ value of 0.6 or higher (8 of 15 factors for naïve raters; 11 of 15 factors for expert raters). For phase 2, all the factors had a total percent agreement value greater than 0.8 and κ values of >0.65 for the expert raters.

Conclusions HUC is an objective, reliable tool for coding the antecedent events and mechanisms of concussions in the NHL.

  • Concussion
  • Ice hockey
  • Injury Prevention

https://doi.org/10.1136/bjsports-2012-092059

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    The sport of ice hockey (hockey) is characterised by speed, strength and endurance, played by individuals within an enclosed venue. Participation in hockey benefits individuals for numerous reasons. However, similar to other contact or collision sports, hockey also involves an inherent risk of injury. The frequency and types of injuries in hockey vary at different levels of participation,1 yet there is concern for the occurrence of concussion across all levels of play.1–4 Concussion has been defined by Concussion in Sport Group as a brain injury and is a complex pathophysiological process affecting the brain, induced by traumatic biomechanical forces.5

    Given the frequency of concussions in hockey,1 ,6–9 the development of concussion prevention strategies is paramount. However, in order to introduce and implement effective prevention strategies, an understanding of the situational context and injury mechanisms is necessary. The widespread availability and accessibility of video replay technology allows for the thorough analysis of injury mechanisms. Therefore, understanding the sequences leading to sports injuries can inform the development of prevention strategies.

    The use of video replay to understand injury mechanisms in sport is not new. Almost 25 years ago Silver and Gill10 used video analysis to analyse the mechanisms underlying cervical spine injuries in rugby players. They found that most injuries occurred in the ‘ruck’ and ‘maul’ situations, and that such injuries were caused by so-called ‘irresponsible actions.’ Investigators have also used video analysis of injuries in other sports including alpine skiing,11 basketball,12 baseball,13 professional football,14–16 lacrosse,17 rugby,18 and soccer.19 ,20

    Digital video records of sports events contain a rich source of valuable but frequently untapped information, and are therefore a promising resource for the analysis of the antecedent events and mechanisms of concussion. However, research that relies on video records must be conducted systematically, in order to reduce the potential for bias that may occur when investigators unintentionally favour a non-random, selective approach to analysis, which could undermine internal validity.21

    With this in mind, the present paper describes the development and reliability of an objective standardised recording tool referred to as the heads-up checklist (HUC), for coding and analysis of National Hockey League (NHL) concussions via video records. Specifically, the aim of this paper was to determine whether independent raters reliably agreed on factors included in HUC.

    Methods

    Through team athletic trainers/therapists and physicians, the NHL maintains comprehensive player medical records, with injury surveillance being an integral component of that process. Oversight of NHL medical and injury management/prevention policy is governed by the NHL Health Management Panel (HMP). The Concussion Working Group (CWG) reports to the HMP and is comprised of healthcare professionals and representatives from the League, member Clubs and the NHL Players’ Association (NHLPA). The CWG has been tracking and studying various aspects of NHL player concussions since 1997–1998, as part of its concussion programme. The broad aims of the NHL concussion programme include the examination of issues and protocols related to postinjury concussion management and evaluation, return to play and, ultimately, concussion prevention.

    For the present study, the NHL isolated digital video records of events in which players had been medically diagnosed with concussions. These video records were made available to the investigators for coding and analysis. Additional descriptive information such as date of injury, time of injury and known characteristics of the players (ie, age, height, weight, etc) were also provided. The research ethics review board of the University of Toronto approved the study (protocol reference #23882).

    Development of the heads-up checklist (HUC)

    An earlier version of HUC (V.1.0) was developed as part of a pilot project examining the relationship between body contact and injuries at the Canadian Interuniversity Sport (CIS) level.22 The current version (V.1.1) of HUC was redesigned for NHL hockey in collaboration with individuals having considerable hockey expertise (ie, coaches, players, referees, etc), including NHL/NHLPA CWG members. HUC (V.1.1) was developed as a tool that provides a standardised framework for reliably coding and analysing video recorded events for the majority of circumstances and mechanisms leading to concussion in hockey.

    HUC (V.1.1) consisted of 15 general factors organised under three broad categories: (1) Event, (2) Game Situation and (3) Equipment. Within each category, there were several domains containing a list of mutually exclusive factors in which the rater selected only one item. HUC content was based on variables of interest generated by initial exploratory video analysis, the research team's extensive research and experience related to hockey and concussions, consultation with players and coaches external to the research team and review of the literature.

    Phases of HUC development

    There were two distinct phases in the development of HUC. Phase 1 was the initial development of HUC and included four raters (two ‘expert’ and two ‘naïve’) who viewed recorded digital video content of NHL concussion-events. The objective of phase 1 was to ensure the utility of HUC with respect to the concepts and terminology specific to hockey and concussion. The inclusion of both naïve and expert raters allowed for the identification of any limitations or lack of clarity with the factors and definitions included in HUC. Naïve raters were defined as individuals with limited experience who might have played or coached hockey at a competitive level. Naïve raters also had no specialised training in concussion identification or management (ie, were not a healthcare professional). Expert raters were defined as individuals with considerable hockey and concussion expertise, including several years of experience playing or coaching hockey at a competitive level. Following phase 1, HUC was modified for phase 2, where the expert raters viewed and coded a larger pool of concussion events using HUC.

    The study period consisted of concussion events from the beginning of the 2006–2007 NHL season, up to – and including – 31 December 2009. In both phases, raters viewed each event using Quicktime Player Pro V.7.6.6 software resident on an Apple MacBook Pro 4.1 (operating software Mac OS X V.10.5.8).23 Each category of HUC was completed for each distinct concussion event. Raters were allowed to view the event an unlimited number of times, at any ‘playback speed’ as deemed necessary to complete all categories on the HUC.

    In phase 1, each rater independently viewed and completed the HUC for each of 25 events chosen randomly from the pool of events for the 2006–2007 NHL season. For phase 2, two expert raters viewed the remaining concussion events chosen randomly from the 2006–2007 NHL season, up to and including 31 December 2009 (n=174). In summary, a total of 199 concussion events were identified and coded with HUC during the study period (phase 1, n=25; phase 2, n=174).

    Statistical analysis

    Inter-rater agreement emphasises the interchangeability or the absolute consensus between judges and is typically indexed via a consensus value.24 ,25 Total per cent agreement (TPA) is a simple method that uses the ratio of the number of ratings for which both the raters agree to the total number of ratings. κ coefficients are similar, albeit with a correction factor that allows for agreement occurring by chance. That is, κ coefficients are calculated by considering the observed percentage (proportion of rater agreement) and the expected proportion (proportion of agreements that are expected to occur by chance as a result of the rater scoring in a random manner).26 Using the interpretations of κ described by Shrout,27 we categorised κ agreement as substantial (0.81–1.00), moderate (0.61–0.80), fair (0.41–0.60), slight (0.11–0.40) or virtually none (<0.10).

    As noted above, in phase 1, four raters completed the HUC for a sample of 25 concussion events chosen randomly from the 2006–2007 NHL season. TPA and κ coefficients were calculated for the expert and naïve raters separately. For phase 2, TPA and κ coefficients were calculated to determine the strength of inter-rater agreement between the two expert raters for the sample of remaining 174 concussion events.

    Results

    Phase 1

    Table 1 provides a summary of the inter-rater reliabilities for both the naïve and expert raters. For the naïve raters, 12 of the 15 factors had a TPA value of 0.7 or higher. κ Values were lower, ranging 0.23–0.95. Applying Shrout's27 classification for κ coefficients for the results of the naïve raters, 6 of the 15 factors of the HUC were considered to have ‘substantial’ agreement, 2 ‘moderate’ agreement, 3 ‘fair’ agreement and 4 ‘slight’ agreement. However, for the expert raters, 14 of the 15 factors had a TPA value of 0.8 or higher. Also, the expert raters had a greater number of κ values classified as ‘substantial’ agreement (8 factors compared to 6 for naïve raters). The remaining HUC factors for expert raters were considered to have the following agreement ratings: 3 ‘moderate’, 3 ‘fair’ and 1 ‘slight’.

    View this table:
    Table 1

    Inter-rater reliabilities for naïve and expert raters reviewing 25 events

    Overall, the majority of the agreement values were classified as moderate or substantial. However, phase 1 highlighted some of the challenges in ascertaining specific variables of interest relating to discrete factors such as Acceleration of Head, Body Location, Mouthguard, Puck Possession and Scenario. The quantitative discordance values and qualitative feedback from the raters resulted in refinement of the HUC (V.1.1) prior to phase 2, which ultimately led to the development of HUC V.1.2.

    Phase 2

    In phase 2, the Mouthguard factor was eliminated from analysis owing to the overall difficulty in reliably identifying the presence or absence of a mouthguard in video recordings. In addition to revising some operational definitions and adding examples of common responses for the HUC factors with fair and slight agreement in phase 1, three additional factors were added to V.1.2: Score, Anticipated Hit and On-ice Medical Attention were incorporated, resulting in 17 factors of interest in HUC V.1.2 (see figure 1 and table 2). Anticipated Contact and On-ice Medical Attention were included to provide relevant information associated with injury severity and potential patterns of injury mechanisms. The Score factor was added to consider the context of the game to be able to make a future determination whether a relationship between the score of a game and frequency of concussion exists. Table 3 presents the agreement findings for each item on the HUC (V.1.2), including TPA and κ values for the two expert raters. All the factors had a TPA value greater than 0.8. In addition, all factors had κ values of >0.65 and considered to have moderate to substantial agreement. Acceleration of Head (κ=0.73, 95% CI 0.65 to 0.82), Region (κ=0.75, 95% CI 0.64 to 0.85) and Anticipated Hit (κ=0.69, 95% CI 0.59 to 0.79) had the lowest κ values. An overall agreement value was calculated based on the 14 subjectively rated components of the HUC. Penalty, Period and Score were not included in the overall calculation as this information was independently obtained and confirmed from additional sources. The overall agreement of the HUC based on TPA and κ value was 0.91 (SD=0.06) and 0.85 (SD=0.10), respectively.

    View this table:
    Table 2

    Summary of Heads-Up Checklist (HUC) factors

    View this table:
    Table 3

    Phase 2 inter-rater reliabilities for expert raters (n=174)

    Figure 1
    Figure 1

    Heads-Up Checklist, V.1.2.

    Discussion

    The purpose of the present study was to develop a reliable instrument for coding and analysing antecedent factors and mechanisms leading to concussions in hockey using digital video records. Although hockey is a complex and fast-paced game in which it is often difficult to identify and classify various playing actions, moderate to substantial inter-rater agreement for the all the factors developed in the current HUC indicates that the systematic process of extracting and coding video information can be performed with high degree of accuracy.

    One of the main objectives of including both naïve and expert raters in phase 1 was to ensure the HUC and corresponding user manual were developed so that individuals with limited hockey knowledge could be trained to reliably code the majority of events and mechanisms leading to concussion. Overall, the data support the utility of HUC as a useful method to code events reliably via video analysis among independent raters with different levels of hockey knowledge.

    Although the κ coefficients from phase 1 indicate that the inter-rater agreement of the majority of the HUC factors ranged from moderate to substantial, the results from phase 1 identified some practical deficiencies. The factors with lower agreement levels in phase 1 (V.1.1) related to Acceleration of Head, Body Location, Mouthguard, Puck Possession and Scenario. The lower inter-rater reliabilities for these factors were likely a reflection of the raters’ inability to clearly differentiate or identify the appropriate coding response. For example, in the development phase of the HUC, mouthguard use was included as a factor, with the expectation that the higher resolution and the ability to ‘zoom-in’ (which the current digital technology allows) would afford viewers a greater degree of visual detail. However, the ability to correctly identify the presence or absence of a mouthguard in a reliable manner was unattainable, even with the current technology. Consequently, the Mouthguard factor was dropped from the HUC.

    Two other factors with lower inter-rater agreement values—Puck Possession and Acceleration of Head—were of greater concern. The relatively poor agreement on these two factors was related to the lack of clarity in the operational definitions that accompanied the initial HUC. Prior to phase 2, operational definitions of Puck Possession and Acceleration of Head factors were revised. More detailed descriptions with examples of each item were added to the most recent version of the HUC manual to illustrate typical behaviour.

    Finally, with respect to phase 1, the results show that expert raters’ TPA values and κ coeffecients were generally higher. Therefore, prior to using HUC for the first time, it would be useful for naïve raters to receive additional training sessions to improve their familiarity with HUC constructs and operational definitions. This does not need to be time consuming or arduous, and would in most cases simply involve eyes-on experience with an expert rater present to explain colloquial terms such as ‘breakout’ or ‘on the rush’ etc, followed by checking for accuracy.

    In phase 2, along with removing the Mouthguard factor and revisions to a few of the variables (ie, initial Contact With, Acceleration of Head and Puck Possession) and corresponding operational definitions, three additional factors were added to HUC V.1.2. Score, Anticipated Hit and On-ice Medical Attention were incorporated into the HUC, resulting in 17 factors of interest. Of these, the two additional factors for which inter-rater agreement was evaluated—Anticipated Contact and On-ice Medical Attention—had moderate and substantial inter-rater agreement, respectively. Both of these factors were added to the HUC to provide relevant information associated with injury severity and potential patterns of injury mechanisms. For instance, are collisions that result in an injured player receiving medical attention on the ice—often seen as spectacular to some viewers—associated with any particular injury mechanism?

    Common sense suggests that the video analysis approach is an appropriate method to record playing situations and athlete–opponent movements, but insufficient to provide detailed biomechanical information. The Acceleration of Head factor was included to qualitatively describe the rotation of the head as an attempt to capture and document the movement of the head related to mechanism of injury. Nevertheless, it was important to attempt to document such an information as the literature suggests that greater severity of brain injury is associated with axonal shearing that accompanies head rotation.28 ,29 However, the Acceleration of Head factor scored fairly low in phase 1, so it was revised prior to phase 2. This factor had the lowest inter-rater agreement coefficient in phase 2. Thus, the analysis of a detailed biomechanical component is likely beyond the functional utility of the HUC.

    V.1.2 of HUC included 17 factors of interest and the results of the second phase indicated that all the factors had a TPA of greater than 0.8, a cut-off commonly regarded as necessary for agreement.25 ,30 In addition, all factors had κ coefficients of greater than 0.6, reflecting moderate-to-substantial agreement. The use of a reliable coding tool with digital video content allows for the investigation of potential patterns of injury and antecedent and situational factors associated concussions in the NHL. This process allows for sport-specific knowledge of the mechanisms of—and the associated risk factors leading to—injury to inform the development and implementation of prevention strategies. A future study might explore the relationship between mechanism of injury and outcome factors including recovery trajectories, symptom clusters, neurocognitive sequelae and return-to-play disposition.

    With a framework established for the video analysis of concussion, it is reasonable to assume that HUC can be applied to various levels of play (eg, university or collegiate) or modified for other sports (eg, football and soccer) when digital video records are available, in order to provide relevant information that informs concussion preventive strategies.

    Limitations

    HUC has practical utility, but some methodological shortcomings warrant discussion. First, although HUC sufficiently captures situational contexts and mechanisms of concussion, it is important to keep in mind—consistent with previous video analysis studies—that the exact moment of injury cannot be determined with this method.31 Second, the amount of high quality video coverage available in the NHL allowed for a relatively unique opportunity to conduct a project of this nature; but, some of the digital video was of less than optimal quality and/or was captured from a limited number of views. Overall, the proportion of lower quality digital video images was low, but this may have contributed to the lack of agreement between raters. With this in mind, the video analysis approach may not be as effective as in other situations in which the quantity and quality of video imaging records may be of variable or lower quality as in the NHL.

    Conclusion

    HUC is a reliable tool for coding and analysing the situational context and the mechanism of concussion for expert raters. Data obtained from the HUC will allow for a better understanding of concussions in professional hockey, which can be used to generate prevention strategies at the NHL level.

    What are the new findings?

    • Although hockey is a complex and fast-paced game in which it is often difficult to identify and classify various playing actions, moderate-to-substantial inter-rater agreement for all the factors developed in Heads-Up Checklist (HUC) indicated that the systematic process of extracting and coding video information can be performed with high degree of accuracy.

    • The use of a reliable coding tool such as HUC allows for the investigation of potential patterns of injury and antecedent and situational events-associated concussions in the NHL.

    How might it impact clinical practice in the near future?

    • Understanding the factors and mechanisms leading to injury will inform the development of effective prevention strategies.

    • Future use of the HUC combined with clinical information will allow researchers to explore the relationship between mechanism of injury and outcome factors including recovery trajectories, symptom clusters, neurocognitive sequelae and return-to-play disposition.

    Acknowledgments

    The authors are grateful to members of the NHL/NHLPA Concussion Working Group: Dr Mark Aubry, Mr Bill Daly, Ms Maria Dennis, Ms Julie Grand, Dr John Rizos, Mr Rob Zamuner and also Mr Jim Ramsay for their insight throughout the research study. We would also like to thank Ms Cynthia Lichter of NHL Operations for her ongoing assistance with the videos. PC and MGH would like to acknowledge Dr Angela Colantonio of the University of Toronto for her ongoing advice and support through the development of the manuscript.

    References

    1. ↵
      1. Flik K,
      2. Lyman S,
      3. Marx RG
      . American collegiate men's ice hockey: an analysis of injuries. Am J Sports Med 2005;33:183–7.
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Goodman D,
      2. Gaetz M,
      3. Meichenbaum D
      . Concussions in hockey: there is cause for concern. Med Sci Sports Exerc 2001;33:2004–9.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Marchie A,
      2. Cusimano MD
      . Bodychecking and concussions in ice hockey: should our youth pay the price? CMAJ 2003;169:124–8.
      OpenUrlFREE Full Text
    4. ↵
      1. Pashby T,
      2. Carson JD,
      3. Ordogh D,
      4. et al
      . Eliminate head-checking in ice hockey. Clin J Sport Med 2001;11:211–13.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. McCrory P,
      2. Meeuwisse WH,
      3. Aubry M,
      4. et al
      . Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Br J Sports Med 2013;47:250–8.
      OpenUrlFREE Full Text
    6. ↵
      1. Benson BW,
      2. Meeuwisse WH,
      3. Rizos J,
      4. et al
      . A prospective study of concussions among National Hockey League players during regular season games: the NHL-NHLPA Concussion Program. CMAJ 2011;183:905–11.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Echlin PS,
      2. Tator CH,
      3. Cusimano MD,
      4. et al
      . A prospective study of physician-observed concussions during junior ice hockey: implications for incidence rates. Neurosurg Focus 2010;29:E4.
      OpenUrlCrossRefPubMed
    8. ↵
      1. Emery CA,
      2. Hagel B,
      3. Decloe M,
      4. et al
      . Risk factors for injury and severe injury in youth ice hockey: a systematic review of the literature. Inj Prev 2010;16:113–18.
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Emery CA,
      2. Kang J,
      3. Shrier I,
      4. et al
      . Risk of injury associated with body checking among youth ice hockey players. JAMA 2010;303:2265–72.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Silver JR,
      2. Gill S
      . Injuries of the spine sustained during rugby. Sports Med 1988;5:328–34.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Ettlinger CF,
      2. Johnson RJ,
      3. Shealy JE
      . A method to help reduce the risk of serious knee sprains incurred in alpine skiing. Am J Sports Med 1995;23:531–7.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Krosshaug T,
      2. Nakamae A,
      3. Boden BP,
      4. et al
      . Mechanisms of anterior cruciate ligament injury in basketball: video analysis of 39 cases. Am J Sports Med 2007;35:359–67.
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Athiviraham A,
      2. Bartsch A,
      3. Mageswaran P,
      4. et al
      . Analysis of baseball-to-helmet impacts in major league baseball. Am J Sports Med 2012;40:2808–14.
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Pellman EJ,
      2. Viano DC,
      3. Tucker AM,
      4. et al
      . Concussion in professional football: reconstruction of game impacts and injuries. Neurosurgery 2003;53:799–812; discussion 12–4.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Viano DC,
      2. Casson IR,
      3. Pellman EJ
      . Concussion in professional football: biomechanics of the struck player—part 14. Neurosurgery 2007;61:313–27; discussion 27–8.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Viano DC,
      2. Pellman EJ
      . Concussion in professional football: biomechanics of the striking player—part 8. Neurosurgery 2005;56:266–80; discussion 66–80.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Caswell SV,
      2. Lincoln AE,
      3. Almquist JL,
      4. et al
      . Video incident analysis of head injuries in high school girls' lacrosse. Am J Sports Med 2012;40:756–62.
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. McCrory PR,
      2. Berkovic SF
      . Video analysis of acute motor and convulsive manifestations in sport-related concussion. Neurology 2000;54:1488–91.
      OpenUrlCrossRef
    19. ↵
      1. Andersen TE,
      2. Floerenes TW,
      3. Arnason A,
      4. et al
      . Video analysis of the mechanisms for ankle injuries in football. Am J Sports Med 2004;32(1 Suppl):69S–79S.
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Arnason A,
      2. Tenga A,
      3. Engebretsen L,
      4. et al
      . A prospective video-based analysis of injury situations in elite male football: football incident analysis. Am J Sports Med 2004;32:1459–65.
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Grimes DA,
      2. Schulz KF
      . Bias and causal associations in observational research. Lancet 2002;359:248–52.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      Hutchison MG, Mainwaring L. Incidence and mechanism of cerebral concussion in varsity ice hockey: development of an observational checklist. National Undergraduate Research Conference; March 28, 2003; University of Toronto.
    23. ↵
      Mac OS X Version 10.5.8 [program]. Cupertino, CA: Apple.
    24. ↵
      1. Fleiss JL,
      2. Levin B,
      3. Paik MC
      . Measurement of interrater agreement. Statistical methods for rates and proportions. 3rd edn. Hoboken, NJ: John Wiley & Sons, Inc, 2004.
    25. ↵
      1. House AE,
      2. House BJ,
      3. Campbell MB
      . Measures of interobserver agreement: calculation formulas and distribution effects. J Behav Assess 1981;3:37–57.
      OpenUrlCrossRef
    26. ↵
      1. Cohen J
      . A coefficient of agreement for nomina scales. Educ Psychol Meas 1960;20:37–46.
      OpenUrlCrossRefWeb of Science
    27. ↵
      1. Shrout PE
      . Measurement reliability and agreement in psychiatry. Stat Methods Med Res 1998;7:301–17.
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Meaney DF,
      2. Smith DH
      . Biomechanics of concussion. Clin Sports Med 2011;30:19–31.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Ommaya A,
      2. Gennarelli T
      . Cerebral concussion and traumatic unconsciousness. Correlation of experimental and clinical observations of blunt head injuries. Brain 1974;97:633–54.
      OpenUrlFREE Full Text
    30. ↵
      1. Hartmann DP
      . Considerations in the choice of interobserver reliability estimates. J Appl Behav Anal 1977;10:103–16.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Krosshaug T,
      2. Nakamae A,
      3. Boden BP,
      4. et al
      . Mechanisms of anterior cruciate ligament injury in basketball: video analysis of 39 cases. Am J Sports Med 2007;35:359–67.
      OpenUrlAbstract/FREE Full Text

    Footnotes

    • Contributors PC, MGH, WHM and RJE were the main contributors to the content of this manuscript and were involved in the study's design. MGH mainly contributed to the statistical analysis. PC and MGH are the guarantors. All authors interpreted the results, drafted the manuscript and decided to submit the manuscript. All authors had full access to all the data in the study.

    • Funding This study received funding by Ontario Neurotrauma Foundation (ONF) and The Pashby Sports Safety Fund. The organisations that contributed funds to the research had no role in the design of the study, the collection, analysis and interpretation of the data.

    • Competing interests PC is a member of the Concussion Working Group and a clinical neuropsychologist consultant with the NHLPA, for which he receives remuneration, but he did not receive any funding from the NHL or the NHLPA to conduct the present research. WHM is a consultant to the NHL. No financial remuneration was received for activities specifically related to this project. RJE is a consultant to the NHL, MLS and the US Soccer Federation for which he receives financial remuneration. No financial remuneration was received for activities specifically related to this research.

    • Ethics approval University of Toronto.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • ▸ References to this paper are available online at http://bjsm.bmj.com

    Linked Articles