Protocol

Studying Contact Replays: Investigating Mechanisms, Management and Game Exposures (SCRIMMAGE) for brain health in the Australasian National Rugby League: a protocol for a database design

Abstract

Concussions in contact sports are challenging for athletes, health professionals and sporting bodies to prevent, detect and manage. Design of interventions for primary prevention, early recognition of concussion and continuing to improve postconcussion management are essential for protecting athletes and promoting brain health. Over the last decade, there have been advancements in video technology for analysing head impact events and improvements in the clinical management of concussions. This study protocol describes how researchers, clinicians and staff from the Australasian National Rugby League (NRL) have brought these advancements together and developed a database of videos with head impact events and clinical outcomes. The intended outputs from this work will enhance the understanding of head impact events in NRL, from biomechanical and gameplay factors to concussion and return to play outcomes. Publishing this protocol increases the transparency of this large-scale effort to better identify head impacts and their relationship to concussions and player movement behaviour to contextualise these variables to generate new knowledge and support the reproducibility of these emerging findings. Between 2017 and 2023, over 5250 head contact cases were recorded in the database, from which >1700 head injury assessments were performed, and >600 concussions were diagnosed. Future studies using these data are planned to inform both primary and secondary injury prevention initiatives, such as risk analysis and prediction of game scenarios that result in concussion, as well as investigation of features and factors that help to inform the duration of recovery and return to play.

What is already known on this topic

  • Initiating the medical evaluation of a potential concussion during sports events occurs when there is suspicion of a possible injury.

  • Clinical suspicion may arise from observing a substantial head impact, observable concussive signs and/or the disclosure of symptoms by athletes.

  • This is difficult in a fast-paced contact sport (ie, professional rugby league match) because neurological signs may not be witnessed, and/or the athlete may minimise symptoms.

  • Video-supported identification of head impact events is now widely used on the sideline of professional rugby league games to facilitate the immediate recognition of a potential concussion.

  • The quick removal of an athlete suspected of having sustained a concussion from play and the initiation of a medical evaluation is designed to promote clinical recovery and encourage a safe and gradual return to play.

What this study adds

  • This protocol describes the development of a research database comprising a 7-year series of head impact event videos with linked findings of subsequent head injury assessments and clinical outcomes for athletes in the National Rugby League.

  • A future research programme to advance the understanding of sport-related concussion is outlined.

How this study might affect research, practice or policy

  • Data-driven insights from this research programme are intended to support new knowledge and robust decision-making related to the primary prevention, timely identification and medical management of concussions.

Introduction

A sport-related concussion is a mild traumatic brain injury that results in clinical signs and symptoms that can vary from person to person.1 In the absence of objective tools and tests, both the identification and diagnosis of concussion are challenging. Standardised tests that include subjective reporting of symptoms and objective measures of neurological functioning, such as those in the Sport Concussion Assessment Tool fifth Edition (SCAT5),2 assist healthcare professionals in the diagnostic process and support clinical decisions. The initiation of a medical evaluation for concussion, such as a head injury assessment (HIA), largely relies on the observation of signs in an athlete that are consistent with a potential traumatic brain injury (eg, following an impact, the player is unresponsive, slow to get up or ataxic) and/or athlete disclosure of symptoms (eg, describing symptoms to the medical team or sports trainer). The reliance on player disclosure or observation by officials and/or health professionals to initiate the medical evaluation process can lead to missed cases during games, particularly because of the fast pace of matches and deliberate minimisation of symptoms by athletes who hope to continue playing.3 4 Video analysis has emerged as a useful adjunct to support the identification of potentially concussive head impacts because it enables the replay of incidents from different angles and at various playback speeds (eg, slow motion).5 6

Video analysis has been used to better understand head impact events for over two decades. For example, in 2000, video playback was used to identify the contact site and calculate the kinematic data of the impact of medically confirmed concussion cases in professional rugby league, rugby union and Australian football.7 This research was an early example of using match videos to analyse head injuries. Still, the study was limited by its input data (ie, video sampled at 25 Hz) and perspective error of two-dimensional positional video data. Since that study was published, the potential to use video analysis in research to inform policy and practice has advanced substantially.

Previous rugby league research has linked signs and symptoms of concussion to video-identified on-field head impact events, demonstrating that valuable biomechanical, contextual and medical insights can emerge from studies involving video review.8–10 Earlier rugby league research was focused on specific, investigator-initiated questions or was limited in sample size: our research programme’s initial work published in 2015 only sampled three clubs from a single NRL season (2013),8 while work published in 2021 used data from two NRL seasons (2017–2018).11

To date, the video-supported identification of head impact events in the NRL and the subsequent clinical decisions and outcomes has been made on a case-by-case basis for players at the time of their injury to inform their clinical management. This study protocol describes creating a central database for head impact events, HIAs and clinical outcomes. Publishing this protocol increases transparency related to this large-scale effort to better identify head impacts, concussions and their contextual variables surrounding these gameplay situations to generate new knowledge and support the reproducibility of these emerging findings. Study registration and prespecification of a protocol are also recommended by consensus research recommendations (eg, Strengthening the Reporting of OBservational Studies in Epidemiology),12 and research and funding agencies (eg, National Institutes of Health). This protocol also allows other research teams to align data collection and coding of information to facilitate future data comparison across settings.

Methods

The described database was created from seven NRL Premiership seasons, 2017 through 2023 inclusive. Data collection concluded at the end of the 2023 season. Coding of the variables from video footage is in progress at the time of publication.

Patient and public involvement

This protocol was developed with NRL medical and performance staff, representatives from the NRL Research Committee and NRL players. Together with an international multidisciplinary research team, these stakeholders have driven the study goals and data collection to ensure the research leads to insights that can improve player safety in the sport.

In accordance with the NRL and Rugby League Players Association Collective Bargaining Agreement, all athletes consented a priori to Rugby League Research Committee endorsed research using deidentified injury data. For this study, professional player contracts vary, and the players’ Collective Bargaining Agreement has changed over time, but these factors have not impacted the design or conduct of this research.13

Setting

In Australia, the National Rugby League (NRL) Premiership is the highest rugby league club competition for men. The number of teams and matches has varied since 2017 (first inclusion year for this study; online supplemental table 1). In 2023, 17 teams (16 from Australia and 1 from New Zealand) competed across 27 rounds, with all teams having three ‘byes’ (non-playing weeks).

Each match has 2 teams, each with 13 players on the field and 4 regular substitute players on the interchange bench. An interchange can take place at any time during a match. From the 2016 season to the present (2024), each team can use up to eight interchanges per game. In the fifth year of the study (2021), an additional medical substitute (‘18th player replacement’) was introduced that can be activated when 2 players in the same team are unable to continue in the same game following a concussion and/or game-limiting physical injury due to foul play from the opposing team (where an opponent is either permanently removed from the game or sent to the ‘sin bin’, a 10 min exclusion from the game where the player must leave the field of play and sideline, usually the dressing room).

Three rule changes have been introduced in the NRL during the observation seasons that are relevant to the occurrence of head impact events, HIAs and concussions. First, in the mid-season of 2020, a new rule for ruck infringements was introduced with ‘six again’ instead of a penalty. The six again rule restarts the tackle count for the attacking team and was intended to increase the speed of play without the need for game stoppage for a penalty. Second, since 2022, the off-site Injury Surveillance Bunker has included an Injury Surveillance Medical Officer (medical doctor independent of teams), supported by two video spotters who review play in real-time to identify potential head impact events. The Injury Surveillance Bunker Medical Officer will notify a specialist camera operator who has radio contact with the referee to stop the game if they believe the player has demonstrated signs of concussion. Finally, shoulder charge definitions and rulings have varied since first being banned in 2012. A shoulder charge, in which a defending player contacts/bumps an opposition player with the upper arm/shoulder while not attempting to tackle or steal the ball, is considered high-risk for injury and was outlawed from the game.

The in-game medical evaluation of a player suspected of having sustained a concussion referred to as the HIA in the Australian NRL, enables the removal of a player from the match for up to 15 min without the replacement player counting towards the team’s limited interchanges. During the HIA, the club doctor conducts a formal medical assessment for concussion using standardised testing (ie, SCAT). In the NRL, an HIA may be initiated when: (i) a player shows signs or discloses symptoms of a concussion; (ii) it is deemed necessary by the club medical doctor and club sports trainer following a brief on-field assessment or (iii) it is recommended by an independent medical doctor with access to multiple camera angles to review any incident from an off-site medical bunker. The independent doctor will contact the team doctor to initiate an HIA if a concerning head impact event is identified by video review. The club sports trainer is responsible for signalling to the referee that a player is being removed from play for an HIA by patting the top of their head with one hand. Video playback of gameplay is available on the sideline at each NRL and NRL Women’s Premiership (NRLW) game to support the team doctor by allowing immediate review of gameplay, mechanism of head impact and the postimpact event behaviour of the athlete.

Case definitions

Head impact event

A ‘head impact event’, defined as direct or indirect contact resulting in sudden contact or change of direction to the head, was the primary criterion for including cases. The operator of the sideline injury surveillance system identifies head impact events. All identified head impact events are included in the database. Not all head impact events require an HIA; medical assessment is only initiated when there is cause for further assessment and follow-up. The recording of a head impact event has evolved over time. Initially (ie, during the 2017–2020 seasons), the recording of head impact events came from one of two sources: (i) sideline surveillance—operators of team surveillance system or club medical staff can tag a player who had an impact to the head but was not removed from play for HIA or (ii) the injury surveillance bunker—the independent medical doctor in the bunker can tag a player who had an impact to the head but was not removed from play for HIA. From the 2021 season, all surveillance was centralised in the injury surveillance bunker. That is, the sideline was only equipped with a monitor, and all operations were managed via the injury surveillance bunker. In the injury surveillance bunker, the spotter tags events that are reviewed by the independent doctor and communication, if required, is made between the injury surveillance bunker and the team medical staff (figure 1).

Figure 1
Figure 1

Visualisation of injury surveillance bunker (ISB) and its relationship with the sideline injury surveillance (SIS) and independent bunker doctor (diagram reproduced from an internal document, shared with permission from National Rugby League).

Head injury assessment

Includes all athletes removed from play under the NRL HIA protocol by the team or independent medical doctor for an in-game medical evaluation.

Concussion

Includes all athletes who enter the NRL concussion management protocols. (ie, cases that were either diagnosed in-game or postgame).

Inclusion criteria and sample size

The video data collected as part of this research programme is a census sample of the identified head impact events, the HIAs and the medically diagnosed concussions from a game. The HIA results and the return-to-play information are available for most of the overall sample.

Data acquisition and storage

Over the course of the research programme, the video data provision process has changed. Initially (ie, 2017–2019), notification of any head impact events and the time at which they occurred was provided by email to the lead researcher by the Chief Medical Officer of the NRL. Access to video footage of the head impact event was obtained through the NRL’s Digital Press Pass subscription.

Subsequently (2020–2023), in-game email alerts for each head impact event were provided to the lead researcher. Video clips of the events were available for review through NRL stored systems. The lead researcher accessed and downloaded the relevant video clips, securely storing them on university-managed hardware and software, with additional encryption protection, and backing them up to secure online platforms accessible only to authorised research team members.

Data processing

Once the case videos were obtained, the coding of player, match, tackle and other event characteristics is completed. The variables of interest are presented in online supplemental tables 2–4, with a description and rationale for their inclusion provided below. Each case is coded by at least one of four possible reviewers. At least two reviewers code video signs of concussion to calculate inter-rater reliability. Each case requires approximately 5 min to review.

Player and game characteristics

The collection and analysis of data on player position and role allowed for a better understanding of patterns related to those who experience head impacts, sustain injuries and the phases of play involved. The player and game variables and options for selection are presented in online supplemental table 2. The incidence of concussion varies by playing position, with forwards tending to have a lower risk of concussion than backs.14 There is evidence of a difference in concussion risk for players in an offensive or defensive phase of play, although the direction (who is at greater risk) is inconsistent.9 15 Characteristics of play, such as time elapsed, may be studied in the future for insight into events and timing of higher risk. Research in rugby unions suggests that different injuries and risks are experienced at various periods in a match.16–18 To enable exploration of the timing of head impact events, the time on the match clock for each case was reported and categorised.

Event and tackle features

Examining the on-field positioning and the first point of physical contact characteristics during head impact events may help to identify potential prevention strategies. To this end, a range of precontact, at-contact and postcontact variables are coded. The intention is to analyse the movement and positions of both the tackler and ball carrier to determine if there are common patterns or movement behaviours across different scenarios that led to a head impact event. The variables and potential responses included in the database are presented in online supplemental table 3. These were selected and refined from several studies in rugby league and rugby union that have investigated tackle technique, head injury, and video assessment.9–11 19–22

Signs and symptoms of concussive injury

On-field signs of a potential concussion were recorded (online supplemental table 4), in alignment with the items outlined by the international consensus definitions of video signs of concussion in professional sports,23 and a study of concussion signs specific to NRL players.24 The symptoms the player experienced during the HIA were collected via self-report using SCAT5. The SCAT5 is relevant for this study, as testing was conducted from 2017 to 2023; SCAT6 was published in 2023.25

Clinical reports (HIA and concussion outcomes)

All HIAs are performed by club medical staff. The results of the in-game HIA are retrospectively added to the database at the conclusion of each season. This ensures that coding of player and game characteristics, event and tackle features and the video signs of potential concussive injury are independent of (ie, blind to) the eventual clinical outcomes. The administration and storage of the HIA data was provided via a smartphone application from CSx (2017–2020) and subsequently by HitIQ (2021–2023).

Planned applications

The early and accurate identification of concussions, along with data-informed primary and secondary prevention strategies, supports an evidence-based approach to managing the care of rugby league athletes and promoting brain health. Stakeholders within the NRL, including players, teams, administrators, medical staff and researchers, have supported the information collection detailed in this study protocol. We have provided a descriptive overview of the video footage and clinical outcomes collected and compiled over 7 years. The subsequent processing and preparation of the database will facilitate multiple studies relying on data from >5250 head impact events, including >1700 HIAs and >600 concussions.

Screening efficiency

The intention of the procedural changes introduced by the NRL, and subsequently captured in this database, was to improve concussion recognition and management of player welfare. With increased knowledge about player, game and tackle characteristics that result in clinical outcomes (eg, diagnosed concussion), there is potential to reduce ‘false alarms’ (ie, removing players from the game for HIA when it is not indicated). The occurrence of a head impact event can be likened to a medical screening test used to identify those players who need to undergo a diagnostic test (ie, an HIA). For certain health conditions and in many healthcare settings, diagnostic testing of everyone is considered inefficient and costly, so screening is used as a preliminary step to identify those with an elevated risk of a condition, which in turn justifies the conduct of a diagnostic test (often more invasive and costly). Therefore, one of the major aims of the Studying Contact Replays: Investigating Mechanisms, Management and Game Exposures in the Australasian National Rugby League (SCRIMMAGE) study is to evaluate and improve screening efficiency. This will be realised through studies identifying if there are common characteristics or behaviours of the player(s), game and/or tackle, or a particular combination of factors consistently associated with specific outcomes (eg, cleared to return to play or confirmed diagnosis).

Observable signs

While video review can assist in identifying potential concussive events, previous studies have shown there are still challenges in discerning signs of head impact from observation. For example, in a recent study involving a video review of 25 medically diagnosed concussions from the Queensland Rugby League, none of the six international consensus video signs of possible concussion were seen in 28% of these cases.26 Similarly, 53% of concussions in the NHL did not have any video signs of concussion.27 These prior findings were from databases with relatively small case numbers (<50 and <1000), and it is envisaged our larger number of cases will allow opportunities for diverse studies.

Concussion incidence and changes in gameplay

Another major aim of SCRIMMAGE is to evaluate if there have been changes in the incidence or clinical features of concussion over the last 7 years. We will analyse the number of cases by different units of time (cases per number of players and cases per hours of play), and for different categories (eg, by player role as ball carrier or tackler). Alongside these studies, we will explore if there have been changes in gameplay characteristics (eg, types of tackles, number of players involved in tackles, speed of the game) that might have influenced outcomes (eg, increased severity of concussion or provided protection against reduced incidence).

Inter-rater and intra-rater reliability

Due to the volume of data, complete coding (100% of variables) for most cases has only been performed by one reviewer. At least two reviewers have coded all video signs of potential concussion for all cases. Methodologically focused research will be undertaken to continue refining the methods and processes of coding to maximise data reliability. For example, investigating the qualifications and training required for video review and coding is important. High inter-rater reliability was reported in an earlier study between expert reviewers who both had at least one season working at an NRL club with responsibility for concussion management (including identifying and assessing athletes with suspected concussions).28 These expert reviewers were compared with those of ‘naïve’ reviewers with limited understanding of rugby league and no experience in concussion management. Low agreement between ‘naïve’ reviewers was reported.28

Novel analyses

Alongside traditional methods of disease frequency and comparisons (ie, reported as incidence, prevalence and relative risks; reliability testing), novel approaches to these data will be explored. We intend to use machine learning (a subtype of artificial intelligence) to identify patterns and insights from the data. One example is to explore the three-dimensional kinematics of players’ movement before and during contact through pose estimate (eg, model-based image matching).29 30 In addition, computational modelling will be used to estimate brain strain and its association with clinical outcomes.31

Strengths and limitations

This paper describes the construction of a head impact event database that includes >80 variables from 6 years of professional rugby league competition. All cases have multiple video angles with high-definition footage provided from up to 10 or more cameras, and a subset of these have medical data gathered from players who undergo subsequent diagnostic screens. The study is focused only on professional NRL players, so findings of these cases will not be immediately generalisable to other settings, such as community rugby league. Separate work is underway for the NRLW Premiership.

Training injuries are not currently included. Recent data from men’s teams in the European Super League suggest concussions can occur in training but are less frequent compared with match play.32 This may be an accurate assumption based on the scheduled training session (ie, the drills) and the level of intensity being less than match play. However, injury surveillance and medical support are often not as thorough during training, and some injuries may go unrecognised unless the athlete self-reports symptoms. Potentially, HIA or concussion outcomes from training could be included in the database. However, video data from training is unavailable and thus could not be included. Finally, we do not have records of players’ data from their junior years, lower grades or any code-switches, as head impacts are limited to NRL career matches only.

Conclusion

Continuing to develop an understanding of the risk factors and mechanisms contributing to concussions in athletes is critical for reducing these injuries during practices and competitions. This multidisciplinary collaborative research programme identifies key variables and data sources to develop a large dataset linking player, game and tackle features with head impact events and associated clinical variables. Planned studies will inform both primary and secondary prevention efforts in the NRL.