Article Text
PDF
Abstract
Background Elevated neck pain prevalence among skydivers is associated with exposure to repeated parachute opening shock (POS). A study is planned to evaluate a preventive free fall acrobatics intervention, but prior assessment of the protocol is necessary given the complex and safety-critical study environment.
Aim To validate an intervention protocol to reduce POS neck loads.
Methods A protocol was developed based on observational data and theoretical calculations. Six experts rated each component of the protocol on a four-point Likert scale, regarding relevance, simplicity/feasibility and safety, and responded to open-ended questions. Two iterations were made, each followed by consensus panel protocol revisions. The content validity index (CVI) was used to quantify ratings. A measure of universal agreement (CVI/UA) was computed as the proportion of components that achieved a rating ≥3 by all raters. For safety, a high-sensitivity CVI/UA was computed with a rating of no <4 (highest score) as acceptable.
Results CVI/UA for relevance increased from 0.80 in the first assessment to 1.00 in the second; for simplicity from 0.50 to 0.63; and for safety from 0.70 to 1.00. High-sensitivity CVI/UA for safety increased from 0.10 to 0.75. Responses to open-ended questions included safety concerns for free fall stability, altitude awareness and concerns over comprehensibility.
Conclusions The proposed protocol has been improved in assessed relevance, simplicity and safety, and is considered validated for the start of the empirical trial. To what degree complex interventions should be preceded by open prevalidation is discussed.
- Risk factor
- Intervention
- Prevention
- Ethics
- Evaluation
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Summary box
Expert opinion was used to systematically validate a complex and safety-critical intervention study protocol before the start of the trial.
In an iterative process, experts were invited to give quantitative scores, qualitative comments and suggestions to each discrete component of the proposed protocol, followed by consensus panel protocol revisions.
As a result of this process, the proposed protocol underwent substantial changes and improved in assessed relevance, simplicity and safety.
Other complex or safety-critical intervention studies may benefit from systematic and open validation prior to the start of the trial.
Introduction
Rationale for an intervention
The processes through which interventions are developed form more or less visible cornerstones of modern healthcare. If possible side effects are unacceptable, their preclusion must take precedence to protect test subjects, who within the trial are first in line to exposure. Among lessons learned from the TGN1412 tragedy was a call for greater transparency throughout the development process.1 Article 16 of the Declaration of Helsinki states that medical research projects involving human subjects should be preceded by careful assessment of risks and that their designs should be publicly available.2 Systematic early integration of expert opinion in study development has been proposed to address why randomised controlled trials have failed to find treatments to sepsis.3 One subject suggested as unfit for a randomised controlled trial because of unacceptable side effects is the impact of the opening of a parachute on the parachutist.4 Unfortunately, parachutes, besides being life-saving, can also be harmful. Parachute opening shock (POS) is a sudden and brutal deceleration of a human being. In skydiving (sport parachuting from aircraft), it slows a free falling skydiver from a velocity >200 to <30 km/h within a few seconds. POS deceleration magnitudes 9–12 times Earth's gravitational acceleration (a dimensionless ratio denoted G) have been measured.5 These hard openings can be painful, and a number of very hard openings have generated injuries visible to healthcare systems.6 During subjectively normal openings, decelerations measured on the human neck exceed 4 G with initial onset rates (jerks) exceeding 20 G/s.7 Considering that active skydivers may perform 10 jumps/day and may accumulate well over a thousand jumps during a parachuting career,8 these are problematic values. Fighter pilots have suffered neck pain after less accelerative exposure.9 ,10 In the Swedish skydiving population, neck pain prevalence is 45%,11 as compared to a general population estimate of 37%.12 Recently published data show that skydivers’ neck muscles are under excessive strain during POS,13 and data from our group (manuscript in preparation) suggest POS as composed of biomechanically discrete phases. A first phase contains an initial jerk in ventral to dorsal direction, that is, ‘pulled backwards’, denoted negative Gx,14 when the skydiver is rapidly rotated from a prone belly-to-earth body position to an upright position. During this phase, the moment arm from the centre of mass of the head to the parachute connection point at the shoulders is long and likely to yield a high torque in the neck. The second phase, denoted positive Gz, contains the bulk of POS-deceleration directed caudally to cranially. Entering the second phase with the neck flexed forward from the jerk would put the neck muscles in a clear disadvantage.
Injury prevention through athletic technique
Physical hazards may be conceptualised as related to technological, environmental or human factors.15 Arguably, an elegant solution to the POS problem is a technological invention, but while waiting for, metaphorically, ‘silver bullet’ equipment, the sum of technological factors in POS will remain at Pareto optimality, affected by considerations made by athletes when purchasing, packing and maintaining their parachutes. The average skydiver in an average skydive leaves the aircraft with a parachute system that will not have the best of possible openings. This real-world POS will over time, and over jumps, yield the accumulated exposure. Deployment altitude (air density) is known to affect POS,16 but in regular skydiving, this variable is standardised at c. 1000 m above mean sea level, where ambient pressure is around 0.9 bar. If we, by human factors, mean variables that are operator dependent, similar to excessive speeding in road traffic or reliable image acquisition in sonography, the question raised is: What can a skydiver at terminal velocity do to have the least harmful parachute opening possible? In a sport, it would seem desirable to prevent injuries by the way the sport is practiced. A number of techniques to reduce POS neck loads have been suggested among athletes,17 two of which are biomechanically appealing: Reducing parachute deployment airspeed and positioning the human body head high prior to main parachute extraction. Whether these techniques actually reduce neck loads during POS has not been systematically evaluated. From an empirically determined relation between maximum POS deceleration and free fall velocity,5 it can be calculated that a decrease in velocity from 220 to 190 km/h may reduce the maximum deceleration and thereby reduce the (constant mass) force 25%. Such a velocity reduction is possible using the human body only. Our static anthropometrical assessments suggest that, unless a forward flexion of the head occurs, pitching up the body, head high to an angle of 45° from the flat-belly-to-relative-wind plane, may reduce the head-neck lever arm 30%. Thus, a successful combination of velocity reduction and head-neck lever arm reduction holds the promise of an approximately halved torque in the neck during POS. Such a substantial mechanical change can be hypothesised to have measurable biological effects.
Rationale for study protocol validation
It is suggested that risk assessment in research with humans should be considered in context.18 The UK Medical Research Council holds that complex interventions work best if tailored to local circumstances, and recommends attention to intended contexts.19 In a report on the Space Shuttle Challenger disaster, Feynman advocated practical engineering judgement in risk assessment.20 Such lines of thought suggested confronting experts of parachuting with the intended study protocol. Conceptually, our sought measure may be perceived as a form of validity, implicating use of expert rating for assessment.21 An iterative dialogue was desired, including expert opinion on the relevance and feasibility of the intended intervention protocol.
Aim
The aim of this study was to validate a free fall acrobatics intervention protocol to reduce parachute opening neck loads.
Methods
Study design
Validation was performed by iterative expert assessments with consensus panel revisions after each loop. Experts rated written athlete instructions of the protocol and responded to open-ended questions. Ratings were quantified using a four-point Likert scale and the content validity index (CVI), with the intervention subdivided into conceptual components: ‘items’.22 Each item was rated in three domains: Relevance, simplicity (ie, feasibility) and safety (table 1).
Rating framework for pre-assessment of a proposed intervention study protocol to reduce parachute opening neck loads among skydivers, undertaken before the start of the trial
Expert raters and consensus panel
Six experienced skydivers, five men and one woman, from different geographic locations in Sweden, were invited and volunteered to participate as independent expert raters of the proposed intervention protocol. Three of these skydivers were clinical experts with research experience (one nurse, one orthopaedic surgeon, one physiotherapist; two of whom held PhD degrees and one who was an advanced PhD candidate) and the other three were world class skydivers with extensive experience in parachuting safety. One was a licensed equipment expert. All of the experts gave their written informed consent before taking part in the study. Confidentiality and the voluntary nature of the study were stressed. The raters were informed that they could withdraw at any time without giving any reason and that no data could be linked to any individual rater. The consensus panel consisted of the two present authors. AW is a parachutist (>2500 jumps) and MD; BOA is a biomechanics expert and registered physical therapist. Both are experienced researchers.
Assessment process
Two iteration loops were made. An initial intervention protocol was proposed in November 2014, viewed by both authors as relevant, feasible and safe before submission by email to the raters. Two discrete major intervention components were categorised: (1) free fall terminal velocity reduction and (2) head high pitched up overall body attitude. Nine detailed intervention components were verbalised and categorised as subsets (1.1–1.3 and 2.1–2.6, given in online supplementary appendix A), including a deliberate anomaly in the first iteration, for calibration purposes.23 The entire intervention as a whole was also assessed, yielding a total of 12 items for the first assessment. Item-level CVI (I-CVI) proportion in agreement was computed for each item as the number of experts giving a rating of either 3 or 4, divided by the total number of assessments of the item.24 An I-CVI ≥0.78 was considered ‘excellent’.24 An I-CVI for the proposed intervention as a whole (treating it as an item in itself) was also calculated. A measure of universal agreement among experts (CVI/UA) was computed as the proportion of individual items that achieved a rating ≥3 by all the experts.24 For the domain of safety, a high-sensitivity CVI/UA was computed with a rating of no <4 (highest score) as acceptable. Computation of CVI/UA excluded the deliberate anomaly and the ‘item’ of the proposed intervention as a whole. The raters were invited to leave free-text comments to every item. Comprehensiveness of the instructions and safety aspects of the proposed intervention were further examined by open-ended questions. Identified problem areas and recommendations for adjustments from the first assessment were analysed by the consensus panel, and a revised intervention protocol was proposed in December 2014 (see online supplementary appendix A). The same six experts were invited to participate as raters in a second assessment of the revised protocol, following the same procedures and methods as the first. One of the experts declined due to reasons unrelated to the intervention protocol or to study participation. Consequently, the second assessment was performed by five experts. Concerns and suggestions in the second assessment were addressed by the consensus panel in a final revised protocol, ending the iterative process.
Basic outlines of the intervention protocol under study
The initial intervention protocol was based on observational data and static anthropometrical calculations, and inspired by anecdotal information.7 ,13 ,25 It was conceived as a combination of velocity reduction and head-neck lever arm reduction, main outcome variables being magnitudes of decelerations and jerks (expressed in G and G/s, respectively) and magnitudes of surface electromyographic signals (expressed as the percentage of reference maximum voluntary electrical activity), and research subjects being experienced skydivers completing two consecutive skydives on the same day with random ordering (one being the control jump and one being the intervention jump).
Results
Relevance
CVI/UA for relevance increased from 0.80 in the first assessment to 1.00 in the second (table 2). One item, instructing participants to follow standard reserve parachute activation procedures in case of a main parachute malfunction, receiving a relevance I-CVI of 0.83 in the first assessment, was omitted from the manoeuvre instructions, its intellectual content instead incorporated in the general instructions for study participation. The deliberate anomaly (item 2.6) received an I-CVI of 0.50 for relevance and was omitted.
Item content validity index (I-CVI) and universal agreement (CVI/UA) for the first and second assessment loops of a proposed intervention study protocol to reduce parachute opening neck loads among skydivers
Simplicity (feasibility)
CVI/UA for simplicity increased from 0.50 in the first assessment to 0.63 in the second. Simplicity concerns in the first assessment included comprehensibility of the instructions; among comments were a caution against using aircraft terminology, for example, pitch, roll and yaw,26 for athletic instruction. In the second assessment, concerns about comprehensiveness and verbosity remained. One expert perceived the revised item 2.2 as similar to the warning text: “…Caution, do not light a fire inside the tent, it might catch fire”, that is, overly detailed. An illustration (item 2.5) that received an I-CVI of 0.80 was omitted in the revised protocol.
Safety
CVI/UA for safety increased from 0.70 to 1.00 between the first and second assessments. High-sensitivity CVI/UA for safety was 0.10 in the first assessment, increasing to 0.75 in the second. Safety concerns in the first assessment included altitude awareness and free fall stability. Adding instructions clarifying that the free fall terminal velocity reduction manoeuvre must be initiated at sufficient altitude and that a stable body position has priority over minimising fall rate, was suggested. Expert responses to the proposed instructions for obtaining a head high pitched up overall body attitude were notably detailed with divergent technique suggestions and safety concerns regarding free fall stability. One expert wrote: “If I just extend my arms and pull my knees in fully, that's how we teach a backflip, right? We obviously need to make sure no one ever does a backflip during [parachute] deployment.” In the second assessment, safety concerns remained regarding the head high pitched up overall body attitude; one expert suggested adding the explicit instruction that “if an unintended backflip or instability occurs, abort the intervention and activate your parachute according to normal procedure—flat belly stable position”. An illustration (item 2.5) that received poor simplicity ratings also received a safety I-CVI of 0.80 in the first assessment and was omitted in the revised protocol. In the second assessment, one expert suggested using photographs to demonstrate the manoeuvres.
Discussion
Safety concerns and improvements
We sought to pre-assess, prior to the start of the trial, the relevance, simplicity (feasibility) and safety of a proposed intervention study of the human neck response to parachute openings, which is a complex and safety-critical event. Six parachuting experts independently rated the proposed study protocol in a systematic, iterative process. The most important results were safety concerns regarding altitude awareness and free fall stability in the first assessment, unlikely to have been pinpointed by a local ethics committee or similar research governance body, as they lack specific expert competence. The protocol was revised and submitted to the same raters for a second iteration loop, yielding substantial assessed safety improvements.
Free fall instability and loss of altitude awareness
The risk for an unintentional backflip during parachute deployment, as a side effect to the head-neck lever arm reduction manoeuvre, is a serious concern. Several fatalities in skydiving have been attributable to unstable parachute activation with subsequent line entanglement.27 It was pointed out in the assessments that the velocity reduction manoeuvre may, as well, produce a less stable body position. Thus, both major components of the proposed intervention may cause, with possible interaction, a high-risk situation for a human test subject. Another perceived risk noted in the first assessment was loss of altitude awareness, which remains, even after the widespread implementation of automatic reserve activation devices, a cause of skydiving fatalities.28 The importance of altitude control was stressed in rater comments, and it was suggested that the intervention, since it may contribute to loss of altitude awareness, should be undertaken at a considerably higher altitude than the lowest required for main parachute activation, set by current Swedish regulations at 700 m above ground level.29 Revisions of the protocol included emphasis in the instructions on maintaining free fall stability, if need be at the cost of not performing the intervention manoeuvres, and explicit instructions to deploy the main parachute no lower than 1200 m altitude.
Relevant but difficult
Scores for relevance were already high in the first assessment, rising further after revisions. Scores for simplicity were, by contrast, low in the first assessment, and did not improve, in absolute values, to the same levels as the other two domains after revisions. This may be interpreted as the proposed intervention study being assessed as a worthwhile but athletically challenging endeavour. Given unfavourable remarks for comprehensiveness, persisting into the second assessment, part of the poor ratings for simplicity may reflect didactic shortcomings. The entire intervention as a whole, assessed as an item in itself, received the highest possible scores for relevance by all raters in both the first and second assessment. The suggestion to use photography to demonstrate the intervention manoeuvres will be considered and possibly supplemented by free fall or wind tunnel videography. Poor simplicity scores for the illustration (omitted in the revised protocol) discourages from the use of schematic drawings.
The content validity index
Quantifications of the iterative reappraisals were made using a simple hand-calculated estimate commonly used for scale validation, the CVI. Within the CVI framework, subject matter experts rate, through multiple iterations, successive designs of a construct before implementation,22 a method noted for effectiveness in refining and discarding individual components.24 CVI has desirable features such as emphasising agreement of relevance rather than agreement per se, but has been criticised for a failure to adjust for chance agreement; this has, however, been addressed.24 In risk management, it was recently used for prior validation of a safety checklist.30 Published recommendations for scale CVI interpretation may not be fully applicable to our results, but the ‘excellent’ criterion, all items having I-CVIs ≥0.78,24 was met. Increasing sensitivity for safety above standard CVI calculations and computing a universal agreement measure (CVI/UA) with a rating of no <4 (highest score) by all the experts as cut-off yielded only 0.10 in the first assessment of safety, increasing to 0.75 in the second assessment. We valued this as one of the most important quantitative outcomes of the iterative process.
Prior validation of complex interventions
It may be debated whether it is worth the time and available resources to scrutinise planned research studies in a systematic process of the kind we employed. A slight ambiguity can be sensed in our work, since the described process strove to improve both scientific value and test subject safety; these are discrete concerns. We termed our approach to integrating expert opinion ‘validation’—other labels may be suggested. In a mature and non-complex field of research without much risk, prior validation may be considered unnecessary. In complex, dangerous, or financially costly research programmes, improvements in any of the three domains examined here would seem to justify the effort of a systematic validation process.
Study limitations
The soundness of an inter-rater agreement estimate is influenced by the selection of raters. We strove to follow published recommendations for CVI methodology,31 including relevant training and experience, a history of publications in refereed journals and national presentations, heterogeneity and clinical expertise. The raters were not viewed as representative samples of the target population, since they were convenience chosen. A greater number of raters would have exposed the protocol to greater scrutiny, and a greater number of iterations may have brought further changes to the protocol, as may a larger consensus panel of greater heterogeneity (consisting of experts other than just the present two authors) have done. Given time and available resources, the validation process described here was considered appropriate for its purpose. Remaining concerns and suggestions from the second assessment were addressed in the final revised protocol that has been submitted to the Local Ethics Committee for ethical evaluation, in accordance with the Declaration of Helsinki.2
Conclusion
The proposed protocol has been improved in assessed relevance, simplicity and safety and is considered validated for the start of the empirical trial. It is suggested that complex or safety-critical intervention studies should be preceded by systematic and open pre-validation.
Acknowledgments
The authors thank the Swedish National Centre for Research in Sports (Centrum För Idrottsforskning, CIF) and Gösta Fraenckel's Foundation for Medical Research, for financial support. They also thank Jenny Nilsson, Kristofer Gladh, Riccardo Lo Martire, David Bengtsson, Marcus Marcusson and Ann Lindberg, for constructive criticism during development of the first intervention protocol.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors Both authors contributed to study design and data analysis. AW was responsible for data collection and preparation of the manuscript, which has been read and approved by BOA.
Funding This study was funded by the Swedish National Centre for Research in Sports (Centrum För Idrottsforskning, CIF) and Gösta Fraenckel's Foundation for Medical Research.
Competing interests None declared.
Ethics approval The Local Ethics Committee of Stockholm has been offered the paper on the present results in its entirety as a part of a formal application for ethical review of the proposed free fall acrobatics intervention study (2015/1189-31).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.