Article Text
PDF
Abstract
Objectives To examine the sensitivity/specificity of the Athlete Psychological Strain Questionnaire (APSQ) in both male and female elite athletes, and also assess internal consistency and convergent/divergent validity, and determine discriminative validity relative to current injury status.
Methods Data were provided by 1093 elite athletes (males n=1007; females n=84). Scale validity and reliability values were benchmarked against validated measures of general psychological distress and well-being. ROC curve analysis determined a range of optimal severity cut-points.
Results Bias-corrected area under curve (AUC) values supported three APSQ cut-points for moderate (AUC=0.901), high (AUC=0.944) and very high (AUC=0.951) categories. APSQ total score Cronbach coefficients exceeded those observed for the Kessler 10 (K-10). Gender ×injury status interactions were observed for the APSQ total score and K-10, whereby injured female athletes reported higher scores relative to males and non-injured female counterparts.
Conclusion By providing a range of cut-off scores identifying those scoring in the marginal and elevated ranges, the APSQ may better facilitate earlier identification for male and female elite athletes vulnerable to mental health symptoms and developing syndromes. Use of the APSQ may support sports medicine practitioners and allied health professionals to detect early mental ill health manifestations and facilitate timely management and ideally, remediation of symptoms.
- athlete
- mental health
- distress
- assessment
- injury
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
What are the new findings?
The Athlete Psychological Strain Questionnaire provides cut-off scores to discriminate between ‘moderate’, ‘high’ and ‘very high’ levels of psychological distress in elite athletes.
Female athletes suffering from an injury tend to experience greater psychological distress relative to uninjured female or male athletes.
The Athlete Psychological Strain Questionnaire can be used to support sports medicine practitioners and allied professionals to detect early mental health symptoms in elite athletes.
There is growing interest into athlete mental health, as evidenced, in part, by the International Olympic Committee’s (IOC) Consensus Statement on mental health in elite athletes.1 While self-report data provided by elite athletes indicates comparable prevalence rates of mental health symptoms to the general population (eg, anxiety/depression and sleep disorders),2 3 athletes are also exposed to various sports-related stressors not observed in the general population (such as susceptibility to serious or chronic injury, performance impairments, popular and social media scrutiny and early or unplanned career termination).1 These stressors tend to be experienced during the peak age of onset for mental health disorders,4 which when combined, may increase athletes’ susceptibility to psychological distress and mental health symptoms.
Despite this risk, there is a lack of well-validated athlete-specific screening tools that can assist with early detection of elite athletes experiencing psychological distress. Such screening tools may facilitate more timely intervention in the course of developing mental health symptoms. Rice et al5 developed the multidimensional Athlete Psychological Strain Questionnaire (APSQ), which was modelled on the widely used and validated Kessler 10 (K-10).6 The APSQ is the recommended triage screening tool within the IOC’s Sports Mental Health Assessment Tool, with a positive APSQ result leading to further symptom specific screening and assessment.7
The APSQ’s psychometric evaluation included analyses demonstrating its reliability and validity among a large, representative sample of 1007 male elite athletes from three national Australian sports (Australian football, cricket and soccer), providing a single cut-off score. However, there is a need to provide support and coaching staff working in the elite sport environment with cut-off scores for less severe cases. Such earlier case identification may better facilitate timely referral and intervention, especially among athletes at elevated risk of impaired well-being, such as injured athletes.8 This short report extends earlier APSQ validation work by providing a range of less severe cut-off scores for mental health symptoms, while also including a female elite athlete sample. The aims of this study were to:
Explore the sensitivity/specificity of the APSQ in both male and female elite athletes with regard to their scores on the K-10.
Assess internal consistency and convergent/divergent validity of APSQ in male and female elite athletes.
Assess the discriminative validity of the APSQ to differentiate between injured and uninjured athletes in male and female athletes.
Method
Participants
Data were collected from a sample of 1091 elite athletes (males n=1007; females n=84) from three national Australian sports; Australian football (males only n=676; 84% response rate), cricket (males n=126; 82% response rate, females n=54; 53% response rate), soccer (males n=203; 78% response rate, females national squad n=30; 100% response rate) N’s do not total 100% due to missing data. All athletes were aged 18 years and over, identified as binary gender, and no exclusion criteria were applied (see table 1).
Sample demographics and scale reliabilities
Materials
Respondents completed three validated self-report scales; the 10-item APSQ5 assessing three domains of self-regulation difficulties, performance concerns and externalised coping, in addition to providing a total score; the 10-item Kessler Psychological Distress Scale6 providing a measure of general psychological distress and the 14-item Warwick-Edinburgh Mental Well-Being Scale assessing mental well-being (WEMWBS).9
Procedure
All contracted athletes for the participating sports were invited to participate in an an anonymous online survey, conducted in group-based sessions. Clinician researchers were present to introduce and oversee the data collection, which occurred via tablets. Data collection occurred nationally in local club-based facilities and was coordinated by the participating sports and clubs.
Statistical analyses
The present study used secondary data analysis for the n=1007 male athletes reported in Rice et al,5 and primary data analysis for the included n=84 female athletes. Cronbach alpha coefficients and Pearson correlations were calculated to assess internal consistency and validity (convergent and divergent validity). A series of receiver operating characteristic (ROC) curve analyses were undertaken to identify the sensitivity and specificity of the APSQ total score in discriminating levels of severity on the K-10. Three analyses were conducted, with the ROC curve state variable either ‘moderate’ distress and above (K-10=16+), ‘high’ distress and above (K-10=22+) or ‘very high’ distress (K-10=30+). For each of these analyses, Youden’s (J) index was identified, which is a coefficient maximising the sensitivity and specificity of the respective cut-point.10 Previously, Rice et al5 reported a single preliminary cut-off score on the APSQ for identifying athletes who may be at risk of psychological distress and warrant mental health intervention. In order to facilitate even earlier detection for less severe cases, we sought to identify a series of APSQ cut-points. To guide interpretation, we used the accepted area under curve (AUC) values of >0.90 (very good), >0.80 (good) and >0.70 (fair).11 Between-group analyses examined differences by injury and gender status. Internal validation of AUC values and between-groups outcomes were undertaken with bootstrapping procedures using 10 000 resamples. Analyses were conducted using SPSS V.26.0, with bias-corrected bootstrapping conducted in STATA V.15.0.
Results
In addition to the sample of n=1007 male elite athletes reported in Rice et al,5 this analysis incorporates data from 84 female elite athletes (of whom n=22 were currently injured). Internal consistency values for the APSQ were all in the acceptable range (see table 1) for both male and female athletes, with the APSQ total score values exceeding those observed for the K-10.
ROC curves and APSQ cut-offs
A series of ROC curve analyses were undertaken to identify the sensitivity and specificity of the APSQ total score in discriminating levels of severity on the K-10. The diagnostic odds ratio (ie, positive likelihood ratio ÷ negative likelihood ratio) of the three cut-off points indicted that the very high APSQ cut-off was better at discriminating cases than was the high or moderate cut-offs, both of which reported lower sensitivity and specificity, with bootstrap results validating the AUC values (see table 2).
APSQ ROC curve and cut-point values (including bias-corrected AUC and 95% CIs)
Convergent and divergent validity
Pearson correlations were evaluated separately by gender (see table 3). Moderate-to-strong positive correlations were observed between the APSQ domains and the K-10, providing support for convergent validity. Conversely, moderate negative correlations were observed between the APSQ domains and the WEMWBS, supporting divergent validity. Bootstrapping indicated these correlations were internally valid (see online supplementary table 1).
Supplemental material
APSQ, K-10 and WEMWBS correlations by gender
Gender and current injury effects
Between-groups analysis was conducted to examine gender (male, female)×injury (injured, uninjured) effects (see table 4). Gender ×injury interactions were observed for the APSQ total score F(1, 1067)=17.82, p<0.001, η2=0.016, (bootstrap p=0.001), and also the K-10 F(1, 1082)=13.38, p<0.001, η2=0.012, (bootstrap p=0.001). There was a significant main effect for gender on the WEMWBS F(1, 1064)=18.31, p<0.001, η2=0.017, (bootstrap p<0.001) and a trend-level effect for injury F(1, 1064)=3.86, p=0.050, η2=0.014, (bootstrap p<0.001) but not a significant interaction (p=0.197; bootstrap p=0.225). For the APSQ and K-10 gender ×injury interactions, females experiencing current injury tended to report higher scores, relative to uninjured females or males.
Means, SDS and 95% CIs and bias-corrected 95% CIs by gender and injury status
Discussion
The need for validated athlete-specific questionnaires that are acceptable and appropriate for measuring mental health symptoms in the athlete environment is increasingly manifest.12 The APSQ was designed specifically with the athletic context in mind, relative to behaviours that may indicate underlying distress in sports settings. The results of the present analyses not only support the use of the APSQ among both male and female elite athletes, but provide a validated range of cut-off points, reflecting moderate, high and very high scores. If measured repeatedly at regular intervals, the moderate cut-off point in particular may facilitate earlier case identification and subsequent intervention for athletes vulnerable to mental health symptoms and developing syndromes. The ROC curve AUC values for the three cut-off scores (all >0.90) indicate that the APSQ is ‘very good’ at correctly discriminating moderate, high and very high distress in both male and female athletes, benchmarked against the K-10. Professional sports organisations may consider use of the APSQ as a periodic screening tool, which could be implemented at key stages throughout the training or competitive season, or coinciding with athlete injury or career transition. Future research is needed to explore the prognostic value of the APSQ including associations with clinician diagnosed mental health disorders, in addition to exploring correlations between APSQ scores and current athletic performance or achievement.
Gender and injury effects
The results also indicated that currently injured female athletes reported the highest APSQ scores, as evidenced by the gender ×injury interactions. While we acknowledge that the sample included a relatively small number of injured female athletes (n=22), this result may in part reflect the less resourced supports and well-being services available to female sports, given their relatively earlier stages of professionalisation.13 In addition, the present sample of female athletes was insufficient to conduct confirmatory factor analysis, where samples of >300 cases are recommended. Nonetheless, scale reliability values indicated satisfactory internal consistency of the APSQ among female athletes, with the Cronbach coefficient for the APSQ total score exceeding the value observed for the K-10 (for both females and males). Additional research using the APSQ in larger, representative female athlete samples is required, as is research exploring gender differences for the APSQ.
Conclusion
Through the use of a range of cut-off scores, the APSQ may assist sports medicine practitioners and allied health professionals to better detect the early manifestations of mental health symptoms experienced by both male and female elite athletes. This in turn can facilitate timely management and ideally, remediation of symptoms.
Acknowledgments
The authors thank the participating professional codes and associations for support with project administration and data collection.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @VGouttebarge
Contributors Study conception and management of ethics approval process was undertaken by RP, AGP, SR. Planning and support for data collection and refining measures was undertaken by RP, AGP, SR, PC, PH, ML, AK and BS. Interpretation of data was led by RP, AGP and SR with support from LO, VG, PC, PH, ML, AK, BS and BB. Data analysis was undertaken by SR. Writing of the manuscript was led by SR with all authors contributing to multiple drafts of the manuscript. All authors approved the final submitted version.
Funding Simon Rice was supported by a Career Development Fellowship from the NHMRC (APP1158881). This study was, in part, made possible from funding from several sporting bodies, all of whom requested anonymity.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval Ethical approval for the study was granted by The University of Melbourne Human Research Ethics Committee (ID: 1442705).
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available.