Article Text
PDF
Abstract
Background Musculoskeletal (MSK) complaints comprise a large proportion of outpatient visits. However, multiple studies show that medical school curriculum often fails to adequately prepare graduates to diagnose and manage common MSK problems. Current standardised exams inadequately assess trainees’ MSK knowledge and other MSK-specific exams such as Freedman and Bernstein’s (1998) exam have limitations in implementation. We propose a new 30-question multiple choice exam for graduating medical students and primary care residents. Results highlight individual deficiencies and identify areas for curriculum improvement.
Methods/Results We developed a bank of multiple choice questions based on 10 critical topics in MSK medicine. The questions were validated with subject-matter experts (SMEs) using a modified Delphi method to obtain consensus on the importance of each question. Based on the SME input, we compiled 30 questions in the assessment. Results of the large-scale pilot test (167 post-clerkship medical students) were an average score of 74 % (range 53% – 90 %, SD 7.8%). In addition, the tool contains detailed explanations and references were created for each question to allow an individual or group to review and enhance learning.
Summary The proposed MSK30 exam evaluates clinically important topics and offers an assessment tool for clinical MSK knowledge of medical students and residents. It fills a gap in current curriculum and improves on previous MSK-specific assessments through better clinical relevance and consistent grading. Educators can use the results of the exam to guide curriculum development and individual education.
- musculoskeletal medicine
- education
- assessment
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Summary points
MSK30 is a new tool to assess the musculoskeletal (MSK) knowledge of medical trainees.
The exam fills a gap in medical school and primary care residency programme curriculums.
Results of the exam guide MSK curriculum development and improve physician knowledge of MSK complaints.
Introduction
Musculoskeletal (MSK) conditions make up a large proportion of outpatient and emergency department visits. Based on National Ambulatory Medical Care Survey data, the United States Bone and Joint Initiative estimates that one in two Americans are affected by MSK conditions which cost $874 billion annually and comprise 18% of all healthcare visits.1–3 Certain populations such as athletes and the military experience an even higher burden of MSK disease and injury. Given the high prevalence and cost, competence in the diagnosis and management of MSK complaints is critical for primary care physicians.
Despite the recognised MSK disease burden, clinical MSK medicine continues to be underrepresented in medical school curriculum and few MSK curriculum evaluation tools have been developed.4 In 1998, Freedman and Bernstein (FB) published a Basic Competency Examination in Musculoskeletal Medicine.5 This exam consists of 25 short-answer questions and was approved by orthopaedic and internal medicine programme directors (PDs) who recommended a passing score of 73.1% and 70%, respectively.6 They administered the exam to all medical and surgical residents on the first day of postgraduate Year 1 at the Hospital of the University of Pennsylvania, representing graduates from 37 different medical schools. The average score was 59.6% and only 18% passed the exam. The authors validated their examination by distributing the final exam to orthopaedic surgery and internal medicine residency PDs nationwide. The PDs rated each question in terms of importance on a scale of 1–10 (not important to very important) and 24 of the 25 questions scored an average of at least five points and determined to be ‘at least important’. The study did not evaluate any psychometric factors. No other MSK competency exams have been investigated and popularised in the literature since the introduction of this exam 20 years ago.
Several medical schools and residencies have used FB’s exam with very similar results. Harvard administered the exam to fourth year medical students and only 26% earned a passing score of 70%.7 Tufts School of Medicine used the same exam to evaluate their new MSK module and had a mean exam score of 40% and only one student (of 405) passed.8 At the University of Arizona, the exam was administered to primary care residents (family medicine, internal medicine, paediatrics and emergency medicine) in their third year of residency as well as a group of fourth year medical students and a control group of orthopaedic residents. They found that the medical students who took an orthopaedic rotation had a mean score of 60.3% compared with 45.1% for those who did not. The primary care residents had an average score of 56.3% and the subgroup of emergency medicine residents had a mean score of 77.5%, whereas the orthopaedic residents had a mean score of 90%.9
The Uniformed Services University (USU) also implemented the FB examination to evaluate the MSK knowledge of its graduating students. USU graduates serve in the active duty military or Public Health Service and proficiency in managing MSK conditions is especially critical because, in the military health system, MSK complaints comprise 25% of all outpatient encounters.10 USU has a comprehensive MSK education curriculum which includes an pre-clerkship 8-week MSK module, 8 additional hours of MSK trauma care in the pre-clerkship period, minimum of 4 hours of MSK training during clerkship period and 11 hours of MSK didactics/training in the post-clerkship period plus any elective rotations. USU fourth year medical students took the FB exam and average score for all 172 students was 68.5% (SD 11.9%).
The MSK curriculum directors expressed concern that the examination did not represent the material necessary for primary care providers to properly triage and manage MSK complaints. To assess this concern, the FB exam was also administered to a group of board certified family medicine, physical medicine and rehabilitation (PM&R), and emergency medicine physicians beginning their primary care sports medicine fellowship; this group of 21 physicians had an average score of 76.8% (76% passed with score >70%). Given that these physicians should have a competency in primary care and have demonstrated specific interest in MSK medicine, these poor results support the concern that the FB exam may not be the most effective tool to assess clinical MSK knowledge in primary care physicians.
Another limitation encountered with FB’s basic competency exam was that the short-answer nature of the exam makes reliable grading more difficult. Exam graders scored the same answers differently based on interpretation of the original grading instructions and acceptance of other responses as correct though they were not the correct answer in the original publication. Therefore, we created a new multiple-choice assessment to evaluate the broad range of clinical MSK problems encountered in clinical practice.
Methods
The questions used for the assessment were developed and refined based on a list of common and critical topics (table 1) in MSK medicine by a core group of stakeholders in orthopaedic surgery, primary care sports medicine and medical students. The goal of the questions was for the learner to correctly identify MSK conditions and proper initial management to include referral for specialty care. This large bank of questions was narrowed to 45 questions based on the feedback of the abovementioned group.
Topics included in MSK knowledge assessment
The importance and quality of each question was determined using a modified Delphi technique to survey subject-matter experts (SMEs) and obtain consensus. These SMEs represented a wide range of specialties with a focus in MSK medicine to include two orthopaedic surgeons, seven primary care sports medicine physicians, one family physician, one physical therapist, one rheumatologist and six primary care sports medicine fellows (board certified in PM&R, family medicine or emergency medicine). The SMEs rated each question on a scale of 1–5 (1, not at all important; 3, moderately important and 5, extremely important) and provided additional written feedback on the overall effectiveness and difficulty of the assessment. These importance ratings were averaged and questions with a rating of 4 or greater were kept for the final question set while ensuring coverage of a broad range of topics. The full question bank (45 questions) was also pilot tested with fourth year medical students at USU to assess question quality and psychometric factors.
After final question selection and editing, the final 30-question exam was piloted with a group of Family Medicine interns. Following this, a full-scale test of the final product under exam conditions was administered to the entire USU class of 2019 after completion of their clerkship year. Online supplementary appendix A contains the complete exam with the correct answers in bold font.
Supplemental material
Results
A total of 18 SMEs participated in the modified Delphi process. The average importance rating for the initial 45 questions was 4.18 (range 3.39–4.89, SD 0.34). The average importance of the final 30 questions used in the exam was 4.32 (range 3.82–4.89, SD 0.28).
The pilot test of 45 questions was completed by 18 fourth year medical students and the average score was 80.9% (SD 7.8%). A group of seven family medicine interns also completed the final 30 question exam with an average score of 75.1% (SD 8.9%).
The final 30-question examination underwent full-scale implementation with the USU class of 2019. These 167 students had just completed their core clerkship rotations and took the new MSK knowledge assessment under testing conditions. The average score was 74% (range 53%–90%, SD 7.8%), equivalent to 22.3 of 30 questions correct. The distribution of scores is shown in figure 1. No passing score was established but the results of the exam will be used to assess the curriculum. Any topics that presented problems for a large proportion of students will be supplemented and reinforced subsequently in the curriculum.
Results of musculoskeletal knowledge assessment in post-clerkship medical students.
Discussion
This new 30-question multiple choice assessment of clinical MSK medicine was created with significant involvement of a variety of stakeholders. All questions were vetted through a multistep process to obtain consensus and ensure the exam represents the most clinically important topics. The goals of the examination are for the learner to identify common and critical musculoskeletal conditions, select appropriate initial management and know when to refer a patient for specialty care. The format of the exam decreases the impact of interpretation (compared with short answer style) and allows for improved analytics. This clinical MSK knowledge assessment tool can be implemented in medical schools and primary care residency programme to assess the individual learner’s MSK knowledge and evaluate the programme’s MSK curriculum. Trainees can evaluate their own weaknesses and fill in knowledge gaps using the accompanying answer explanation guide and included references. Programme can trend overall scores between classes as the curriculum evolves or look at individual questions responses. If a low percentage of trainees correctly answered a particular question, then that topic is likely not addressed effectively by the current MSK curriculum.
It is impossible to cover every important topic in MSK medicine with an exam short enough for easy implementation but the most relevant and important questions were selected based on the input of SMEs. This exam only contains 30 questions and does not come with alternate versions so it likely cannot be administered to the same learner repeatedly. However, it can be implemented at a defined point in the medical school curriculum or residency programme so that a programme can evaluate its trainees at that point and trend its results over time.
The next goal for this project is broader implementation in other medical schools and primary care residencies to validate its reproducibility and generalisability. MSK conditions will continue to make up a large proportion of primary care visits in the future and the next generations of physicians must be prepared for this challenge.
Footnotes
Contributors All authors contributed to conception/design of the work and have read and approved this manuscript. DC, BM and MS were responsible for data collection and analysis. DC drafted the manuscript and JL provided critical revision of the article.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval The study was designated 'IRB-exempt' by the Uniformed Services University Institutional Review Board.
Provenance and peer review Not commissioned; internally peer reviewed.
Data sharing statement Raw data will be made available upon request, please contact the corresponding author.