Methods to assess validity of positioning systems in team sports: can we do better?

Data acquisition

Data from a consumer-grade standalone GNSS (OptimEye S5, Catapult Sports, Melbourne, Australia) that is typically applied in team sports were collected and compared with two different reference systems: a high-end dGNSS and a tape measure. The study was conducted outdoors (59°58'01.8"N, 10°43'37.8"E) on a level sports field in open-sky conditions with no obstructions that could block or interfere with GNSS signals. The average horizontal dilution of precision was 0.63±0.08, and the average number of satellites was 16.8±0.9, measured by the consumer-grade standalone GNSS during the trials. Participants completed three courses (figure 1): course 1—Straight line sprinting and deceleration to stop (SLC); course 2—straight line sprinting, change of direction (CoD) and then deceleration to stop; and course 3—zigzag course with multiple changes of direction (mCoD). The start, end and turning points in each course were marked with cones. Participants were instructed to follow a straight path between the cones and stop exactly at the final cone, and to give maximal effort to complete the courses in the shortest time possible. Two participants executed six trials on each course, while one participant executed three trials on each course.

Request permission

The standalone GNSS (OptimEye S5, Firmware V.7.18) has a 10 Hz GNSS (internal GNSS antenna) and an inertial measurement unit in a casing with dimensions 96×52×13 mm, with a mass of approximately 67 g. In the current study the device was placed in a backpack (figure 2). The device was oriented in an erect position and was located on the midline between the shoulder blades.

Request permission

The dGNSS reference system consisted of a differential multifrequency and multi-GNSS receiver. The base station consisted of a GNSS antenna (Grant-G3T, Javad, San Jose, California, USA) and a receiver (Alpha-G3T, Javad) and was placed <50 m from the location where the three tasks were executed to allow short baseline differential solution calculations. The base station of the reference system was positioned in a global reference frame using data from the Norwegian Mapping Authority for postprocessing use with the precise ephemerides data and a baseline of 3 km to the closest GNSS station using geodetic postprocessing software (Justin, Javad GNSS). The participants carried a GNSS rover with an antenna (160 g, G5Ant-2AT1, Antcom, Torrance, California, USA) and a GNSS receiver (430 g, Alpha-G3T, Javad) during the data collection (figure 2). The sampling frequency was set to 20 Hz and downsampled by removing every second measurement to 10 Hz, to match the frequency of the standalone GNSS. For the dGNSS reference system, instantaneous position solutions were calculated using data from the local dGNSS base station and the dGNSS rover carried by the athlete. From these data, kinematic carrier phase double difference solutions using L1 and L2, GPS and GLONASS (the Russian GNSS) GNSS data were calculated using geodetic postprocessing software (Justin, Javad GNSS) in kinematic mode.3 The dGNSS has previously been shown to have an accuracy of ≈5 cm in highly dynamic applications.3 Both dGNSS systems and the standalone GNSS were switched on 30 min prior to the measurements to ensure ephemerides download and enough time to establish stable GNSS solutions.

In addition to the three courses, a static position accuracy test was done to compare static and dynamic (measurements in the three courses) position accuracy. Two standalone GNSS were mounted 1.5 m above ground on a tripod and put in an upright position ensuring a similar GNSS antenna orientation to that which would occur if it was mounted on an athlete. To ensure similar GNSS measurement conditions between the static and dynamic GNSS applications the static position data were collected for 15 min during the same time period as the dynamic tests were conducted. To assess the accuracy of the static standalone GNSS, the true global position of the static standalone GNSS device was measured using the same geodetic high-end dGNSS receiver, with correction data from the Norwegian Mapping Authority, and the same postprocessing method as for the local base station, as described above.

Data analysis

Horizontal plane position data (X and Y coordinates) were extracted from both the standalone GNSS and the dGNSS reference system using their respective software (OptimEye S5: Catapult Sprint, software V.5.1.7; dGNSS reference system: Justin (Javad)). All further processing and analysis were conducted in MATLAB V.R2021a (MathWorks, Natick, Massachusetts, USA). All GNSS data were transformed from WGS84 to UTM32N coordinates using ‘Geodetic Toolbox’,18 and all calculations were conducted in UTM32N. Reference system data were handled as suggested in previous research with a weighted spline interpolation.2 Position data from the standalone GNSS were recorded both raw and filtered with a second-order Butterworth filter (forward and backward) with a cut-off frequency of 4 Hz. This made it possible to assess the raw data and the impact of filtering, as filtering is commonly done in validation studies.

From the dynamic GNSS position-time measurements of the standalone GNSS and the dGNSS reference system, the following criterion measures were calculated: distance travelled, mean speed, instantaneous speed and instantaneous dynamic position. Distance travelled per trial was calculated as the sum of the Euclidean distances between consecutive horizontal plane positions along the trajectory, excluding the vertical error, as this is commonplace in team sports. Instantaneous speed in the horizontal plane (hereafter called speed) was calculated from horizontal plane position data using a four-point finite central difference formula.19 Speed was filtered with a fourth-order Butterworth filter in a forward and backward direction with a cut-off frequency of 2.5 Hz for both the standalone GNSS and the dGNSS reference system. Mean speed was calculated over the length of each trial.

The standalone GNSS and the dGNSS reference system data were time synchronised using cross-correlation of speed data in two steps: (1) position data in the horizontal plane (X and Y coordinates) were time differentiated to obtain horizontal plane speed for both the Catapult S5 and the reference system; (2) standalone GNSS and dGNSS reference system data were time synchronised using cross-correlation20 of horizontal plane speed data. After time synchronisation, data were trimmed to reflect only the time athletes were performing the trials by using a speed threshold of 0.5 m/s (determined from the standalone GNSS) to identify the start and end of trials. For the trials to be included in the data analysis, the dGNSS reference system had to have fixed ambiguities in the differential solution3 for the entire trial length. The numbers of included trials were 11 (SLC), 14 (CoD) and 11 (mCoD).

To assess position accuracy, instantaneous position differences between the standalone GNSS and the reference system were calculated for both the dynamic and static measurements. In addition, instantaneous position of the static standalone GNSS was compared with the mean position over the trial duration for the same standalone GNSS. Accuracy of distance travelled was calculated as the difference in the sum of the Euclidean distances between each position along the trajectory from start to finish. The error distributions for static and dynamic positions were visualised in a histogram, displaying the percentage distribution in each bin. Differences in distance were calculated as mean±SD. To assess the effect of the time window over which speed was averaged, the mean speed was calculated over various lengths of time windows, ranging from one point up to the entire duration of the respective trial.

Position

Mean errors in raw dynamic position for the standalone GNSS compared with the dGNSS reference system were 1.62±0.84 m, 1.57±0.99 m and 1.46±0.67 m for SL, CoD and mCoD, respectively. Position data from the standalone GNSS and the dGNSS reference system are depicted in figure 3. The mean error in the dynamic position (1.53±0.81 m) was larger than that in the static position (1.10±0.60 m; figure 4).

Request permission

Request permission

For the static position, the maximal error of the standalone GNSS was 1.72 and 3.20 m from the dGNSS reference system. The maximal error of the standalone GNSS from the mean standalone GNSS position was 1.59 and 1.97 m (figure 5). The total distance measured by the standalone GNSS in the static condition was 20.66 m (figure 5A) and 29.05 m (figure 5B).

Request permission

Distance

The distance measured by dGNSS varied from 6% longer to 17% shorter compared with the predefined course that was measured by tape measure (table 1). None of the distances measured with the dGNSS systems in the CoD and mCoD courses during the trials were greater than the distances of the predefined courses measured with the tape measure. When comparing errors in the standalone GNSS to the two reference systems, the differences in errors were larger in the mCoD course compared with the other courses (figure 6).

Request permission

Speed

The distribution of errors in mean speed and instantaneous speed differed both in mean (absolute error 0.10 vs 0.28 m) and in range (figure 7). Large reduction in error was found when longer time windows were used for mean speed calculations (figure 8).

Request permission

Request permission

Position

The position error of tracking systems is typically investigated in two different ways: (1) with static measurements21 22 or (2) with dynamic measurements.11 23 For the assessment of static position, two methods were used as a reference: either the use of a known point in a global reference frame (eg, using a reference system to measure the true position)22 24 25 or use of the time-averaged position from the same measurement system being validated.21 The latter approach will not yield information about the accuracy of the tracking system but is instead a measure of precision (variability around the mean). This is comparable to investigations of interdevice reliability of tracking systems, where typically the data from several units mounted on the same athlete are compared with each other to assess whether different units of the same type provide similar results. Such testing should not be confused with system accuracy assessment. Therefore, accuracy of position (both static and dynamic) must be assessed against a global reference position. In addition, position accuracy assessment should be conducted for a period of at least several hours, especially when assessing standalone GNSS (such as OptimEye S5), since the position error of such devices drifts substantially over time and a long measurement time is needed to cancel out coincidental results. Examples of such global drift over time and drift from trial to trial can be seen in figures 3 and 5, and in the study by Jølstad et al.7

Our data showed that dynamic position measurements had both a larger mean error and a larger range in errors compared with errors from static position testing. This implies that static position accuracy cannot be used as a proxy for dynamic position accuracy of tracking systems. Hence, in order to not mislead the users of position tracking devices, position error should be validated against a global reference system and frame that allow instantaneous position tracking in a dynamic application. Dynamic position is usually not included in validation studies for team sport tracking devices.15 However, its inclusion would provide reliable assessment of validity of a system across time periods, when software and firmware are updated by the manufacturer, which may change the parameter calculation methods such as position measurement, and the calculation of propagation parameters (eg, distance and speed). Future studies should also validate dynamic position to include spatial-tactical behaviour analysis. Additional guidance for instantaneous position validation using dGNSS has previously been provided by Specht and Szot.26

Distance

Our data show differences between the tape measure and dGNSS reference systems for the three courses, with larger differences in the courses incorporating CoD (table 1). As previously mentioned, there are deviations in human locomotion from predefined straight line paths. Using tape measure as a reference precludes measurements of the deviations from the straight line, which might be an important factor for the differences in this study. Hence, the choice of tape measure over dGNSS or another instantaneous position tracking device as a reference system can affect the outcome of distance validation, leading to an underestimation or overestimation of the accuracy of a tracking system, depending on the sport task. Since team sports include a large number of CoD, it is therefore recommended to use a position tracking device to validate distance, instead of a tape measure.

Speed

Reference systems such as tape measures and timing gates only allow for validation of mean speed. However, mean speed over longer time periods is not a good representation of intensity in team sports, since an athlete’s speed fluctuates substantially.27 In comparison, tracking of instantaneous speed allows speed to be presented as a distribution of time or distance covered in different speed zones,1 which provides better insight into the speed characteristics of a match or training session, and is now commonly used in team sports. The use of mean speed instead of instantaneous speed to represent team sport intensity is therefore a critical shortcoming for sport analysis and validation studies. Figure 7 illustrates how the use of mean speed as a measure of speed accuracy is misleading, since the mean error drops substantially when only a few epochs are averaged. Hence, in system validity assessments, mean speed cannot be used as a proxy for instantaneous speed, even when averaged over short time periods. Reference systems should therefore be able to measure instantaneous speed and be properly time synchronised with the system being assessed. The use of such reference systems is already common practice in other sports domains, such as winter sports.2–8 26 28 29

Methodological considerations

This study used dGNSS and a tape measure as reference systems to illustrate the consequences of the choice of criterion measure and reference system on validation outcomes. Other reference systems such as timing gates, radar guns, trundle wheels and video-based and infrared camera-based motion capture systems should also be assessed to establish a more complete overview of the differences between reference systems. The main goal of this study, however, was to highlight the importance of the appropriate choice of reference system for use in validation studies.

In our study, the dGNSS reference system antenna was placed on the head while the standalone GNSS antenna was placed between shoulder blades (figure 1). This causes a deviation in the horizontal plane position in the scale of a few decimetres, which could introduce some error. However, this error is small compared with the expected position error of the standalone GNSS used in this study (scale of metres). The present set-up did not allow us correction of the offset between the two systems. However, this is an aspect that should be taken into consideration when conducting validation studies.

The error characteristics (accuracy and precision) of position, distance and speed will be different for measurement systems other than the ones used in this study. For example, drift of position error over time will be different for an LPS. The essence of this study is not the exact values for the specific GNSS system validated in this study, but the principles of validity assessment that will also apply to other measurement systems.

Filtering of data can play a large role in parameter calculation30 and caution is therefore indicated when using an unknown filtering technique (eg, at the manufacturer’s discretion) in validation studies. We have therefore displayed both the raw and filtered data to show the typical effect of filtering. The cut-off frequencies used in this study can be considered conservative, and were chosen since previous studies have shown that liberal filtering can influence parameter calculations to a large extent.31 Future validation studies should also investigate the effect of different filters on parameter accuracy in typical team sport applications.

Perspectives

This study has shown that the choice of reference system to validate tracking systems can substantially affect the outcome of validation studies. The reference systems should therefore be a true gold standard to assess the criterion measure in question. The use of tape measures to validate distance travelled along a trajectory of the participant is not recommended since this method does not account for the deviations an athlete makes from the straight line path between the marked positions on the course.

To assess standalone GNSS, as in this study, the reference should typically be a high-end dGNSS or a video-based or infrared camera-based motion capture system, which allows instantaneous and sufficiently accurate quantification of an athlete’s path. Use of an appropriate reference system may have a wider application, be more appropriate to assess team sport-specific parameters and keep track of changes in parameter calculation across firmware/software versions, which might change over time.