The information on this page is related to the paper:
Sandberg S, Fauskanger P, Johansen JV, Keller T, Budd J, Greenberg N, Rej R, Panteghini M, Delatour V, Ceriotti F, Deprez L , Camara JE, MacKenzie F, Lyle AN, Hagen E, Burns C, Miller WG. Recommendations for setting a criterion for assessing commutability of sample materials used in external quality assessment/proficiency testing schemes (in preparation).
As described in this paper an effective method for evaluating EQAMs in practice is by a user-friendly application. To use the application, one will need two excel files, one containing the data for the clinical sample measurements (CSs) and one for the measurements of the external assessment materials (EQAMs). Example dataset, explanations and a link to the application is given below.
If using the application from this website for any purpose, it should be referenced as:
Sandberg S, Fauskanger P, Johansen JV, Keller T, Budd J, Greenberg N, Rej R, Panteghini M, Delatour V, Ceriotti F, Deprez L , Camara JE, MacKenzie F, Lyle AN, Hagen E, Burns C, Miller WG. Commutability assessment of external quality assessment materials. https://www.noklus.no/en/a-practical-tool-for-commutability-evaluation-of-external-quality-assessment-material/ [time of access].
Structure requirements of the uploaded datasets used in the application
It is crucial to ensure that both Excel files contain columns with similar structure and names. The data files must include the identifier columns SampleID and ReplicateID. SampleID distinguishes between different samples, while ReplicateID identifies replicate measurements within a single sample. While CSs and EQAMs may or may not be measured in replicate, it is necessary to include both identifier columns. In addition, all remaining columns in both Excel files must be numeric and include measurement results for the IVD-MDs being considered. The names of the IVD-MDs must also be consistent between the CSs data and EQAM data.
Outliers should be removed from the data sets before they are loaded into the application. A reference to how that can be done is given in the paper.
While some degree of missing data is acceptable, excessive missing data in one or more IVD-MDs can lead to serious consequences. Specifically, comparisons involving IVD-MDs with a significant amount of missing data can compromise the statistical strength of the commutability evaluation results.
It is essential to interpret and use these results with care, as limited data can impact the reliability and validity of conclusions drawn from the data. Therefore, it is crucial to minimize missing data in order to obtain accurate and robust commutability evaluation results. Missing data is detected in the application and is part of the validation tests routinely performed on the uploaded CS data and EQAM data.
Interpretation of the tabled results of the application
Upon uploading valid datasets, the commutability evaluation data analysis can be initiated. In order to effectively comprehend the resulting commutability evaluation outcomes, a detailed understanding of the output tables and plots is essential. These final tables can be presented in two formats: expanded and compact.
If the user selects the compact data form, the following columns will be generated (see example at the bottom of this page):
IVD-MD comparison: This column indicates which two IVD-MDs are being compared.
Zeta + confidence interval: This column displays the calculated zeta value for the given IVD-MD comparison. The corresponding 95% bootstrap confidence interval for zeta is presented within parentheses.
Zeta upper: This column provides the upper acceptable zeta value, which is automatically determined based on the user's study design and choice of M (see how to choose M in the paper).
Differences in non-selectivity is: This column presents the conclusion regarding the existence of statistical evidence for differences in non-selectivity for the considered IVD-MD comparison based on the chosen study design and zeta value. The column can display either "deemed acceptable" or "deemed excessive".
EQAM ID: This column comprises the same values as the SampleID column of the uploaded EQAM Excel file. In other words, the sample identifier of the different EQAMs is presented in this column.
Measurements for IVD-MD 1 and IVD-MD 2: This column displays the average of the replicated measurements of the EQAM for both IVD-MDs. The first result within the parentheses is the average of the IVD-MD measurements along the x-axis, while the second result within the parentheses is the average of the IVD-MD measurements along the y-axis.
Prediction + prediction interval: This column presents the predicted value for the average of the IVD-MD 2 measurement of EQAMs results, given the average measurement result of IVD-MD 1. The prediction value found outside the parentheses is the value of the Deming regression line. The values inside the parentheses correspond to the upper and lower bounds of the corresponding prediction interval.
EQAM is: This column presents the conclusion regarding the location of the EQAM sample relative to the estimated prediction interval, found in the "Prediction + prediction interval" column. If an EQAM has a replicate average measurement of IVD-MD 2 within the prediction interval, the material is deemed to be inside the prediction interval. This column can display either "inside PI", when the EQAM is inside the estimated prediction interval, or "outside PI", when it is outside.
Conclusion strength %: This column represents the conclusion strength (in percent) of the EQAM. The conclusion strength is estimated using a simple Bootstrap resampling algorithm. The replicate means of the CSs are resampled 1,000 times, and for each resampled dataset, the prediction interval limits are estimated and the EQAM is checked whether it is outside or inside the prediction interval limits. For example, if the original data produces a prediction interval that encloses the average of the replicate measurements, the conclusion is "inside PI". Then we resample the CSs and repeat the process 1,000 times, finding that 800 of these resulted in "inside PI". The conclusion strength would then be 80%.
Interpretation of the plot results of the application
Commutability evaluation plots serve as an additional means of representing commutability evaluation results and are an important complement to the tabular data analysis results. Specifically, each unique comparison between two IVD-MDs, is depicted in its own panel, distinguished by a blue strip above. The CSs’ replicate means are represented by black circles with dark blue fill. The green or gray regions represent pointwise prediction intervals, or prediction bands/ribbons. These regions are gray when difference in nonselectivity (DINS) is deemed excessive, and green when DINS is deemed acceptable. All other shapes, which may be light blue or red, represent the replicate means of EQAMs. The shapes corresponding to the EQAMs are enclosed by circles with the same color as the shape itself, indicating the strength of the conclusion as mentioned in the interpretation of the tabulated results above. The higher the opacity of the circles the higher the strength of the conclusion.
Example Excel sheets for glucose
- Glucose clinical samples measurements 3
- Glucose external quality assessment materials measurements 3
The EQAM commutability evaluation application
Commutability evaluation of EQAMs, Vers. 1.0, 2023. Shinyapps.io,
https://qualitylife.shinyapps.io/Commutability-evaluation-of-EQAMs/