ABSTRACT
Despite the adaptation of international standards, quantitative viral load testing of transplant-associated viruses continues to be limited by interlaboratory disagreement. Studies have suggested that this disagreement and the poor commutability of standards may, in some cases, be linked to amplicon size and the fragmentation of circulating viral DNA. We evaluated target fragmentation as a cause of noncommutability and pretest fragmentation of quantitative standards as a potential means of increasing commutability and interassay agreement. Forty-two cytomegalovirus (CMV)-positive and 41 Epstein-Barr virus (EBV)-positive plasma samples, together with two different quantitative standards for each virus, were tested as unknowns using 10 different quantitative PCR assays at 5 different laboratories. Standards were tested both intact and after intentional fragmentation by ultrasonication. Quantitative agreement between methods was assessed, together with commutability, using multiple statistical approaches. Most assays yielded results within 0.5 log10 IU/ml of the mean for CMV, while for EBV a greater variability of up to 1.5 log10 IU/ml of the mean was shown. Commutability showed marked improvement following fragmentation of both CMV standards but not after fragmentation of the EBV standards. These findings confirm the impact of amplicon size and target fragmentation on commutability for CMV and suggest that for some (but not all) viruses, interlaboratory harmonization can be improved through the use of fragmented quantitative standards.
INTRODUCTION
As quantitative measures of circulating virus or DNAemia have become integral to patient care, increasing scrutiny has been focused on the causes of interlaboratory result variability. Several studies have demonstrated marked discrepancies between quantitative values produced by different assays; such disparities have been seen across a spectrum of viruses, particularly among those commonly thought of as transplant-related viruses, including cytomegalovirus (CMV), Epstein-Barr virus (EBV), adenovirus (ADV), human herpesvirus 6 (HHV6), and BK virus (BKV) (1–5). Such assays are often laboratory-developed, real-time PCR methods and as such are complex processes, with accurate and consistent results depending on a wide variety of factors. Many of those factors have been shown to potentially contribute to result variability, among them being the nucleic acid extraction method, the amplification target, and the quantitative calibrator used to calculate the final results (3). Such calibrators are intuitively a fundamental aspect of achieving consistent and accurate results, and therefore, the development of both primary international consensus standards and secondary commercial standards for routine use has become a principal focus of the effort to improve DNAemia testing and, therefore, clinical test utility (6–8). Such improvements could, in turn, provide benefits of result portability (eliminating the need to reestablish baselines for patients when switching tests or institutions) and the development of consensus quantitative interpretive breakpoints, as well as allowing the comparability of published data from different institutions.
However, efforts at reducing interassay variability have thus far met with limited success. International consensus standards have been developed by the World Health Organization (WHO) for CMV, EBV, adenovirus, HHV6, and BKV. Subsequent studies have shown some improvement in agreement among laboratories, but differences in the detected DNA copy number remain high in some cases (2, 9). The assays utilized may still show widely disparate results, although the degree of disparity and the apparent improvement afforded by the use of WHO standards depend upon the virus being tested. The assays investigated in these studies used several different nucleic acid targets, extraction methods, and calibrators. Even among those normalizing to WHO standards, various commercially prepared secondary standards might be used. The latter fact may lead to disparities, in that differences have been shown between such secondary standards (10). By and large, however, one might expect that other sources of variability (for example, assay efficiency, target sequence and structure, the extraction methodology) would be compensated for by normalizing to a common calibrator, yet this is still not always the case.
In fact, the ability of consensus calibrators to improve interassay agreement depends upon the presupposition that those calibrators behave faithfully in the assay system in a manner similar to that in which patient samples behave. This concept of commutability has been of demonstrable importance over time, first, in quantitative clinical chemistry assays and, more recently, in molecular microbiological testing (11, 12). The use of commutable standards may improve agreement between assays, while noncommutable standards may actually have the opposite effect (13). Commutability reflects not only the calibrators in use but also the interplay between those calibrators and the entire assay system. So, for example, if the calibrators amplify with an efficiency different from that for patient samples, this, by definition, is a demonstration of noncommutability, which could potentially be resolved by altering the primer design, amplification conditions, or the calibrators themselves. The noncommutability of the WHO CMV standards has been shown in the context of some, but not all, assay systems (14). That noncommutability, in turn, correlated with those assay systems showing an increased disparity of results compared to other, commutable assay systems. While, as already mentioned, a myriad of factors could have led to this picture, an interesting feature of the methods showing poor commutability (and quantitative agreement) was noted to be an amplicon size in excess of 100 bp (9). Other work has shown that circulating CMV in vivo exists largely as fragmented nucleic acid (15). The WHO standard is composed of whole virus. Larger amplicon assays might then systematically underamplify fragmented circulating nucleic acid compared to the level of amplification of standard material, resulting in the observed noncommutable state. If this is the case, then fragmenting the CMV standard prior to amplification should reduce or eliminate the issue, resulting in a commutable assay system and improving interassay quantitative agreement. Here we sought to evaluate the commutability of several common CMV and EBV quantitative PCR assays with both primary WHO standards and secondary standards normalized to WHO international units (IU). We further sought to evaluate target fragmentation as a cause of noncommutability and as a potential means of increasing commutability and interassay agreement.
MATERIALS AND METHODS
Patient samples.A total of 42 clinical plasma samples, most of which originally assayed as CMV positive by routine clinical testing, and another 41 plasma samples, most of which originally assayed as EBV positive by routine clinical testing, were included for study. All samples represented material remaining after clinical testing, performed at the University of Washington Medical Center (Seattle, WA) between June 2012 and July 2016 and stored for quality assurance purposes at −80°C until they were utilized for the study. The study was assessed for the need for formal IRB review and was deemed IRB exempt. The samples were thawed, coded, and deidentified, with remnant consecutive samples from each patient being pooled as necessary to reach a sufficient volume (final sample). Patient samples negative across five or more assays were excluded from analysis (CMV sample numbers 5, 28, and 34 and EBV sample numbers 11, 25, 30, and 39).
Quantitative standards.Both WHO international standards and commercially prepared quantitative standards were used. The 1st WHO international standards for CMV (standard 09/162) and EBV (standard 09/260) were purchased from the National Institute for Biological Standards and Control, United Kingdom, and reconstituted in 1 ml of Ambion reverse transcription-PCR-grade water (Thermo Fisher, MA) to a final concentration of 6.7 log10 IU/ml and diluted in EDTA plasma to give final concentrations of 2.7, 3.3, 4, 4.6, and 5.6 log10 IU/ml for CMV and 2.7, 3.3, 4, 4.7, and 5.7 log10 IU/ml for EBV. The CMV and EBV verification panels (Exact Diagnostics, TX) were utilized as intact (nonfragmented) secondary standards and are referred to here as the Exact standards. Both the WHO international standards and the secondary standards for CMV and EBV utilize the Merlin and B95-8 strains, respectively (1, 2). The traceability of the secondary standards to the 1st WHO international standard was established using a QX200 droplet digital PCR (dPCR) system (Bio-Rad, CA) and the primers and probes from the RealStar CMV and EBV analyte-specific reagents (ASRs; Altona Diagnostics, Germany). The CMV and EBV secondary standards contained five concentrations in EDTA plasma at 2.6, 3.6, 4.6, 5.6, and 6.6 log10 IU/ml for CMV and 2.7, 3.7, 4.7, 5.7, and 6.7 log10 IU/ml for EBV.
CMV and EBV Exact standards were sheared by ultrasonication using a Covaris S2 focused ultrasonicator (Covaris, MA) at settings of a duty cycle of 10%, an intensity of 5, 200 cycles per burst, and 430 s to target a base pair size of 150 bp. Fragmentation sizes were confirmed using a Agilent TapeStation high-sensitivity D1000 kit (Agilent, CA) with an average fragment size of 166 bp (see the data in the supplemental material). The fragmented EBV and fragmented CMV were then assigned values against the respective international standard by use of the QX200 droplet digital PCR system (Bio-Rad, Pleasanton, CA) and diluted in EDTA plasma to concentrations of 2.6, 3.6, 4.6, 5.6, and 6.6 log10 IU/ml for CMV and 2.7, 3.7, 4.7, 5.7, and 6.7 log10 IU/ml for EBV. After reconstitution in nuclease-free water, the WHO international standards were also sheared prior to dilution in EDTA plasma to concentrations of 2.7, 3.3, 4, 4.6, and 5.6 log10 IU/ml for CMV and 2.7, 3.3, 4, 4.7, and 5.7 log10 IU/ml for EBV. All fragmented material was stored in low-adhesion tubes (Simport, Canada).
Nucleic acid extraction and quantitative PCR.For assay systems that consisted only of detection reagents (all laboratory-developed tests [LDT] and ASR assays), fragmented and intact EBV and CMV WHO and Exact secondary standards for EBV and CMV were extracted on a MagNA Pure 96 instrument with an input volume of 0.2 ml and an eluate volume of 0.1 ml. The 42 patient specimens positive for CMV and the 41 specimens positive for EBV were also extracted in tandem with the standards. Eluates from each final sample and each standard concentration were then pooled, mixed, and divided for use by each study site. Aliquots of the plasma, unextracted standards, and eluate were frozen at −20°C and then shipped frozen on dry ice to each study site. Unextracted material was used for the Abbott RealTime CMV assay, the EliTech EBV assay, the Qiagen Artus RGQ CMV and EBV assays, the Roche Cobas AmpliPrep/Cobas TaqMan CMV assay (referred to here as Roche CMV), and the Roche Cobas 6800/8800 CMV assay. All testing by each instrument was performed in singlet, using the instrumentation, reagents, cycling conditions, and quantitative calibrators specific to each laboratory and assay, as described in Table 1, with the WHO and Exact standards (intact and fragmented) being tested as unknowns. The results of all real-time PCR assays are reported as the number of IU per milliliter and with calibration assignment also being reported as the number of IU per milliliter (as noted in Table 1).
Test assays and sites
Statistical methods.(i) Definition of terms. Commutability (16) is defined as a measure of the similarity of the behavior of a reference material in a given assay system with that of human specimens. A qualitative measure of commutability offers only a yes-or-no answer, while a quantitative measure of commutability yields a numeric to quantify the degree of commutability. When a cutoff is given, a quantitative commutability measure can be categorized as a qualitative measure. Such cutoffs preferably should be based on clinical tolerance. Relative commutability describes the commutability of a reference material in one assay relative to that in another assay. In contrast, absolute commutability characterizes the commutability of one assay in relation to that of a reference standard assay, which is considered close to truth.
Linear regressions and correspondence analysis are two popular qualitative approaches for assessing commutability. Both yield solely a qualitative result, though confusion can arise when a series of dilutions of a reference material is evaluated.
We proposed elsewhere a combined quantitative approach to measure commutability (17, 18). With this approach, one is able to (i) quantify the degree of commutability and compare the commutability of a material across assay systems and (ii) relate commutability with accuracy and interassay agreement.
(ii) Linear regressions. Linear regressions with 95% prediction intervals were utilized to investigate the pairwise commutability between assays (13). In brief, a regression line was estimated by plotting the measured values for patient samples obtained by a pair of assays. A 95% prediction interval was then generated. Measured values for reference material were plotted to check whether they fell within the patient sample-generated interval. If all dilutions of a reference material fell within the interval, the result was classified as commutable. If less than half of the dilutions fell within the interval, the result was classified as noncommutable. If half or more but not all dilutions fell within the interval, the result was defined as marginally commutable. In each instance, only a pair of assays was evaluated.
(iii) Correspondence analysis. Correspondence analysis was performed as previously described (19). Patient data from all lab assays under consideration were projected onto a factorial plane, with the axes representing two dominant extracted factors. Both patient sample values and assays were reflected on the extracted plot, and a 95% confidence ellipse was generated. Then, the values for reference material dilutions (run as unknowns) measured using all assays were projected onto the plane. If all dilutions fell within the ellipse, the reference material was determined to be commutable for all included assays. Otherwise, those falling outside of the ellipse were considered either noncommutable or marginally commutable (half or more but not all dilutions fell within the interval) as described in “(ii) Linear regressions,” above.
(iv) Combined quantitative approach. The combined quantitative approach consists of three quantitative components, as previously described (17, 18), deviation from ideal (DFI), deviation from equality (DFE), and deviation from agreement (DFA), which characterize the departure from commutability, accuracy, and agreement, respectively.
DFI can be regarded as an absolute measure of commutability. DFI measures the discrepancy of two regressions. Measured patient sample values and measured reference material values are regressed against their true values, with all samples being evaluated as unknown samples using a reference standard method, which was digital PCR (dPCR) in the present study. The discrepancy of the patient sample regression line from the reference material regression line is calculated, with components characterizing measurement variability, as well as the numeric difference between two lines. DFI is reported in the same units used for the measured samples (for example, number of copies per milliliter). If the DFI value is close to 0, the reference material is determined to behave similarly to the human samples.
DFE was defined as the departure of a measured value in relation to the true value offered by the reference standard system. In other words, for any given sample with an unknown concentration, the concentration is measured both by dPCR and by a given lab assay under evaluation. A regression line of measured values against true values is fitted, and the difference of that line from the equality line is calculated. Similar to DFI, DFE also takes on the same reporting unit as the measured samples. A small DFE value indicates that the measured value is close to the expected value (truth).
DFA was defined as the departure of the results of a pair of assays from perfect agreement with one another. It was calculated from the difference of the values measured by two given assays under comparison. Again, DFA takes on the same units used for the measured sample, and a large value suggests that the two assays under consideration would likely yield different values for a common unknown sample. In other words, those two assays lack interassay agreement.
Taking advantage of resampling techniques, formal statistical comparisons of DFI values from various lab assays can be made (20). Furthermore, by using bootstrap analysis, the confidence intervals of DFI and DFA can be plotted. In practice, the upper bound of the confidence interval can be compared against a prespecified clinical threshold to further determine whether the noncommutability or interassay disagreement exceeds clinical allowance. The resampling statistical framework proposed here applies to all situations.
To facilitate the combined quantitative approach, dPCR (Altona [digital]) was used as the reference standard assay.
(v) Recalibration. A linear regression of the nominal values versus the threshold cycle (CT) values reported from each assay for reference material was fitted. The values reported for patient samples by each assay were then predicted by their own CT values using the regression coefficients fitted when using dilutions of reference materials. Patient values were plotted for each assay using nonfragmented and fragmented reference materials, with the results being plotted against dPCR values as a reference standard.
Data availability.The data utilized in this study will be made fully available, without restriction, upon request.
RESULTS
CMV.(i) Quantitative agreement. Ten lab assays were considered for CMV quantification, including Abbott, Qiagen, Altona (7500), Roche, Roche 6800/8800, Luminex, DiaSorin, LDT, Altona (digital), and LDT (digital). Most assays (Abbott, Altona [7500], Roche 6800/8800, Luminex, DiaSorin, LDT, and LDT [digital]) yielded results within 0.5 log10 unit of the mean observed value (Fig. 1a and b). Among them, Roche almost always undermeasured the results for patient samples relative to the other tests, while Altona (7500) and Altona (digital) seemed to report higher values more often. In general, measured values for CMV in all 10 assays had a small variation within 0.5 log10 IU/ml, except that Roche, Luminex, and Altona (digital) had slightly greater variations in the reported results. The overall mean was 3.41 log10 IU/ml, the standard deviation (SD) was 0.72 log10 IU/ml, and the range was 3.71 log10 IU/ml.
(a) Overall summary of reported patient (PT) sample CMV values by 10 assays, including Abbott, Qiagen, Altona (7500), Roche, Roche 6800/8800, Luminex, DiaSorin, LDT, Altona (digital), and LDT (digital). The results of the real-time PCR and digital PCR assays are reported as the number of log10 international units per milliliter and the number of log10 copies per milliliter, respectively. (b) Overall summary of the deviation from the mean CMV values reported by the 10 assays. Each box plot represents the difference in the value reported by the assay from the overall mean for the 10 assays, while the blue boxes are real-time PCR results and the red ones are digital PCR results. The black plus sign represents the mean difference for all patient samples, the line in the middle of the box indicates the median, the interquartile range is indicated by the top and bottom (the 25th and 75th percentiles, respectively), and the whiskers are ±1.5 the interquartile range.
(ii) Commutability analyses. (a) Linear regression. In the case of CMV, nonfragmented reference material (both WHO and Exact) showed poor commutability (Tables 2 and 3, top), with many noncommutable pairs, and Qiagen was the least commutable, followed by Roche and Roche 6800/8800, as the viral load values reported by them were likely slightly lower than those reported by the other assays (Fig. 1b). In contrast, fragmentation appeared to markedly improve commutability, with all assay pairs appearing to be commutable with fragmented WHO material and only 5 pairs showing noncommutability with fragmented Exact material (although a larger number showed marginal commutability). The Exact material tended to exhibit more noncommutable assay pairs than the WHO material. When using the nonfragmented Exact reference material, many pairs were not commutable, with Roche 6800/8800 being the least commutable compared with the others (Tables 2 and 3). The fragmented Exact version appeared to be better, with Roche showing the fewest commutable assay pairs.
Commutability of CMV assay pairs by linear regression method—WHO reference material
Commutability of CMV assay pairs by linear regression method—Exact reference material
(b) Correspondence analysis. Similar to the results shown by regression analysis, the nonfragmented WHO standard had commutability across all assays except at the lowest dilution (Fig. 2a), while fragmented CMV WHO reference material showed slightly improved commutability across the 10 assays at all dilutions (Fig. 1b). Roche was the most discordant in terms of quantitative results. This is consistent with the earlier observation that Roche tended to underreport viral load values (Fig. 2b). The use of Exact standards resulted in results similar to those obtained with the WHO standard (Fig. 2c and d), in that Roche was the least commutable and the fragmented standard increased the commutability compared to that achieved with the nonfragmented standard.
In the case of cytomegalovirus (CMV), the commutability of WHO and Exact standard material was examined across all 10 assays with correspondence analysis. Clinical sample projections, WHO and Exact standard dilution projections, and method projections are shown on the first significant factorial plane. The axes indicate how much of the total information in the clinical samples is accounted for by the first two factors, with most of the contribution going to the first factor. The proportion of the variation represented by each dimension is reported in parentheses. The variability in the clinical samples in terms of all 10 methods is described using the black ellipse of the 95% prediction area. (a) Nonfragmented WHO standard material; (b) fragmented WHO standard material; (c) nonfragmented Exact standard material; (d) fragmented Exact standard material. Observations from any samples below the assays’ linear ranges were excluded from the analysis.
(c) Quantitative commutability (DFI). The DFI values obtained when using WHO material were generally small, indicating high overall commutability. Using the nonfragmented WHO material, most assays had a DFI value not statistically exceeding 0.3 log10 IU/ml, as suggested by Fig. 3a, with the 95% confidence interval of the DFI for Abbott, Altona (7500), Luminex, DiaSorin, LDT, and LDT (digital) all including 0.3 log10 IU/ml. Roche 6800/8800 had a DFI value with a 95% confidence interval beyond 0.3 log10 IU/ml but less than 0.6 log10 IU/ml. Qiagen and Roche had the largest DFI values, which were significantly greater than 0.3 log10 IU/ml but also not significantly greater than 0.6 log10 IU/ml. Qiagen was the only assay whose DFI confidence interval did not overlap that of any other assays except Roche, suggesting that Qiagen and Roche had significantly larger DFI values than the other assays (Fig. 3a). The use of fragmented WHO material seemed to improve commutability; almost all assays had DFI values not significantly greater than 0.3 log10 IU/ml, with the exceptions being Roche and Luminex. The usage of fragmented WHO material also helped to bring the DFI values of all assays closer, as their confidence intervals looked more compact (Fig. 3b). By computing a 95% confidence interval for the DFI difference between nonfragmented and fragmented material, it is seen that the use of fragmented material resulted in a significant increase in commutability with Qiagen, Altona (7500), and Roche (see Table S1 in the supplemental material).
In the case of cytomegalovirus (CMV), DFI (blue dot) assessment with two-sided 95% bootstrap confidence intervals (CI; gray line) for eight real-time PCRs and one digital PCR, while digital PCR by the St. Jude Altona method is regarded as the reference standard assay. The red and green dashed lines are two hypothetical thresholds of 0.3 log10 IU/ml and 0.6 log10 IU/ml, respectively. An assay is considered less commutable than the predefined threshold if the whole 95% bootstrap confidence intervals of DFI are above the threshold. When the 95% bootstrap confidence interval of one assay has no overlap with the 95% bootstrap confidence interval of the other assay, the two assays have significantly different commutability performance. (a) Inference plots of nonfragmented WHO standard material; (b) inference plots of fragmented WHO standard material.
Exact reference materials exhibited a similar pattern. When using nonfragmented material, Abbott, Qiagen, Roche, and Roche 6800/8800 had DFI values significantly greater than 0.3 log10 IU/ml. Abbott, Qiagen, and Roche even had DFI values significantly greater than 0.6 log10 IU/ml. Qiagen and Roche had DFI values significantly greater than those of most other assays, as their confidence intervals did not overlap the others, except for Abbott (Fig. 4a). In contrast, when using fragmented Exact material, the pattern looked more concordant. No assay except Roche had a DFI value significantly greater than 0.6 log10 IU/ml, but even Roche had a smaller DFI value of 0.82 log10 IU/ml compared to the DFI value of 1 log10 IU/ml obtained using the nonfragmented material. There was also more overlapping in all assays, suggesting that commutability was not significantly different among them (Fig. 4b). A consistent conclusion was reached when computing confidence intervals for DFI differences. The use of a fragmented material resulted in significantly improved commutability with Abbott, Qiagen, Roche 6800/8800, and LDT (digital) (Table S2).
In the case of cytomegalovirus (CMV), DFI (blue dot) assessment with two-sided 95% bootstrap confidence intervals (gray line) for eight real-time PCRs and one digital PCR, while digital PCR by the St. Jude Altona method is regarded as the reference standard assay. The red and green dashed lines are two hypothetical thresholds of 0.3 log10 IU/ml and 0.6 log10 IU/ml, respectively. An assay is considered less commutable than the predefined threshold if the whole 95% bootstrap confidence interval of DFI is above the threshold. When the 95% bootstrap confidence interval of one assay has no overlap with the 95% bootstrap confidence interval of the other assay, the two assays have significantly different commutability performance. (a) Inference plots of nonfragmented Exact standard material; (b) inference plots of fragmented Exact standard material.
(iii) Quantitative accuracy (DFE) and interassay agreement (DFA). Detailed DFE and DFA reports are summarized in Data Set S1 in the supplemental material.
Consistent with the commutability results, Roche had the lowest accuracy, as quantified by DFE, followed by Qiagen, with DFE values being greater than 1 log10 IU/ml when patient samples were measured, and both reported lower viral loads in the same samples than the other assays. As shown in Fig. 3 and 4, Roche seemed to perform differently from the DFI perspective. Figure S1 shows the confidence intervals of the DFE values for all assays, with Roche being the only assay having a DFE value significantly greater than 1 log10 IU/ml.
Pairwise DFA values along with resampling-based confidence intervals are presented in Data Set S1. Consistent with the findings in Fig. 3 and 4, where Roche was the assay overlapping with the other assays the least, it was found that Roche was the assay that had the largest DFA values compared against those of many of the other assays. For example, the DFA values of Roche versus Altona (7500), LDT, and Luminex were greater than 0.6 log10 IU/ml, as their DFA confidence intervals bypassed 0.6 log10 IU/ml, suggesting lower interassay agreement of Roche with those assays. These DFA values tended to be lower for the newer Roche assay (Roche 6800/8800).
(iv) Recalibration. With recalibration based on available CT values (all assays except Roche and DiaSorin), a shift toward the true values was noticed for several CMV assays when utilizing fragmented reference materials rather than nonfragmented materials. Figures 5a and b show two examples of the results obtained with CMV Exact material and the Abbott and Qiagen reagents. When the reference material was fragmented, there was a noticeable shift in the measured values for the patient samples toward the true (dPCR-generated) values. Recalibration data for all assays on which recalibration was performed are shown in Fig. S2 to S4.
Cytomegalovirus (CMV) Abbott (a) and Qiagen (b) assay values for patient (PT) samples predicted by CT values. The blue solid line indicates recalibrated values determined using the linear relationship between nominal values and CT values for nonfragmented Exact reference material, while the red solid line shows recalibrated values obtained by fitting the nominal values on the CT values for fragmented Exact reference material. Digital PCR (Altona [digital]) (green dashed line) was used as the reference standard assay.
EBV.(i) Quantitative agreement. Ten lab systems were used for EBV quantification: EliTech, Qiagen, Altona (7500), Luminex, DiaSorin, LDT (EBER), LDT (IR-1), Altona (digital), LDT (EBER-digital), and LDT (IR-1-digital) (Fig. 6a). It was noted that although most assays (EliTech, Qiagen, Altona [7500], Luminex, LDT [IR-1], Altona [digital], and LDT [IR-1-digital]) had measures falling within 0.5 log10 IU/ml from the overall mean, the values from different assays could differ by as much as approximately 1.5 log10 IU/ml. Plotting the difference of the observed values from each assay from the mean of all measured values for each patient sample, Fig. 6b shows that LDT (EBER) tended to undermeasure EBV by about 1 log10 IU/ml or more relative to the other assays, while DiaSorin tended to overreport the measured values by more than 0.5 log10 IU/ml. The variation for EliTech seemed to be the largest across all assays (Fig. 6b), followed by Qiagen and LDT (EBER-digital). Qiagen, Luminex, and Altona (7500) tended to agree with each other better. The overall mean was 3.96 log10 IU/ml, the SD was 0.82 log10 IU/ml, and the range was 4.33 log10 IU/ml.
(a) Overall summary of reported patient (PT) sample EBV values by 10 assays, including EliTech, Qiagen, Altona (7500), Luminex, DiaSorin, LDT (BEBR), LDT (IR-1), Altona (digital), LDT (EBER-digital), and LDT (IR-1-digital). The results of the real-time PCR and digital PCR assays are reported as the number of log10 international units per milliliter and the number of log10 copies per milliliter, respectively. (b) Overall summary of the deviation from the mean EBV values reported by 10 assays. Each box plot represents the difference in the value reported by the assay from the overall mean for 10 assays, while the blue boxes are real-time PCR results and the red ones are digital PCR results. The black plus sign represents the mean difference for all patient samples, the line in the middle of the box indicates the median, the interquartile range is indicated by the top and bottom (the 25th and 75th percentiles, respectively), and the whiskers are ±1.5 the interquartile range.
(ii) Commutability analyses. (a) Linear regression. When testing EBV, both reference materials, WHO and Exact, showed commutability with most assay pairs. In general, Exact standards showed more noncommutable assay pairs (Tables 4 and 5). Fragmentation appeared to have little impact on commutability using either of the standards, yielding similar results in the top and bottom halves of Tables 4 and 5. Altona (digital) and LDT (EBER) showed more noncommutability with other assays using the WHO standard. Using the Exact standard, LDT (IR-1-digital) and LDT (IR-1), as well as Altona (digital), showed fewer commutable assay pairs. This observation was consistent with the results presented in Fig. 6b, as LDT (EBER), LDT (IR-1-digital), and LDT (IR-1) were the three assays which tended to underreport test values compared to the values reported by the other assays.
Commutability of EBV assay pairs by linear regression method—WHO reference material
Commutability of EBV assay pairs by linear regression method—Exact reference material
(b) Correspondence analysis. The commutability pattern was similar for all EBV assays, using the WHO and Exact standards and nonfragmented and fragmented materials (Fig. 7). The results of EliTech and Qiagen were less concordant with those of the other assays. When comparing the pattern of the nonfragmented WHO standard to that of the fragmented WHO standard (Fig. 7a and b), the patterns seemed to be similar and fragmentation showed no evident impact. Similar findings for Exact standard material are shown in Fig. 7c and d. EliTech and Qiagen yielded results more extreme than those of the other assays (Fig. 7).
In the case of Epstein-Barr virus (EBV), the commutability of WHO and Exact standard material was examined across all 10 assays with correspondence analysis. Clinical sample projections, WHO and Exact standard dilution projections, and method projections are shown on the first significant factorial plane. The axes indicate how much of the total information in the clinical samples is accounted for by the first two factors, with most of the contribution going to the first factor. The proportion of the variation represented by each dimension is reported in parentheses. The variability in the clinical samples in terms of all 10 methods is described using the black ellipse of the 95% prediction area. (a) Nonfragmented WHO standard material; (b) fragmented WHO standard material; (c) nonfragmented Exact standard material; (d) fragmented Exact standard material. Observations from any samples below the assays’ linear range were excluded from the analysis.
(c) Quantitative commutability (DFI). With nonfragmented WHO standard material, Altona (7500) seemed to be the most commutable, with a DFI value of 0.23 log10 IU/ml (95% confidence interval, 0.19, 0.28 log10 IU/ml), which is significantly smaller than 0.3 log10 IU/ml (Fig. 8a). Except for Altona (7500) and DiaSorin, all other assays had DFI values significantly greater than 0.3 log10 IU/ml, while EliTech had DFI values significantly greater than 0.6 log10 IU/ml. Comparing all assays, Altona (7500) and DiaSorin seemed to be significantly more commutable than the other assays, while the confidence intervals of EliTech, LDT (EBER), and LDT (EBER-digital) had limited overlap with those of the other assays and seemed to be less commutable than the others (Fig. 8a). The use of fragmented WHO material seemed to bring the results closer, suggested by more overlap. EliTech was the only assay with a DFI value significantly greater than 0.6 log10 IU/ml. Altona (7500) and DiaSorin showed the lowest DFI values (0.2 and 0.35 log10 IU/ml, respectively). The assays’ commutability performance looked more concordant, with EliTech being less commutable than the other assays (Fig. 8b). It should be noted that the DFI for DiaSorin increased slightly with the use of fragmented WHO material. The use of fragmented material did not seem to significantly improve commutability, except with LDT (EBER-digital) (Table S3).
In the case of Epstein-Barr virus (EBV), DFI (blue dot) assessment with two-sided 95% bootstrap confidence intervals (gray line) for seven real-time PCRs and two digital PCRs, while digital PCR by the St. Jude Altona method is regarded as the reference standard assay. The red and green dashed lines are two hypothetical thresholds of 0.3 log10 IU/ml and 0.6 log10 IU/ml, respectively. An assay is considered significantly less commutable than the predefined threshold if the whole 95% bootstrap confidence interval of DFI is above the threshold. When the 95% bootstrap confidence interval of one assay has no overlap with the 95% bootstrap confidence interval of the other assay, the two assays have significantly different commutability performance. (a) Inference plots of nonfragmented WHO standard material; (b) inference plots of fragmented WHO standard material.
Data for the EBV Exact standard were similar to those described above. Nonfragmented Exact standard material showed Altona (7500) and DiaSorin to be the most commutable, with DFI values being significantly smaller than 0.3 log10 IU/ml, followed by LDT (EBER) and Luminex. Except for Altona (7500), DiaSorin, and Luminex, all other assays had DFI values that were significantly greater than 0.3 log10 IU/ml, while EliTech had DFI values that were significantly greater than 0.6 log10 IU/ml. Looking across assays, Altona (7500), DiaSorin, Luminex, and LDT (IR-1) were the most commutable, while EliTech and LDT (EBER-digital) were the least commutable (Fig. 9a). The use of fragmented Exact material improved the commutability slightly, suggested by slightly more overlap between confidence intervals. DiaSorin was most the commutable, with a DFI value significantly less than 0.3 log10 IU/ml, and the DFI value for Altona (7500) was comparable to that value. EliTech was the only assay with a DFI value significantly greater than 0.6 log10 IU/ml. EliTech showed the lowest commutability, followed by LDT (IR-1-digital), LDT (EBER-digital), and Qiagen (Fig. 9b). Again, the use of fragmented material did not seem to significantly improve commutability, except with LDT (EBER-digital) (Table S4).
In the case of Epstein-Barr virus (EBV), DFI (blue dot) assessment with two-sided 95% bootstrap confidence intervals for seven real-time PCRs and two digital PCRs, while digital PCR by the St. Jude Altona method is regarded as the reference standard assay. The red and green dashed lines are two hypothetical thresholds of 0.3 log10 IU/ml and 0.6 log10 IU/ml, respectively. An assay is considered significantly less commutable than the predefined threshold if the whole 95% bootstrap confidence interval of DFI is above the threshold. When the 95% bootstrap confidence interval of one assay has no overlap with the 95% bootstrap confidence interval of the other assay, the two assays have significantly different commutability performance. (a) Inference plots of nonfragmented Exact standard material; (b) inference plots of fragmented Exact standard material.
(iii) Quantitative accuracy (DFE) and interassay agreement (DFA). Detailed DFE and DFA reports are summarized in Data Set S1.
EliTech, LDT (EBER-digital), and LDT (EBER) yielded the least accurate results with patient samples, and the discrepancy was greater than 0.6 log10 IU/ml (Fig. S5). This was consistent with the observation that those assays had the greatest departure from perfect commutability.
Consistent with the findings presented in Fig. 8 and 9, the DFA values of EliTech were larger than 0.6 log10 IU/ml and were larger than those of all other assays, followed by LDT (EBER). This is an indication that the interassay agreement of EliTech with the other assays was reduced. The finding is consistent with the data in Fig. 8 and 9, in which EliTech was far apart from the other assays.
(iv) Recalibration. With recalibration based on available CT values (all assays except DiaSorin), a shift toward true values was not apparent with EBV assays when utilizing fragmented reference materials rather than nonfragmented materials. Recalibration data for all assays on which recalibration was performed are shown in Fig. S6 to S9.
DISCUSSION
The findings presented here further support the complexity in achieving harmonization of DNAemia assays, how the development of commutable quantitative reference materials is critical to that effort, and the challenges involved in achieving and measuring that commutability. Earlier findings have demonstrated the limitations of currently available quantitative reference material in achieving result harmonization and how the diminished commutability of such material in given assay systems correlates with a lack of agreement (5, 9, 11, 14, 21). Subsequent work has shown that CMV often exists in circulating plasma as fragmented nucleic acids (15) and allowed the hypothesis that poor commutability may be a result of such fragmentation, when whole virus is used as a quantitative standard in real-time PCR assays (14). The findings presented here support that hypothesis, in that use of the fragmented WHO standard showed a clear improvement in commutability for some assays, particularly among those measuring CMV.
Despite this apparent correlation with earlier studies, only one (Roche) of the three assays (Abbott, Qiagen, Roche) with the poorest commutability with nonfragmented samples had an amplicon exceeding 100 bp. This suggests that the relationship between diminished commutability, amplicon size, and fragmentation is more complex than was previously thought. Potentially, the 100-bp threshold, suggested previously (9), is not fully accurate, with some patients experiencing a higher degree of fragmentation. Data on CMV fragmentation in the literature remain limited regarding the exact size of typical fragments and the degree to which such fragmentation varies from patient to patient. Nonetheless, an even more stringent view of the ideal amplicon size for quantitative assays would not cleanly distinguish those with poor commutability that would be improved by fragmentation from those with a higher degree of commutability when nonfragmented reference materials are used. Another explanation might be that fragmentation varies based on the genetic region of the virus and that it is a complex combination of target region and amplicon size that predicts commutability. At present, these data are insufficient to prove the latter point but still support fragmentation as a means of improving commutability and harmonization, at least for assays targeting certain viruses.
While assays for both CMV and EBV showed some changes in commutability when fragmented reference materials were utilized, improvements were primarily seen here in the case of assays for CMV. This may reflect less in vivo fragmentation of EBV in clinical samples; data regarding the fragmentation seen with CMV have not been published to date for EBV. Changes in commutability with EBV assays were somewhat more evident using the quantitative (DFI) method of assessment. This method of commutability analysis appears here to be more sensitive to changes in commutability than the other means of analysis (linear regression and correspondence analysis) and is also the only method of providing a quantitative metric for commutability.
The linear regression method is also disadvantaged in that only pairwise commutability can be assessed, making it more difficult to characterize the overall pattern of commutability when more than two assays are under consideration. Correspondence analysis is capable of pooling data from several assays to make an overall assessment of behavior but remains a qualitative method whose results may be distorted by extreme performers. DFI allows assessment and comparison of commutability in a nonrelative manner, allowing single, paired, or group-wise comparison of methods in a quantitative manner (Fig. 3, 4, 8, and 9). Although the approximate relationships between DFA, DFI, and DFE hold only when there is evidence that certain assumptions are met (and, therefore, it is not always feasible to set upper and lower bounds of DFA values with the values of DFI and DFE), DFA, DFI, and DFE values are always valid and informative. The proposed quantitative framework always holds and offers a way to evaluate accuracy, interassay agreement, and commutability together, as seen here. This work does highlight the inherent limitations of all three approaches to determining commutability. While the DFI method has advantages, as mentioned, it depends on its full value for certain underlying assumptions to be met. It also requires a reference standard method, typically dPCR. However, dPCR is an imperfect method itself, and though earlier work has shown that for DNA targets, digital methods are generally in agreement, others have shown that some assays can be subject to dropout and other potential sources of inaccuracy, generally reducing apparent quantitative results (22). Therefore, it is important that any dPCR method used as a reference standard be fully validated. Here, only one of the included dPCR methods (previously well characterized) was used as a reference standard for statistical determination of DFI. While there was some lack of concordance with other dPCR results, typical sources of inaccuracy result in lower quantitation, and the method used here had the higher value in cases of disagreement. It seems unlikely, therefore, that the choice of reference standard interfered with the conclusions reached. One caveat is that the digital reference standard used here was run using Altona reagents; the real-time assay using those same reagents appeared from the DFI analysis to have (quantitatively) the highest degree of commutability. While these data are included for instructive purposes, bias cannot be ruled out, and the relative commutability of this specific reagent set may need further confirmation. Beyond that, the overall agreement between all three commutability methods supported the general accuracy of each.
Other potential limitations included the number of assays utilized, the number of clinical samples included, and the heterogeneity of the sample extraction methods. The last potential limitation was the necessary result of including in vitro diagnostic (IVD) assays, wherein nucleic acid extraction was an integral part of the assay. While this did not directly limit the assessment of commutability, it did increase the number of variables to consider when assessing potential reasons for interassay differences. Previous work has shown variability in the yield of different extraction methods, with some such methods showing a reduced nucleic acid yield from fragmented input DNA (23), illustrating how extraction method choice could potentially affect both interassay agreement and commutability. The number of assays was relatively large but could not be exhaustive, based on access to reagents and particularly on the available sample volume, again limiting the ability to generalize beyond the tests examined. Finally, the lack of availability of large numbers of patient samples with a sufficient spectrum of viral load in a sufficient volume for analysis by numerous tests also limited the power of this study to some degree. Also, it was not possible to recalibrate some of the systems that showed the most difference in commutability (e.g., Roche CMV), since these were internally calibrated and did not generate CT values that could be used for recalibration. However, it was indeed noted that the use of the fragmented material pushed the curve closer to truth, improving the apparent accuracy in those systems where recalibration was possible.
A lack of commutability and diminished interlaboratory agreement may result from various factors (3, 14); here, only the fragmentation of the reference material was examined. Work to better define the fragmentation of transplant-associated viruses is under way and may help further the understanding gained here regarding the use of fragmented reference materials. The impact of various extraction methods and other aspects of assay design and performance must also be evaluated over time. Commutability, interassay agreement, and the value of international standards should also be further defined for other viruses.
The use of a fragmented reference material appears to offer some improvement in commutability and interassay agreement, particularly in the case of CMV. This improvement was accompanied by an apparent improvement in assay accuracy, based on the recalibration of patient sample results. However, it is clear from this study and others that the causes of interassay disagreement and the paths to harmonization are multifaceted and may vary significantly among viruses. Continued efforts to define both the causes of disagreement and ways to ameliorate them are needed to further improve the predictive value and the clinical utility of these important diagnostic tools.
ACKNOWLEDGMENTS
Ederlyn Atienza is gratefully acknowledged for her work in coordinating sample processing, performing nucleic acid extractions, and performing PCR assays at the University of Washington Medical Center.
R.T.H. and A.M.C. have served on advisory boards for Roche Diagnostics.
FOOTNOTES
- Received 3 June 2019.
- Returned for modification 22 July 2019.
- Accepted 7 October 2019.
- Accepted manuscript posted online 16 October 2019.
Supplemental material is available online only.
- Copyright © 2019 American Society for Microbiology.
