Previous Article | Next Article ![]()
Journal of Clinical Microbiology, January 2008, p. 157-163, Vol. 46, No. 1
0095-1137/08/$08.00+0 doi:10.1128/JCM.01252-07
Copyright © 2008, American Society for Microbiology. All Rights Reserved.

Departments of Pathology,1 Biostatistics, St. Jude Children's Research Hospital, Memphis, Tennessee,3 University of Minnesota Medical Center, Minneapolis, Minnesota,2 Department of Microbiology, Diagnostic Laboratory Services, Inc., and the Queens and Kuakini Health Systems, Honolulu, Hawaii,4 Viromed (LabCorp) Laboratories Minnetonka, Minnesota,5 Department of Pathology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio,6 Department of Pathology, Johns Hopkins Medicine, Baltimore, Maryland,7 Departments of Pediatrics and Pathology, Children's Hospital of Philadelphia and University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania,8 University of Utah School of Medicine, Salt Lake City, Utah,9 University of Minnesota Medical Center, Fairview, Minneapolis, Minnesota,10 ARUP Institute for Clinical & Experimental Pathology, Salt Lake City, Utah,11
Received 21 June 2007/ Returned for modification 7 September 2007/ Accepted 25 October 2007
|
|
|---|
|
|
|---|
While the utility of molecular testing for EBV (most commonly PCR) is well accepted (16), and many have implemented it on a routine basis, there is a lack of uniformity and consistency among the currently available assays. The lack of standardized methods extends to all aspects of testing, including selection of specimen type, specimen collection methods, initial processing, nucleic acid extraction, molecular amplification, result interpretation, and reporting. Within the area of assay design, relevant issues include probe chemistry, target selection, cycling conditions, internal control (selection of target and mode of use), thermocycling platform, and quantitative calibration methods. Although many have moved to real-time methodologies, the implementation of these assays is by no means uniform or universal. Furthermore, and perhaps of greatest impact, there is no universally accepted quantitative standard for EBV, as has been adopted for other viruses (5, 12-14). All of these issues have grown in importance with the increased use of EBV quantitative analysis in patient management, which has become the standard-of-care in some clinical settings.
The relative effect of the different variables noted above on assay results and their impact on clinical utilization of test results when viewed over time within a given institution and when viewed across institutions have not been well defined. The optimal interpretation of studies performed at different centers using different test methodologies and the ability to monitor patients who transfer their care between different institutions becomes increasing dependent on a better understanding of these variables. Evaluating the sources of test variability should improve our ability to interpret such results and may also help establish a more uniform standard for performing these tests. This multicenter study is the first published evaluation of the variability of quantitative real-time PCR for EBV across a wide variety of institutions, testing platforms, and methodologies. Whole blood was chosen over other peripheral blood components for this analysis, based on data showing a higher degree of sensitivity in cellular compartments (whole blood or peripheral blood mononuclear cells) compared to plasma (3, 18). Also, whole blood was the most commonly used specimen type for this assay among participating laboratories.
(The results of this study were presented in part 30 April to 3 May 2006 at the 22nd annual meeting of the Pan American Society for Clinical Virology, Clearwater Beach, FL.)
|
|
|---|
|
View this table: [in a new window] |
TABLE 1. Comparison of selected assay and procedural characteristics among different test sites
|
|
View this table: [in a new window] |
TABLE 2. Test panel characteristics
|
Patient samples. Nineteen de-identified whole blood specimens (totals of 4, 6, and 9 for the first, second, and third test panels, respectively) were studied. Nucleic acid was extracted from multiple 200-µl aliquots of each sample using a QIAamp DNA blood minikit (Qiagen, Inc., Valencia, CA), with elution into 100 µl of 10 mM Tris-Cl-0.5 mM EDTA buffer (pH 9.0). All eluates from each sample were subsequently pooled and divided again into aliquots prior to distribution to ensure the uniformity of the samples among participant laboratories. All samples were centrally prepared, de-identified, coded, and then stored at –80°C until distribution and testing.
Test panel instructions and result reporting. Each laboratory was instructed to test samples using their usual standard operating procedures, reagents, and instrumentation. All samples were tested blindly, with each laboratory testing their usual number of replicates (one to three sample replicates and two to three calibrator replicates). The results were reported in "copies EBV per µl of input DNA" and submitted to an independent facility for tabulation prior to unblinding. Additional assay run data, such as cycle threshold (CT) values and calibration curve equations, were solicited for all panels, as were key facets of each site's EBV quantification protocols. To assess inter-run variability, sites performed three independent runs with panel 3. All testing was performed within a 3-day time window in order to limit the effects of specimen stability on interlaboratory result variability.
Statistical methods and data analysis. Classical least-squares regression with CT as the y variable and log10 copies/µl as the x variable was used to obtain each standard curve. The root mean square error (RMSE) was used to summarize the deviation of the calibrator CT values from the fitted line. The reported viral load result for each unknown sample was determined by averaging the CT values across replicates and mapping the result against the calibration curve. For each unknown sample, the range and interquartile range were used to summarize the variability of reported results across laboratories.
To statistically compare the interlaboratory deviation of regression parameter estimates (slope, intercept, or RMSE) of one panel to those of another panel, the absolute deviation of each lab's regression estimate from the across-lab mean for that parameter was determined. Next, for each lab, the difference between the two panels' estimates was computed, and the Wilcoxon signed-rank test was applied to the set of differences. Pagano and Gauvreau (10) describe all of these statistical methods. No adjustments for multiple testing were performed. S-Plus software (Insightful Corp, Seattle, WA) was used to perform all analyses.
|
|
|---|
Calibration curves for panel 1 were generated using both site-specific calibrators (site-specific calibrator curves) and using calibrators common to all testing sites (common calibrator curves). Summary statistics showing calibration curve variability for both calibrator types are shown in Table 3. The intercept estimates, slope estimates, and RMSE values showed less variability across laboratories when common calibrators were used than when site-chosen calibrators were used (P = 0.0742, P = 0.0273, and P = 0.0039). For instance, the intercept estimates across laboratories ranged from 28.84 to 49.36 with a standard deviation of 5.91 when site-chosen calibrators were used; but when common calibrators were used, the intercept estimates ranged from 33.2 to 40.6 with a standard deviation of only 2.19. Regression curves generated using common calibrators also demonstrated improved interlaboratory consistency compared to those generated with site-specific reagents (Fig. 1, panel 1 plots).
|
View this table: [in a new window] |
TABLE 3. Interlaboratory summaries of calibration regression estimates
|
![]() View larger version (23K): [in a new window] |
FIG. 1. Comparison of calibration curves for all test panel runs.
|
Panel 3 was analyzed using a single set of common calibrators but differed from panels 1 and 2 in that each laboratory ran the panel (calibrators and patient samples) on three separate occasions. Intralaboratory variability of calibration curve slope, y intercept, and RMSE differed markedly among the eight test sites (data not shown). There was further evidence of this variability when cross-panel comparisons were performed. For example, the intercept estimates for one laboratory ranged from 37.00 to 41.33 across panels, while the intercept estimates for another laboratory ranged from 28.84 to 41.70 across panels (Table 4). While expected discrepancies were seen between calibration curves generated from the site-specific and common calibrators curves in panel 1, calibration curve plots generated by some laboratories were more widely splayed than others, when all runs from all panels (Fig. 2, laboratories C and H) or when replicates from panel 3 test runs were compared (data not shown).
|
View this table: [in a new window] |
TABLE 4. Calibration curve parameter estimates summarized across panels for each lab
|
![]() View larger version (21K): [in a new window] |
FIG. 2. Comparison of calibration curves by laboratory, comparing all test panel runs.
|
![]() View larger version (15K): [in a new window] |
FIG. 3. Variation in viral load values for patient samples. Each point gives the result from a specific laboratory for the indicated panel and sample number. The horizontal line at y = 1 corresponds to the threshold used to qualitatively interpret the findings as positive or negative. The numbers at the bottom of each plot indicate the number of laboratories that did not report a CT value and interpreted the result as negative (zeroes are not shown).
|
![]() View larger version (17K): [in a new window] |
FIG. 4. Consistency of results for negative and low viral load samples. Each solid line shows the results of a negative or low viral sample for one panel across labs. The plot includes all instances with a mean viral load less than or equal to 2 on the log-scale. The results deemed negative by the lab are shown as –4 on this plot. A dashed horizontal line at y = 1 is included for purposes of qualitative interpretation.
|
|
|
|---|
Standardized calibration materials have been developed for a number of commonly tested analytes (5, 6, 11-15). These international standards, combined with assay improvements, including broadly available commercial reagents and automated systems for specimen preparation/processing, have played important roles in the development of quantitative tests and have greatly facilitated their broad implementation (8, 17). In addition, for some analytes, the availability of international standards and test systems with excellent reproducibility has allowed the development of international consensus guidelines on quantitative thresholds to guide patient management (4). Finally, standardized diagnostics with excellent reproducibility can be critical for the management of individual patients across geographical boundaries and in the accurate interpretation of data generated at various sites.
The data presented here suggest that similar improvements in quantitative precision can be achieved for EBV PCR using real-time methods. The generation of internationally agreed upon calibration standards would be a first step toward such a goal. However, these data also show substantial remaining variability. That variability can be ascribed to differences in assay design and laboratory technique. The latter aspect of test performance likely explains much of the intralaboratory variation seen here. Although such run-to-run changes may be mitigated somewhat by increased training or experience, it is likely that automated technologies will be more effective in reducing interlaboratory variability.
The present study was not designed to assess the relative contributions of various assay characteristics to result variability. Nor could certain variables such as the effects of transportation on test panel samples be totally obviated. However, the distribution of lyophilized material for common calibrators, the use of a single extraction method, and in panel 3 the specification of testing dates and procedures for storing and thawing specimens were all measures intended to minimize the effects of specimen handling on interlaboratory variability.
It is hoped that our findings will stimulate efforts to further standardize quantitative assays for EBV and other viruses. The problems demonstrated here exemplify the challenges in this still-developing diagnostic field. The implementation of a replicable paradigm for developing quantitative viral controls or calibration standards is clearly an unmet need. Furthermore, the availability of commercially produced and marketed assays and the introduction of automation and reagents produced using good manufacturing practices should contribute to assay precision and should allow more widespread implementation of quantitative testing.
This study was supported in part by the American Lebanese Syrian Associated Charities, the Minnesota Medical Foundation, and the University of Minnesota International Center for Antiviral Research and Epidemiology.
Published ahead of print on 7 November 2007. ![]()
|
|
|---|
This article has been cited by other articles:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Copyright © 2009 by the American Society for Microbiology. For an alternate route to Journals.ASM.org, visit: http://intl-journals.asm.org | More Info»