Previous Article | Next Article ![]()
Journal of Clinical Microbiology, April 2006, p. 1283-1287, Vol. 44, No. 4
0095-1137/06/$08.00+0 doi:10.1128/JCM.44.4.1283-1287.2006
Copyright © 2006, American Society for Microbiology. All Rights Reserved.
Robert Koch Institute, Berlin,1 Bundeswehr Institute of Microbiology, Munich,2 Bernhard Nocht Institute for Tropical Medicine, Hamburg, Germany3
Received 10 October 2005/ Returned for modification 5 December 2005/ Accepted 2 February 2006
|
|
|---|
|
|
|---|
External quality assurance (EQA) has improved the general diagnostic performance of laboratories for a number of viruses (7, 8, 13, 14, 17, 18). This is not only because EQA identifies weaknesses but also because it provides virus material which is necessary for assay optimization. Because no EQA program was available for rare and emerging viruses, we have recently organized the first proficiency studies in this field, focusing on Ebola virus, Marburg virus, Lassa virus, and orthopoxviruses in one study and on severe acute respiratory syndrome coronavirus in another (5, 10).
To find out whether our prior studies had an effect on diagnostic quality, we have now conducted a new such study after 2 years. Focusing on orthopoxviruses this time, we have extended the scope of tested performance criteria. Sensitivity, rates of false-negative results, and the ability of laboratories to detect the presence of PCR inhibitors were queried in order to provide a comprehensive picture on how well laboratories are prepared for a real diagnostic scenario.
|
|
|---|
Test specimens. To obtain test specimens, genotyped MPX, CPX, and VAC stocks were grown on Vero cells. Freeze-thawed preparations were heated to 56°C for 1 h, followed by gamma irradiation with 30 kGy to inactivate viruses. Residual infectivity was excluded by inoculation in Vero cell cultures (three passages). Inactivated virus stock solutions were aliquoted, lyophilized, and redissolved, and virus DNA was quantified by two different real-time PCR assays (11, 12). Quantification with other methods was not attempted after inactivation, but virion integrity was confirmed morphologically by electron microscopy. The final panel of test samples was generated by diluting inactivated virus stock solutions in human fresh-frozen plasma testing negative for human immunodeficiency virus type 1, hepatitis B virus, hepatitis C virus, and orthopoxviruses by PCR or reverse transcription-PCR. Aliquots of 100 µl each were lyophilized and shipped at ambient temperature to the participating laboratories. Each participant received a coded panel of seven virus-positive samples containing between 4,000 and 40,000,000 DNA copies per ml after resuspending in 100 µl of water. Six negative plasma samples were also included. A negative and a positive plasma sample were included which contained 10% (wt/vol) humic acid (Sigma, Munich, Germany), as well as one negative plasma sample containing 10,000 U/ml of heparin (Sigma, Munich, Germany). Both additives represented components which cause strong inhibitory effects in PCR (unpublished observations). The presence of a detectable inhibitory effect in both samples was confirmed in the reference laboratory using a commercial PCR kit (RealArt Orthopox PCR kit; Artus, Hamburg, Germany) after extracting nucleic acids with a commonly used method of sample preparation (QIAGEN silica-based columns).
Statistical analysis. All statistical analysis was carried out using the Statgraphics 5.1 software package (Manugistics, Dresden, Germany). Analysis of variance (ANOVA) analysis used type III sums of squares to obtain evaluations of each influence factor without the effect of the remaining factors.
|
|
|---|
Sensitivity. In our prior EQA study, the overall sensitivity of laboratories was characterized on the basis of the detection rate that was cumulatively achieved in all virus-containing samples. Seventy-eight percent of all tests in virus-containing samples and 44% of tests in samples containing less than 10,000 copies per ml were positive in the first study (10). In the current study, the corresponding numbers were 85% and 58%, respectively. However, as the composition of the test panel in the second study was not identical to that of the first one, these numbers were not directly comparable. To achieve a precise comparison, data from both studies were evaluated using a dose-response model (Probit analysis). As shown in Fig. 1, there was no obvious difference in overall sensitivities in both studies. In the earlier EQA, the average laboratory achieved detection with 50% probability when at least 1,380 copies per ml were present in a sample, compared to exactly 1,000 copies in the current study. Eighty percent detection chances were achieved at 97,723 and 56,234 copies per ml, respectively, in the old and the new study. When both data sets were compared in a Wilcoxon two-sample test, contrasting the input copy numbers required to achieve given detection chances, starting at 20% and ending at a 99.9% chance, the differences between the studies were not significant (P = 0.694). Using the same criteria, the 15 laboratories which had already participated in the earlier study did not achieve significantly higher sensitivities than the 18 first-time participants (Wilcoxon two-sample test; P = 0.76). Among the five laboratories which had achieved the worst results in the first study, three were now among the top performers, while two others still remained far below an acceptable level: no virus detection was achieved in any sample containing less than 11,000,000 virus genome copies per ml.
![]() View larger version (22K): [in a new window] |
FIG. 1. Proportions of positive results (y axis) achieved globally by participants in each test sample (virus concentration per sample is depicted on the x axis). Letters at datum points identify virus strain present per sample (V, vaccinia virus; CM, camelpox; C, cowpox; no letter, monkeypox). (A) Reevaluated results from the first EQA study on bioterrorism-relevant viruses (orthopoxvirus data points only). (B) Global results from this study. (C) Results from first-time participants in this study (see the text). (D) Results from second-time participants in this study.
|
A sample containing 28,000 copies of MPX was typed as CPX by three laboratories. One of these laboratories and one additional laboratory mistook a sample containing 11,000,000 copies of CPX for VAC. To test for the ability of laboratories to discriminate orthopoxviruses from other poxviruses, one sample containing a yatapox virus (tanapox) and one sample containing a parapoxvirus (orf virus) were also included in our sample panels. Of 27 laboratories, four mistook tanapox for VAC or an unspecified orthopoxvirus, and three mistook orf virus for either MPX, CPX, or an unspecified orthopoxvirus. All wrong genotyping results in total were contributed by seven laboratories, using either real-time PCR (n = 2), sequencing (n = 1), or RFLP analysis (n = 4) for typing.
Quantification. The relevance of poxvirus quantification in patients has not been evaluated clinically. However, such data have proven useful in the management of other rare virus infections in the past (16). Of all 33 participants in our study, 9 provided information on the virus concentration in test samples. Figure 2 summarizes a comparison of observed and expected quantification results. At the 95% confidence level, laboratories achieved quantification with a deviation of ±1.4 log10 from the expected value. Quantification of VAC was the most accurate, followed by MPX and CPX (significant differences in mean deviations from expected values; F test; P = 0.037). This observation is possibly associated with the fact that many laboratories used VAC for establishing their assays.
![]() View larger version (14K): [in a new window] |
FIG. 2. (A) Predetermined virus concentration in all samples containing orthopoxviruses (x axis) versus the virus concentrations in the respective samples determined by nine different laboratories. The resulting linear regression followed the equation Y = 0.281879 + 0.93 x X. The correlation coefficient was 0.87 (R2 = 0.764). (B) Deviations of determined DNA concentrations from the values expected according to prequantification data for three different orthopoxviruses.
|
PCR inhibition. Clinical samples may contain substances which interfere with PCR. Appropriate spiking of samples with internal or external inhibition controls is mandatory when critical diagnoses have to be reached by PCR (6). To test the ability of laboratories to detect PCR inhibitors, two negative plasma samples were spiked with substances of known inhibitory influence on PCR. One sample contained heparin, a common anticoagulant for blood samples. It can be eliminated from samples to a certain degree by efficient sample preparation (1, 4). The other sample contained humic acid, a substance which according to our experiences is notoriously difficult to remove by various methods of preparation.
Whether an inhibitor is detected depends on two factors: on the one hand, efficient sample preparation may remove the inhibitor; on the other hand, control techniques may fail to display an inhibitor due to breaches in technology, e.g., a too-high concentration of control DNA (15). This study intended to test the inhibition control techniques applied by laboratories, also taking the effect of sample preparation methods into consideration. Twenty-six of 33 laboratories used silica-based affinity chromatography methods from major commercial manufacturers which were specified for use on viruses (20 laboratories used QIAGEN methods, 6 laboratories used those of Roche; no discrimination was made between manual and automated methods). Six laboratories used silica-based in-house protocols or commercial methods from other manufacturers which are not widely applied in the field. Only one laboratory used organic extraction with phenol-chloroform. ANOVA analysis on the three main categories of methods (QIAGEN, Roche, and in house) revealed a significantly smaller rate of inhibited samples in laboratories using in-house methods (8.3% samples) than in those using either QIAGEN or Roche methods (41% or 50%, respectively; the difference was not significant). Because in our hands commercial extraction methods are much more efficient at removing inhibitors than in-house methods, we concluded that those laboratories in the first group which did not detect an inhibitor probably missed it due to insufficient inhibition control technique. For the commercial assays, the presence of an inhibitor after extraction had been confirmed in the reference laboratory. It was thus assumed that with either of these methods an inhibitor should have been detected in both samples. Given this, 55% (18 laboratories) of participants could detect the strong inhibitory effect of humic acid, and only 12% (4 laboratories) were able to detect the weak effect of heparin in addition.
Influence factors. To determine which technical factors had general influence on laboratory performance, all applied variants of sample preparation and PCR methods were categorized according to five major criteria, as listed in Table 1. For each laboratory, the influences of these factors on the detection rates in positive samples were analyzed by multifactor ANOVA. Samples which contained more than 105 copies per ml were excluded, because they were unlikely to reflect general lack of sensitivity. Those five laboratories which reported false-positive results in more than half of all negative samples were also excluded: their results in low-positive samples may have been due to contamination rather than due to true virus detection. As shown in Table 1, the use of real-time PCR protocols or commercial test kits had a significant positive influence on sensitivity. This correlated well with observations made in earlier studies (5, 10). The use of commercial sample preparation methods from major suppliers did not provide significant influence, and neither did the use of automated platforms for sample preparation.
|
View this table: [in a new window] |
TABLE 1. Factors influencing the performance of laboratoriesd
|
|
|
|---|
Though the study was widely announced, mainly European laboratories participated, responding to a call for participation from the European organization ENIVD. However, since the diversity in Europe regarding regulations, funding, and training is comparably high, we believe that the European setting will be similar to many other regions of the world.
The major finding in our study is an only marginal improvement of performance: detection sensitivity in general remained at the same level, and second-time participants did not show better sensitivity than first-time participants. Other general diagnostic skills, such as the detection of inhibitory substances, have been systematically queried in this study for the first time. In view of experiences with molecular diagnostics of severe exotic infections (6), the rate of laboratories which were capable of detecting at least a strong PCR inhibitor was too low at 55%. Current methods of inhibition control should be adapted from published PCR-based assays for more common viruses. On the other hand, laboratories in general seemed to have good command in avoiding cross-contamination, which is probably the most important skill in this context. It is interesting that contamination was associated with only a few single laboratories; rates seemed to correlate positively with the application of non-real-time PCR methods and simple nucleic acid extraction procedures. Obviously, general improvements in PCR technology are taking effect, even in this very specialized field of molecular diagnostics. This is also documented by the fact that the majority of participants were able to genotype and quantify virus. Real-time PCR was the technical basis for quantification in the majority of laboratories (n = 23). Remarkably, quantification of VAC was more accurate than that of other viruses, probably because it is the only virus widely available for the establishing of assays. Genotyping was also done by real-time PCR in most laboratories, although results obtained from PCR product sequencing seemed more reliable. Interestingly, those laboratories that used classical RFLP genotyping generated the most typing errors.
The expertise for application of sophisticated molecular diagnostic technology is now available in most laboratories. However, the costs of such methods still exceed those of conventional techniques severalfold. In view of the described outcomes, investment into the improvement of methodology does seem well justified. On the other hand, most western countries have already provided generous funding for bioterrorism-related diagnostics. In this light, the observed outcome in second-time participants in our study appears disappointing, and improvements should be implemented very soon. Other issues in improving diagnostics, e.g., the availability of reference materials, cannot be accomplished as simply. Our study demonstrates the relevance of reference materials and provides a useful contribution in this area: all inactivated test samples are available for a nonprofit charge from ENIVD.
We are grateful to the following participants, listed by country, for making this study possible: INEI Anlis "Carlos G. Malbran," Buenos Aires, Argentina; Victorian Infectious Diseases Reference Laboratory, Melbourne, Australia; Katholieke Universiteit Leuven, Belgium; Bureau of Microbiology Health Canada, Winnipeg, Canada; Statens Serum Institut, Copenhagen, Denmark; Haartman-Institute, Helsinki, Finland; UIBVE, Lyon, France; Unité de Virologie, CRSSA, La Tronche, France; Artus GmbH, Hamburg, Germany; Bernhard Nocht Institute for Tropical Medicine, Hamburg, Germany; Institut für Mikrobiologie der Bundeswehr, Munich, Germany; Institut für Lebensmittel, Arzneimittel und Tierseuchen, Berlin, Germany; Ludwigs-Maximilians-Universität, Munich, Germany; Medizinische Hochschule, Hannover, Germany; Niedersächsisches Landesgesundheitsamt, Hannover, Germany; Philipps Universität, Marburg, Germany; Robert Koch-Institut, Berlin, Germany; Wehrwissenschaftliches Institut für Schutztechnologien-ABC-Schutz, Munster, Germany; Aristotelian University, Thessaloniki, Greece; University College, Dublin, Ireland; Army Medical and Veterinary Research Center, Rome, Italy; Instituto Nazionale per le Malattie Infective, Rome, Italy; Erasmus MC, Rotterdam, The Netherlands; RIVM, Bilthoven, The Netherlands; Military Institute of Hygiene and Epidemiology, Warsaw, Poland; Russian Academy of Sciences, Novosibirsk, Russia; VECTOR, Koltsovo, Russia; University of Ljubljana, Ljubljana, Slovenia; Instituto de Salud Carlos III, Madrid, Spain; Swedish Institute for Infectious Disease Control, Stockholm, Sweden; Spiez Laboratory, Spiez, Switzerland; Siriraj Hospital Mahidol University, Bangkok, Thailand; Health Protection Agency, Porton Down, United Kingdom; and Health Protection Agency, London, United Kingdom.
Supplemental material for this article may be found at http://jcm.asm.org/. ![]()
|
|
|---|
This article has been cited by other articles:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Copyright © 2009 by the American Society for Microbiology. For an alternate route to Journals.ASM.org, visit: http://intl-journals.asm.org | More Info»