Previous Article | Next Article ![]()
Journal of Clinical Microbiology, July 2002, p. 2357-2362, Vol. 40, No. 7
0095-1137/02/$04.00+0 DOI: 10.1128/JCM.40.7.2357-2362.2002
Copyright © 2002, American Society for Microbiology. All Rights Reserved.
McMaster University, St. Joseph's Healthcare, Hamilton, Ontario, Canada,1 University of California, San Franscisco, California,2 University of Louisville, Louisville, Kentucky,3 Abbott Laboratories, Abbott Park, Illinois,4 University of Washington, Seattle, Washington,5 Johns Hopkins University, Baltimore, Maryland,6 State University of New York Health Sciences Center, Brooklyn, New York7
Received 7 December 2001/ Returned for modification 2 March 2002/ Accepted 15 April 2002
|
|
|---|
|
|
|---|
Current approaches to help interpret variable findings in clinical studies using PCR assays for C. pneumoniae have focused on standardizing the PCR protocols, including choice of target genes, primers, PCR conditions, detection systems, and nucleic acid extraction techniques (1, 14). It has been recognized that a commercially produced PCR assay might enable standardization of C. pneumoniae PCR results. To test the agreement or variability of different assays produced, five laboratories in North America, experienced in conducting PCR assays for C. pneumoniae, participated in this study. They donated positive and negative clinical specimens which were propagated, titrated, and used to create panels of coded specimens. The panel specimens were first DNA extracted and then sent to the participating laboratories to be tested according to their standard routine PCR protocols. An industry-developed LCx C. pneumoniae PCR assay kit (Diagnostics Division, Abbott Laboratories, North Chicago, Ill.) intended for research use only (RUO) was used in one center to test the same panels in a blinded fashion and was compared to the "in-house" PCR tests. We report comparisons of sensitivity, specificity, and reproducibility from these studies.
|
|
|---|
The first shipment of coded specimens consisted of two panels (A and B), each of which had 30 samples containing 75 µl each. After submission of the results for the 60 samples in panels A and B, each laboratory received two similar panels (C and D), which contained the same specimens as the previous panels A and B but with the order changed (for a total of 120 samples).
Specimen panels. C. pneumoniae isolates from humans (collected under the guidelines of approved institutional ethics review boards) or animals were solicited from the participating Chlamydia research laboratories. The panels included respiratory isolates (n = 5), lung tissues (n = 2), bronchial washes (n = 3), nasopharyngeal swabs (n = 2), buffycoats (n = 2), carotid tissues (n = 2), and sputa (n = 2) collected from the laboratory sites participating in this study; ATCC C. pneumoniae strains VR-1310 and AR39; and strain PS32 from the University of Washington Research Foundation. The isolates were shipped to the McMaster University Chlamydiology Research Laboratory at St. Joseph's Hospital, Hamilton, Ontario, Canada, and inoculated into HEp-2 cell lines for propagation (14). C. pneumoniae was detected by staining of the monolayers with a genus-specific anti-lipopolysaccharide monoclonal antibody (Pathfinder; Bio-Rad Diagnostics, Montreal, Quebec, Canada). Contamination with Mycoplasma species or C. trachomatis was excluded by PCR (15, 24). Of the 48 specimens received, 16 were successfully propagated in HEp-2 cells and confirmed as uncontaminated C. pneumoniae. DNA was extracted from all samples with the QIAamp DNA Mini-kit (Qiagen, Mississauga, Ontario, Canada) according to the manufacturer's instructions and was eluted in a final volume of 50 µl. Tenfold serial dilutions (10-1 to 10-8) of the purified DNA from all the C. pneumoniae-positive samples were subsequently prepared and tested in triplicate by an established nested PCR (23) to determine an approximate end point. The last two positive dilutions and the following two negative dilutions from each end point and titration were integrated into the panels. The propagated specimens were used at four different dilutions, representing different DNA copy numbers, as specimens for the multicenter study.
In addition to the 48 C. pneumoniae specimens, 12 negative specimens were incorporated into the panel. These included specimens with water only, with related organisms such as C. trachomatis and Simkania negevensis (ATCC VR1471), and clinical specimens (sputum or blood) which had tested negative in multiple determinations (10 or more) by an established nested PCR (23). Aliquots (75 µl) for each panel specimen were sent, frozen on dry ice, to each participating laboratory. Data report sheets were included in all shipments, as were instructions for testing and reporting panels A and B (60 specimens), followed 1 month later by panels C and D (60 specimens).
The orders of the two sets of 60 specimens were randomized and coded, and all participating laboratory technologists were blinded to the specimen status. The study analysis was initially performed, and ambiguities in reporting were resolved, before the code was broken.
Quantitation of panel specimens. Selected specimens were quantitated by real-time PCR. An aliquot of 5 µl of extracted DNA from the panel specimens was added to 15 µl of a reaction mixture containing 4 mM MgCl2, 0.5 µM primers CPNA and CPNB (7), and SYBR Green dye (LC DNA FastSTART Master SYBR Green 1 kit). Specimens were amplified on a Lightcycler (Roche Molecular Biochemicals) under the following cycling conditions: an initial 10 min at 95°C for FastStart Taq DNA polymerase activation, followed by 40 cycles of 2 s of denaturation at 95°C, 5 s of annealing at 55°C, and 19 s of extension at 72°C. Data were obtained after the extension period in the "single" mode. Serially diluted cloned plasmid controls containing the CPNA-CPNB PCR product (10 to 106 copies) were used to generate a standard curve for quantitation of C. pneumoniae DNA.
PCR methods. Table 1 summarizes the characteristics of the five in-house PCR methods used by the study participants. Each laboratory director was instructed to detail the laboratory's method in advance. Interpretation of results and photographs of the agarose gels were submitted with the results reported for each panel.
|
View this table: [in a new window] |
TABLE 1. Summary of in-house PCR methods for detection of C. pneumoniae
|
Statistical methods.
SPSS for Windows, version 10.0 (SPSS Inc., Chicago, Ill.), was used for all statistical analyses. For the 24 negative panel specimens, specificity was determined for each of the tests. For the 96 potentially positive specimens, analyses were performed to examine the sensitivity by a measure for correlated dichotomous outcomes (Cochrane Q), followed by pairwise comparisons between the LCx C. pneumoniae PCR RUO assay and all other tests by use of the McNemar
2 test. For the Cochrane Q test, which measures overall differences between the six tests, an alpha of 0.05 was established for determining statistical significance, whereas for the five pairwise comparisons, an alpha of 0.01 was established to account for multiple testing (Bonferroni's correction).
Reproducibility was examined by contingency table methods, overall agreement, and Cohen's kappa (agreement beyond chance). Multiple linear regression models were constructed to examine the relationship between C. pneumoniae concentration, test positivity, test volume, and number of replicates.
The study was designed, and all statistical analyses were performed, independently of the manufacturer of the LCx RUO PCR.
|
|
|---|
|
View this table: [in a new window] |
TABLE 2. Comparison of sensitivities of C. pneumoniae PCRs by copy number for 96 coded samples
|
![]() View larger version (14K): [in a new window] |
FIG. 1. Box-and-whisker plot of the number of test centers (n = 6) reporting a specimen positive versus the quartile of the C. pneumoniae DNA copy number for 96 specimens. N, number of specimens tested in each quartile. Thick lines, median values; rectangular boxes, 25th to 75th percentiles; "whiskers," 10th and 90th percentiles (SPSS for Windows 10.0). Quartile 1, <0.05 copies/µl; quartile 2, 0.05 to 0.49 copies/µl; quartile 3, 0.5 to 4.99 copies/µl; quartile 4, 5.0 copies/µl.
|
Impact of replicate testing on the sensitivity of the RUO assay.
The data in Table 2 suggested that laboratories doing replicate testing had a better chance of identifying positives with a low level of C. pneumoniae nucleic acid. We performed an analysis of the impact of replicate testing on the LCx C. pneumoniae PCR assay and the in-house PCR from laboratory B. Both assays tested three aliquots of each sample. Figure 2 illustrates the impact of the number of replicates on the percentages of samples positive in the LCx C. pneumoniae RUO assay and the in-house PCR from laboratory B according to the DNA copy number in each sample. When the 24 samples contained higher numbers of C. pneumoniae DNA copies (
1.0 DNA copy/µl), testing more replicates resulted in a minimal increase in the percentage of positive results with the in-house PCR of laboratory B and failed to increase that percentage in the LCx PCR (Fig. 2). Our observations were similar when the samples were very dilute (<0.1 copy/µl). The advantage of replicate testing was demonstrated by an increased percentage of positive samples in the 0.1-to-0.99 copy/µl interval. For these samples, the sensitivity of the in-house PCR test increased from 21% (5 of 24) for single testing to 42% (10 of 24) for a criterion of at least one positive out of two replicates and 54% (13 of 24) for at least one positive out of three replicates. For the LCx C. pneumoniae PCR assay, the results were: 21% (5 of 24), 63% (15 of 24), and 67% (16 of 24), respectively, indicating that a single LCx RUO test could have missed more than half of the positive specimens containing these levels of C. pneumoniae DNA.
![]() View larger version (15K): [in a new window] |
FIG. 2. Impact of number of replicates tested on the percentage of samples testing positive in the Abbott LCx C. pneumoniae RUO assay (broken lines) and the in-house PCR from laboratory B (solid lines), by DNA copy number per microliter. A sample was considered to test positive if one or more positive results were obtained for one, two, or three replicates tested.
|
Test specificity. The LCx C. pneumoniae PCR assay demonstrated a sample specificity of 100% (24 of 24) for the negative specimens. In total, this assay was performed six times on the 12 negative specimens during the multicenter panel challenges (72 determinations) and an additional three times in a reproducibility panel (36 tests), and all 108 tests were scored negative. The specificities of the five in-house assays on the negative specimens were as follows: 100% (24 of 24) for centers B, D, and F and 95.8% (23 of 24) for centers A and E. Approaching test specificity from a test point of view, during the first challenge these negative samples were tested one to four times per center (total, 11 x 12 = 132) and a single run was positive; thus, the overall specificity for all the tests was 99.2% (131 of 132). In the second challenge, laboratory D had two false-positive replicates of one sample, laboratory A had one false-positive result, and laboratory E switched to double testing. Therefore, the overall test specificity in the second challenge was 97.9% (141 of 144).
Reproducibility of testing results between panels. Since the samples in panels C and D were the same as those in A and B, but were numbered differently, we were able to determine the reproducibility of testing results for each in-house PCR and the RUO assay. Table 3 shows the reproducibility or agreement within each laboratory between the two panels of 60 specimens, tested 1 month apart. Calculation of the kappa values showed the LCx C. pneumoniae PCR assay to have the best agreement (0.77) and the in-house PCR in laboratory E to have the least agreement (0.19). The majority of laboratories showed agreement above 0.48. Examination of agreement according to amount of DNA in the specimen (data not shown for the in-house PCR tests) indicated the best agreement in all laboratories for specimens having the greatest or least amount of DNA in the samples, except for laboratory E. Analysis of intralaboratory reproducibility of the LCx C. pneumoniae PCR assay for the same day showed no statistically significant differences between three runs of 60 specimens each time (run AB yielded 29, 31, and 28 positives and a Cochrane Q of 1.6 on 2 df [P = 0.46]; run CD yielded 28, 28, and 26 positives and a Cochrane Q of 0.9 on 2 df [P = 0.64]). Analysis of intralaboratory reproducibility for the LCx RUO PCR assay 1 month apart was possible because three runs of 60 specimens were done at two different times (6 x 60). Testing reproducibility indicated no statistical difference between runs (Cochrane Q = 4.4 with 5 df; P = 0.49).
|
View this table: [in a new window] |
TABLE 3. Within-center agreement between two C. pneumoniae panels of 60 specimens performed 1 month aparta
|
I 2I) were found negative by six replicates of testing by the in-house PCR assay of laboratory B, and the other five discordant samples were positive in that assay in one to three of the six replicates. Out of 60 specimens and using a positive result in at least one of the three runs as the criterion for detection of a positive specimen by each of the two technologists (data not shown) there were 28, 28, and 26 positives reported by the original LCx PCR-testing technologist, compared to 27, 29, and 31 positives reported by the other technologist, indicating no significant differences (Cochrane Q = 4.3 with 5 df; P = 0.50). |
View this table: [in a new window] |
TABLE 4. Testing profiles of seven specimens found discordant when tested by two technologists in replicates of three in the LCx C. pneumoniae RUO assay
|
|
|
|---|
Our interlaboratory comparison of the detection of C. pneumoniae DNA in a panel of 60 preextracted specimens assayed in parallel by six PCR tests found that the LCx C. pneumoniae PCR assay was more sensitive than all of the in-house NAA tests, especially when the samples contained >1 copy of C. pneumoniae DNA per µl. Of the 24 samples that were diluted 1:100 past the end point, the five in-house tests and the LCx C. pneumoniae PCR RUO assay were in agreement that 8 were negative in panels A and B and 12 in panels C and D. Of these 20 samples, 6 were determined negative both times by consensus of all six assays. If these 12 (6 + 6) consensus negatives were moved from the positive group of specimens, the overall adjusted sensitivity for each test would increase (data not shown); that of the LCx C. pneumoniae PCR RUO assay would increase from 68 to 77% compared to those of the in-house PCR tests, which would range from 37 to 71%.
The specificity of the LCx RUO PCR was 100%, and few false positives were observed in the other PCR tests. Negative specimens were purposely placed in sequence after strong positives, but the study results showed high specificity, with little or no contamination, for these assays performed by experienced laboratory technologists. These results may not be generalizable to all laboratory settings.
Apfalter et al. (1) conducted a nine-center comparison of 16 PCR test methods on identical sets of 15 experimental atheroma samples and 5 spiked controls. The majority of specimens with 1 inclusion body of tissue homogenate were identified by all of the participants, but only 19% of the test methods reported specimens with 0.01 of an inclusion body as positive. In the Apfalter study, 3 out of 16 negative samples were rated as positive, whereas in our study, a minimal number of false positives were reported by the in-house assays (overall specificity, 97.9%). All three of the false positives were reported by two laboratories, and the LCx PCR assay had no false positives. Regardless of previous experience, a high specificity needs to be continually demonstrated by instituting measures to minimize carryover contamination, including detection of contamination through an adequate number of negative controls in each run. The LCx C. pneumoniae PCR RUO assay kit has a nonnested automated format, with fewer steps where contamination might occur.
It was surprising that the overall sensitivity varied markedly for the six testing sites, because extraction was standardized, three laboratories used the same nested PCR primers, and the laboratory technologists performing the assays were highly experienced. The panel was intentionally designed to maximize differences between tests by diluting specimens beyond the end point of detection. Thus, a large number of samples had low DNA copy numbers. These differences in sensitivity may or may not reflect on how well these tests will perform with clinical specimens. With a high C. pneumoniae DNA copy number, all of these assays would be expected to perform well on clinical specimens. Based on our previous observations (21), the C. pneumoniae DNA copy number can be low in clinical specimens such as sputum, nasopharyngeal aspirates, and peripheral blood mononuclear cells, as shown by replicate testing and probit regression modeling. The observations in that study indirectly confirm the findings in this study, but multicenter comparisons need to be repeated with clinical specimens.
We demonstrated the critical dependence of test sensitivity on relative DNA copy number in each sample (Table 2 and Fig. 2). Figure 3 plots the percentage of positives in the LCx RUO PCR assay against the DNA copy number per microliter and illustrates the ability of this assay to reproducibly detect samples with lower copy numbers, based on testing nine replicates of 20 µl each and using the criterion that at least one of three replicates must test positive for the sample to be declared positive. A very high percentage of positives can be expected when the number of DNA copies is above 1.0 per 1 µl of sample. The percentage varies from 77 to 12% when the number of DNA copies ranges from 0.99 to 0.01 per µl. Reproducibility was also assessed by testing the same specimens a second time in a different random order. All laboratories showed high kappa agreement except for one. The LCx C. pneumoniae PCR assay had the highest kappa agreement at 0.77. This level of intraobserver agreement remained high as larger numbers of runs in the RUO assay were factored into the calculations. The reproducibility of the RUO assay between two different technologists performing three runs each (180 samples) was also excellent, and presumably when this assay becomes more universally used, similarly good agreement may be expected.
![]() View larger version (19K): [in a new window] |
FIG. 3. Relationship between the reproducibility of the Abbott LCx C. pneumoniae RUO PCR and the C. pneumoniae DNA copy number among 48 specimens assayed nine times each. Dashed lines indicate DNA copy numbers of 1 (left) and 10 (right)/µl.
|
|
|
|---|
This article has been cited by other articles:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Copyright © 2009 by the American Society for Microbiology. For an alternate route to Journals.ASM.org, visit: http://intl-journals.asm.org | More Info»