ABSTRACT
In a comparison of 4 second-generation nucleic acid amplification tests performed with self-collected vaginal swab (SCVS) and first-void urine (FVU) specimens from 575 women, SCVS specimens indicated more infections than did FVU specimens in all assays. The prevalence rates were 9% (53/575 patients) for Chlamydia trachomatis and 2% (11/575 patients) for Neisseria gonorrhoeae. The clinical sensitivities for testing SCVS specimens for C. trachomatis were 98.1% on a Tigris system and 96.2% on a Panther system for the Aptima Combo 2 assay (Hologic Gen-Probe), 98.0% for the RealTime CT/NG assay on an m2000 instrument (Abbott), 90.6% for the ProbeTec CT/GC Qx assay on the Viper system (Becton Dickinson), and 84.6% for the cobas CT/NG assay on the cobas 4800 platform (Roche). Clinical sensitivities for C. trachomatis in FVU specimens were 88.7% (Tigris) and 88.0% (Panther) for the Aptima Combo 2 assay, 76.9% for the RealTime CT/NG assay, 75.5% for the ProbeTec CT/GC Qx assay, and 81.1% for the cobas CT/NG assay. Clinical sensitivities of the assays for N. gonorrhoeae, with limited positive results, ranged from 63.6% to 100%. Specificities for both infections ranged from 98.4 to 100%. Differences in analytical sensitivities and levels of molecular targets in clinical samples but not inhibitors of amplification may explain the differences in clinical sensitivities.
INTRODUCTION
Chlamydia trachomatis and Neisseria gonorrhoeae infections of the female genital tract present a diagnostic challenge because many patients are asymptomatic (1, 2). This has led to implementation of screening strategies using less-invasive sampling procedures (3, 4), and screening programs often increase the number of samples received in the laboratory (5). Nucleic acid amplification tests (NAATs) for C. trachomatis and N. gonorrhoeae diagnosis have been available from commercial sources for over 15 years. The first PCR assay from Roche was the semiautomated Amplicor CT/NG test (6), which evolved into the automated cobas Amplicor assay. The ligase chain reaction assay (Abbott LCX) was the second NAAT to be used extensively but was withdrawn globally in 2003. A first-generation strand-displacement assay called ProbeTec ET, from Becton Dickinson, became available for evaluation in 1999. During the same time frame, a first-generation transcription-mediated amplification test was developed by Gen-Probe and evolved into the second-generation Gen-Probe Aptima Combo 2 (AC2) assay, which was evaluated with female specimens and was reported in 2003 (7). More recently, the U.S. Food and Drug Administration (FDA) cleared three additional second-generation assays and automated instruments, namely, the Abbott Molecular RealTime CT/NG assay on the m2000 instrument, the Becton Dickinson ProbeTec CT/GC Qx assay on the Viper XTR instrument, and the Roche Diagnostics cobas CT/NG assay on the cobas 4800 instrument.
Examination of the package inserts for these newer second-generation assays indicated that FDA clearance was granted by comparing the new assays with patient-infected status (PIS) results, with infection being based on positive results from at least two cleared assays when 2 different specimen types were tested. The cobas CT/NG assay was compared with two other second-generation assays (AC2 and ProbeTec CT/GC Qx assays) (8, 9). The AC2 test and the first-generation ProbeTec ET assay were chosen for the RealTime CT/NG performance studies (10) and the ProbeTec CT/GC Qx clinical evaluations (11). Since the AC2 test is clinically more sensitive than the first-generation ProbeTec ET assay (12), comparisons that include it in PIS analyses may result in misleading performance estimates for the investigational assay, because the extra positive results detected by the more-sensitive AC2 reference test are classified as false-positive results. Head-to-head comparisons of several assays with more than one sample type have yielded more-accurate reflections of sensitivity and specificity (12, 13). The objectives of this study were to compare the performance of the four second-generation assays with first-void urine (FVU) samples and self-collected vaginal swab (SCVS) specimens from patients attending women's health clinics and to detect inhibitors when a proportion of each sample was spiked with known concentrations of C. trachomatis organisms.
MATERIALS AND METHODS
Study design.Between July 2012 and August 2013 a total of 575 women attending sexual health centers in Hamilton and Toronto, Ontario, Canada, self-collected FVU (initial 20 ml of urine) specimens and then four SCVSs, using collection kits from Abbott Molecular, Becton Dickinson, Roche Diagnostics, and Hologic Gen-Probe. Each woman signed a consent form approved by the St. Joseph's Healthcare and Hamilton Health Sciences Research ethics boards. Illustrated instructions were provided, and the order of collection of the SCVSs was randomized. Following collection, the samples were transported the same day to the Infections Research Laboratory (IRL) at St. Joseph's Healthcare.
Determination of assay endpoints for detection.Before any clinical specimens were processed, uninfected SCVS and FVU specimens were used to make serial 10-fold dilutions of C. trachomatis strain L2 434, which had been propagated in McCoy cell cultures, using 10 replicates for each dilution. C. trachomatis elementary bodies (EBs) were counted by direct fluorescent antibody staining with monoclonal antibodies specific for major outer membrane proteins, as described previously (12). The panels were shipped by overnight courier and assayed at the testing sites. The endpoint dilution of detection was selected as 10 of 10 replicates reading positive in the assay for each specimen type. A dilution 100-fold above the endpoint of detection was used to spike clinical specimens to determine whether inhibitors were present. Percent rate of inhibition was calculated as the number of spiked specimens testing negative divided by the number of spiked specimens times 100. Probit regression analysis was performed to calculate the 50% levels of detection, with 95% confidence intervals (CIs) (SPSS version 20).
Laboratory testing.For each patient, spiked and unspiked FVU and SCVS samples were tested in a blinded fashion, within 2 weeks, with the AC2 test on Tigris and Panther instruments (Hologic Gen-Probe) at the IRL. Samples were shipped, according to the package insert guidelines, to Trois-Rivières, Quebec, Canada, for RealTime CT/NG testing on an m2000 system, to Halifax, Nova Scotia, Canada, for BD ProbeTec CT/GC Qx testing on a Viper system, and to St. John's, Newfoundland, Canada, for cobas CT/NG testing on a cobas 4800 system. All tubes were bar coded, handled in a blinded fashion, and tested according to each manufacturer's instructions within 2 weeks after collection. Results from each testing site were sent electronically to the IRL, where they were transferred into a Microsoft Excel database. During this study, workflow and maintenance characteristics of each automated instrument were determined (18).
For the AC2 test, each patient self-collected a SCVS and placed it in an Aptima collection and transportation tube containing 2.9 ml of transport medium. For FVU samples, 2 ml of urine was pipetted into an Aptima transport tube containing 2 ml of medium, in the IRL. The two specimen types were split into two tubes each and spiked with an appropriate concentration of C. trachomatis. The tubes were placed directly in the Tigris system, and 400 μl of each sample was tested in the assay. The same samples were tested separately, in a blinded fashion, with a Panther instrument.
For the RealTime CT/NG test, each patient placed a self-collected Abbott swab directly in an Abbott multi-Collect tube containing 1.2 ml of specimen transport buffer. For FVU samples, 3 ml of urine was placed in a multi-Collect tube containing 1.2 ml of buffer, in the IRL. The two types of samples were split into two tubes each, and one of each was spiked. At the testing site, 400 μl of each sample was tested with the m2000 instrument.
For the ProbeTec CT/GC Qx test, patients placed a self-collected BD swab into a transport tube, which was transported dry. At the IRL, SCVS specimens were eluted into a tube containing 2 ml of BD swab diluent, and 4 ml of the FVU samples was placed in an empty BD tube. Each sample type was split into two tubes, and one of each was spiked. At the testing site, 150 μl of each sample was tested with the Viper instrument.
The Roche SCVS was placed by the patient in a cobas tube containing 4.3 ml of cobas PCR medium. At the IRL, 5 ml of the FVU specimens was pipetted into a cobas tube containing 4.3 ml of PCR medium. The two sample types were split into two tubes each, and one of each was spiked. After shipping to the testing site, 800 μl of each sample was processed with the cobas 4800 system.
Data analysis.The results of each assay were compared with PIS findings based on results from all four assays with both SCVS and FVU samples. Although AC2 testing was performed on two platforms, results from the Tigris instrument were used in the PIS analysis. A patient was considered infected if at least two of the four assays yielded positive results for any specimen type. The clinical performance of tests with SCVS and FVU specimens was determined by calculating sensitivity, specificity, and predictive values, with 95% confidence intervals. In addition, because FVU specimens from four patients were negative in all assays, the clinical sensitivity of tests with FVU specimens was also determined in a PIS analysis that excluded those four patients. McNemar's test was used to compare sensitivity estimates for matched samples. P values of <0.05 were considered statistically significant.
RESULTS
As indicated in Table 1, the final dilutions recording 10 of 10 replicates as positive for SCVS and FVU specimens were 10−7 and 10−7, respectively, for AC2 testing with the Tigris system, 10−5 and 10−5 for RealTime CT/NG testing, 10−6 and 10−5 for ProbeTec CT/GC Qx testing, and 10−5 and 10−6 for cobas CT/NG testing. On the basis of these results, all samples were spiked with C. trachomatis at dilutions 100-fold above each assay's endpoint of detection. Arbitrarily considering calculated percent rates of inhibition of less than 3% as expected variations related to spiking (Table 1), results from the spiked tubes showed no inhibition in either specimen type for all assays, except for SCVS specimens in the ProbeTec CT/GC Qx assay (4.5%).
Endpoints of detection of C. trachomatis dilutions in normal SCVS and FVU samples for the AC2 (Tigris), RealTime CT/NG, ProbeTec CT/GC Qx, and cobas CT/NG assays
The C. trachomatis prevalence was 9% (53/575 patients), and 60.4% of infected patients (32/53 patients) tested positive with both specimen types in all tests on all instruments. A comparison of the C. trachomatis clinical sensitivities, specificities, and predictive values of each assay for SCVS specimens is shown in Table 2. The sensitivities were 98.1% for the AC2 assay on the Tigris system and 96.2% on the Panther system, 98.0% for the RealTime CT/NG assay, 90.6% for the ProbeTec CT/GC Qx assay, and 84.6% for the cobas CT/NG assay. Three of the C. trachomatis-infected cases the cobas CT/NG assay identified as negative did not have sufficient sample volume for testing with the RealTime CT/NG assay and so had to be excluded from the McNemar test, the results of which did not reach significance (P = 0.063).
Sensitivities, specificities, and positive and negative predictive values of the AC2 assay on the Tigris and Panther systems, the RealTime CT/NG assay, the ProbeTec CT/GC Qx assay, and the cobas CT/NG assay for SCVS specimens
Using the FDA-defined PIS findings, FVU testing for C. trachomatis (Table 3) yielded sensitivities for the AC2 assay of 88.7% on the Tigris system and 88.0% on the Panther system; the values were 76.9% for the RealTime CT/NG assay, 75.5% for the ProbeTec CT/GC Qx assay, and 81.1% for the cobas CT/NG assay. The clinical sensitivity of the AC2 assay was significantly greater than those of the RealTime CT/NG assay (P = 0.039) and the ProbeTec CT/GC Qx assay (P = 0.039). Examination of the C. trachomatis FVU testing profiles for 15 women with discordant results (Table 4) showed one patient to be FVU positive and SCVS negative in all assays (1.9% of the 53 C. trachomatis-positive patients). Fourteen vaginal PIS-positive patients had variable FVU results across the four assays, and four of them were FVU negative in all assays (7.5% of the C. trachomatis-positive patients). Performance calculations for FVU testing without the four patients whose urine samples were negative in all tests resulted in the following sensitivity values: 95.9% for the AC2 assay on the Tigris system and 95.7% on the Panther system, 83.3% for the RealTime CT/NG assay, 81.6% for the ProbeTec CT/GC Qx assay, and 87.8% for the cobas CT/NG assay. All of the assays demonstrated specificities above 98%. Occasionally single-sample, single-test, false-positive results (specificity) were seen in the total study (Tables 2 and 3).
Sensitivities, specificities, and positive and negative predictive values of the AC2 assay on the Tigris and Panther systems, the RealTime CT/NG assay, the ProbeTec CT/GC Qx assay, and the cobas CT/NG assay for FVU specimens
C. trachomatis FVU testing profiles for 15 women with discordant results
The N. gonorrhoeae prevalence was 2% (11/575 patients). For SCVS specimens, N. gonorrhoeae sensitivities were 90.9% for the AC2 assay on the Tigris system and 100% on the Panther system, 70.0% for the RealTime CT/NG assay, 100% for the ProbeTec CT/GC Qx assay, and 63.6% for the cobas CT/NG assay. For FVU samples, N. gonorrhoeae sensitivities were 72.7% for the AC2 assay on the Tigris system and 66.7% on the Panther system, 70.0% for the RealTime CT/NG assay, 80.0% for the ProbeTec CT/GC Qx assay, and 70.0% for the cobas CT/NG assay. All specificities were above 99.5%. The sample size for N. gonorrhoeae infection was too small for statistical comparisons.
DISCUSSION
When clinical sensitivity and specificity calculations were made according to FDA guidelines, SCVS testing identified more C. trachomatis-positive patients than did FVU testing with all four second-generation assays (range, 3% to 21%). Similar observations for these two specimen types with the individual assays have been made in several other studies (3, 7, 9, 12). In the current study, the AC2 assay performed on the Tigris or Panther system identified significantly more C. trachomatis-infected women (98.1% to 96.2%) than did the cobas CT/NG test (84.6%; P = 0.016). The values in our study for cobas CT/NG testing of SCVS specimens are considerably lower than those reported by Van Der Pol et al. (9), who showed 93.5% of the positive samples being detected by cobas CT/NG testing. That study did not test SCVS specimens in the reference assays; therefore, a true head-to-head comparison of the new assays with SCVS specimens was not possible. We have conducted an internal investigation of the cobas CT/NG assay-negative SCVS samples from C. trachomatis-positive patients. The assay runs that contained those samples exhibited normal positive- and negative-control values, and none of the patient samples spiked with C. trachomatis showed inhibitors. The lot numbers of the reagents used in those runs have not been identified as problematic. Examination of the order of collection of SCVSs for eight false-negative results in cobas CT/NG testing showed that three were collected first, one was second, four were third, and none was fourth. Splitting of samples to enable spiking for inhibitors might have had a slight dilution effect on sensitivity for samples already near a test's cutoff value, but examination of test results in relation to assay cutoff values for the false-negative samples did not confirm this hypothesis.
In the multicenter clinical trial reported by Gaydos et al. (10), the RealTime CT/NG assay on the m2000 system was compared with the AC2 assay and the first-generation ProbeTec ET test with SCVS and clinician-collected vaginal swab samples and showed sensitivities similar to those of the AC2 assay for C. trachomatis. Comparison of the AC2 assay and the RealTime CT/NG assay on the m2000 system with SCVS specimens in the current study confirms those observations (Table 2).
In C. trachomatis clinical evaluations of the ProbeTec CT Qx assay on the Viper system with XTR technology, Taylor et al. (11) compared its performance with that of the AC2 assay and the first-generation ProbeTec ET test and showed good sensitivity (96.5%) with SCVS specimens. In the current study, the sensitivity of the ProbeTec CT/GC Qx assay on the Viper system was lower (90.6%) but was not statistically different from values for the other assays. The C. trachomatis FVU sample sensitivity rates from the cobas evaluations (8) were 92.3% for the cobas test, 95.8% for the AC2 assay, and 94.8% for the ProbeTec CT/GC Qx assay, in comparison with 81.1%, 88.7%, and 75.5%, respectively, in the current study (Table 3), but the specificities for all assays were above 99.4%.
AC2 testing performed with FVU specimens on the Tigris or Panther system in the current study identified significantly more C. trachomatis-infected patients than did the ProbeTec CT/GC Qx assay or the RealTime CT/NG assay. In the ProbeTec CT/GC Qx assay clinical trial publication (11), the sensitivities reported for FVU specimens were 93.0% for the ProbeTec CT/GC Qx assay, 96.5% for the AC2 assay, and 86.7% for the first-generation ProbeTec ET assay. In the RealTime CT/NG assay trials (10), the sensitivities for FVU specimens ranged from 92.6% to 95.7% for the RealTime CT/NG assay, which was much higher than 76.9% in the current study (Table 3).
Cheng et al. (14) compared the RealTime CT/NG assay and the first-generation Amplicor CT/NG assay, showing very high levels of agreement for FVU specimens. Similarly, Rockett et al. (15) showed that the cobas CT/NG assay for C. trachomatis detected 94.5% of the FVU specimens found to be positive by the first-generation Amplicor test. These studies (14, 15) suggest that the two newer second-generation assays are performing comparably to the first-generation tests with FVU samples.
Mushanski et al. (16) conducted a head-to-head comparison of the BD ProbeTec CT/GC Qx assay on the Viper system with the AC2 assay on the Tigris system for C. trachomatis with FVU samples and found strong agreements of positive and negative results. Analysis of the discordant samples showed that all of the AC2 assay-positive samples were confirmed as positive by the Aptima C. trachomatis alternate primers assay, whereas only one of the four extra ProbeTec CT/GC Qx assay-positive samples tested positive in repeat testing, which suggests that three of the specimens might have been false-positive findings originally. If these confirmatory results had been taken into consideration for calculation of sensitivity, then the ProbeTec CT/GC Qx assay would have had lower sensitivity with FVU specimens and the findings would agree more closely with findings in the current study.
Differences seen in the current head-to-head study for detection of positive findings among SCVS and FVU samples, in comparison with previously reported clinical trials, may be due to several factors, including (i) failure to include vaginal swabs for the reference tests in the calculated comparisons for vaginal swabs for the new test, (ii) differences in methods used for PIS calculations, and (iii) the use of first-generation assays for PIS analyses. These problems can be compounded if multiple specimen types are used for PIS analyses, because female C. trachomatis infections do not always yield analytical material from all sample sites. In the present study, four women were clearly infected with C. trachomatis by SCVS testing but their FVU samples were negative in all assays (Table 4). Calculations without the results for these four patients provide a clearer understanding of the assay sensitivities for FVU specimens (95.9% for the AC2 assay on the Tigris system and 95.7% on the Panther system, 83.3% for the RealTime CT/NG assay, 81.6% for the ProbeTec CT/GC Qx assay, and 87.8% for the cobas CT/NG assay), in comparison with calculations made following FDA PIS guidelines (88.7%, 88.0%, 76.9%, 75.5%, and 81.1%, respectively) (Table 3).
A weakness of our study is the limited number of patients infected with N. gonorrhoeae, which prevented meaningful calculations to compare the performance of the four assays with the two specimen types. Similar to C. trachomatis data, SCVS testing identified more N. gonorrhoeae infections than did FVU testing. Four of the 11 patients were dually infected with C. trachomatis.
A previous study comparing the AC2 assay with first-generation ProbeTec and Amplicor tests showed considerable inhibition rates of 27.2% for FVU samples with the ProbeTec assay and 10.4% for SCVS specimens and 12.1% for FVU specimens with the Amplicor assay (12). The present study showed that inhibitors of C. trachomatis amplification in SCVS and FVU specimens are not playing a role in any of these second-generation assays, suggesting that internal inhibitor controls are unnecessary. Internal controls for specimen adequacy may be useful but may create a dilemma for clinicians regarding repeat sample collection from patients. Endpoint-of-detection experiments in the earlier study (12) showed that the AC2 assay detected C. trachomatis to a dilution of 10−8 or 1.0 to 0.1 elementary bodies (EBs) per 100 μl, in contrast to the first-generation cobas Amplicor and ProbeTec ET tests, which detected 100 EBs per 100 μl. The current study found the values to be 1 EB per 100 μl for the AC2 assay and 10 to 100 EBs per 100 μl for the cobas CT/NG, ProbeTec CT/GC Qx, and RealTime CT/NG assays. Cheng et al. (14) showed that the RealTime CT/NG assay was 4 times more sensitive analytically than the first-generation cobas Amplicor test. These studies suggest that differences in analytical sensitivity may translate into differences in clinical sensitivity. The differences in clinical sensitivities between C. trachomatis rRNA detection with the AC2 assay and detection of cryptic plasmid DNA with PCR or strand-displacement assays are more likely due to differences in the endpoints of detection of the assays (analytical sensitivity), as shown in the dilution series, and are expressed clinically by differences in rRNA versus DNA amounts in the clinical samples. Michel et al. (17) showed significant differences of 733 C. trachomatis EBs/100 μl for SCVS specimens versus 47 EBs/100 μl for FVU samples, which may explain the differences seen in the present study between SCVS and FVU samples for the DNA tests.
In conclusion, the high specificities observed for the four assays are important to ensure minimal reporting of false-positive results. Inhibitors of nucleic acid amplification did not account for differences in the comparisons of sensitivities for SCVS and FVU specimens. The clinical sensitivity differences seen in this head-to-head comparison study are more likely determined by analytical sensitivities and the levels of C. trachomatis target analytes in clinical samples.
ACKNOWLEDGMENTS
This study was supported by a research grant from Hologic Gen-Probe Inc. (San Diego, CA).
Hologic Gen-Probe had no input in the study design, choice of instruments studied, data analysis, or preparation of the report.
We thank the following individuals for meticulously reviewing assay processing steps, running assays, and providing support and cooperation throughout the study: Christine Peterson, Caroline Marchand, and Jocelyn Blouin, Centre de Santé et de Services Sociaux de Trois-Rivières, Trois-Rivières, Quebec, Canada; Elizabeth Oates and Sandy Tucker, Public Health Laboratory, St. John's, Newfoundland and Labrador, Canada; and Catherine Roberts, Gale Naugle, and Joan MacDonald, Queen Elizabeth II Health Sciences Centre, Halifax, Nova Scotia, Canada. We also thank Laura Gilbert and Sandra Pike, Public Health Laboratory, St. John's, Newfoundland and Labrador, Canada, for assistance with preparation of the manuscript.
S.R. has received research funding and speaker honoraria from and has consulted for Hologic Gen-Probe. M.C. has received research funding and speaker honoraria from Hologic Gen-Probe and has served as a consultant for and received research funding from Abbott Molecular.
FOOTNOTES
- Received 30 December 2013.
- Returned for modification 31 January 2014.
- Accepted 27 March 2014.
- Accepted manuscript posted online 2 April 2014.
- Copyright © 2014, American Society for Microbiology. All Rights Reserved.
The authors have paid a fee to allow immediate free access to this article.
