ABSTRACT
Screening for chlamydial and gonococcal infection has been strongly recommended for all sexually active women under the age of 26. Advances in the ability to detect infection by nucleic acid detection techniques have improved access to screening methods in routine clinical practices. To meet the increasing demand for testing, a high-throughput system is desirable. We evaluated the performance of the Hybrid Capture 2 CT/GC (HC2) assay with the Digene Rapid Capture System (HC2-RCS). The results of HC2-RCS for endocervical samples from 330 women were compared to those of culture and the COBAS Amplicor PCR. For detection of chlamydial infection, HC2-RCS had a sensitivity and a specificity similar to those of PCR (P > 0.5) and an improved sensitivity compared to that of culture (P = 0.007). For identification of gonococcal infections, all assays performed similarly (P > 0.5). The performance of HC2-RCS was also compared to that of the manual HC2 format (HC2-M) with these samples and with 911 endocervical samples collected previously. The performance of HC2-RCS was equivalent to that of HC2-M; the overall concordance rates for the detection of chlamydia and gonorrhea were 99.7% (κ = 0.97) and 99.8% (κ = 0.97), respectively. When the HC2 assay was performed with a semiautomated system application designed for high throughput, it demonstrated high sensitivity and a high specificity for detection of both Chlamydia trachomatis and Neisseria gonorrhoeae.
An estimated 3 million chlamydial infections and over 600,000 gonococcal infections occur annually in the United States (1, 7, 8, 11). Both Chlamydia trachomatis and Neisseria gonorrhoeae frequently cause asymptomatic infections in women that may result in undetected and, therefore, untreated disease. Untreated chlamydial and gonococcal infections have both been implicated in long-term sequelae which include pelvic inflammatory disease, salpingitis, ectopic pregnancy, adverse outcome of pregnancy, and neonatal infection. The health care burden of the consequences of untreated infections with either of these organisms is estimated to be $2.7 billion annually (14, 24, 32).
The Centers for Disease Control and Prevention has recommended that all sexually active women under the age of 26 be screened for C. trachomatis infection (9), and the European guidelines for the management of chlamydial and gonococcal infections recommend similar targeted screening for both of these organisms (4, 27). Beginning in the year 2000, the Health Employer Data Information Set, a system used nationwide to assess the performance of managed care health services in the United States, adopted this screening recommendation as a criterion measure for evaluation of the performance of insurance programs (19). As a result of these recommendations and of studies demonstrating the cost-effectiveness of chlamydial screening programs (19, 21), increases in the number of patients screened have been seen throughout the country (5, 23).
Universal screening practices are now more feasible since highly sensitive commercially available nucleic acid amplification tests allow transport of specimens from clinical settings to remote laboratories. Although these assays have demonstrated excellent sensitivities and specificities in clinical trials (6, 28, 29, 30), they require extreme technical precision in order to achieve maximum performance (18, 22, 28). With the assays available at present, this translates into labor-intensive diagnostic methods, often with low-throughput capacity (2, 3, 16). In this study we evaluated a semiautomated system application of the signal amplification-based Hybrid Capture 2 CT/GC (HC2) assay that allows testing of 352 samples for both C. trachomatis and N. gonorrhoeae during an 8-h shift.
The Rapid Capture System (RCS) is a programmable 96-well microplate processor that integrates liquid handling, plate handling, incubation, shaking, and washing via software specifically designed to run the HC2 assay (Digene Corporation, Gaithersburg, Md.). We evaluated the performance of the HC2 assay with the RCS application (HC2-RCS) using prospectively collected endocervical samples tested by the HC2 assay performed manually (HC2-M), the COBAS Amplicor (Roche Diagnostic Corporation, Indianapolis, Ind.) CT/NG assay (PCR), and culture. Here we present the first evaluation of the performance characteristics of HC2-RCS for detection of C. trachomatis and N. gonorrhoeae.
MATERIALS AND METHODS
Study populations and samples.Endocervical samples used for evaluation of HC2-RCS were collected from two distinct populations. Consenting women attending a sexually transmitted disease (STD) clinic in Indianapolis, Ind., were enrolled in a prospective evaluation of HC2-RCS. Three endocervical samples were collected. The first was collected with a Dacron swab and plated on modified Thayer-Martin agar for N. gonorrhoeae culture. The other two samples were collected with the Digene cervical sampler (25), a brush used for sample collection for the Digene Hybrid Capture II HPV DNA assay, and a Dacron swab for C. trachomatis culture. Specimens for HC2 assays were placed in Digene sample transport medium (STM), while those for C. trachomatis culture were placed in chlamydia transport medium (CTM). The CTM used in this study (an in-house preparation of sucrose, phosphate, and glutamic acid [15]) is a sample collection medium approved by the Food and Drug Administration for use with the COBAS Amplicor CT/NG assay (22, 29). The order of sample collection for placement in STM and CTM was random; the sample collection order was not controlled but was recorded. Samples placed in STM first were collected from 45.5% of study subjects. Samples for N. gonorrhoeae culture were immediately placed in a CO2 environment, while samples placed in CTM for C. trachomatis culture were refrigerated prior to transport to the laboratory. All samples in STM were stored at −20°C or below prior to testing.
The second population consisted of consenting women attending an obstetrics and gynecology clinic in Porto Alegre, Brazil, who were enrolled in a study to evaluate the Digene Hybrid Capture II HPV DNA assay. Diagnostic testing of a separate specimen for STDs was included in the design of the original study for which these samples were collected. The results of tests performed at Digene for human papillomavirus and C. trachomatis-N. gonorrhoeae were reported to the clinic in Brazil if the specimen had been kept under stable conditions for testing (at ambient temperature for up to 2 weeks and at −20°C for up to 3 months). Archived specimens from Brazil tested at Indiana University were stored at ambient temperature for 5 to 8 days, shipped at ambient temperature to the United States, and subsequently stored at −20°C for 3 to 6 months. The only identifiers on these samples were specimen numbers, and testing at Indiana University was approved by the Indiana University Institutional Review Board.
HC2-RCS and HC2-M.The HC2 assay is performed as a combined test for C. trachomatis and N. gonorrhoeae (CT/GC assay), followed by identification of the infecting pathogen: C. trachomatis (CT-ID assay) and N. gonorrhoeae (GC-ID assay). The HC2 assays detect DNA-RNA hybrids with signal amplification technology. Therefore, these assays are not affected by DNA amplification inhibitors and do not include a DNA amplification control. The CT/GC, CT-ID, and GC-ID assays follow a single testing protocol with assay-specific reagents. The initial denaturation step is performed manually for both HC2-RCS and HC2-M. Samples were processed in batches of up to 96 by addition of a denaturation reagent and incubation at 65°C for 45 min. After denaturation, samples were either tested immediately or held at −20°C for subsequent testing.
HC2-M was performed as described previously (10) with endocervical samples, the only specimen type approved in the United States for use in the HC2 assay for testing for C. trachomatis and N. gonorrhoeae. For HC2-RCS, denatured samples were loaded onto the RCS, in which all assay steps prior to signal detection were performed automatically by the RCS application. Each run of the RCS has the capacity to test up to four 96-well plates, consisting of 352 samples (88 per plate) and 32 controls (8 per plate), but has a flexible format that allows any number of specimens from 1 to 352 to be run. To maximize the work flow with the RCS application, testing was done with batches of 352 samples whenever possible. The total run time for the HC2 assay with the RCS application is approximately 7.5 h, with 3.5 h of uninterrupted hands-free time.
Test results were calculated by using assay-specific software accompanying the DML 2000 luminometer. Relative light units (RLUs) for the positive and negative controls are used to calculate the run-specific cutoff (CO), and specimen results are reported as RLU/CO ratios. In clinical testing settings, samples that yield an RLU/CO value of ≥1.0 by the CT/GC assay would routinely be retested by the CT-ID and GC-ID assays to identify the infecting organism. In these studies, both the CT-ID and GC-ID assays were performed regardless of the CT/GC assay result, allowing interpretation of results by one of the two methods described below. RLU/CO values between 1.0 and 2.5 by the CT-ID or GC-ID assay were considered equivocal, and the samples were retested in duplicate. These assay results were interpreted by using an equivocal-zone retesting algorithm and were considered positive if the results of at least two of the three tests (the initial test and the two retests) had RLU/CO values ≥1.0. Denatured samples in STM contain sufficient volume for 15 to 18 assays, which allows retesting to be performed, if necessary.
The performance of HC2-RCS was evaluated by using two separate analyses with different interpretive criteria based on the results of the different tests. The first analysis used results based on the outcome of the CT/GC assay followed by the identification assays (CT/GC algorithm), while the second analysis was based on the results of the individual identification assays, including those obtained with the equivocal-zone retesting algorithm, for specimens interpreted as equivocal after initial testing. Therefore, in one analysis, if the CT/GC assay result was negative, the identification assay results were not considered. In the alternate analysis, the results of the CT/GC assay were not considered for any samples; only the identification assay results, including equivocal zone resolution, were compared.
C. trachomatis culture.Culture for C. trachomatis was performed as described previously (15). McCoy cell monolayers in 96-well plates were inoculated with samples within 24 h of collection. Inoculated plates were fed a growth medium consisting of Eagle's minimal essential medium supplemented with 10% fetal calf serum, 4.4% glucose, and antibiotics. The plates were centrifuged for 1 h at 1,750 × g and were then incubated at 37°C for 48 to 72 h prior to staining. Cultures were stained with a fluorescein-labeled monoclonal antibody to C. trachomatis prepared in-house (15).
N. gonorrhoeae culture.Culture for N. gonorrhoeae was performed with modified Thayer-Martin agar in a 5% CO2 environment at 37°C. Cultures were assessed at 24 and 48 h for growth. The presence of N. gonorrhoeae isolates was confirmed if the colonies consisted of gram-negative diplococci, reacted to oxidase reagent, and stained positively with a fluorescent monoclonal antibody (Microtrak; Trinity Biotech, Bray, Ireland) specific for N. gonorrhoeae (17).
PCR.Specimens in CTM were stored at −70°C immediately following inoculation for culture. PCR was performed with a portion of the sample remaining after C. trachomatis culture. The instructions in the COBAS Amplicor CT/NG assay package insert were followed for all PCR tests. One hundred microliters of CTM was added to 100 μl of lysis buffer. After 10 min at room temperature, 200 μl of specimen diluent was added to the sample. Samples processed in this manner were then added to the master mixture reagent that contains enzymes and deoxynucleoside triphosphates in a buffer optimized for amplification. The samples were amplified and the DNA products were detected with the COBAS semiautomated system. Amplified products that bound to organism-specific oligonucleotides on magnetic beads were detected by using a biotin-streptavidin-mediated fluorescent readout. The internal control assay, which detects the amplification of a sequence of DNA included in the master mixture reagent, was used to rule out the possibility of inhibition of amplification. Interpretation of all PCR results was performed according to the algorithms in the manufacturer's package insert.
Statistical analyses.The performance of HC2-RCS compared to that of HC2-M was analyzed by using the kappa statistic (κ). This measure of agreement in categorical analyses is generally interpreted as follows: <0.20, poor agreement; 0.21 to 0.40, fair agreement; 0.41 to 0.60, moderate agreement; 0.61 to 0.80, good agreement; and >0.80 very good agreement (26). The infection status of patients from whom samples had been collected and placed in a repository could not be determined due to a lack of alternate samples for comparator testing. Therefore, the agreement between the two tests was the only measure of the performance of HC2-RCS calculated for these samples.
Since HC2-M has been cleared by the Food and Drug Administration for detection of these organisms from endocervical specimens and the objective of these studies was to evaluate the performance of HC2-RCS, HC2-M served as a component of the reference standard for the identification of infections in the prospective study. Patients were defined as being infected if they had a positive culture result or both a positive PCR result and a positive HC2-M result. Specimens positive only by HC2-M or only by PCR were considered false positive. McNemar's χ2 test was used to compare the performance of HC2-RCS to those of both culture and PCR (α = 0.05).
RESULTS
HC2-RCS compared to HC2-M.Endocervical samples were collected from 354 women for prospective comparison of diagnostic assays. However, since this would constitute little more than one assay run on the RCS, additional samples from a repository were used in order to more fully evaluate the comparability of the manual and automated portions of the assay. Stored endocervical samples were available from 1,000 women. Eighty-eight samples were excluded from the analysis because of potential cross-contamination because a specimen rack was dropped. One additional sample lacked sufficient volume for testing. The CT/GC assay was performed with the remaining 911 repository samples. Positive results were obtained by HC2-RCS, HC2-M, or both assays for 62 (17.5%) and 32 (3.5%) of prospective study specimens and repository samples, respectively. When the two populations were considered separately, despite the significant difference in the prevalence of infection (P < 0.0001), the agreement between the test formats was similar. The agreement was greater than 99% for all assays for both the prospective study specimens and the repository specimens, and κ scores ranged from 0.965 to 0.983 (Table 1). The agreement between the HC2-RCS and HC2-M formats was observed not only for interpreted results but also for the RLU/CO values observed for individual samples (Fig. 1).
(a) Comparison of initial results of HC2-RCS and HC2-M; (b) results of HC2-M and HC2-RCS for C. trachomatis; (c) HC2-M and HC2-RCS results for N. gonorrhoeae.
Comparison of HC2-RCS and HC2-M
HC2-RCS compared to culture and PCR for C. trachomatis detection.Although HC2-M was used to define infected patients in the prospective study, due to the high level of agreement between the two formats, only the results obtained by HC2-RCS are presented here. The HC2 CT/GC and CT-ID assays and culture for C. trachomatis were performed with 354 samples. Remaining CTM with samples from 333 subjects was available for PCR. The PCR results for three samples, from patients culture negative for C. trachomatis and N. gonorrhoeae, were uninterpretable due to failure of the internal control to amplify the target DNA, thus indicating the presence of PCR inhibitors. All three samples were negative by the CT-ID assay, while one was positive by the GC-ID assay on both platforms. It is unclear whether this was a false-positive result by the HC2 assay or whether the result would have been confirmed by PCR. Since a positive PCR result was required as a part of the definition of infection, the number of subjects from whom samples were available for comparison of all assays was 330.
Forty-five (13.6%) subjects were found to have C. trachomatis infection, which was defined by either a positive culture result or positive results by both PCR and the HC2-M CT-ID assay (Table 2). Had a positive HC2-M result been excluded from the definition of infection, two additional patients, positive by PCR alone, would have been defined as being infected with C. trachomatis. When the CT-ID assay alone was used, HC2-RCS yielded positive results for 43 of 45 (95.6%) samples. When the results for the specimens were interpreted on the basis of the initial result of the CT/GC assay and verification of positive results by the CT-ID assay, 42 of 45 (93.3%) samples were positive. The decrease in sensitivity was not significant (P > 0.5) when the CT/GC assay and identification verification algorithm were used instead of the CT-ID assay alone. There were no false-positive results by HC2-RCS. The use of the CT/GC assay did not change the interpretation of the results for any of the negative subjects. The sensitivity and specificity of the HC2-RCS CT-ID assay were equivalent to those of PCR (P > 0.5), and the sensitivity was significantly (P = 0.007) better than that of culture (Table 2).
Performance characteristics for detection of chlamydial infections
HC2-RCS compared to culture and PCR for N. gonorrhoeae detection.The same 330 samples were available for comparison of the HC2 GC-ID assay and PCR for detection of N. gonorrhoeae. N. gonorrhoeae infections, defined by a positive culture or positive results by both PCR and the GC-ID assay of HC2-M, were identified in 21 (6.4%) of the study subjects (Table 3). Had HC2-M not been used to define infected patients, three additional samples positive only by PCR would have been included in the infected group. All 21 samples from infected patients were identified as positive by the GC-ID assay. Three samples gave false-positive results by HC2-RCS (specificity, 99.0%). Use of the CT/GC assay did affect the specificity of the final result for N. gonorrhoeae by the HC2 assay. One sample with a false-positive result by the GC-ID assay had been negative by the CT/GC assay and therefore would not have been tested by the GC-ID assay and would have been interpreted as negative. Although the specificity was increased to 99.4% by use of the screening assay, this was not significant (P > 0.5). The overall performance of the GC-ID assay was similar to that of culture or PCR for this population (P = 0.125 and P > 0.5, respectively) (Table 3).
Performance characteristics for detection of gonococcal infections
Resolution of equivocal results.In the prospective study, the results for three (0.9%) specimens were interpreted as equivocal on the basis of the CT-ID or GC-ID assay results. Of the three specimens, one had an equivocal result for C. trachomatis that was resolved as positive and that was confirmed to be positive by PCR, although the specimen was culture negative. For the two specimens with equivocal results for N. gonorrhoeae, the result for one was resolved as positive after retesting, and the specimen was positive by culture and PCR. The result for the second specimen was resolved as positive, although it was negative by both culture and PCR and the result was classified as false positive.
Samples excluded from analysis.In the subset of 24 specimens that were not included in the analysis described above due to unavailable PCR results, 1 (4.2%) was positive for chlamydia by culture, while one other sample was positive for N. gonorrhoeae by culture. The prevalence of infection in this subset of samples excluded from the analysis is not statistically different from that in the remaining set of samples for either C. trachomatis or N. gonorrhoeae (P > 0.1 and P > 0.5, respectively). Each of the culture-positive samples was positive by the identification assays. All of the samples culture negative for N. gonorrhoeae were also negative by the GC-ID assay. Two samples culture negative for chlamydia were positive by the CT-ID assay, and the remaining 21 were negative by HC2-RCS. DNA amplification was inhibited for three of the samples (<1%) excluded from the analysis. Since the HC2 assay is a signal amplification assay, it does not include a control for DNA amplification inhibition. All of the samples that were inhibitory for PCR were culture negative for C. trachomatis and N. gonorrhoeae.
DISCUSSION
Verification that a new platform does not negatively affect the performance of an assay is an essential component of the validation process. Therefore, we compared the performance of HC2-RCS to that of HC2-M with samples from two populations with very different prevalences of sexually transmitted infections (3.5 and 17.5% [P < 0.001]). The HC2 CT/GC, CT-ID, and GC-ID assays all showed excellent agreement: above 99% between the two formats for all assays (κ ≥ 0.95 for all comparisons). This level of reproducibility is evidence of a highly robust test method for both C. trachomatis and N. gonorrhoeae.
The proportion of samples that yielded results in the equivocal zone was low for both the high- and low-prevalence populations: 0.91 and 0.11%, respectively (data not shown). Retesting of these specimens in duplicate by the HC2 assay resulted in final interpretations for all samples and provided results in agreement with that of either culture or PCR for two of three samples. Resolution of equivocal results by retesting by either the HC2 assay or another test method may improve the specificity and reproducibility around the cutoff of the HC2 tests. Since only three samples had equivocal results by HC2-RCS in this study, further evaluation of the impact of retesting is necessary.
A limitation of the prospective study was the difficulty associated with defining infected patients. All evaluations of assays with potentially improved sensitivities compared to that of the “gold standard” assay face this issue. The bias inherent in discrepant analysis has been described mathematically (13; W. C. Miller, Editorial Response, Clin. Infect. Dis. 27:1186-1193, 1998). For this reason, discrepant analysis was not a part of the prospective study design. Instead, two reference tests, culture and PCR, were performed on all samples, and the results were included in the analysis of sensitivity and specificity. However, although we arbitrarily assigned a specificity of 100% to culture, this cannot be done for PCR. Therefore, we felt that the positive PCR results must be confirmed by another highly sensitive nucleic acid detection test in order to define infected patients with negative cultures. Darwin et al. (10) reported the relative sensitivities of HC2-M for the identification of C. trachomatis and N. gonorrhoeae to be 97.2 and 92.2%, respectively, compared to culture with discrepant resolution. The specificities were 99.2 and 99.8%, respectively, for identification of C. trachomatis and N. gonorrhoeae. Girdner et al. (12) also showed that HC2-M has 95.4% sensitivity and 99% specificity for the detection of C. trachomatis compared to the results of culture or direct fluorescent-antibody assay and PCR. On the basis of these performance parameters, we decided to use the results obtained from the identification assays of HC2-M to verify infection for culture-negative, PCR-positive patients. A positive HC2 assay result alone was not sufficient to define an infection. The specificity of HC2-RCS reported here was not affected by inclusion of a positive HC2-M result in the definition since samples positive by HC2-M alone were considered to be false positive in this analysis. Had we defined infection as a positive result by either culture or PCR alone, the performance estimates of the HC2 identification assays would not have changed significantly. The sensitivity of the CT-ID assay would have dropped from 93.3 to 91.5% (P > 0.2), and the sensitivity of the GC-ID assay would have dropped from 100 to 87.5% (P > 0.1); the specificity of either test would not have changed. However, exclusion of a positive HC2-M result from the definition of infection would have artificially set the specificity of PCR to 100%. The specificity measured by using the interpretation defined for this study resulted in PCR specificities of 99.3 and 99.0% for C. trachomatis and N. gonorrhoeae, respectively. These values are well within the range reported for this assay (12, 18, 22, 29, 30).
Another limitation of the data obtained by the HC2 assays is the storage and handling of the samples that was necessary for the multiple tests by both HC2-RCS and HC2-M. Samples from the repository were stored at −20°C for times longer than the limit recommended in the package insert. However, since this affected the sample itself, the effect should have been equivalent for both assay platforms. Since sensitivity and specificity calculations were not performed with data for these samples, there is no measurable impact as a result of extended storage. With the exception of samples with results in the equivocal zone, the samples used in the prospective study underwent three freeze-thaw cycles, the maximum allowable according to the package insert. Those with initial equivocal results underwent an additional thaw. However, this would be expected to reduce the sensitivities of the assays. Therefore, any impact from multiple freeze-thaw cycles would cause the data presented here to be a conservative estimate of the true performance of the HC2 assays.
For laboratories that test large volumes of samples for C. trachomatis and N. gonorrhoeae, the performances of the CT/GC assay and the subsequent organism-specific identification assays are of particular interest. The ability to perform a sensitive assay that will provide final results for the vast majority of specimens is highly desirable, especially for low-prevalence populations. In a population with an overall prevalence of infection with C. trachomatis and N. gonorrhoeae combined of 4%, as few as 15 of 352 samples would require testing by the CT-ID and GC-ID assays. In a population with a prevalence of nearly 18%, like that seen in the STD clinic population, as few as 62 of 352 samples would require additional testing. Since use of the CT/GC assay did not significantly affect the interpretation of results, each laboratory would have to evaluate the utility of screening by the CT/GC assay with confirmation by the organism-specific identification assays versus the use of the CT-ID and GC-ID assays as stand-alone tests. The HC2 assay system will allow laboratories the flexibility to choose the most economical testing scheme for the populations that they serve.
Due to the number of controls per plate, the HC2 assay may not be cost-effective in laboratories that perform less than 88 tests per run. However, this is similar to the number of controls required for other automated or semiautomated assays for C. trachomatis and N. gonorrhoeae combined. PCR by the COBAS Amplicor assay requires 8 controls for every 88 patient samples tested, the LCx assay (Abbott Diagnostics, Abbott Park, Ill.) requires 24 controls for every 72 samples, and the BD ProbeTec assay (Becton Dickinson Biosciences, Sparks, Md.) requires 6 controls for every 90 samples.
One RCS System instrument can test 352 samples plus 32 controls during a single work shift. During this testing procedure, 3.5 h of continuous hands-free time is available for technicians to perform other tasks. The availability of a high-throughput assay with excellent performance characteristics such as HC2-RCS is obviously desirable for large clinical laboratories and may influence programmatic decisions related to screening in less traditional settings. Although many studies have reported the cost-effectiveness of screening (5, 14, 19, 23), especially for chlamydial infections, these results are often based on targeted screening of high-risk populations. Use of the recommendations in both the United States and Europe and screening of women solely on the basis of age as a risk factor have not been found to be cost-effective when the lower-throughput systems available at present are used (20, 31; K. H. Webb, Letter to the Editor, Sex. Transm. Dis. 25:403-405, 1998). These analyses are strongly affected by the specificities of molecular assays. Increasing the specificity from 99.0 to 99.9% in a population with a 2% prevalence of infection reduced the cost per major outcome averted by up to 40% in one study (31). The impact on cost-benefit ratios of an assay that has a high sensitivity and a high specificity and that is designed to be compatible with high-throughput automation needs to be evaluated in future studies.
FOOTNOTES
- Received 4 February 2002.
- Returned for modification 11 April 2002.
- Accepted 2 July 2002.
- Copyright © 2002 American Society for Microbiology
