Evaluation of the Clinical Performance of the HPV-Risk Assay Using the VALGENT-3 Panel

ABSTRACT Human papillomavirus (HPV) testing is increasingly being incorporated into cervical cancer screening. The Validation of HPV Genotyping Tests (VALGENT) is a framework designed to evaluate the clinical performance of various HPV tests relative to that of the validated and accepted comparator test in a formalized and uniform manner. The aim of this study was to evaluate the clinical performance of the HPV-Risk assay with samples from the VALGENT-3 panel and to compare its performance to that of the clinically validated Hybrid Capture 2 assay (HC2). The VALGENT-3 panel comprises 1,300 consecutive samples from women participating in routine cervical cancer screening and is enriched with 300 samples from women with abnormal cytology. DNA was extracted from original ThinPrep PreservCyt medium aliquots, and HPV testing was performed using the HPV-Risk assay by investigators blind to the clinical data. HPV prevalence was analyzed, and the clinical performance of the HPV-Risk assay for the detection of cervical intraepithelial neoplasia grade 3 or worse (CIN3+) and CIN2 or worse (CIN2+) relative to the performance of HC2 was assessed. The sensitivity of the HPV-Risk assay for the detection of CIN3+ was similar to that of HC2 (relative sensitivity, 1.00; 95% confidence interval [CI], 0.95 to 1.05; P = 1.000), but the specificity of the HPV-Risk assay was significantly higher than that of HC2 (relative specificity, 1.02; 95% CI, 1.01 to 1.04; P < 0.001). For the detection of CIN2+, similar results were obtained, with the relative sensitivity being 0.98 (95% CI, 0.93 to 1.02; P = 0.257) and the relative specificity being 1.02 (95% CI, 1.01 to 1.03; P < 0.001). The performance of the HPV-Risk assay for the detection of CIN3+ and CIN2+ was noninferior to that of HC2, with all P values being ≤0.006. In conclusion, the HPV-Risk assay demonstrated noninferiority to the clinically validated HC2 by the use of samples from the VALGENT-3 panel for test validation and comparison.

problematic due to the difficulties with obtaining an appropriate set of clinical specimens. The Validation of HPV Genotyping Tests (VALGENT) framework facilitates the comparison and validation of HPV tests by providing a set of samples obtained from women attending routine screening enriched with cytologically abnormal samples (5)(6)(7)(8)(9). In order to allow comparison with other HPV tests, each VALGENT panel includes a comparator assay that is clinically validated for cervical cancer screening purposes.
The HPV-Risk assay is a real-time PCR-based assay that targets the E7 region of 15 (probable) high-risk HPV types and enables partial genotyping for HPV16 and HPV18. A previous evaluation of the HPV-Risk assay relative to the clinically validated comparator GP5ϩ/6ϩ-PCR showed that its clinical performance and reproducibility met the international validation criteria for HPV tests for use in cervical cancer screening (10).
This study was performed to evaluate the clinical performance of the HPV-Risk assay for detection of high-grade CIN using the VALGENT-3 panel and to compare its performance to that of the clinically validated Hybrid Capture 2 assay (HC2) as a comparator test (4,11).

RESULTS
HPV-Risk assay results. Of the 1,600 samples tested by the HPV-Risk assay, 30 (1.9%) samples were considered invalid for PCR amplification and were excluded from further analysis. HPV-Risk assay and HC2 results are shown in Table 1. Of the 1,570 (98.1%) valid results, 1,272 were from the screening population (including 9 CIN grade 2 [CIN2] and 11 CIN grade 3 or worse [CIN3ϩ] cases) and 298 were from the enrichment population (including 36 CIN2 and 71 CIN3ϩ cases). Of 1,272 women in the screening population with valid HPV-Risk assay results, 133 (10.5%) women tested HPV positive with the HPV-Risk assay. The prevalence of HPV16 and HPV18 was 2.6% (33/1,272) and 0.9% (12/1,272), respectively. Of 298 women in the enrichment population with valid HPV-Risk assay results, 205 (68.8%) women tested HPV positive. The prevalence of HPV16 and HPV18 was 27.2% (81/298) and 7.4% (22/298), respectively. For comparison, 159 (12.5%) women in the screening population and 214 (71.8%) women in the enrichment population tested HPV positive with HC2.
Clinical performance of the HPV-Risk assay. The absolute sensitivity and specificity of the HPV-Risk assay for the detection of CIN3ϩ and CIN grade 2 or worse (CIN2ϩ) are shown in Table 2, both for the total study group (n ϭ 1,316) and for women Ն30 years old (n ϭ 1,088). Within the total study group, the sensitivity of the HPV-Risk assay for the detection of CIN3ϩ was 97.6% (95% confidence interval [CI], 91.5 to 99.7%) and the specificity was 89.0% (95% CI, 87.1 to 90.7%). For the detection of CIN2ϩ, the sensitivity and specificity were 93.7% (95% CI, 88.0 to 97.2%) and 91.8% (95% CI, 90.1 to 93.3%), respectively. For women aged Ն30 years (n ϭ 1,088), the sensitivity of the HPV-Risk assay for the detection of CIN3ϩ was 97.0% (95% CI, 89.5 to  Table 2. Cross-tabulations of the results of the HPV-Risk assay and HC2 are shown in Table 3. The corresponding relative sensitivities of the HPV-Risk assay versus those of HC2 for the detection of CIN3ϩ and CIN2ϩ as well as the relative specificities of the HPV-Risk assay for the detection of cervical intraepithelial neoplasia grade 1 or lower (ՅCIN1) are shown in Table    noninferior to that of HC2 with respect to sensitivity and specificity for the detection of CIN3ϩ (P value for the noninferiority of the HPV-Risk assay to HC2 [P ni ] ϭ 0.003 and P ni Ͻ 0.001, respectively) and also for CIN2ϩ (P ni ϭ 0.006 and P ni Ͻ 0.001, respectively). When we performed the analyses for women aged Ն30 years, results similar to those obtained for the total study group were obtained (Table 4). Relative sensitivity and specificity for the detection of CIN3ϩ in women aged Ն30 years were 1.00 (95% CI, 0.94 to 1.06; P McN ϭ 1.000) and 1.02 (95% CI, 1.01 to 1.04; P McN ϭ 0.002), respectively. Relative sensitivity and specificity for the detection of CIN2ϩ were 0.97 (95% CI, 0.92 to 1.02; P McN ϭ 0.257) and 1.02 (95% CI, 1.01 to 1.03; P McN ϭ 0.005), respectively. In women aged Ն30 years, the performance of the HPV-Risk assay was clinically noninferior to that of HC2 with respect to the sensitivity and specificity for the detection of CIN3ϩ (P ni ϭ 0.007 and P ni Ͻ 0.001, respectively) and CIN2ϩ (P ni ϭ 0.010 and P ni Ͻ 0.001, respectively).

DISCUSSION
In this study, we evaluated the clinical performance of the HPV-Risk assay for the detection of high-grade CIN and compared its performance to that of a clinically validated test (HC2) using samples from the VALGENT-3 panel. The results of this study show that the sensitivity of the HPV-Risk assay for the detection of CIN3ϩ and CIN2ϩ was comparable to that of HC2 and that it had a significantly higher specificity, both in the total study group and in women aged Ն30 years. The HPV-Risk assay showed a noninferior clinical performance compared to that of HC2.
Clinical validation studies of HPV tests are crucial because the HPV assays that are used for primary cervical cancer screening must ensure an optimal distinction between HPV infections associated with CIN2ϩ/3ϩ and clinically irrelevant transient HPV infections (4). The current study, as an independent clinical validation in a different study cohort with a different clinically validated comparator test (HC2), further supports previously published clinical validation data for the HPV-Risk assay (10).
An advantage of the HPV-Risk assay, in comparison with other commercially available HPV assays, is that no specific laboratory equipment is needed since the HPV-Risk assay can run on different real-time PCR platforms and is compatible with various collection media and different DNA extraction procedures. Furthermore, the HPV-Risk assay can be performed reliably on self-sampled material, both lavage-based and brush-based material, in addition to physician-taken cervical scrape samples (10,12,13). The turnaround time of the HPV-Risk PCR method is about 1 h. However, the turnaround time of the total procedure, from sample collection to HPV test result, depends on the duration of the DNA extraction method that is used. Finally, the HPV-Risk assay targets a conserved region within the E7 open reading frame of the respective high-risk HPV types. A recent study has shown that the E7 region of HPV16, in contrast to most other open reading frames, is highly conserved in cervical cancer and precancer (14). Therefore, the use of the HPV-Risk assay might be associated with a lower risk of nondetection of cervical cancer and precancer caused by HPV16 than for assays that target other open reading frames which are more variable in cancer and precancer.
The strength of this study is the large sample set from the VALGENT-3 framework, which makes it possible to evaluate the clinical performance of various HPV assays compared to that of the validated and accepted comparator test (HC2) in a formalized and uniform manner. In addition, a network of test comparisons, which can later be pooled in multiple-testing meta-analysis, can be developed (4,5,11). Another strength is that testing by the HPV-Risk assay was performed by investigators completely blind to the clinical data and cytology findings. Furthermore, in accordance with the VALGENT framework criteria, aliquots of original, nonisolated cervical scrape material were sent to the test laboratory; therefore, not only the assay itself but also the entire laboratory process from sample processing to the HPV test result was evaluated. Finally, in VALGENT-3, women were managed on the basis of cytology and HPV screening test results, whereas in a previous VALGENT round, VALGENT-2, detection of (pre-)cancerous lesions was triggered only through cytology (6)(7)(8)(9). Therefore, the VALGENT-2 panel might have included a circumstantial advantage for sensitivity and a disadvantage for specificity. On the other hand, a relatively short active follow-up period in VALGENT-3 (12 to 48 months) may be considered a limitation. However, since HC2 was used as a comparator test that was validated through randomized trials with follow-up over 8 years, its cross-sectional accuracy is considered sufficient for validation. Nevertheless, VALGENT-3 results will soon most likely be linked to the outcomes of subsequent cervical cancer screening rounds (continuous passive follow-up through the Slovenian National Cervical Cancer Screening Registry and National Cancer Registry), which will provide information on long-term safety.
In conclusion, the HPV-Risk assay has a high sensitivity and a high specificity for the detection of CIN3ϩ and CIN2ϩ and has a demonstrated noninferiority to the clinically validated HC2. Results from this study provide additional evidence that the HPV-Risk assay can be considered clinically validated and therefore can be safely used for primary cervical cancer screening.

MATERIALS AND METHODS
VALGENT-3 panel. The VALGENT-3 panel was collated in Slovenia (5). Ethical approval was obtained from the Medical Ethics Committee of the Republic of Slovenia (consent numbers 83/11/09 and 109/08/12). The panel is standardized, comprising 1,300 consecutive samples from women participating in the organized national cervical cancer screening program with more than 70% coverage (screening population), enriched with 300 samples from women with abnormal cytology (enrichment population). The characteristics of the two study populations are shown in Table 5.
Cervical scrape samples from the screening population were collected from December 2009 to August 2010, and samples from the enrichment population were collected from January 2014 to May 2015. All samples were collected in ThinPrep PreservCyt medium (Hologic, Bedford MA, USA) according to European Union guidelines (15). Samples were transported to the University of Ljubljana (Ljubljana, Slovenia) weekly. Several aliquots were prepared from each cervical specimen. One aliquot was used for HC2 testing, and the remaining aliquots were immediately stored at Ϫ70°C and later sent to participating laboratories for HPV testing. HC2 was performed according to the manufacturer's instructions (Qiagen, Gaithersburg MD, USA) at the University of Ljubljana (Ljubljana, Slovenia) during the period from December 2009 to August 2010 (16). HC2 testing was performed within 2 weeks after sample collection. Briefly, HC2 uses a cocktail of captured RNA probes to detect 13 high-risk HPV types (i.e., HPV16, -18, -31, -33, -35, -39, -45, -51, -52, -56, -58, -59, and -68). The probes hybridize with viral DNA, and the captured HPV RNA/DNA hybrids are identified by a secondary capture system that yields a light signal. The intensity of the light signal can be semiquantitatively related to the viral load (17)(18)(19).
Management of the women was based on the results of cytology and HPV testing (by HC2 and the Abbott RealTime High Risk HPV test), as previously described in detail (16). Women with atypicalsquamous-cell (the possibility of a high-grade squamous intraepithelial lesion cannot be excluded) (ASC-H) or atypical-glandular-cell (AGC) cytology or worse were invited for colposcopy, according to the criteria of the Slovenian National Cervical Cancer Screening Program and European Union guidelines for quality assurance for cervical cancer screening (3,16,20). In addition, women who tested positive for HPV16 or HPV18 were invited for colposcopy irrespective of their cytology result. Women who tested HPV-non-16/18 positive were invited for either colposcopy or a control visit after 6 to 12 months, at the physician's discretion. During colposcopy, biopsy specimens were taken from any regions suspicious for CIN. Histological samples were classified as CIN grade 0, 1, 2, or 3 or as invasive cancer (21,22).
HPV-Risk assay. For testing with the HPV-Risk assay, ThinPrep aliquots were transported to the VU University Medical Center (Amsterdam, The Netherlands) in June 2016. Extraction of DNA from the ThinPrep aliquots was performed using an automated extraction system (Macherey-Nagel, Duren, Germany) during the period from August to December 2016, with subsequent HPV-Risk assay testing being performed according to the manufacturer's instructions (Self-screen BV, Amsterdam, The Netherlands). Accordingly, the time between sample collection and HPV-Risk assay testing was approximately 7 years. The HPV-risk assay targets the E7 region of 15 (probably) high-risk HPV types. Detection is based on hydrolysis probes with 3 spectrally unique fluorescent dyes, providing genotype information for HPV16 and HPV18 and a pooled detection of the 13 other HPV types (i.e., HPV31, -33, -35, -39, -45, -51, -52, -56, -58, -59, -66, -67, and -68) (10). The fourth channel enables detection of the human ␤-globin gene using a probe labeled with a different fluorescent dye, intended for assessment of the sample quality control (internal control) and potential inhibition. Samples were considered HPV positive when the threshold cycle (C T ) values for HPV16, HPV18, and/or other HPV types were Յ36. If no HPV signals were obtained and the C T value for the ␤-globin target was Յ33, samples were considered HPV negative. Samples were considered invalid when the C T value for HPV was Ͼ36 and that for ␤-globin was Ͼ33.
All HPV testing was performed by investigators blind to the clinical data and cytology findings. Results obtained by testing by both HPV assays were sent to the Unit of Cancer Epidemiology, Scientific Institute of Public Health (Brussels, Belgium), where analysis and evaluation of the data were performed.
Statistical analysis. On the basis of the HPV-Risk assay results, the overall prevalence of high-risk HPV infection and the type-specific prevalence of HPV16 and HPV18 infection were assessed in the screening population of 1,300 women. For clinical validation, women with histologically confirmed CIN2, CIN3, and cervical cancer within 48 months after sample collection were classified as having high-grade disease (disease group). Women with two consecutive negative cytology results (at enrollment and at 12 to 48 months of follow-up) were classified as having no evidence of disease (control group). The clinical performance of the HPV-Risk assay for the detection of both CIN3ϩ (primary endpoint) and CIN2ϩ (secondary endpoint) was assessed in the total study group and in women aged Ն30 years, and the clinical performance of the HPV-Risk assay was compared to the clinical performance of HC2 as the standard comparator assay. Sensitivity and specificity were calculated from cross-tabulation of the test results with exact 95% confidence intervals (CIs). Relative sensitivities (ratios of the sensitivity of one test to the sensitivity of another test) and relative specificities (ratios of the specificity of one test to the sensitivity of another test) with 95% CIs were calculated. The McNemar (McN) test was applied to assess differences between matched proportions. A P McN value of Ͼ0.05 indicated that the sensitivity or specificity of the HPV-Risk assay was not significantly different from that of HC2. Finally, the noninferiority (ni) of the HPV-Risk assay to HC2 was assessed, according to the international validation criteria (4,23). In order to demonstrate noninferiority, the sensitivity of the HPV-Risk assay had to be at least 90% and the specificity had to be at least 98% compared to the results of HC2. A P ni value of Ͻ0.05 indicated that the sensitivity or specificity of the HPV-Risk assay was not significantly lower than that of HC2.

ACKNOWLEDGMENTS
We thank S. Doorn and M. van der Salm for excellent technical assistance and Self-screen B.V. for supplying the HPV-Risk assay kits. VALGENT is a researcher-induced study network initiated by the Scientific Institute of Public Health (Brussels, Belgium) with the aim of performing independent evaluations of HPV assays in collaboration with academic partners (5)(6)(7)(8)(9). Manufacturers can have their test assessed through VALGENT when the costs for logistics and statistical analysis are covered and test kits/equipment are delivered to a recognized academic laboratory and if independent reporting is assured. The researchers of the Scientific Institute of Public Health did not receive personal advantages from the manufacturers of the HPV tests included in VALGENT. The Scientific Institute of Public Health did not receive any funding for the current evaluation of the HPV-Risk assay. P.J.F.S. has been on the speakers' bureaus of Roche Diagnostics, Gen-Probe, Abbott, Qiagen, and Seegene and has been a consultant for Crucell B.