ABSTRACT
Invasive aspergillosis is a difficult-to-diagnose infection with a high mortality rate that affects high-risk groups such as patients with neutropenia and hematologic malignancies. We performed a bivariate meta-analysis of diagnostic data for an Aspergillus sp. PCR assay with blood specimens from high-risk hematology patients. We included all studies involving human subjects that assessed the performance of any PCR assay for invasive aspergillosis in whole blood or serum and that used the European Organization for the treatment of Cancer/Mycoses Study Group criteria as a reference standard. Three investigators independently searched the literature for eligible studies and extracted the data. Out of a total of 37 studies, 25 met strict quality criteria and were included in our evidence synthesis. Twenty-five studies with 2,595 patients were analyzed. The pooled diagnostic performance of whole-blood and serum PCR assays was moderate, with a sensitivity and specificity of 84% (95% confidence interval [CI], 75 to 91%) and 76% (95% CI, 65 to 84%), respectively, suggesting that a positive or negative result is unable, on its own, to confirm or exclude a suspected infection. The performance of a PCR assay of serum was not significantly different from that of whole blood. Notably, at least two positive PCR test results were found to have a specificity of 95% and a sensitivity of 64% for invasive infection, achieving a high positive likelihood ratio of 12.8. Importantly, the European Aspergillus PCR Initiative (EAPCRI) recommendations improved the performance of the PCR even further when at least two positive specimens were used to define PCR positivity. In conclusion, two positive PCR results should be considered highly indicative of an active Aspergillus sp. infection. Use of the EAPCRI recommendations by clinical laboratories can further enhance PCR performance.
INTRODUCTION
Despite advances in treatment and supportive care, invasive aspergillosis (IA) is associated with significant morbidity and mortality rates, especially among patients with hematologic malignancies and hematopoietic stem cell transplant (HSCT) recipients (1, 2). Systemic antifungal prophylaxis is widely used in this patient population (3, 4), and patients with recurrent or persistent fever and prolonged neutropenia frequently require empirical coverage with antifungal agents (5). Timely and accurate diagnosis of an active infection is needed in order to initiate targeted antifungal therapy and avoid unnecessary antifungal treatment, which is often accompanied by a multitude of side effects and the cumulative risk of resistance.
There is a need for the development of newer diagnostic techniques that would ideally be able to identify IA rapidly, noninvasively, and at an early stage. To aid in this endeavor, the European Organization for the treatment of Cancer/Mycoses Study Group (EORTC/MSG) developed specific criteria for the diagnosis of IA in 2002 (6), which were later revised in 2008 (7), in an attempt to provide the elusive “gold standard” to which any newer diagnostic test should be compared. Dominant among the novel diagnostic methods is the PCR assay, which allows pathogen DNA detection and identification to the species level in a variety of clinical samples. A number of PCR assays are now available to detect Aspergillus spp. in various clinical specimens and have a high diagnostic yield in vitro, but their clinical performance is debated because of contradictory reports from clinical trials. Previous systematic reviews have assessed Aspergillus sp. PCR assay performance with promising findings (8) but did not succeed in convincing the guideline-issuing organizations to incorporate the test in their algorithms (6, 7). The main argument has been that differences in the PCR protocols used in different studies do not allow sufficient interlaboratory comparisons to be made and thus preclude widespread implementation (9). In an effort to determine the value of the Aspergillus sp. PCR assay in clinical diagnostics as a screening or a confirmatory tool and to evaluate different parameters that could contribute to contradictory reports regarding PCR performance from the existent literature, we performed a meta-analysis of clinical trials that evaluated the accuracy of the PCR assay for IA performed with serum and whole blood from high-risk patients.
MATERIALS AND METHODS
Literature search.A systematic literature search of the MEDLINE (1951 to December 2013) and Cochrane Central Register of Controlled Trials (2010 to June 2013) databases was conducted to identify all of the published studies involving human subjects and evaluating the performance of a PCR assay for Aspergillus spp. using as a reference standard the EORTC/MSG criteria. We searched only full publications and not unpublished studies or conference abstracts. We used the search term “Aspergil* AND (PCR OR PCR).” The date of last access was 20 January 2014.
Three investigators (M.A., I.M.Z., and F.N.Z.) independently identified and scrutinized studies for potential inclusion. Studies published in languages other than English were excluded. Our meta-analysis is in line with PRISMA (preferred reporting items for systematic reviews and meta-analyses) recommendations (10).
Eligibility criteria and definitions.We included all studies involving human subjects that assessed the performance of any PCR assay for IA in whole blood or serum and that used as a reference standard the EORTC/MSG criteria (either the 2002 [6] or the revised 2008 [7] criteria). Studies of series of patients with a positive-only (probable/definite) or negative-only (no infection) reference outcome were not considered because it is impossible to extrapolate all of the pertinent diagnostic estimates from these studies. We defined as “true positive” the cases that were positive by PCR and were classified as definite or probable by the EORTC/MSG criteria and as “true negative” the cases that were negative by PCR and were classified as possible or unlikely infection by the EORTC/MSG criteria. Possible IA cases were categorized as negative in accordance with the recommendations of the Cochrane group (11). For our primary analysis, we defined as PCR positive any case that had at least one positive PCR result. In multiarm studies that used the same patients more than once, we included the strata with the highest reported performance estimates. Also, in trials that examined PCR assay performance in both a high-risk group and a control group of patients at low risk for IA, we excluded the control arm to avoid a case-control design that could spuriously increase performance estimates.
Outcomes of interest.The primary outcomes of interest were the summary sensitivity and specificity of an Aspergillus PCR assay of blood products compared to the reference standard. Our secondary goal was to evaluate whether the choice of sample, the type of PCR, the choice of primers, or any of the methodological aspects proposed by the European Aspergillus PCR Initiative (EAPCRI) (12, 13) for optimum PCR assay performance had a significant impact on the test's accuracy. Finally, we performed a sensitivity analysis to evaluate how PCR assay performance would be affected if two positive results were used to define a positive PCR outcome.
Data extraction and quality assessment.Data from eligible studies were independently extracted by three reviewers (M.A., I.M.Z., and F.N.Z.). Discrepancies between authors were resolved by consensus. The methodological quality of each trial was evaluated independently by three of us (M.A., I.M.Z., and F.N.Z.) for potential sources of bias by using standard criteria. The QUADAS-2 score was used to test for potential sources of bias in each study (14). To have high-quality estimates, we excluded from further analysis the studies with high-bias QUADAS-2 elements. Specifically, we excluded studies that had a nonrandom population or a case-control design, and we also excluded studies that did not have a prespecified threshold of PCR positivity. We chose not to eliminate trials in which the investigators had knowledge of the EORTC/MSG classification of the patient before PCR interpretation because this knowledge cannot alter the interpretation of a positive or negative finding as long as the positivity threshold is prespecified. Finally, we excluded studies that had a >10% loss to follow-up.
Statistical analysis.We calculated the independent sensitivity and specificity of each study by using a two-by-two contingency table. Each table consisted of true positives, true negatives, false positives, and false negatives according to the results of the Aspergillus sp. PCR assay (the index test) compared to the classification of the patient as definite/probable versus possible/unlikely aspergillosis on the basis of the EORTC/MSG criteria (the reference standard). We estimated the combined (pooled) sensitivity, specificity, likelihood ratios (LRs), and diagnostic accuracy (area under the curve) by using a bivariate mixed-effects binomial regression model that accounts for variability both within a study and between studies (15–17). We used empirical thresholds of >10 for LR+ and <0.1 for LR− to rate the test as of high value in the decision to rule in (i.e., probable/definite aspergillosis) or rule out infection (18). Publication bias was assessed by using the Deeks regression test for asymmetry, with P < 0.05 for the slope coefficient denoting significant asymmetry (19). We did not use heterogeneity tests (16, 20), as such tests can mislead systematic reviews of diagnostic test accuracy, are not recommended by the Cochrane diagnostic test accuracy group (16), and may be misleading in diagnostic accuracy studies, since computing separate I2 statistics for sensitivity and specificity will usually overestimate statistical heterogeneity. Effects were plotted as summary receiver operating characteristic (SROC) curves (15).
We calculated subgroup estimates to account for variation in laboratory PCR methodologies, provided that we had at least four pertinent studies in each group to draw the quadrature points and provided that we had data on the magnitude and difference of average sensitivity and specificity estimates between groups. We also evaluated whether the use of the old (2002) or the revised (2008) EORTC/MSG criteria as a reference standard affected PCR performance estimates. The significance of differences between average specificities or sensitivities was assessed by metaregression, and a P value was reported.
The bivariate meta-analysis was performed with Stata v11 (StataCorp, College Station, TX) by using the Midas set of commands (21, 22). The metaregression was performed with MetaAnalyst software (23).
RESULTS
Two thousand six nonduplicate studies were identified by the initial search, of which 37 met the inclusion criteria (Fig. 1). Twenty-three studies provided data for PCR based on the 2002 EORTC/MSG criteria (24–46), and 14 did so by using the revised (2008) criteria (47–60) (Table 1).
Flow diagram of the inclusion criteria used in this study.
Characteristics of studies included
Quality assessment.Most of the studies were of high quality, as determined by the QUADAS-2 assessment tool (Table 2). After all of the low-quality studies were excluded, 25 studies remained and were further analyzed (25, 26, 28–37, 43–48, 50, 52–54, 56–58).
Quality assessment and final determination of inclusion versus exclusion based on QUADAS-2 toola
The 25 eligible studies (26 strata) included a total of 2,595 patients. The number of patients enrolled varied from 8 to 218 (median, 91). The target patient population was patients with hematologic malignancies and/or HSCT recipients in 17/25 studies, bone marrow transplant recipients in 1/25, and a mixed population of patients with hematologic malignancies and other immunocompromised individuals in 7/25. Six studies performed PCR assays of serum (23%), and the rest used whole-blood assays (77%). The setting was heterogeneous regarding quality laboratory criteria. Of note, only 3 studies (all 3 using a whole-blood PCR assay) were found to be fully compliant with all EAPCRI criteria, while 7 studies (5 using whole blood and 2 using serum) were compliant with all of the criteria but one, 10 (9 using whole blood and 1 using serum) deviated by 2 criteria, and 6 (4 using whole-blood and 2 using serum) deviated by 3 or more.
Pooled diagnostic estimates.Across the 25 eligible studies, the individual sensitivity estimates ranged from 0 to 1.0, while the individual specificity estimates ranged from 0.29 to 0.98 (Fig. 2). In bivariate meta-analysis, the pooled sensitivity was 0.84 (0.75 to 0.91) and the corresponding pooled specificity was 0.76 (0.65 to 0.84) (Table 2; Fig. 3). There was no evidence of significant funnel plot asymmetry (Deeks' bias = −0.71, P = 0.9), indicating that small study effects were not present. The combined effects show that the Aspergillus sp. PCR assay has a positive LR of 3.5 and a negative LR of 0.21, suggesting that its diagnostic performance is moderate (61). Specifically, in settings where the expected pretest probability of IA is low (≤5%), PCR will yield a positive predictive value (PPV) of 0.08 (range, 0.06 to 0.11) and a negative predictive value (NPV) of 0.99 (range, 0.97 to 1.00), while in settings where IA is strongly suspected (pretest probability, ≥10%) PCR is expected to yield a PPV of 0.76 (range, 0.71 to 0.81) and an NPV of 0.71 (range, 0.66 to 0.77), respectively.
SROC curve of PCR performance.
Subgroup analysis.We performed a subgroup analysis to evaluate changes in sensitivity and specificity caused by a series of different parameters in each study. We also assessed whether the effect of these covariates on the average sensitivity estimates were significant. Specifically, across studies where the updated EORTC/MSG 2008 criteria were implemented, the diagnostic estimates were 0.83 (range, 0.72 to 0.90) for sensitivity and 0.79 (range, 0.67 to 0.87) for specificity, respectively. These estimates did not differ from those of studies that used the EORTC/MSG 2002 criteria as the reference index. Interestingly, studies that used PCR assays of serum yielded higher specificity (85% versus 73%) and lower sensitivity (78% versus 86%) estimates than those that used whole-blood assays, but these differences did not reach statistical significance. It is of note that the average specificity of serum was significantly higher than that of whole blood without bead beating (P = 0.04).
Regarding the assay methodology used, no methodological parameter significantly affected PCR performance (Table 3). Further, when the studies that deviated by no more than one of the EAPCRI criteria were compared to studies that deviated by two or more, no significant changes in sensitivity (87% versus 82%, respectively) or specificity (77% versus 75%, respectively) were noted. Of note, we could not assess the effect of white blood cell and red blood cell lysis steps, as these were undertaken by almost all of studies in which whole blood was used. We also were unable to evaluate the effect of the use of mitochondrial versus ribosomal primers because only two studies used the former (36, 44), while the difference in PCR performance among different ribosomal primers (18S rRNA versus others) was not significant.
Results of subgroup analysisc
Sensitivity analysis.Thirteen of our 26 studies (12 using whole blood and 1 using serum) included data on the diagnostic performance of the PCR assay when at least two PCR results were considered positive. By analyzing the performance estimates of these studies using differential criteria for PCR positivity (one versus two or more positive PCR results), we found that PCR specificity was greatly increased (73% versus 95%, respectively) with the latter method, giving a high positive LR of 12.8, whereas sensitivity decreased (85% versus 64%, respectively). Moreover, two or more positive assays for low-risk individuals (≤5% pretest probability) give a PPV and an NPV of 0.23 and 0.99, respectively, while among high-risk patients (≥10% pretest probability), the PPV increases to 0.90 while the NPV drops to 0.62.
Interestingly, when we compared the effect of the EAPCRI recommendations on this sensitivity analysis (Table 4), we found that bead beating significantly improved the specificity of the test (96% versus 86%, metaregression P = 0.006), with a nonsignificant drop in sensitivity (50% versus73%). Further, we found that compliant studies (no more than one deviation from the EAPCRI criteria) showed higher specificity (98% versus 85%, metaregression P = 0.003) with an additional nonsignificant increase in sensitivity (67% versus 61%).
Results of sensitivity analysis
DISCUSSION
Despite the fact that multiple Aspergillus sp. PCR assays of blood specimens are now available, a consensual conclusion about the role that this test can play in the diagnosis of IA has yet to be reached. Shortly after its introduction in the 1990s, it was thought that this method could, in fact, revolutionize the diagnosis and management of this severe disease (62). However, the multitude of clinical studies performed since then failed to prove beyond doubt whether the test can rule an active infection in or out, thus often leaving clinicians baffled when trying to interpret a positive or negative result. Our findings indicate that whole-blood and serum Aspergillus sp. PCR assays have moderate diagnostic performance, which suggests that a positive or negative result is unable, on its own, to confirm or exclude a suspected infection in high-risk patients. We should note, however, that the pooled PCR performance estimates in our study were not inferior to the performance estimates of serum galactomannan (sensitivity, 71%; specificity, 89%) (63) or beta-glucan (sensitivity, 76.8%; specificity, 85.3%) (64) tests obtained by previous meta-analyses.
Our choice to define as PCR positive all episodes that had at least one PCR-positive specimen, coupled with the fact that most of the studies tested more than one specimen per episode, may have overestimated the average sensitivity and underestimated the specificity of the method. To see whether this is true, we performed a sensitivity analysis of studies that had extractable data about at least two versus one PCR-positive specimen per patient. Indeed, we found that PCR specificity was increased dramatically by this approach to 95%, which leads to a PPV of 90% for high-risk individuals. Therefore, the presence of at least two positive whole-blood PCR specimens in a high-risk patient should be considered very indicative, if not confirmatory, of IA. On the other hand, the sensitivity of this approach was lower at 64%, which suggests that it would be more valuable as a confirmatory tool than as a screening tool. Although our results are in accordance with previous reports (8), we observed a significant variation of results reported by individual studies involved in our analysis, which led us to search for reasons behind these inconsistencies.
One of the potential explanations is the choice of the specimen, namely, whole blood or serum, with which the PCR assay is performed. Two previous clinical studies comparing Aspergillus sp. PCR assays performed with serum and whole blood from the same high-risk patients failed to show significant differences in accuracy (49, 51), with one study suggesting a nonsignificant trend toward increased sensitivity and reduced specificity of the whole-blood PCR assay (51). In accordance with these studies, our results suggest that PCR assays performed with serum had nonsignificantly lower sensitivity and higher specificity than those performed with whole blood. Taking into consideration the facts that a serum PCR assay is faster and easier to perform and that serum can also be used for other biomarker diagnostic tests, such as galactomannan, serum has the potential to be the preferred specimen for PCR testing.
Another potential moderator of effect among different trials could be the use of different versions of the EORTC/MSG criteria for the diagnosis of IA. Compared to the old version, the revised criteria (7) kept the terminology but expanded the definition of probable while reducing the scope of possible IA. The adoption of the new classification may have an unclear impact on outcomes in future trials. Indeed, a recent retrospective evaluation stated that the majority of “possible” IA cases can be downgraded to “unclassifiable” but also probable cases showed 75% reductions (65). While this impact may or may not hold true in clinical practice, our data indicated that PCR performance was unaffected.
Finally, the use of different methodological parameters in DNA extraction and amplification is considered one of the most important sources of inconsistencies between different trials (66). To circumvent this problem and standardize Aspergillus sp. PCR assays among different laboratories, the EAPCRI has recently issued a series of recommendations on how to perform Aspergillus PCR assays with whole blood and serum on the basis of the results of two multicenter studies with spiked samples (12, 13). These include the use of a larger sample volume, smaller elution volumes, an internal control, and thorough cell lysis with bead beating and enzymatic white and red blood cell lysis steps for whole blood. Notably, in the clinical setting we studied, none of these methodological characteristics significantly altered the sensitivity or specificity estimates in our primary analysis.
The interpretation of this finding is challenging, mainly because we do not know in what form Aspergillus spp. circulate in the blood. Of note, many studies suggest that most of the circulating Aspergillus DNA is present not in the form of conidia but rather as free DNA (67, 68). This provides an explanation for the fact that blood cultures have such a low sensitivity for aspergillosis (69), as well as our finding that whole-blood testing is not superior to serum testing. This would also explain why sequential cell lysis methods could not improve PCR assay performance for patients with IA, in contrast to what would be expected for samples spiked with Aspergillus sp. conidia (13). In addition, it is unclear whether any form of Aspergillus DNA can be found in the circulation at all times during an active infection. As suggested by a recent in vitro study, release of DNA is intermittent and happens only during certain stages of fungal growth (68). If, in fact, circulating fungal DNA is released intermittently only during mycelial breakdown, it is plausible that its levels in the blood would have such a wide fluctuation that it could make differences in sample size or elution volume relatively insignificant.
Nevertheless, it is of note that compliance with the EAPCRI recommendations significantly increased the specificity of PCR and was also accompanied by an increase in sensitivity when at least two positive results were used to define PCR positivity. An explanation for this seemingly contradictory finding would be that other factors that we were unable to assess (such as PCR testing algorithms and the use of prophylactic antifungal agents) could have served as potential confounders in the assessment of the effect of the EAPCRI recommendations on our primary analysis, and those factors should be evaluated in future trials. Compliance with these simple methodological guidelines should, in fact, be encouraged since it would allow interlaboratory comparisons and has the potential to improve PCR performance in clinical practice.
Several systematic reviews of Aspergillus sp. PCR assays of bronchoalveolar lavage fluid have been previously performed, with very promising findings (70). However, despite the large number of new studies on the issue, to our knowledge, there is only one meta-analysis of whole-blood and serum PCR performance to date. That study, by Mengoli et al. (8), reported that PCR assay performance is moderate but specificity can be increased when two PCR results are used to define a positive PCR finding. That study was published before the EAPCRI recommendations were issued and included a smaller number of trials, so the authors could not reach any conclusions regarding the performance of different PCR assay protocols. It is of note that our study confirms the finding that a single positive PCR assay result has moderate diagnostic accuracy, whereas a strategy that uses at least two positive PCR assay results per suspicious episode to define positivity is able to achieve superior specificity. Moreover, by investigating the effects of different protocols on PCR assay performance, our study shows that a serum PCR assay is not inferior to a whole-blood PCR assay and that compliance with the EAPCRI recommendations has the potential to improve PCR assay performance in clinical practice.
Important limitations of our study are the facts that (i) only a subset of the studies included had extractable data regarding PCR assay performance for at least two positive specimens and (ii) the vast majority of these studies used whole blood and not serum as the test specimen. Therefore, the results of this sensitivity analysis should be interpreted with caution. Also, all of the studies included in our analysis enrolled individuals who had a high risk of invasive fungal infection, as this is the patient population for which the Aspergillus sp. PCR test will be the most valuable. Therefore, our performance estimates may not be accurate for the general population.
In summary, for high-risk individuals, two whole-blood or serum specimens PCR assay positive for IA should be considered very indicative, if not confirmatory, of a clinically suspected infection. Given the superior specificity of this approach, it has the potential to be used along with other circulating biomarker detection assays, such as galactomannan, as a criterion to define a probable infection. Compliance with the EAPCRI Aspergillus sp. PCR assay recommendations can further increase the specificity of this approach and will allow better interlaboratory comparisons. However, given the complexity of IA diagnosis, it is unlikely that a single noninvasive test will be able to be used alone in clinical decision making. Consequently, the focus of future clinical trials should include the development of decision algorithms that would take into account multiple parameters to guide the management of high-risk groups. In this context, serum and whole-blood PCR assays have a concrete potential to improve our ability to detect and diagnose IA.
ACKNOWLEDGMENTS
A.M.C. is an active member of the Aspergillus Technology Consortium (AsTeC) group.
A.M.C. and E.M. have received financial support from T2 Biosystems for research unrelated to this report. The rest of us have no potential conflicts of interest to report.
FOOTNOTES
- Received 13 May 2014.
- Returned for modification 9 June 2014.
- Accepted 1 August 2014.
- Accepted manuscript posted online 13 August 2014.
- Copyright © 2014, American Society for Microbiology. All Rights Reserved.
