ABSTRACT
Management of invasive aspergillosis has been improved by biomarker assays, but limited accessibility and batch testing limit the impact. Lateral flow assays (LFA) are a simple method for use outside specialist centers, provided performance is acceptable. The objective of this study was to determine the performance of the recently released IMMY sona Aspergillus LFA when testing serum samples. The study took the form of a retrospective, anonymous case/control study comprising 179 serum samples from 136 patients with invasive fungal disease, previously documented using recently revised internationally accepted definitions. The LFA was performed following the manufacturer’s instructions using a cube reader to generate a galactomannan index (GMI). Performance parameters were determined, and receiver operator characteristic (ROC) analysis was used to identify an optimal threshold. Concordance with the Bio-Rad Aspergillus Ag assay (GM-EIA) was performed. At the recommended positivity threshold (GMI ≥ 0.5), LFA sensitivity and specificity were 96.9% (31/32) and 98% (98/100), respectively. ROC analysis confirmed the optimal threshold and generated an area under the curve of 0.9919. Qualitative agreement between LFA and GM-EIA was 89.0%, generating a Kappa statistic of 0.698, representing good agreement, with most discordance arising due to false-negative GM-EIA samples that were positive by LFA. The median GMI generated by the LFA was significantly greater than that generated by the GM-EIA. The IMMY sona Aspergillus LFA, when used with a cube reader, provides a rapid alternative to the well-established GM-EIA, potentially detecting more GM epitopes and enhancing sensitivity.
INTRODUCTION
The development of assays to detect biomarkers (Galactomannan antigen [GM] and Aspergillus DNA) in clinical samples from patients at risk of invasive aspergillosis (IA) has significantly improved the management of IA. Multicenter randomized control trials have shown that persistent GM and Aspergillus PCR negativity are sufficient to exclude infection and withhold or cease antifungal therapy without detrimentally affecting patient prognosis (1, 2). A systematic review and meta-analysis confirmed the feasibility of combined testing, generating a sensitivity of 99%, but also showed that when patients were positive for both biomarkers, there was a high likelihood of disease, as specificity was 98% (3). Until recently, the detection of GM through enzyme immunoassay (GM-EIA) was only available from a single manufacturer (Bio-Rad, California, USA), and despite the recent release of alternative kits (e.g., FungiXpert Aspergillus galactomannan enzyme-linked immunosorbent assay [ELISA] kit; AGMAg ELISA kit; Aspergillus galactomannan Ag Virclia Monotest, and Dynamiker Aspergillus galactomannan assay), the evidence base in relation to clinical performance remains significantly weighted toward the original test. Nevertheless, performance of this assay varies between centers, despite it being a standardized testing method (4, 5). PCR testing is generally limited to specialist centers of excellence, and the recent availability of commercial molecular tests has yet to expand the testing repertoire of most centers. This has resulted in Aspergillus biomarker testing being consolidated in reference facilities. In addition, testing is usually batched to minimize costs; this hampers turnaround time and potentially limits the influence of these tests on patient management. The complex methodological nature of these tests also limits their availability in resource-limited settings.
Lateral flow devices to assist in the diagnosis of IA have been described for over a decade (6). Performance as determined by meta-analysis generated pooled sensitivity and specificity of 68% and 87%, respectively, when serum was tested, and 86% and 93%, respectively, when bronchoalveolar lavage (BAL) fluid was tested and was primarily focused on a single device (OLM Diagnostics, Newcastle Upon Tyne, UK) (7). The development of a cryptococcal antigen test (IMMY, Oklahoma, USA) markedly improved the diagnosis of cryptococcosis, providing a near-patient test that has excellent performance (sensitivity and specificity, >95%) yet is suitable for use in resource-limited settings (8). The release of the IMMY sona Aspergillus galactomannan lateral flow assay (LFA) represents a potentially exciting development in the field of mycology. To date, studies have focused on the performance of the LFA when testing respiratory samples (e.g., BAL fluid), generating sensitivity and specificity of 83 to 92% and 91 to 92%, respectively (9, 10). This article describes the first evaluation of LFA performance when testing serum from a large cohort of patients at risk of IA.
MATERIALS AND METHODS
Study design and patient population.The study took the form of a retrospective, anonymous case/control study comprising 179 serum samples taken from 2012 to 2019 for routine diagnostic investigations for invasive fungal disease (IFD), with 97.2% of samples originating from the past 2 years. Samples were stored at –80°C for quality control and performance evaluation purposes not requiring ethical approval. The recent second revision of the European Organization for Research and Treatment of Cancer and the Mycoses Study Group Education and Research Consortium (EORTC/MSGERC) definitions were used to classify the certainty of IFD in each patient, using GM-EIA and/or Aspergillus PCR positivity in blood or BAL fluid as a mycological criterion when defining IA (11). In addition, probable IFD was defined in patients with host factors, clinical features typical of IA, and positive (1-3)-β-d-glucan results as the mycological criterion. Under the recently revised EORTC/MSGERC definitions, a single galactomannan index (GMI) of ≥1.0 in serum/plasma or BAL fluid is required to use this test as a mycological criterion for defining probable IA. In five of the cases of probable IA, the samples evaluated in the current study had a GMI of ≥0.5 but <1.0; in three patients, probable IA was defined, as the GMI in previous serum samples was ≥1.0; in another, BAL fluid had a GMI of ≥1.0, with the remaining patient being Aspergillus PCR positive in blood. In one probable IA case, the serum GMI was consistently <0.5, but the patient had BAL fluid with a GMI of ≥1.0. In four patients, serum GMI was consistently <0.5, and all patients achieved a diagnosis of probable IA due to Aspergillus PCR positivity in blood and/or BAL fluid. The LFA result played no role when defining IFD.
Routine investigations for invasive fungal disease.As part of the routine prospective diagnostic testing, samples were tested using the Bio-Rad Aspergillus Ag assay following manufacturer’s instructions and using a positivity threshold, at the time of testing, of a GMI of ≥0.5. Aspergillus PCR testing was performed on serum/plasma following the recommendations of the European Aspergillus PCR Initiative (now known as the Fungal PCR Initiative) using a well-validated in-house qualitative real-time PCR assay (12, 13). The detection of (1-3)-β-d-glucan was performed using the Associates of Cape Cod Fungitell assay following the manufacturer’s instructions, with a positivity threshold of 80 pg/ml, which was used as the mycological criterion when defining probable IFD.
IMMY sona Aspergillus galactomannan lateral flow assay.The LFA was performed by a single operator following the manufacturer’s instructions, using a GMI of 0.5 as a threshold for positivity. To remove subjectivity, confirm validity, and provide a GMI, the sona LFA cube reader (IMMY Diagnostics, Oklahoma, USA) was used when reading each LFA.
Statistical analysis.When determining the clinical accuracy of the LFA, the positivity rate in samples originating from cases was compared to the false positivity rate in control samples. Clinical performance was determined by the construction of 2 × 2 tables to calculate the sensitivity, specificity, positive and negative likelihood ratios, and diagnostic odds ratio of the LFA. The Youden’s statistic (sum of sensitivity and specificity minus one) was calculated to determine the combined performance of a dichotomous test. Given the case control study design and artificially high incidence of proven/probable IA (23.0%), predictive values are provided for completeness but should be interpreted with an appreciation for the influence of disease incidence on these parameters. For each proportionate value, 95% confidence intervals and, where required, P values (Fisher’s exact test; P = 0.05) were generated to determine the significance of the difference between rates. Receiver operator characteristic (ROC) analysis was performed to determine the overall performance of the LFA and to identify an optimal positivity threshold. Qualitative agreement between the LFA and GM-EIA was demonstrated through the generation of observed agreement (accuracy) and a Kappa statistic. Quantitative agreement was performed by determining a Spearman correlation between the GMI calculated by the LFA and GM-EIAs. Median values were compared using a Mann-Whitney t test for pairwise analysis or Kruskal-Wallis one-way analysis of variance (ANOVA) when comparing multiple median values. Statistical analysis was performed using GraphPad Prism 5.02.
RESULTS
The study involved 27 cases of proven/probable IA, 4 cases of probable IFD, 4 cases of possible IA, and 100 patients with no evidence of fungal disease. One additional patient with chronic aspergillosis defined by Aspergillus IgG and clinical presentation was also included. Most patients (82.2%) were being treated for a hematological malignancy; the median age was 58 years, and the male/female ratio was 1.4/1 (Table 1). A total of 57 samples were from the 32 cases of aspergillosis (16 patients had multiple samples [n = 41] tested; 9 patients, 2 samples; 5 patients, 3 samples; and 2 patients, 4 samples). Ten samples were from the four cases of possible IA (one patient each contributing one, two, three, and four samples, respectively). A total of 112 samples were from control patients (10 patients had multiple samples [n = 22] tested; 8 patients, 2 samples and 2 patients, 3 samples).
Patient demographics and associated IMMY sona Aspergillus lateral flow assay galactomannan index valuesa
Sample positivity rates.Of the 179 samples, 54 (30.2%) were positive (GMI ≥ 0.5) by the LFA. Positivity rates for samples originating from cases (proven/probable IA/IFD and chronic aspergillosis), possible IFD, and control patients were 91.2% (52/57; 95% CI, 81.1 to 96.2), 0% (0/10; 95% CI, 0.0 to 27.8), and 1.8% (2/112; 95% CI, 0.5 to 6.3). The positivity rates for samples originating from cases was not significantly different whether a single or multiple (2 to 4) samples were tested per patient (single sample tested per patient positivity rate, 93.75% [15/16; 95% CI, 71.7 to 98.9]; multiple samples tested per patient positivity rate, 90.2% [37/41; 95% CI, 77.5 to 96.1]). The two false-positive LFA results were associated with control patients from whom multiple samples had been tested. Positivity rates for samples originating from cases were significantly greater than those for samples from both possible IFD and control patients (Fisher’s exact test; P < 0.0001). The median GMI for cases, possible IFD, and control samples was 1.21, 0.21, and 0.17, respectively (Table 1). The median value for case samples was significantly higher than that for the others (one-way ANOVA; P < 0001).
Clinical performance.The LFA performance when serum was tested is shown in Table 2. At the manufacturer’s recommended positivity threshold (GMI ≥ 0.5), sensitivity and specificity were 97% and 98%, respectively, and the assay could be confidently used to both confirm (positive likelihood ratio, 48.44) and exclude (negative likelihood ratio, 0.03) a diagnosis of IA. Receiver operator characteristic (ROC) analysis confirmed the optimal positivity threshold to be 0.5, but by lowering the threshold to 0.33, sensitivity was increased to 100% (Fig. 1). Conversely, increasing the threshold to 0.6 provided a specificity of 100%. The area under the ROC curve was 0.9919 (95% CI, 0.9834 to 1.0).
Clinical performance of the IMMY sona Aspergillus lateral flow assay testing serum samples from cases with proven/probable/chronic IA/IFD (n = 32) and control patients with no evidence of invasive fungal disease (n = 100)a
Receiver operator characteristic curve of the IMMY Aspergillus lateral flow assay testing serum samples from cases of proven/probable invasive aspergillosis/invasive fungal disease and control patients with no evidence of invasive fungal disease. When considering all samples, at the current 0.5 galactomannan index (GMI) threshold, sensitivity is 91.2% and specificity is 98.2%. Using a 0.33 GMI threshold, sensitivity is 100% and specificity is 85.7%. Using a 0.6 GMI threshold, sensitivity is 86% and specificity is 100%. Please note that these values differ slightly from those in Table 1, as they consider all samples, whereas in Table 1 performance is calculated on a patient basis. AUC, area under curve.
Concordance between the LFA and GM-EIA.The overall qualitative observed agreement between LFA and GM-EIA was 89.0% (95% CI, 83.5 to 92.9), generating a Kappa statistic of 0.698 (95% CI, 0.515 to 0.881), representing good agreement. The qualitative observed agreement for samples originating from cases was 66.7% (95% CI, 53.0 to 78.0), with most (94.1%) of the discordance arising due to samples that were GM-EIA negative but positive by the LFA. The qualitative observed agreement for samples originating from controls was 98.2% (95% CI, 93.7 to 99.5). The quantitative correlation between the GMI calculated by the GM-EIA and the LFA was moderate (Spearman’s coefficient, 0.64; P < 0.0001; Fig. 2a). The GMIs calculated by the LFA were significantly higher than those generated by the GM-EIA (all samples LFA median GMI, 0.27; GM-EIA median GMI, 0.05; Mann-Whitney t test P < 0.0001). The median GMI for case samples tested using the LFA (1.21) was significantly greater than that for case samples tested using the GM-EIA (0.7; P = 0.0083). The median GMI value for control samples tested using the LFA (0.17) was also significantly greater than that generated using the GM-EIA (0.04; P < 0.0001), but the LFA 90% percentile (0.37) was below the current positivity threshold.
Correlation between the galactomannan index (GMI) determined by the Bio-Rad Aspergillus Ag and IMMY sona Aspergillus lateral flow assays (a) considering all samples and (b) excluding three samples where the GMI generated by the Bio-Rad Aspergillus Ag assay was above the upper limit of the optical density reader at 450/620 nm. R2, Spearman coefficient.
DISCUSSION
This article represents the first evaluation of the IMMY sona Aspergillus LFA to assist in the diagnosis of IA when testing serum samples. Performance was excellent, and ROC analysis confirmed a GMI threshold of 0.5 to be optimal, but with slight adjustments, the assay was capable of confidently both confirming and excluding IA (Fig. 1 and Table 2). Interestingly, and despite the retrospective evaluation of the LFA performance, it outperformed the well-established GM-EIA, detecting cases missed by the GM-EIA, associated with the generation of a higher GMI. The GM-EIA utilizes the single rat monoclonal antibody (MAb) EB-A2 to bind and detect GM. Based on the design of the IMMY CrAg LFA, the Aspergillus LFA has been developed to incorporate two MAbs, which has the potential to provide greater sensitivity (10). Indeed, the ME-A5 MAb likely binds to a similar GM epitope as EB-A2, so any GM-EIA positivity would be expected to be confirmed by the LFA. Only one sample that was GM-EIA positive (GMI, 0.7) was negative by the LFA (GMI, 0.33). The other undisclosed MAb utilized in the LFA could detect other epitopes of GM that would be missed by GM-EIA. The LFA was positive in five cases of IA, four cases of IFD, and one case of chronic aspergillosis that were negative by GM-EIA and was the main reason for discordance between the LFA and GM-EIA. The median LFA GMI for these cases was 0.71. Given that the overall median LFA GMI is approximately double that of the GM-EIA, the predicted GMI (0.35) for the GM-EIA would be expected to be below the positivity threshold. In reality, the median was significantly less (GM-EIA GMI, 0.09), indicating that the EB-A2 MAb was not binding any significant quantity of antigen. Alternatively, cross-reactivity with non-Aspergillus pathogens has been noted and could explain the positivity in the four cases of probable IFD (9).
The performance in cases of possible IFD also provides significant clinical value, given that these cases are frequently encountered in the clinic. Unfortunately, the ambiguity in diagnosis hinders the interpretation of results, and with the increasing range of available diagnostic tests, the number of possible IFD cases should be limited. Interestingly, the 10 samples from the four possible IFD cases (by definition, GM-IA negative) were also negative by the LFA. Given the sensitivity of the LFA, and the fact that other biomarker assays (1-3-β-d-glucan and Aspergillus PCR) were also negative, it is likely that these patients had alternative fungal infection (e.g., Mucorales infection) or another clinical reason for their chest radiology being suggestive of IA. Because possible IFD is more regularly encountered in the clinic than proven/probable IFD, this study would have benefitted from evaluating performance when testing additional possible cases.
This first evaluation of the LFA demonstrates remarkable clinical performance when serum is tested, comparable to that shown for CrAg LFA. Despite excellent performance in the two previous studies evaluating the LFA when BAL fluid is tested, the performance of the LFA was inferior to that of GM-EIA (9, 10). This could reflect the effect of testing different specimen matrices or the absence of cases of probable IFD (GM-EIA negative/BDG positive) in the previous studies. Interestingly, in the study of Mercier et al., the sensitivity of LFA was superior when testing cases of probable IA that excluded GM as the defining mycological criterion, indicating that the LFA did indeed detect cases of IA that were negative by GM-EIA (10). The difference in reported performance could also reflect the retrospective case/control single-center design of this current study as a limitation, and performance needs to be ratified in a large-scale multicenter prospective study.
The use of the cube reader is critical to the success of the test, as it removes subjectivity and provides a GMI, which permits direct comparison with the GM-EIA and may allow the opportunity to use the test to monitor the response to therapy or determine patient prognosis. Indeed, if the reader had not been utilized, the ability to detect positives at or around the positivity threshold would have been compromised. In a previous study of the LFA, when BAL fluid was tested, the use of the digital reader was critical to overcoming disagreement between independent visual inspections (10). The cube reader is critical to correlating GMI generated between the LFA and GM-EIA, and moderate correlation was determined in this study but was influenced by the enhanced sensitivity of the LFA, which generated positive GMI (≥0.5) in cases of proven/probable IA/IFD that were negative (<0.5) by GM-EIA. One caveat for consideration is that if the GMI is exceptionally high, then saturation of the LFA may influence the accuracy of the reading. However, this applies equally to the GM-EIA, and three samples in this study generated a GMI by GM-EIA that was above the upper limit of the optical density reader at 450/620 nm. All three samples had a GM-EIA GMI of >18 (maximum possible optical density at 450 and 620 nm [OD450/620 nm] of 9.0 divided by the mean OD450/620 nm of the threshold control ≤0.5), and the GMI determined by the LFA ranged between 3.0 and 8.0, significantly less than that generated by the GM-EIA and negatively impacting the correlation between the two assays (Figure 2b). Given that the GM-EIA GMI was not specifically reported, including these data points for comparative correlation analysis could skew the data.
Unfortunately, given the nature of a reference laboratory receiving samples referred from a range of remote clinical institutions, it was not possible to retrieve all pertinent information, such as the use of antifungal therapy, to determine its influence on assay performance. That information would have also been important when interpreting the difference in performance between the Bio-Rad GM-EIA and LFA.
One limitation of the test format is that if large numbers of samples are to be tested (as in a high-throughput screening strategy), then LFA testing becomes cumbersome in comparison to plate-based EIA formats, which can also be automated.
To conclude, the IMMY sona Aspergillus LFA represents an exciting development in the field of Aspergillus biomarker detection. When used with a cube reader, it provides an alternative to the well-established GM-EIA, particularly for use in resource-limited settings, potentially detecting more GM epitopes. Given the retrospective design of the study, the fact that the LFA outperformed the GM-EIA is testament to its design, based on knowledge and experience gained from development of other lateral flow assays.
ACKNOWLEDGMENTS
We thank IMMY for providing the kits and cube reader required to perform this study at no charge.
P.L.W. performed diagnostic evaluations and is a founding member of the European Aspergillus PCR Initiative.
P.L.W. received meeting sponsorship from Bruker, Dynamiker, and Launch Diagnostics; speaker fees, expert advice fees, and meeting sponsorship from Gilead; speaker and expert advice fees from F2G; and speaker fees from MSD and Pfizer. M.B. received speaker fees, expert advice fees, and meeting sponsorship from Gilead and meeting sponsorship from Abbvie. J.S.P., R.P., M.C.-V., and L.V. disclose no conflict of interest.
FOOTNOTES
- Received 10 January 2020.
- Returned for modification 3 February 2020.
- Accepted 10 March 2020.
- Accepted manuscript posted online 18 March 2020.
- Copyright © 2020 American Society for Microbiology.
