ABSTRACT
Standard culture methods for diagnosis of Streptococcus pneumoniae pneumonia take at least 24 h. The BinaxNOW urine-based test for S. pneumoniae (BinaxNOW-SP) takes only 15 min to conduct, potentially enabling earlier diagnosis and targeted treatment. This study was conducted to assess whether the use of BinaxNOW-SP at the time of hospital admission would provide adequate sensitivity and specificity for diagnosis of community-acquired pneumonia (CAP) in adult patients. We searched PubMed, EMBASE/OVID, Cochrane Collaboration, Centre for Reviews and Dissemination, INAHTA, and CADTH for diagnostic or etiologic studies of hospitalized predominately adult patients with clinically defined CAP that reported the diagnostic performance of BinaxNOW-SP versus cultures. Two authors independently extracted study details and diagnostic two-by-two tables. We found that 27 studies met our inclusion criteria, and three different reference standards were used between them. A bivariate meta-analysis of 12 studies using a composite of culture tests as the reference standard estimated the sensitivity of BinaxNOW-SP as 68.5% (95% credibility interval [CrI], 62.6% to 74.2%) and specificity as 84.2% (95% CrI, 77.5% to 89.3%). A meta-analysis of all 27 studies, adjusting for the imperfect and variable nature of the reference standard, gave a higher sensitivity of 74.0% (CrI, 66.6% to 82·3%) and specificity of 97.2% (CrI, 92.7% to 99.8%). The analysis showed substantial heterogeneity across studies, which did not decrease with adjustment for covariates. We concluded that the higher pooled sensitivity (compared to culture) and high specificity of BinaxNOW-SP suggest it would be a useful addition to the diagnostic workup for community-acquired pneumonia. More research is needed regarding the impact of BinaxNOW-SP on clinical practice.
INTRODUCTION
Streptococcus pneumoniae pneumonia is believed to be the most common cause of community-acquired pneumonia (CAP) in adults, which in turn is the most common infection-related cause of death in developed countries (1). Diagnosis is usually established by observation of S. pneumoniae in a Gram-stained sputum sample or growth of S. pneumoniae in a culture of blood, sputum, pleural fluid, or other respiratory sample. Although highly specific, culturing is known to be insensitive, with diagnostic yields reported to be <30% for blood culture (2–7) and 57% for sputum culture (the latter in patients who had an etiologic diagnosis established) (8). Cultures also require 24 h or more to produce results. In the absence of a reliable rapid test for pneumonia caused by S. pneumoniae, initial treatment of pneumonia must be empirical, based upon knowledge of local pathogens, patient risk factors and comorbidities, and severity of presentation (7). Empirical therapy is generally effective (7), but there is increasing interest in improved targeting of antibiotics, due to the understanding that this may decrease the community prevalence of antibiotic-resistant bacteria and individual risks of antibiotic-associated Clostridium difficile infection (9–11).
The BinaxNOW Streptococcus pneumoniae test (BinaxNOW-SP; Binax, Inc.) is an immunochromatographic test for the presence of the pneumonococcal C-polysaccharide coat protein in urine. It produces a result within 15 min of a urine sample being obtained, and therefore it can be used as a rapid diagnostic test for S. pneumoniae infection in patients presenting with pneumonia. A number of studies have reported comparisons of BinaxNOW-SP with culture methods. A challenge in reviewing this literature is that a number of these studies were etiologic studies that may have incorporated BinaxNOW-SP in the diagnostic standard, thus artificially overestimating its sensitivity and specificity. Another challenge is that, since culturing has poor sensitivity, an analysis that assumes the reference standard (i.e., the test to which the new test is compared) is perfect may produce an underestimate of the sensitivity and specificity of BinaxNOW-SP (12). The problem of bias is worsened in a meta-analytic setting due to the diversity in reference standards across studies.
We undertook a systematic review and diagnostic meta-analysis of the sensitivity and specificity of BinaxNOW-SP in comparison with established culture methods for the diagnosis of S. pneumoniae infection in patients admitted to the hospital with community-acquired pneumonia. In addition to the standard bivariate model for meta-analysis, we also used a latent class meta-analysis method, which allowed adjustment for an imperfect reference standard and accommodated variable reference standards across studies (13).
MATERIALS AND METHODS
Data sources and searches.We searched PubMed (inception to 10 July 2012), EMBASE (Ovid; 1996 to 2012 week 27), the Cochrane Collaboration, the Centre for Research and Dissemination, the Canadian Agency for Drugs and Technology in Health (CADTH and a CADTH confederated search across other Canadian health technology assessments), and the Centre for Reviews and Dissemination (DARE) for systematic reviews, health technology assessments, and studies that had assessed sensitivity and specificity for BinaxNOW-SP against any reference standard. The search strategies are detailed in Table S1 in the supplemental material. The search was last updated on 10 July 2012. We also hand-searched the citation lists of articles and review articles retrieved.
Study selection.Two investigators (A.S. and X.X.) independently screened articles by title and abstract to produce a list of articles for full text review, resolving differences by discussion. We included studies of adult patients admitted to the hospital with suspected CAP that provided data that could be used to construct a two-by-two cross-tabulation for BinaxNOW-SP against a reference test. Studies had to have recruited an identifiable cohort of patients with CAP, defined by clinical signs and symptoms of lower respiratory tract infection or pneumonia, and an X-ray with new abnormalities described as an infiltrate or consolidation or otherwise consistent with pneumonia. We excluded studies that had a case-control design that used patients without CAP as controls or that predominately or exclusively included children, patients with nosocomial pneumonia, or outpatients. The BinaxNOW-SP test performs poorly with children because of their high level of asymptomatic nasal carriage of S. pneumoniae (14). We did not exclude studies with patients with HIV, AIDS, or other forms of immunosuppression, as BinaxNOW-SP measures a bacterial coat protein, the presence of which does not depend upon the patient mounting an immune response.
We included studies that reported BinaxNOW-SP results for nonconcentrated urine that had been collected as part of the initial investigation prior to or at admission or within 48 h of admission. Urine could be frozen prior to assay, provided that storage was not prolonged (greater than 3 years), as the coat protein is considered stable to freezing. If the urine was stored, our risk of bias assessment acknowledged the possibility that the index test results might have been interpreted with knowledge of the reference test. We included studies that reported results for BinaxNOW-SP against a reference standard that consisted of culture of samples from blood alone or from a respiratory site (sputum, pleural fluid, bronchiolar lavage, transthoracic needle aspirate, nasopharyngeal) with or without Gram stain of sputum or pleural fluid. All reference standards had to include blood culture. We excluded studies in which the BinaxNOW test was applied to samples other than urine. We excluded studies that incorporated the results of BinaxNOW-SP in the reference standard and did not provide the data to separate patients diagnosed solely by BinaxNOW-SP.
Data extraction and quality assessment.Data extracted from each study included the ages of patients, admitting diagnoses, and location of the study, the clinical and X-ray criteria for diagnosis of CAP, and the criteria for diagnosis of S. pneumoniae pneumonia (sites of cultures and other tests conducted). Where definite and probable diagnoses of S. pneumoniae pneumonia were reported separately (usually on the basis of samples from normally sterile sites versus samples from nonsterile sites), we confirmed that the categories were mutually exclusive and combined the results under a single diagnosis of pneumonia. Where BinaxNOW-SP results had been incorporated into the reference standard but sufficient information had been included to separate them, patients who had positive BinaxNOW-SP tests but negative culture results were reclassified as false positives for the purposes of our analysis.
We assessed the risk of bias in each study by using the QUADAS tool (15). Three authors (A.S., X.X., and N.D.) carried out both data extraction and risk of bias assessment independently and discussed any discrepancies to arrive at a consensus assessment.
Data synthesis and analysis.The reference standards (i.e., comparators for BinaxNOW-SP) in the selected studies were often based on a composite of multiple tests, such that a subject with a positive result for at least one of these tests was classified as reference test positive. We grouped the reference standards into three types according to the etiologic agent identified by each: (i) reference standard type A, a composite of blood culture, sputum (smear or culture), and culture of any other respiratory sample; (ii) reference standard type B, a composite of a blood culture and sputum (smear or culture); (iii) reference standard type C, a blood culture alone. We used hierarchical summary receiver operating characteristic curve (HSROC) meta-analysis models to summarize sensitivity and specificity estimates of BinaxNOW-SP with respect to each reference standard (13, 16, 17). These models assumed that each reference standard had 100% sensitivity and specificity.
As the sensitivity of these reference standards is believed to be poor, we also considered an extension of the meta-analysis model by allowing the reference test to be imperfect via a latent class model and also to be different across studies (13). A latent class model recognizes that the true disease status (i.e., the S. pneumoniae pneumonia status in the current application) is “latent” or not observed. Each cell of the two-by-two table for comparison of BinaxNOW-SP versus the reference standard was assumed to be a mixture of S. pneumoniae pneumonia-positive and S. pneumoniae pneumonia-negative patients. The percentage of patients who were positive or negative in each cell was determined based on the prevalence of S. pneumoniae pneumonia and the sensitivity and specificity of the BinaxNOW-SP test and the reference standard. We used a hierarchical structure that allowed for consideration of between-study variability in determining the sensitivity and specificity of each reference standard.
We estimated the unknown parameters for all models by using a Bayesian approach with noninformative prior distributions that would allow the observed data to dominate the final estimates of sensitivity and specificity. We carried out a meta-regression analysis by extending the latent class meta-analysis model to investigate whether the heterogeneity of sensitivities and specificities of BinaxNOW-SP across individual studies could be explained by study design (retrospective versus prospective), the purpose of the study (diagnostic versus etiologic), or type of hospital (tertiary university-affiliated center versus other). We also studied the impact of adjusting this model for conditional dependence, i.e., a correlation between BinaxNOW-SP and the reference test within the groups of S. pneumoniae-positive and -negative individuals in each study. We considered models with different degrees of correlation (18) and compared them by using the deviance information criterion (19).
From all meta-analysis models, we obtained estimates of the median and 95% credible interval (CrI) of the pooled sensitivity and specificity of BinaxNOW-SP across studies, the predicted and observed specificity in an individual study, and a summary receiver operating characteristic curve. Analyses were carried out using WinBUGS 1.4.3 (20) and R version 2.14.2 (21). The WinBUGS programs used for the meta-analysis are available from the corresponding author.
RESULTS
Search results and patient characteristics.We identified 27 studies that provided sufficient information on BinaxNOW-SP test performance in patients with CAP to contribute to a meta-analysis (Table 1) (3, 4, 6, 22–45). Detailed results of the literature search, selection, and reasons for exclusion are summarized in the flow chart in Fig. S1 of the supplemental material. Patients in the included studies were predominantly middle-aged or elderly, with the exception of studies that included HIV-positive or AIDS patients. The mean/median age ranged from 43 to 79 years, with the proportion of men from 47% to 79%. Based on the reference standard in individual studies, 4.4% to 38.1% received a diagnosis of S. pneumoniae pneumonia. The proportion with severe disease, as indicated by pneumonia severity index (PSI) class IV or V, ranged from 23 to 61%. One study reported on a cohort of patients admitted to an intensive care unit (ICU) (36). Prior use of antibiotics ranged from 16 to 76%, although some studies assessed antibiotic use postadmission and so may have included in-hospital as well as prior use. Not all studies reported these covariates.
Details of studies of diagnosis of Streptococcus pneumoniae community-acquired pneumonia by using BinaxNOWa
Risk of bias assessment.A summary of the risk of bias assessment results are shown in Table 2. Items according to QUADAS-1 were grouped into domains according to the approach of the recently published QUADAS-2 (46). No study met the requirement for a perfect reference standard. In a number of studies, the assessment of risk of bias was affected by unclear reporting. Few studies, even those that were primarily diagnostic in design, explicitly declared blinding of the index test and reference test relative to each other, although in some instances the described workflow (e.g., where the urine test was conducted on a fresh sample in the emergency room) implied blinding of the index test. Timing of the two tests relative to each other, and importantly, relative to the start of antibiotic adminstration, was frequently not described. The risk of bias assessment did not identify a subset of higher-quality studies; therefore, we did not attempt to adjust the meta-analysis on the basis of quality.
Summary of risk of bias in studies reporting diagnosis of S. pneumoniae community-acquired pneumonia based on BinaxNOWa
Meta-analysis.The etiologic agent was identified as S. pneumoniae by either positive sputum Gram stain or positive culture of blood, sputum, or other body fluid in 11 studies (reference standard type A), by either positive sputum Gram stain or positive blood culture in 12 studies (reference standard type B), and by positive blood culture alone in 4 studies (reference standard type C). Sensitivity and specificity estimates with respect to the reference standard in each study ranged from 29% to 100% and 61% to 99%, respectively (Fig. 1). Based on the 12 studies that used reference standard A, sensitivity of BinaxNOW-SP was 68.5% (95% CrI, 62.6% to 74.2%) and specificity was 84.2% (95% CrI, 77.5% to 89.3%). Based on the 11 studies that used reference standard B, sensitivity was 60.3% (95% CrI, 46.4% to 74.4%), and specificity was 89.2% (95% CrI, 82.5% to 94.4%). Finally, based on the four studies with reference standard C, sensitivity was 76.7% (95% CrI, 49.0% to 93·0%) and specificity was 79.6% (95% CrI, 56.3% to 93.1%).
Forest plot, showing sensitivities and specificites of BinaxNOW-SP with respect to the reference standard in the studies included in our meta-analysis. Studies are ordered by date in descending order and grouped according to reference classes: A (11 studies), B (12 studies), C (4 studies). Footnotes: a, definite and probable S. pneumoniae pneumonia cases were combined into a single category of S. pneumoniae pneumonia; b, authors' definition of S. pneumoniae included a positive BinaxNOW-SP result. and patients diagnosed solely on the basis of a positive BinaxNOW-SP were reclassified as having false-positive results; c, results from the total number of CAP cases derived from the summation of the authors' categories of “pneumococcal infection, pneumonia,” “pneumococcal infection, probable pneumococcal pneumonia,” “nonpneumococcal infections, pneumonia,” and “unknown etiology pneumonia”; d, data included for those patients with lower respiratory tract infections (LRTIs); e, analysis restricted to a subset of patients with complete data; f, data used from those patients with CAP, and data from control patients were omitted; g, complete data to construct a two-by-two table provided only for positive blood culture as a reference standard; h, results for the total number of CAP cases were derived from the summation of the authors' categories of “pneumococcal bacteremia, with pneumonia” and “nonbacteremic pneumonia, combined subtotal.”
According to our latent class model, based on all 27 studies, the pooled sensitivity of BinaxNOW-SP was 74.0% (95% CrI, 66/6% to 82.3%) and the pooled specificity was 97.2% (95% CrI, 92.5% to 99.8%). Figure 2 provides the summary receiver operating characteristic curve from the different meta-analysis models. As anticipated, assuming the reference standard was perfect resulted in lower estimates of the pooled sensitivity and especially the pooled specificity of BinaxNOW-SP irrespective of the reference standard. The latent class meta-analysis model also provided estimates of the pooled sensitivity and specificity (and 95% credible intervals) of the three reference standards, as follows: (i) reference standard A, sensitivity of 59.4% (43.9% to 76.3%), specificity of 98.6% (95.1% to 99.8%); (ii) reference standard B, sensitivity of 56.2% (35.9% to 80.5%), specificity of 97.4% (93.8% to 99.4%); (iii) reference standard C, sensitivity of 50.3% (24.6% to 78.8%), specificity of 98.3% (91.2% to 99.8%). Figure S2 in the supplemental material shows a forest plot of the latent class model-based estimates of the prevalence of S. pneumoniae and the sensitivities and specificities of BinaxNOW-SP in the studies that were included in the meta-analysis.
Summary receiver operating curves for BinaxNOW-SP with respect to each reference standard, as well as that based on a latent class model adjusting for the imperfect nature of all three reference standards.
The predicted sensitivity and specificity in a new individual study were 74.3% (95% CrI, 48.8% to 90.9%) and 97.2% (95% CrI, 84.4% to 100.0%), respectively. The 95% credible intervals are much wider than with the pooled estimates, reflecting the heterogeneity among the 27 studies even after adjusting to for differences in the reference standard used (see Fig. S2 in the supplemental material).
Meta-regression analyses that separated diagnostic from etiologic studies, prospective from retrospective studies, and studies within versus outside North America or Europe (on the assumption that seasonal cycles and strains of S. pneumoniae, as well as hospital practice, would be similar within Western institutions), gave similar results and no reduction of heterogeneity. We also considered the effect of institution type, e.g., large urban, tertiary care, or university-associated settings. Although it appeared that there was less heterogeneity in diagnostic (compared to etiologic) studies, in studies using a prospective design and in studies based in a university center there were no statistically significant differences in the pooled sensitivity and specificity between the subgroups. We also examined the effects of prior antibiotic use and average severity of pneumonia in the subgroups of studies that reported these variables, but we did not find any significant effects.
Models adjusting for correlation had a similar fit to the latent class meta-analysis model described above. The best-fitting model adjusting for correlation between BinaxNOW-SP allowed for up to 20% of the maximum correlation. This resulted principally in lowering the pooled sensitivity (67.6% [95% CrI, 58.7% to 77.0%]) but did not affect the pooled specificity (98.1% [95% CrI, 91.8% to 99.9%]).
DISCUSSION
Our systematic review of the medical literature identified 27 studies that reported the sensitivity and specificity of the BinaxNOW test for Streptococcus pneumoniae in hospitalized patients with suspected community-acquired pneumonia. Using a meta-analysis model that adjusted for the lack of a perfect reference test, we estimated the pooled sensitivity of BinaxNOW-SP in the detection of S. pneumoniae infection in patients with CAP to be 74.0% (95% CrI, 66.6% to 82.3%) and the pooled specificity to be 97.2% (95% CrI, 92.5% to 99.8%).
A previous meta-analysis by Boulware et al. (47), which was based on 24 studies and assumed a single perfect reference test, estimated that the pooled sensitivity of BinaxNOW-SP was 74% (95% confidence interval [CI], 72% to 77%) and the pooled specificity was 94% (95% CI, 93% to 95%). Although the pooled results from this earlier meta-analysis may appear numerically similar to the values we estimated from our adjusted model, the two analyses had only 16 studies in common. In addition, Boulware et al. included only those patients in whom etiology had been established, excluding those with an unknown organism, while we included all patients with clinically suspicious CAP, including those with an unknown organism. Finally, the older meta-analysis model used by Boulware et al. did not adjust for the negative correlation between the sensitivity and specificity of BinaxNOW-SP or the imperfect reference standard. This is reflected in the wider credibility intervals around the pooled and predicted sensitivity and specificity obtained with our model.
Compared to more-naive diagnostic meta-analysis models (48), our model allowed for (i) correlations between sensitivity and specificity across studies due to differences in thresholds or diagnostic accuracies, (ii) heterogeneity in BinaxNOW-SP performance between studies due to observed study-level covariates as well as unexplained variation, (iii) the imperfect nature of the reference standard, which would result in higher estimates of both sensitivity and specificity, (iv) three different types of reference standards in individual studies, and (v) heterogeneity in the performance of each type of reference standard across studies.
Based on the input of our expert consultant (our coauthor Marty Teltscher) and a nonsystematic review of the literature (2–4, 6, 7, 49), we determined plausible ranges for the sensitivity and specificity of the three reference standards: reference standard A, sensitivity from 40 to 70%, specificity from 80 to 100%; reference standard B, sensitivity from 30 to 60%, specificity from 80 to 100%; reference standard C, sensitivity from 10 to 40% and specificity from 90 to 100%. These ranges agree well with the estimates of sensitivity and specificity of the reference standard from our latent class model, thus supporting its validity.
Possibility of risk of bias within individual studies.Recent antibiotic use is known to reduce the diagnostic yield of cultures (7, 9), with S. pneumoniae blood cultures sensitive even to a single dose of antibiotic (7). In the absence of an effect of antibiotics with the BinaxNOW-SP test, this would artificially increase the rate of discordant results for BinaxNOW-SP-positive and culture-negative samples. The effect of prior antibiotic treatment on the sensitivity of the BinaxNOW assay is unclear, with some studies reporting decreased sensitivity (4, 36, 39, 41) while others did not (4, 36, 40, 50).
Some studies differentiated between definite from possible S. pneumoniae, with the former category being restricted to samples from normally sterile sites. For the purposes of this analysis, we combined the two, which potentially increased the rate of discordant culture-positive, BinaxNOW-SP-negative results.
Clinical experience with BinaxNOW-SP.Two randomized controlled trials (51, 52) and two observational studies (24, 33) have examined the impact of use of BinaxNOW-SP in treatment decision-making on outcomes in hospitalized patients. In the randomized controlled trials, patients were randomized to empirical or targeted therapy, and those receiving targeted therapy who tested positive with BinaxNOW-SP received therapy specific for S. pneumoniae. Neither study showed a difference in important clinical outcomes, although one showed more relapses in the targeted therapy group. Given that patients had to be randomized into the targeted group and test positive via BinaxNOW-SP, the studies were underpowered to detect a difference.
Conclusions.The higher pooled sensitivity of BinaxNOW-SP compared to culture, and also its high specificity, suggest it could be a useful addition to the workup for diagnosis of community-acquired pneumonia. While this is an important finding, the current work does not address whether rapid diagnosis with BinaxNOW-SP would impact the initial management of CAP patients or changes to the initial management of CAP patients. More research is needed regarding the potential impact of BinaxNOW-SP on clinical practice, particularly in the context of other interventions, such as an antibiotic stewardship program, and taking into account the cost-effectiveness of antibiotics and the longer-term costs of antibiotic resistance and antibiotic-associated Clostridium difficile infection.
ACKNOWLEDGMENTS
We acknowledge the assistance of Ian Schiller, Division of Clinical Epidemiology, McGill University Health Centre Research Institute, Montréal, Quebec, Canada, for assistance with the analyses related to the conditional dependence models and SROC plots.
Nandini Dendukuri holds a salary award from the Fonds de la Recherche en Santé du Québec. This study was originally undertaken in response to a request to the Technology Assessment Unit of the McGill University Health Centre.
We have no conflicts of interest to declare.
FOOTNOTES
- Received 16 January 2013.
- Returned for modification 3 March 2013.
- Accepted 2 May 2013.
- Accepted manuscript posted online 15 May 2013.
Supplemental material for this article may be found at http://dx.doi.org/10.1128/JCM.00137-13.
- Copyright © 2013, American Society for Microbiology. All Rights Reserved.
