ABSTRACT
Several point-of-care (POC) molecular tests have received emergency use authorization (EUA) from the Food and Drug Administration (FDA) for the diagnosis of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The test performance characteristics of the Accula (Mesa Biotech) SARS-CoV-2 POC test need to be evaluated to inform its optimal use. The aim of this study was to assess the test performance of the Accula SARS-CoV-2 test. The performance of the Accula test was assessed by comparing results of 100 nasopharyngeal swab samples previously characterized by the Stanford Health Care EUA laboratory-developed test (SHC-LDT), targeting the envelope (E) gene. Assay concordance was assessed by overall percent agreement, positive percent agreement (PPA), negative percent agreement (NPA), and Cohen’s kappa coefficient. Overall percent agreement between the assays was 84.0% (95% confidence interval [CI], 75.3 to 90.6%), PPA was 68.0% (95% CI, 53.3 to 80.5%), and the kappa coefficient was 0.68 (95% CI, 0.54 to 0.82). Sixteen specimens detected by the SHC-LDT were not detected by the Accula test and showed low viral load burden, with a median cycle threshold value of 37.7. NPA was 100% (95% CI, 94.2 to 100%). Compared to the SHC-LDT, the Accula SARS-CoV-2 test showed excellent negative agreement. However, positive agreement was low for samples with low viral load. The false-negative rate of the Accula POC test calls for a more thorough evaluation of POC test performance characteristics in clinical settings and for confirmatory testing in individuals with moderate to high pretest probability of SARS-CoV-2 who test negative on Accula.
INTRODUCTION
The importance of diagnostic testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has been strongly emphasized by both the World Health Organization (WHO) and the United States Centers for Disease Control and Prevention (CDC) (1–3). In the United States, most SARS-CoV-2 testing has been conducted using high-complexity molecular-based laboratory-developed tests (LDTs) that have received emergency use authorization (EUA) from the Food and Drug Administration (FDA) in centralized laboratories certified to meet the quality standards of the Clinical Laboratory Improvement Amendments of 1988 (CLIA) (4, 5). Currently, 3 CLIA-waived point-of-care tests (POCT) are EUA approved for SARS-CoV-2 testing: the Cepheid Xpert Xpress, the Abbott ID NOW, and the Mesa Accula (6). Compared to high-complexity LDTs, POCT have the potential to reduce turnaround time of testing, optimize clinical management, and increase patient satisfaction (7). The Accula SARS-CoV-2 test is a POCT that requires only 30 min from sample to answer and utilizes the existing palm-sized Accula dock system originally developed for rapid influenza and respiratory syncytial virus testing. Despite the multiple potential benefits of POC assays, concern has been raised regarding their lower sensitivity for COVID-19 diagnosis than standard high-complexity molecular-based tests (8–10). It remains unclear whether this decreased sensitivity is due to test validation studies being limited to in silico predictions and contrived samples using reference materials, as is the case currently for the Accula SARS-CoV-2 test.
The aim of this study was to evaluate the test performance characteristics of the Accula SARS-CoV-2 test in a clinical setting against a high-complexity reference standard.
MATERIALS AND METHODS
Nasopharyngeal (NP) swabs were collected in viral transport medium (VTM) or saline from adult patients from Stanford Health Care (SHC) and from pediatric and adult patients from surrounding hospitals in northern California. Testing for this study was performed at the SHC Clinical Virology Laboratory using samples collected between 7 April 2020 and 13 April 2020. The same NP specimen was used for both the reference assay (tested first) and Accula test (tested subsequently) for comparison. Clinical data on the presence of symptoms were extracted from the electronic medical record for individuals presenting for care at SHC or an affiliated hospital. This study was approved by the Stanford Institutional Review Board (protocol number 48973).
RT-PCR assays.The reference assay for this study was the Stanford Health Care Clinical Virology Laboratory real-time reverse transcriptase PCR LDT (SHC-LDT) targeting the E gene (11–13). The Accula SARS-CoV-2 POCT (Mesa Biotech, Inc., San Diego, CA) is a sample-to-answer nucleic acid amplification test that can yield a diagnostic result within 30 min of specimen collection. This test uses reverse transcription-PCR (RT-PCR) to target the nucleocapsid protein (N) gene and is read out via lateral flow (Fig. 1) (14). The manufacturer’s instructions comprise the following steps: collection of NP swab, lysis of viral particles in SARS-CoV-2 buffer, transfer of nucleic acid solution to a test cassette that contains internal process positive and negative controls, reverse transcription of viral RNA to cDNA, nucleic acid amplification, and detection by lateral flow. Due to biosafety regulations and hospital-mandated protocols for sample collection at SHC, NP swabs were directly placed into VTM or saline at the patient bedside after collection. Each test was performed at the laboratory, where a volume of 10 μl of VTM or saline was transferred to 60 μl of SARS-CoV-2 buffer and added to the test cassette. These steps were performed within a biosafety cabinet to protect against aerosolization. All remaining steps were followed per the manufacturer’s instructions (14). Testing was repeated once for invalid results on initial testing, and the second result was interpreted as final if valid.
Images of the Accula SARS-CoV-2 lateral-flow readout. (A) Positive patient specimen; (B) negative patient specimen. C, internal positive process control; T, SARS-CoV-2 test; NC, internal negative process control.
Statistics.Overall percent agreement, positive percent agreement (PPA), negative percent agreement (NPA), and associated 95% confidence intervals (CI) were calculated. Cohen’s kappa coefficient (κ) of qualitative results (detected/nondetected) between the Accula SARS-CoV-2 test and the SHC-LDT was also calculated with 95% CI. Cohen’s kappa values between 0.60 and 0.80 were interpreted to indicate substantial agreement, and kappa values above 0.81 were interpreted as excellent agreement (15). All analyses were performed using Stata version 15.1.
RESULTS
We included 100 samples (50 positive, 50 negative) previously tested by the SHC-LDT and subsequently tested with the Accula SARS-CoV-2 POCT. A total of 45 samples were collected in VTM (21 positive, 24 negative), and 55 were collected in saline (29 positive, 26 negative). Data on the presence of clinical symptoms were available for 26/50 individuals with positive results. Of these, 24 individuals were symptomatic, and 2 were asymptomatic and tested for follow-up. Positive samples determined by the SHC-LDT included a range of cycle threshold (CT) values, with a median CT of 28.2 (interquartile range [IQR], 20.4 to 36.3). A total of 3 samples resulted as invalid on initial testing by Accula were retested once. One of these samples was positive for SARS-CoV-2 on repeat testing, and the other 2 samples were negative.
The Accula SARS-CoV-2 test correctly identified 34/50 positive samples and 50/50 negative samples, corresponding to an overall percent agreement of 84.0% (95% CI, 75.3 to 90.6%), (Table 1). The positive percent agreement was 68.0% (95% CI, 53.3 to 80.5%), the Cohen’s kappa coefficient was 0.74 (95% CI, 0.61 to 0.87), indicating substantial agreement, and the NPA was 100% (95% CI, 92.9% to 100%). The positive percent agreement varied by CT values and transport medium used, with higher performance in samples with low-CT samples and in VTM (Table 2). The 34 samples that were detected by both assays had a median CT value of 23.5 (IQR, 19.7 to 28.7). The 16 samples that were positive by SHC-LDT but negative by the Accula test had a median CT value of 37.7 (IQR, 36.6 to 38.2), consistent with lower viral loads. Restricting the analysis to the 24 symptomatic individuals, the positive percent agreement was 66.7% (95% CI, 44.7% to 84.4%), and the median CT value was 26.5 (IQR, 19.8 to 37.3). The lateral-flow read-out on the Accula test was considered easy to interpret for all samples, with the exception of a single known positive sample that showed a faint positive test line. Repeat testing of this sample showed the same faint test line and was interpreted as positive.
Comparison of the SHC-LDT for SARS-CoV-2 and the Accula SARS-CoV-2 PCR test
PPA of the Accula SARS-CoV-2 PCR test compared to the SHC-LDT for SARS-CoV-2, stratified by CT values and transport medium type
DISCUSSION
Although SARS-CoV-2 testing capacity has improved in many countries, a global shortage of diagnostic infrastructure and consumable reagents has limited testing efforts. Point-of-care tests offer the potential advantages of improved access to testing and reduced turnaround time of results. Of the multiple EUA assays for diagnosis of SARS-CoV-2, only the Xpert Xpress, the ID NOW, and the Accula tests are CLIA-waived (6). Recent data support the test performance of the Cepheid Xpert SARS-CoV-2 assay, with agreement of over 99% compared to high-complexity EUA assays (8, 16, 17). In contrast, some studies have raised concerns regarding the diagnostic accuracy of ID NOW, with positive percent agreement ranging from 75% to 94% compared to reference assays (8–10, 18). Given the poor diagnostic performance of ID NOW and uncertainty regarding the availability of Xpert Xpress cartridges, the Accula system has been touted as an interesting POCT alternative, but data were previously lacking on its clinical performance. In this study, we showed that, similar to ID NOW, the Accula SARS-CoV-2 test has a lower sensitivity for diagnosis of COVID-19 than an EUA LDT. The false negatives obtained from the Accula SARS-CoV-2 test were predominantly observed with low-viral-load specimens. The exact reason for the low sensitivity of the Accula test is unclear at present. The primer and probe sequences are not publicly available for this assay to identify which region of the N gene is targeted; previous comparative data support similarly high sensitivity of the N2 and E gene targets but lower sensitivity of the N3 target for the diagnosis of SARS-CoV-2 (19).
Given the accumulating evidence on lower diagnostic performance with 2 of the 3 CLIA-waived SARS-CoV-2 assays, it is now important to consider how best to integrate these tests in diagnostic workflows and to identify groups of individuals for whom POCT use should be prioritized. Furthermore, reagents and kits have been limited, which limits POCT capacity. Certain groups, such as individuals requiring urgent preoperative assessment, including transplantation, patient-facing symptomatic health care workers, and individuals waiting for enrollment in a SARS-CoV-2 therapeutic trial, have been identified as key groups in whom to prioritize POCT. However, for each of these scenarios and depending on the POCT used, the risk of missing a case due to low sensitivity must be considered. In individuals with moderate to high pretest probability of SARS-CoV-2, reflex testing of negative samples on a separate EUA assay should be performed. Education of health care professionals on the limitations of SARS-CoV-2 POCT should also be implemented to ensure the optimal interpretation and management of negative results.
Our study has several limitations. First, NP swabs were placed in VTM or saline at the patient bedside before loading the Accula test cassette, which may have decreased sensitivity by diluting the viral inoculum. Although this is discordant with the best recommended practice by the manufacturer, it is in line with the practice at multiple institutions with clinical laboratories that have assessed SARS-CoV-2 POCT due to biosafety concerns from the risk of aerosolization (8–10, 18, 20). Second, it is possible that the use of saline instead of VTM led to poorer performance of the Accula. However, aliquots from the same sample were used for parallel testing with the EUA method, which minimizes sources of variation, and represents a pragmatic comparison given widespread VTM shortages. Finally, the lateral-flow read-out of the Accula test is generally easy to interpret; however, faint lines may be more challenging to interpret and lead to result discrepancies.
In summary, this study demonstrated that the Accula POCT lacks sensitivity compared to a reference EUA SARS-CoV-2 LDT. Careful consideration should be given to balance the potential advantages of rapid POCT to lower diagnostic accuracy. Individuals with moderate to high pretest probability who initially test negative on the Accula test should undergo confirmatory testing with a separate EUA assay.
ACKNOWLEDGMENTS
We thank the members of the Stanford Health Care Clinical Virology Laboratory, Department of Emergency Medicine, and Department of Medicine, Division of Infectious Disease, for their hard work and dedication to patient care.
We have no conflicts of interest to declare.
FOOTNOTES
- Received 11 May 2020.
- Returned for modification 21 May 2020.
- Accepted 26 May 2020.
- Accepted manuscript posted online 27 May 2020.
- Copyright © 2020 American Society for Microbiology.
