ABSTRACT
As the coronavirus disease 2019 (COVID-19) pandemic second wave is emerging, it is of the upmost importance to screen the population immunity in order to keep track of infected individuals. Consequently, immunoassays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) with high specificity and positive predictive values are needed to obtain an accurate epidemiological picture. As more data accumulate about the immune responses and the kinetics of neutralizing-antibody (nAb) production in SARS-CoV-2-infected individuals, new applications are forecast for serological assays such as nAb activity prediction in convalescent-phase plasma from recovered patients. This multicenter study, involving six hospital centers, determined the baseline clinical performances, reproducibility, and nAb level correlations of 10 commercially available immunoassays. In addition, three lateral-flow chromatography assays were evaluated, as these devices can be used in logistically challenged areas. All assays were evaluated using the same patient panels in duplicate, thus enabling accurate comparison of the tests. Seven immunoassays examined in this study were shown to have excellent specificity (98 to 100%) and good to excellent positive predictive values (82 to 100%) when used in a low (5%)-seroprevalence setting. We observed sensitivities as low as 74% and as high as 95% at ≥15 days after symptom onset. The determination of optimized cutoff values through receiver operating characteristic (ROC) curve analyses had a significant impact on the diagnostic resolution of several enzyme immunoassays by increasing the sensitivity significantly without a large trade-off in specificity. We found that spike-based immunoassays seem to be better correlates of nAb activity. Finally, the results reported here will add to the general knowledge of the interlaboratory reproducibility of clinical performance parameters of immunoassays and provide new evidence about nAb activity prediction.
INTRODUCTION
A novel variant of coronavirus termed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was promptly identified as the etiologic agent of a pneumonia outbreak in Wuhan, China, in December 2019 (1). This zoonotic virus spread rapidly among the immunologically naive human populations of the world in only a few months, creating the biggest pandemic crisis since the swine flu in 2009-2010 (2). SARS-CoV-2 is classified in the beta-coronavirus group upon phylogenetic analysis, sharing high genomic identity (96%) with the bat coronavirus RaTG13 and 80% nucleotide identity with the 2002-2003 pandemic SARS-CoV (3). Conversely, its genetic relatedness to other human beta-coronaviruses, such as OC-43 and HKU-1, and the human alpha-coronaviruses 229E and NL63 is significantly lower (4). The laboratory diagnosis of SARS-CoV2 infection is routinely made by the detection of viral RNA in nasopharyngeal samples in symptomatic patients by real-time reverse transcriptase PCR assays (5). The sensitivity of the molecular assays is good in the acute phase of the disease (6, 7), but they cease to be useful once the virus is cleared by the immune response of the patient, around 2 weeks following the onset of symptoms (8). The lack of sensitivity of the molecular assay as a function of time could impair the correct clinical assignment of patients, and this could contribute to the spread of the virus. In addition, some reports indicated that asymptomatic carriers who are not routinely tested could also contribute to spreading the virus (9).
Serological methods were proposed for use as screening tools for the epidemiological assessment of the immune profiles of the general population (10). Serological assays could also help to assess the immune status of convalescent-phase sera used to treat severely ill patients or for the validation of the effectiveness of vaccination trials. While serology is an indirect assay, it can provide useful insights about the exposition of asymptomatic individuals by the detection of specific antibodies. Thus, the challenge posed by this novel viral variant is the selection of highly specific antibody detection assays which could discriminate this virus from other related viruses and also be sensitive enough to be useful in the acute phase of the disease. Because of the necessity to screen immune profiles of several thousands of individuals in a short period of time, the validation of several assays available on high-throughput serology analyzers is a prerequisite to accomplish this task. In this study, we sought to evaluate the clinical performance and the virus neutralization activity prediction properties of 10 high-throughput serological enzyme immunoassays (EIA). We also evaluated three lateral flow rapid antibody detection devices, since these tests could be useful for screening purposes in distant settings.
MATERIALS AND METHODS
Study design.This multicenter evaluation study, initiated by the Québec provincial health authorities, was a collective effort led by the Laboratoire de Santé Publique du Québec (LSPQ). A total of 10 clinical laboratories were selected for this study, including the reference laboratory (LSPQ). These laboratories were grouped in separate testing sites according to their affiliations to university hospital centers, as follows: site 1, Centre Hospitalier de l’Université de Montréal (CHUM) and Centre Hospitalier Universitaire Sainte-Justine; site 2, Centre Hospitalier de l’Université Laval, Hôpital de l’Enfant-Jésus, and Hôpital du Saint-Sacrement; site 3, Hôpital Cité de la Santé; site 4, McGill University Health Center–Glen Site and St. Mary’s Hospital; site 5, Centre Hospitalier de l’Université de Sherbrooke; and site 6, LSPQ.
This study was conducted in accordance with the Declaration of Helsinki in terms of informed consent and approval by an appropriate institutional board and with the regulations concerning ethical reviews in Quebec provincial jurisdiction, particularly those specified in the Civil Code (http://legisquebec.gouv.qc.ca/fr/ShowDoc/cs/CCQ-1991) and in subsequent internal review board practice. Informed consent was obtained for all participating subjects, and the study was approved by Quebec Public health authorities.
Ten commercial antibody assays running on high-throughput immunoanalyzer platforms currently found in hospital laboratories and three lateral-flow chromatographic immunoassays (LFCIA) were selected for the study and tested at six designated sites. Each of the assays was tested in two separate laboratories to assess the reproducibility of the data except for the LFCIA which were tested only once at site 6. All of the selected assays had received the emergency usage authorization (EUA) by the Food and Drug Administration of the United States, while seven commercial enzyme immunoassays (Abbott IgG, Bio-Rad total antibody, DiaSorin IgG, Euroimmun IgG, Ortho-Clinical total antibody, Roche total antibody, and Siemens total antibody) had been authorized for use in clinical settings by Health Canada.
Patient cohorts.SARS CoV-2 samples (sensitivity panel; n = 176) were collected from patients with various clinical symptoms, including 156 sera and 20 plasma samples (Table 1). All 176 patients were confirmed positive for SARS CoV-2 infection by real-time reverse transcription-PCR (RT-PCR) on nasopharyngeal specimens; 170 of these patients were symptomatic and six were asymptomatic. The average age of the patients was 61 years. Fifty-six (32%) were male, 100 (57%) were female, and 20 (11%) were of unspecified gender. Forty-six samples were obtained at 1 to 7 days, 36 between 8 and 14 days, 60 between 15 and 34 days, and 34 more than 5 weeks after symptom onset. For asymptomatic patients as well as few symptomatic patients without a precise date of symptom manifestations, the time of onset was estimated as 48 h before the first positive SARS CoV-2 RT-PCR result. Among the patients with complete clinical information (n = 48), 32 (67%) were hospitalized, 29 (60%) required oxygen support, 8 (17%) were admitted to intensive care units, and 42 (88%) reported one or more symptoms (Table 1). The reported symptoms included one or more of the following: fever, myalgia, anosmia, ageusia, nasal congestion, pharyngitis, anorexia, and diarrhea.
Baseline characteristics of SARS CoV-2 patients (n = 176)
The first 67 SARS-CoV-2 samples collected for the sensitivity panel were selected for virus neutralization activity measurements. Twenty-one samples were obtained at 1 to 7 days, 14 between 8 and 14 days, 26 between 15 and 34 days, and 6 more than 5 weeks after symptom onset.
Specificity and cross-reactivity of the serological assays were evaluated using samples from patients with other laboratory-confirmed acute infections (n = 64): common cold human coronavirus (n = 20), adenovirus (n = 8), enterovirus (n = 2), influenza virus A (n = 3), and B (n = 1), parainfluenza virus 2/3 (n = 2), rhinovirus (n = 6), syncytial respiratory virus (n = 1), dengue virus (n = 20), and varicella-zoster virus (n = 1). In addition, patient samples with autoimmune antibodies (n = 51) and those collected from the general population prior to January 2020 (n = 38) were tested. Samples collected in 2020 from suspected COVID-19 patients with a negative RT-PCR result (n = 13: April, n = 11, and May, n = 2) were also included for specificity assessment. The first confirmed locally acquired COVID-19 cases in Quebec were in March 2020.
A panel to assess test sensitivity was prepared by collecting serum samples from confirmed COVID-19 patients prospectively in sufficient amounts to accommodate all serology platforms at each site. Because of the small volumes of several patient samples dedicated to the specificity panel, we opted to pool two to four samples from patients sharing the same laboratory findings. With the exception of sera obtained from patients with autoimmune diseases, all other samples were prepared by a pooling procedure.
Evaluation of serological assays.Testing of the commercial immunoassays using various reagent chemistries (enzyme immunosorbent assay [ELISA], chemiluminescence immunoassay [CLIA], electrochemiluminescence immunoassay [ECLIA], and chemiluminescent microparticles immunoassay [CMIA]) and different SARS-CoV-2 antigens (nucleocapsid, spike, and spike domains/epitope) was performed according to the manufacturer’s instructions (Table 2). Qualitative results were derived based upon the clinically based cutoff values determined by the manufacturers and described in the package inserts. All platforms used their built-in associated programmed protocols, calibrators, and controls for quality control assessment for each assay. All assays were validated to utilize both serum and plasma samples. Five assays were evaluated for the detection of IgG against SARS-CoV-2, while four assays were examined for both IgM and IgG. Finally, one assay was evaluated for the detection of IgA. All sites received the same panels of samples for performance assessment. All assays were performed in two separate sites in order to measure the interlaboratory reproducibility. Qualitative results (positive, negative, and equivocal) were inferred using the manufacturer’s specified cutoff values. Equivocal results were considered negative in the calculation of sensitivity and specificity.
Selected serological assays and platformsa
Three lateral-flow rapid antibody detection devices were tested per the manufacturer’s indications at only one site (site 6). All tests were designed to detect both IgM and IgG antibodies in serum or plasma samples. All results were recorded by visual inspection of reactive bands by two technologists. Quality control was assessed by the observation of a colored band on the migration matrices at the control position. A positive result was scored by one technologist when a strong colored band for either IgM and/or IgG appeared at the expected positions, while weak signals needed a consensus between two technologists for a final positive result. Invalid assays with a negative-control band were repeated until a valid result was obtained. We obtained 13 and six invalid tests for BTNX and Biocan, respectively, and none for NADAL.
Virus neutralization activity assessment of SARS-CoV-2 patient sera.Sixty-seven COVID-19 patients’ samples in our sensitivity panel were assessed for virus neutralization activity using a pseudovirus neutralization protocol described previously (11–14). Target cells were infected with single-round luciferase-expressing lentiviral particles. Briefly, 293T cells were transfected by the calcium phosphate method with the lentiviral vector pNL4.3.Luc.R-E-(NIH AIDS Reagent Program) and a plasmid encoding SARS-CoV-2 spike at a ratio of 5:4. Two days posttransfection, cell supernatants were harvested and stored at −80°C until use. 293T-ACE2 target cells were seeded at a density of 1 × 104 cells/well in 96-well luminometer-compatible tissue culture plates (Perkin Elmer) 24 h before infection. Recombinant viruses in a final volume of 100 μl were incubated with various serum dilutions (1/50, 1/250, 1/1,250, 1/6,250, and 1/31,250) for 1 h at 37°C and were then added to the target cells followed by incubation for 48 h at 37°C; cells were lysed by the addition of 30 μl of passive lysis buffer (Promega) followed by one freeze-thaw cycle. A LB942 TriStar luminometer (Berthold Technologies) was used to measure the luciferase activity of each well after the addition of 100 μl of luciferin buffer (15 mM MgSO4, 15 mM KPO4 [pH 7.8], 1 mM ATP, and 1 mM dithiothreitol) and 50 μl of 1 mM d-luciferin potassium salt (Thermo Fisher Scientific). The neutralization half-maximal inhibitory dilution (ID50) represents the serum dilution that inhibited 50% of the infection of 293T-ACE2 cells by recombinant viruses.
Calculations and statistical analyses.Contingency tables were used to compute sensitivity (Se), specificity (Sp), positive predictive values (PPV), and negative predictive values (NPV) as a function of prevalence and total agreement. All statistical analyses were calculated using GraphPad Prism version 8.4.3 (GraphPad, San Diego, CA, USA). The exact binomial proportion 95% confidence intervals (Clopper and Pearson method) were calculated for sensitivity and specificity parameters. Spearman correlation analysis was performed to assess the neutralization prediction capabilities of the commercial assays using data generated by the neutralization and enzyme immunoassays. The determination of optimized cutoff values of the serological assays was accomplished using receiver operating characteristic (ROC) curves and the Youden index. The interrater agreement (Cohen’s kappa) coefficient and total agreement were used to compare the interlaboratory reproducibility of enzyme immunoassays.
RESULTS
Clinical performances.As expected from other studies, the Se of all serological assays tested was relatively low in the first 2 weeks following the onset of symptoms and gradually increased afterwards (Table 3; also, see Fig. S1 in the supplemental material). The ELISA-based assays EuroImmun IgA and Bio-Rad total antibody were the most sensitive tests in the acute phase of COVID-19 (0 to 7 days after symptom onset) with sensitivities of 37% to 39% and 24% to 26%, respectively, while all the chemiluminescence-based assays were less sensitive in the same time frame (9% to 20%). In the 8- to 14-day period, all the assays showed increased sensitivities varying from 44% to 83%, with the ELISA-based assays still being the most sensitive (81% to 83%) compared to chemiluminescent assays. In the 15- to 34-day period, the sensitivity continued to increase for all assays (72% to 92%) with the Bio-Rad total antibody (87%), Ortho-Clinical total antibody (85% to 88%), and Euroimmun IgA (85%) being the most sensitive enzyme immunoassays.
Analytical performance for enzyme immunoassays and lateral-flow chromatographic assays
The maximum sensitivities observed following the 2-week acute window (≥15 days) were achieved by the Bio-Rad total-antibody assay, with an Se of 90% (confidence interval [CI], 83 to 96), and the Ortho-Clinical Diagnostics total-antibody assay, with an Se of 93% (CI, 85 to 97). The NADAL rapid antibody detection device was even more sensitive in the same time period, with an Se of 95% (CI, 88 to 98), while another rapid test, the BTNX assay, also yielded good Se (91%; CI, 84 to 96). Detection of antibodies from all COVID-19 patients was seemingly achieved 5 weeks after the onset of symptoms by only two assays, namely, the Ortho-Clinical total-antibody and NADAL IgM/IgG assays (100%; CI, 90 to 100). In fact, the NADAL rapid device was the only assay which met the minimal acceptable target Se values of ≥95% 2 weeks after the onset of symptoms set by Health Canada (https://www.canada.ca/en/health-canada/services/drugs-health-products/covid19-industry/medical-devices/testing/serological/notice-sensitivity-specificity-values.html). This level of Se was achieved only much later (≥5 weeks after symptom onset) by the other enzyme immunoassays, such as the Roche total-antibody, Bio-Rad total-antibody, Siemens total-antibody, Ortho-Clinical IgG, EuroImmun IgA, and BTNX IgM/IgG rapid tests. Antibody type-specific Se variations were also noted; for example, the EuroImmun IgA assay, although not the most sensitive overall, was the most sensitive (37 to 39%; CI, 23 to 55) of all enzyme immunoassays tested (9 to 26%) in the first week. Furthermore, the Ortho-Clinical assays showed different Se values based upon the antibody kit version, with the total-antibody assay being more sensitive at all time periods than its IgG counterpart. Interestingly, the NADAL and BTNX assays showed equivalent sensitivities to most enzyme immunoassays in all time periods. However, the NADAL assay seems to be the most sensitive of all rapid lateral-flow devices.
The Sp of all assays was assessed by testing patients diagnosed with an disease unrelated to COVID-19 (infectious or autoimmune) and patients from the pre-COVID-19 general population. The chemiluminescence-based assays (Abbott, DiaSorin, Beckman, Ortho-Clinical, Siemens, and Roche) were the most specific platforms, reaching 98% to 100% specificity. According to our results, these assays met the specificity threshold fixed at ≥98% by Health Canada. Conversely, the enzyme-linked immunosorbent assays (Bio-Rad and Euroimmun) were less successful than the former at excluding non-COVID-19 immune profiles, with Sp values ranging from 77% to 97%. Despite the fact that the Euroimmun IgA assay showed good sensitivity, it lacked Sp compared to all other tests. The rapid lateral-flow devices NADAL (98%) and Biocan (99%) had excellent specificities and met the Health Canada threshold specificity criteria.
Positive reactions were noted with samples from the pre-COVID-19 general population (n = 38) for the Bio-Rad assay (5/38) and the Euroimmun IgA (6/38) and IgG (2/38) assays (Table 4). The BTNX assay also yield a positive reaction (1/38) within this population. Of interest, three assays obtained a reactive result with one sample from the COVID-19 RT-PCR negative population (n = 13), namely, the Bio-Rad (1/13), EuroImmun IgA (3/13), and IgG (1/13) assays. No positive reactions with control populations were noted for the chemiluminescence-based platforms.
Cross-reactivity of enzyme immunoassays and lateral flow devices with sera from pre-COVID general population and patients diagnosed with unrelated disease
Cross-reactions with sera from patients of the other disease panel were mostly observed for the Euroimmun IgA and the BTNX assays (Table 4). The Euroimmun IgA assay was reactive against 4 of 20 seasonal coronavirus samples and 5 of 23 samples from patients diagnosed with other respiratory virus infections. The BTNX assay exhibited a lack of Sp when challenged with autoimmune sera, with 35% (18/51) of these samples yielding a positive reaction. The interference of autoantibodies was also observed for the EuroImmun IgA assay but to a lesser extent (23%; 9/39). Interestingly, nonspecific reactions were noted for six serological assays with samples containing antibodies against the dengue virus.
The PPV and NPV as a function of COVID-19 prevalence are presented in Table 5. These values were calculated using the data from ≥15 days after symptom onset. The four assays which are 100% specific (at two sites) are expected to accurately identify all COVID-19 patients at every specified prevalence level. However, the relatively small population that provided samples for our study precludes such interpretation. In fact, the 95% confidence interval calculated for 100% Sp indicates that the true value is probably between 98 and 100%. (Table 3). A decrease of only 1% in Sp has a profound effect on the PPV in a low-prevalence setting. As expected, the PPVs of all assays which are not 100% specific are quite low at 1% prevalence and increase as the prevalence rises. At 5% prevalence, the best PPV values were those of the Beckman (100%), Abbott (100%), DiaSorin (100%), Roche (100%) and Ortho-Clinical total-antibody (100%) assays, followed by Siemens (82%), Ortho-Clinical IgG (82%), and Biocan (82%).
Positive and negative predictive values of serological assays at ≥15 days after symptom onset as a function of COVID-19 seroprevalence
The interlaboratory reproducibility of enzyme immunoassays was assessed by the determination of the total agreement percentages and kappa values using the sample panel for sensitivity evaluation. The total agreement was ≥94% for all assays but Euroimmun IgA (90%). The kappa coefficient, which is a better statistic to compare the performance of different laboratories, yielded values between 0.763 and 1.00, which indicates very good to perfect agreement between two observers (15). The Abbott assay was the only one to reach 100% reproducibility between distant sites.
Virus-neutralizing-activity predictive ability of enzyme immunoassays.In order to determine whether the selected serological assays could detect antibodies with virus neutralization properties, we performed nonparametric correlation studies with 67 COVID-19 patient samples with titrated virus neutralization activities (ID50). Pearson’s correlation analysis was accomplished using the signal-to-cutoff (S/CO) values of the EIAs and the ID50 values obtained with the virus neutralization assay for each sample (Fig. 1). Only six samples with binding antibodies but no neutralization antibodies were found for all assays (Fig. 1, upper left quadrants of the graphs). In addition, 14 samples with no binding or neutralization antibodies were identified in all assays (Fig. 1, lower left quadrants (and 25 samples were positive for both binding and neutralization antibodies for all assays (Fig. 1, upper right quadrants). The remaining 22 samples showed different distribution patterns between assays. Spearman r correlation statistics between the serological assays and the neutralization assay are presented in Fig. 2 for each assay at both sites. No perfect correlation was found among all enzyme immunoassays and the neutralization assay (assessed by neutralization titer, i.e., the highest dilution at which the input virus signal was reduced by 50% [NT50]). However, we noted that assays designed with spike protein or domains derived from spike tend to be better correlates of neutralizing activities (0.67 to 0.76) than nucleocapsid-based assays (0.53 to 0.64). A Mann-Whitney test between paired data revealed a statistically significant difference (P = 0.0095) between spike protein and nucleocapsid-based assays (Fig. 3). The sole exception to this finding was the DiaSorin assay, which utilizes spike domains 1 and 2 (S1 and S2 domains) and was not found to be significantly different from the nucleocapsid-based assays. Correlation analyses between all immunoassays’ S/CO values were also performed. As expected, higher correlations were observed between assays using similar antigens.
Evaluation of the virus neutralization predictive ability of 10 enzyme immunoassays. Spearman correlation analysis of virus neutralization ID50 and immunoassay S/CO values for 67 COVID-19 samples collected between 1 and 43 days after symptom onset. Dotted lines represent the cutoff values for each assay. Data points are presented for two testing sites.
Impact of antigen type on the correlation between antibody binding activity and virus neutralization antibody titers. Spearman r values derived from correlation analysis of ID50 values versus S/CO values for enzyme immunoassays (67 paired data sets from the Se panel). Ag, antigen; N, nucleocapsid; S1, spike domain 1; S1-S2, spike domains 1 and 2; RBD, spike receptor binding domain; S, spike.
Spike-based immunoassays are better predictor of virus neutralization activity. Plotting of Spearman r statistics versus antigen type used by enzyme immunoassays. **, significantly different at a P value of 0.0095; ns, not significant.
Receiver operating characteristic curve analysis.We used ROC curve analyses and the Youden index method at two different time points to optimize either the Se or the Sp. Keeping in mind that specificity and PPV are critical parameters in a low-prevalence setting, we tried to set the cutoff levels according to the minimal limit values recommended by Health Canada (Se ≥ 95%; Sp ≥ 98%; ≥15 days). In some specific clinical cases, such as multisystem inflammatory syndrome in children (16), antibody detection assays are used as adjunct tests to molecular testing for patients with a negative SARS-CoV-2 RT-PCR result (17). However, most of the serological tests are poorly sensitive in the acute phase of COVID-19. Therefore, the optimization of Se at the expense of a slight decrease in Sp could yield enhanced assays to support the diagnosis of special clinical cases.
We performed the ROC curve analyses on assays exhibiting excellent Sp. Assays with lower Sp did not have enough flexibility to increase the Se significantly, since the optimization process is a trade-off mechanism. Also, only the Ortho-Clinical total-antibody assay was analyzed, since its IgG counterpart was less sensitive. The optimized cutoff values and the calculated areas under the curves (AUC) are presented in Table 6. All ROC curves yielded excellent AUC (≥0.95) for all time points.
Optimization of enzyme immunoassays cutoff values using ROC curve analyses
Most of the assays are very specific (≥97%) using the company’s cutoff values, and the optimization is therefore mostly oriented toward adjusting Se. Reducing the cutoff to >0.13 for the Roche assay significantly increased the Se to 96% and, as a trade-off, resulted in a 2% loss in Sp in the ≥15-day period. All the other assays had a remarkable increase in Se (89% to 94%) without reaching the minimal value of 95%. The Bio-Rad assay exhibited a slight increase in Sp to 99% at the optimized cutoff value set at >1.2, while the Se remained at 90%.
At the ≥35-day time point, the assays from Roche, Bio-Rad, and Siemens showed good performance at the company’s cutoff values, while the Ortho-Clinical total-antibody assay was 100% sensitive and specific without any modification of the cutoff. Six of seven assays attained a Se of 100% following a modification of the cutoff values, with a slight decrease in Sp (1% to 2%) for five assays. The biggest gain in Se (+21%) was seen with the Beckman assay, which achieved 100% Se with an adjusted threshold of >0.28. Again, the Bio-Rad assay showed a 2% increase in Sp when its threshold value was slightly raised to >1.2 without any substantial loss of Se.
DISCUSSION
SARS-CoV-2 serological assays should exhibit excellent sensitivities and specificities to be useful as clinical tools for public health investigations or as a diagnostic aid in particular clinical cases. Health Canada recommends that the Se and Sp of serological assays be higher than or equal to 95% and 98%, respectively, in samples collected 2 weeks or more after the onset of symptoms. Unfortunately, none of the evaluated commercial enzyme immunoassays achieved this level of accuracy using the cutoff values suggested by the companies. Within the rapid-device category, only the NADAL assay was able to comply with the minimal requirements in this time window.
SARS-CoV-2 seroprevalence studies in low-prevalence settings such as those seen in the general population need highly specific assays to rule out potential false-positive reactions resulting from past or present unrelated infections or health conditions. Public health studies of the COVID-19 prevalence in various cohorts from distinct regions in the province of Quebec are ongoing, but preliminary data indicate a seroprevalence of 3% in blood donors 18 to 69 years old in the greater Montreal area (www.hema-quebec.qc.ca). Seven immunoassays examined in this study are predicted to have excellent specificity (98% to 100%) and good to excellent PPV (82% to 100%) when used in a low (5%)-seroprevalence setting (Table 5). Accordingly, these assays could potentially be helpful to get a more precise epidemiologic assessment of SARS-CoV-2 exposition following outbreaks in different population settings, such as nursing homes and schools, to name a few.
Similarly to other studies (18–21), we observe a direct relationship between the sensitivity rates of serological assays and time of sample collection following the onset of symptoms. Long et al. reported a median time for IgM and IgG seroconversion of 13 days after symptom onset (22). This finding is consistent with the lack of Se observed for most assays in the acute phase. Overall, we observed Se values as low as 74% and as high as 95% at ≥15 days, while this range increases to 79% to 100% at ≥5 weeks after symptom onset. According to a meta-analysis of the accuracy of serological tests, the Se of different types of serological assays was highest when they were used with samples collected at least 3 weeks after symptom onset (23). The authors reported Se values ranging from 79.7 to 98.9%, 15 days or more following the onset of symptoms.
We also noted that most of our time-stratified Se values were lower than those reported in the company package inserts. For instance, we obtained 85% and 83% sensitivities for the Roche and Abbott assays in the ≥15-day period, while the companies claimed 100% sensitivity for the same time frame. Similar to our findings, other studies reported lower Se values, varying from 73% to 89% and 82% to 92% for the Roche and Abbott assays, respectively (19–21). Clinical performance characteristic variations between serological studies are not unusual and are highly dependent on the patient cohort used for evaluation. Our patient cohort consisted essentially of symptomatic patients with limited available clinical information. Among the 176 patients, only 48 had complete clinical information, including 32 hospitalized patients and 8 patients at the intensive care unit (ICU). Therefore, it was not possible to stratify our data according to the patient setting (outpatient, ward, ICU). Furthermore, it was demonstrated in several studies that the levels of antibodies in COVID-19 patients are significantly higher in severely ill patients (24, 25). It is possible that our cohort of patients presented a moderate level of disease, which could translate to a lower level of antibody detection.
Regarding the rapid antibody detection devices, we obtained relatively good Se and Sp values which were lower than those claimed by the manufacturers but similar to those in published studies (19, 26).
Considering the low Se values reported to date, it is quite clear that serology cannot be used to diagnose acute infections and should rather be used to assess SARS-CoV-2 exposure, as recommended by public health authorities. The lack of Se documented in the acute phase of COVID-19 disease represents a yet-unresolved problem that could be bypassed by several proposed strategies. For example, an orthogonal algorithm using two consecutive enzyme immunoassays with improved global analytical accuracy (27) was proposed by the Centers for Disease Control and Prevention (28). Another path toward better clinical performance could be achieved by the development of second-generation immunoassays with recalibrated cutoff values by manufacturers, as proposed in numerous validation studies (18, 20). Collectively, the determination of optimized cutoff values in our study through ROC curve analyses had a significant impact on the diagnostic resolution of several enzyme immunoassays by increasing the sensitivity significantly without a large trade-off in specificity.
Another medical application for serology is the assessment of the immune status of convalescent-phase plasma collected from recovered SARS-CoV-2 patients. Immune-plasma therapy is considered a therapeutic option to treat very ill patients for whom there is no curative treatment (29). However, assays designed to measure the virus-neutralizing activities of antibodies are tedious and necessitate biosafety level 3 (BSL3) conditions. Alternatively, pseudovirus neutralization assays are performed in BSL2 research laboratories but cannot be implemented easily in clinical laboratories. The virus neutralization assays typically used the spike protein to assess the presence of neutralizing antibodies (nAb). The spike protein was shown to be crucial for SARS-CoV and SARS-CoV-2 to bind to angiotensin-converting enzyme 2 (ACE2) as the entry receptor on mammalian cells via the receptor binding domain (RBD) epitope of their S1 domain (30). Intuitively, serological assays which utilize spike protein should be able to detect antibodies having virus neutralization characteristics more efficiently than nucleocapsid-based assays. We titrated 67 samples from COVID-19 patients for the presence of nAb by pseudovirus neutralization assays and performed correlation statistics in order to determine whether the enzyme immunoassays routinely used in clinical settings could predict the presence of nAb in patient samples. Correlation analysis using a paired data set for each sample using the ID50 values of the neutralization assay and the signal-to-cutoff values of enzyme immunoassays indicated a moderate (nucleocapsid-based assays) to high (spike-based assays) strength of association between nAb and binding antibodies. In fact, the spike-based assays showed the highest coefficient of association (r; 0.67 to 0.77). Therefore, these results indicate that spike-based immunoassays could be useful as correlates for the detection of nAb in patient samples or alternatively to evaluate the immune status of vaccinated populations. In accordance with our data, the study by Muecksch et al. (31) showed that commercial spike-based serological assay measurements were better predictors of virus neutralization potency than results of nucleocapsid-based assays. Another study reported that commercially available serological tests have various degrees of accuracy in predicting nAb activity in convalescent-phase plasma (32). However, the authors of that study did not observe a correlation between antigen type used in serological assays and the predictability of nAb activity levels. Clearly, additional investigations, including functional and structural studies about the native conformation of the whole antigen or subdomains or the glycosylation status of the recombinant spike protein could potentially help to develop enhanced assays.
Furthermore, it will be important to develop assays sensitive enough to detect antibodies in patients enrolled in SARS-CoV-2 vaccination trials. Vaccine development research groups are mostly focusing one spike protein or the RBD, since these proteins/peptides were shown to elicit the production of potent nAb and to confer protection in nonhuman primates challenged with SARS-CoV-2 (33–35).
An important limitation of this study is related to the possible dilution effect of using a pooling procedure for the preparation of the samples for the specificity panel. Due to potential variation of antibody levels among patients, this could potentially reduce the final concentration of cross-reacting antibodies below the detection threshold for any particular kit and so increase the apparent specificity of that assay. Conversely, cross-reacting antibodies were detected in pooled or nonpooled samples (autoimmune panel) with the Euroimmun assays (IgA and IgG). In this case, other factors, such as antigen preparation (peptide conformation and glycosylation status), may contribute to the high false-positive rates observed with these assays.
In this study, we evaluated 10 enzyme immunoassays and three lateral-flow chromatography assays. The specificity was excellent for the majority of the assays, as they were probably designed to be very specific by the manufacturers at the expense of sensitivity. However, several scenarios exist to increase their clinical performance. Orthogonal algorithm testing and optimized enzyme immunoassays should be validated promptly and implemented in clinical settings. As more data accumulate, spike-based immunoassays seem to be better correlates of neutralization activity and could potentially be used to characterize plasma samples from recovered patients in routine clinical laboratories. In the face of the putative second wave of COVID, it will be important to investigate the immunity of different populations. With this knowledge at hand, it will be possible in the long term to examine the protection levels conferred by nAb in immunized individuals from natural infection or vaccination as the pandemic continues to spread.
ACKNOWLEDGMENTS
We thank all members of the SARS-CoV-2 serology provincial committee for helpful discussion about the overall logistic organization of this study. We also thank all the medical laboratory technologists for their excellent work.
We declare no conflicts of interest.
V.M.-L. is funded by the FRQS Clinical Research Scholar-Junior 1. R. Gasser is supported by a MITACS Acceleration postdoctoral fellowship.
FOOTNOTES
- Received 28 September 2020.
- Returned for modification 27 October 2020.
- Accepted 8 December 2020.
- Accepted manuscript posted online 10 December 2020.
Supplemental material is available online only.
- Copyright © 2021 American Society for Microbiology.
This article is made available via the PMC Open Access Subset for unrestricted noncommercial re-use and secondary analysis in any form or by any means with acknowledgement of the original source. These permissions are granted for the duration of the World Health Organization (WHO) declaration of COVID-19 as a global pandemic.
