ABSTRACT
During the ongoing coronavirus disease 2019 (COVID-19) outbreak, robust detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a key element for clinical management and to interrupt transmission chains. We organized an external quality assessment (EQA) of molecular detection of SARS-CoV-2 for European expert laboratories. An EQA panel composed of 12 samples, containing either SARS-CoV-2 at different concentrations to evaluate sensitivity or other respiratory viruses to evaluate specificity of SARS-CoV-2 testing, was distributed to 68 laboratories in 35 countries. Specificity samples included seasonal human coronaviruses hCoV-229E, hCoV-NL63, and hCoV-OC43, as well as Middle East respiratory syndrome coronavirus (MERS-CoV), SARS-CoV, and human influenza viruses A and B. Sensitivity results differed among laboratories, particularly for low-concentration SARS-CoV-2 samples. Results indicated that performance was mostly independent of the selection of specific extraction or PCR methods.
INTRODUCTION
As of December 2020, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has infected over 66 million individuals and caused more than 1,500,000 deaths globally (https://covid19.who.int/; accessed 7 December 2020). As both specific medications and approved vaccines are not available yet, public health strategies need to focus on containment and mitigation measures. Robust detection of acute SARS-CoV-2-infected individuals, typically done by real-time reverse transcription PCR (rRT-PCR), is crucial for clinical management, surveillance and to interrupt transmission chains (1). An established tool to improve and support diagnostic accuracy for clinical management and surveillance use is the conduction of external quality assessments (EQA) (2–5). An EQA of molecular detection of SARS-CoV-2 was organized for the expert laboratories of the Emerging Viral Diseases–Expert Laboratory Network (EVD-LabNet) and/or the European Reference Laboratory Network for Human Influenza (ERLI-Net).
MATERIALS AND METHODS
Panel composition.The EQA panel was composed of 12 samples containing either SARS-CoV-2 at different concentrations to evaluate sensitivity or other respiratory viruses to evaluate specificity of SARS-CoV-2 testing (Table 1).
EQA panel composition and correct test results of participating laboratories
Specimen preparation.SARS-CoV-2 samples were quantified using an E-gene-based in-house assay recommended by the WHO (6) that relies on an in vitro-transcribed RNA standard. Samples were diluted in Dulbecco’s modified Eagle’s medium (DMEM). All samples were heat-inactivated following Matheeussen et al. (4) (65°C for 4 h for SARS-CoV, SARS-CoV-2, MERS-CoV, and adenovirus, and for 2 h for all other viruses listed in Table 1), and 200 µl of each sample was freeze-dried.
Panel validation, testing instructions, and panel dispatch.Successful inactivation of panel samples was confirmed by the absence of viral growth in three consecutive cell culture passages. Along with the EQA panels, participants received detailed reconstitution and testing instructions. The panels were shipped at ambient temperature. If national legislation of a country prohibited receipt of inactivated SARS-CoV and MERS-CoV samples, the laboratories received a modified panel that excluded these sample(s). Two independent laboratories validated the composition of the EQA panel and the result submission platform before distribution to all other EQA participants.
Evaluation of results.Result reporting was done via an online submission form by which laboratories could score their samples (positive, negative, or inconclusive) and provide information such as cycle threshold (CT) values, extraction methods and type of PCR(s). Laboratories were asked to treat and evaluate the EQA samples according to their routine diagnostic workflow, which could include the use of multiple assays. To ensure uniform interpretation, a sample was scored as positive if at least one SARS-CoV-2 test gave a positive result. No CT threshold was applied.
Sample no. 9, containing SARS-CoV, was not considered a “core sample,” and was therefore not included in the laboratory performance analysis. Core samples are defined as clinically relevant samples for proficiency testing. Clinical relevancy was given for the detection of all SARS-CoV-2 samples and for distinction of SARS-CoV-2 samples from other human respiratory viruses that are circulating in the human population. Since SARS-CoV has not been circulating in the human population since 2004, the capability to distinguish between SARS-CoV and SARS-CoV-2, which are closely related, is informative but unnecessary (7).
EQA participants were only asked to report whether samples were SARS-CoV-2 positive or not.
Statistical analysis.Statistical analyses were conducted to examine if the overall EQA performance correlated with specific technical details provided by the EQA participants. The information about workflows and protocols provided by the EQA participants was reviewed to be conclusive regarding applied kits or tests. If needed, participants were contacted to confirm or specify the provided information. Nevertheless, the reported information was not complete for all participants. The utilized data sets did thus differ slightly in size among statistical analyses of different variables. Before conducting statistical analyses, variables were tested for normal distribution using the Shapiro-Wilk test. All statistical analyses were conducted in R version 4.0.2.
Considering the incompleteness of reported information and the high diversity of reported workflows (see Fig. S1 in the supplemental material), different steps of the diagnostic workflow were analyzed individually. For this purpose, each variable was analyzed statistically, including the results of all EQA participants that provided reliable information for it. For some variables, such as applied PCR assays, we both tested for a general trend of correlation between the variable and the number of correct results and compared the amounts of correct results between specifically selected fractions of the variable, e.g., comparing the results between two specific real-time RT-PCR assays.
RESULTS
Sixty-eight laboratories from 35 countries, i.e., 28 of 30 European Union/European Economic Area countries, five of seven European Union preaccession countries, and two other European countries, reported EQA results (Fig. 1A).
Participating laboratories in SARS-CoV-2 molecular EQA and laboratory performance in relation to specific factors. (A) Overview of countries of the participating laboratories in SARS-CoV-2 molecular EQA, June and July 2020. (B) Numbers of correct, false-negative, and false-positive samples per laboratory. (C) Reported CT values per SARS-CoV-2-positive sample considering all results reported, including multiple tests conducted by one participant. Median CT values are indicated by bars, quartiles by boxes, and interquartile range by whiskers. Median CT values were 25.9 (95% confidence interval [CI], 25.7 to 26.6) for sample 11 (2,500 copies [cps]/µl), 31.6 (95% CI, 31.1 to 32) for sample 12 (180 cps/µl), 34.0 (95% CI, 33.6 to 34.6) for samples 2 and 10 (12.5 cps/µl), and 36.4 (95% CI, 35.6 to 36.9) for samples 1 and 6 (2 cps/µl). Single events are indicated by gray (positive) and purple (negative) dots. (D) EQA performance, represented as a fraction of correct results (1.0 = 100% correct and 0.8 = 80% correct), depending on different rRT-PCR targets. For box plot explanation, see panel B.
Twenty-seven of 68 participating laboratories tested all core samples correctly (Fig. 1B), 21 reported correct results for 10 samples, and 20 for nine samples or fewer. Notably, only 37 result submissions were based on the outcome of a single test method. The remaining participants scored their results based on the outcomes of two or more test methods. Incorrect results were reported for 9.7% (72/746) of all samples, with 8.6% (64/746) false-negative results reported for SARS-CoV-2 samples and 1.1% (8/746) false-positive results reported for specificity controls (Table 1). The risk of false-negative tests increased significantly with lower SARS-CoV-2 concentrations (P < 0.001; Spearman correlation test) (Fig. 1C). CT values of correctly tested SARS-CoV-2 samples increased with decreasing concentration (Fig. 1C). Sixty-three laboratories tested all specificity samples correctly, while 5 laboratories reported either one or two false-positive results. No single included respiratory virus was more prone to false-positive test results than others, pointing to contamination issues rather than issues intrinsic to the primer/probe sets used (Table 1).
As the performance in an EQA is the outcome of different steps in the diagnostic workflow combined, preferably different workflows should be compared. These steps comprise, e.g., resuspension of the panel samples, extraction method/kit used, change in nucleic acid (NA) concentration during extraction, and type of PCR assay, including the genomic target and the number of different genomic targets/tests performed. Among the participants of this EQA, there was an extensive variety of rRT-PCR methods that was used (Table 2; see also Table S2 in the supplemental material). The number of diagnostic workflows, which further included the extraction method/kit used and the RNA concentration change during extraction, was so high that a joint analysis of complete workflows was not possible (see Fig. S1 in the supplemental material), only allowing for analysis of individual components of the workflow.
Performance of different rRT-PCR assaysa
The performance among laboratories conducting manual NA extraction (92.4% correct tests) and laboratories conducting automated extraction (89.8% correct tests) did not differ significantly (P = 0.350; Mann-Whitney U test). The EQA performance was not correlated with the type of extraction kit used (P = 0.938; Kruskal-Wallis test) and ranged between 87.9% and 93.8% correct results (see Table S1 in the supplemental material). In most extraction protocols NA concentration is increased. However, the extent of NA concentration was not correlated with diagnostic sensitivity (P = 0.898; Spearman correlation test).
In total, 26 commercial assays and six in-house assays were used by the EQA participants. Among those assays performed by at least five EQA participants, correct results ranged between 78% and 92% (Table 2). The performance was not significantly correlated with a type of PCR assay in general (P = 0.525; Kruskal-Wallis test). However, results were significantly better with the best-performing test (Institute Pasteur, RdRp gene; Table 2) compared to “Others” (P = 0.042; Wilcoxon rank sum test), Corman N assay (P = 0.008), and the Viasure ORF1 assay (P = 0.011). The performances among commercial tests (84.6% correct results) and in-house tests (85.7% correct results) did not differ significantly (P = 0.969; Wilcoxon rank sum test). As transcription in coronaviruses typically varies among different genomic and subgenomic regions, target sites of the applied assays may affect the diagnostic sensitivity (8). However, the proportion of correct results did not differ significantly among different genomic targets (P = 0.852; Kruskal-Wallis test) (Fig. 1D).
DISCUSSION
The overall performance within this EQA was variable. Most false results were reported for low-concentration SARS-CoV-2 samples. While infectious SARS-CoV-2 patients commonly have high viral titers around the first day of illness (9, 10) an optimized diagnostic sensitivity is important to identify patients outside the optimal window for detection or when sampling or transport were suboptimal (11). Low concentrations in clinical samples are also often seen at a later infection stage that coincides with seroconversion and a drop in infectivity (12). The inclusion of serological tests such as enzyme-linked immunosorbent assays in diagnostic algorithms when a diagnosis is needed for proper patient management and infection prevention measures, e.g., in hospitalized patients under strong suspicion of a SARS-CoV-2 infection but with repeated rRT-PCR-negative results, may reduce the risk of false-negative tests.
Ten laboratories showed problematic EQA performances, including five laboratories which had false-positive results. Notably, the laboratories reporting false-positive results applied both extraction kits and rRT-PCR tests also used by other laboratories. False-positive test results could thus be a consequence of contamination during sample handling and extraction or lot-specific contamination of rRT-PCR kits or oligonucleotides, as recently reported (13). Either way, laboratories should adapt workflows to ensure good specificity, which in general was excellent among participants.
Compared to the performance of first-line, routine clinical laboratories in a recently published SARS-CoV-2 EQA, the sensitivity of participants in our EQA was lower (4). However, comparing both EQAs is difficult, as sample quantification and preparation were done differently, in this study more precisely using DMEM instead of transport medium for sample preparation and quantifying concentrations with classic rRT-PCR instead of droplet digital PCR. Based on detection limits for commercial and in-house rRT-PCR tests (14), the participants in the recently published SARS-CoV-2 EQA (4) seem to overperform, which may indicate in reality slightly higher sample concentrations than those quantified by the authors before shipment. Both routine clinical laboratories and expert laboratories performed well, considering that suboptimal sensitivity in EQAs for molecular diagnostics of recently emerged viruses is not uncommon (2, 15, 16).
Notably, the results indicate that performance can be increased by harmonized workflows, rather than by the selection of specific extraction or rRT-PCR kits. None of the individual components of the workflow that were analyzed for this study were able to significantly influence the overall performance of a laboratory.
The conduction of follow-up EQAs will be essential to further support European laboratories to systematically improve and maintain diagnostic capabilities, while the need for a robust global detection capability requires global EQA programs.
ACKNOWLEDGMENTS
We thank all EQA participants, namely, the Center for Virology, Medical University of Vienna, Austria; Department for Pathobiology, University of Veterinary Medicine Vienna, Austria; Clinical Reference Laboratory, Institute of Tropical Medicine, Belgium; National influenza Centre, Sciensano, Belgium; Department of Virology, National Centre of Infectious and Parasitic Diseases, Bulgaria; Molecular Department, Croatian Institute of Public Health, Croatia; Research Unit, University Hospital for Infectious Diseases “Dr. Fran Mihaljević,” Croatia; Department of Biological Sciences, University of Cyprus, Cyprus; Department of Molecular Virology, Cyprus Institute of Neurology and Genetics, Cyprus; Microbiology Department, Nicosia General Hospital, Cyprus; NRL for Influenza and other Respiratory Viruses, National Institute of Public Health, Czech Republic; Diagnostic department, SA Pärnu Hospital, Estonia; Laboratory of Communicable Diseases, Health Board, Estonia; Microbiologia Laboratory of Rakvere Hospital, Estonia; Clinical Microbiology, Helsinki University Hospital, Finland; Expert Microbiology, National Institute for Health and Welfare, Finland; Biology department, University of Corsica Pasquale Paoli, France; Unité des virus émergents, Aix-Marseille University, France; Central Diagnostics Unit, Bundeswehr Institute of Microbiology, Germany; FG17 Influenza and Other Respiratory Viruses, Robert Koch Institute, Germany; Institute of Microbiology and Virology, Brandenburg Medical School Theodor Fontane, Germany; Institute of Novel and Emerging Infectious Diseases, Friedrich-Loeffler-Institute, Germany; ZBS1 Highly Pathogenic Viruses, Robert Koch Institute, Germany; Molecular Diagnostics Laboratory, Central Public Health Laboratory, Greece; National Influenza Reference Laboratory for N. Greece, Aristotle University of Thessaloniki, Greece; National Influenza Reference Laboratory of Southern Greece, Hellenic Pasteur Institute, Greece; National Reference Laboratory for Respiratory Viruses, Nemzeti Népegészségügyi Központ, Hungary; Medical Virology, Landspitali University Hospital, Iceland; National Virus Reference Laboratory, University College Dublin, Ireland; Department of Infectious Diseases, Istituto Superiore di Sanità, Italy; Infectious Diseases, Amedeo di Savoia Hospital Torino, Italy; Microbiology and Virology Unit, Padova University Hospital, Italy; Molecular Virology, Fondazione IRCCS Policlinico San Matteo, Italy; Scientific Department, Army Medical Center, Italy; U.O. Microbiology, Az. Ospedaliero-Universitaria di Bologna, Italy; Virology Laboratory, National Institute for Infectious Diseases “Lazzaro Spallanzani” IRCCS, Italy; Department of Microbiology, National Institute of Public Health of Kosovo, Kosovo; Microbiology and Virology Department, National Microbiology Reference Laboratory, Latvia; Clinical Testing Department, National Public Health Surveillance Laboratory, Lithuania; Microbiology Department, Laboratoire National de Santé, Luxembourg; Pathology Department, Mater Dei Hospital, Malta; Department for Molecular Diagnostics, Institute of Public Health of Montenegro, Montenegro; Laboratory for Virology and Molecular Diagnostics, Institute of Public Health, North Macedonia; Department of Medical Microbiology, St. Olavs University Hospital, Norway; Department of Microbiology, Oslo University Hospital, Norway; Department of Virology, Norwegian Institute of Public Health, Norway; Virology Department, National Institute of Public Health, Poland; Infectious Diseases, National Institute for Health, Portugal; National Influenza and Other Respiratory Viruses Reference Laboratory, National Institute of Health, Portugal; Molecular Diagnostics Laboratory, National Institute for Infectious Diseases, Romania; Viral Respiratory Infections Laboratory, Cantacuzino National Military-Medical Institute for Research and Development, Romania; Department for Microbiology, Clinical Center of Serbia, Serbia; Department of Medical Microbiology, Public Health Authority of the Slovak Republic, Slovak Republic; Department of Virus Ecology, Biomedical Research Center of the Slovak Academy of Sciences, Slovak Republic; Laboratory for Public Health Virology, National Laboratory of Health, Environment and Food, Slovenia; Laboratory for Zoonoses, Institute of Microbiology and Immunology, Slovenia; Laboratorio Microbiología, Hospital Clinic de Barcelona, Spain; National Influenza Centre, Valladolid, National Influenza Centre, Spain; Respiratory Virus and Influenza Laboratory National Center of Microbiology-Instituto de Salud Carlos III, Spain; Servicio de Microbiología, Hospital Universitario Virgen de las Nieves, Spain; Department of Microbiology, The Public Health Agency of Sweden, Sweden; Biology Department, Spiez Laboratory, Switzerland; Laboratory of Virology, HUG, Switzerland; Department Viroscience, Erasmus MC, The Netherlands; Diagnostics and Laboratory Surveillance (IDS), National Institute for Public Health and the Environment (RIVM), The Netherlands; National Virology Reference Laboratory, Public Health General Directorate of Turkey; Rare and Imported Pathogens Laboratory, Public Health England, United Kingdom; and Wales Specialist Virology Centre, Public Health Wales, United Kingdom. In addition, we thank the technicians providing laboratory support at RKI, CUB, and RIVM.
This work was supported by the European Centre for Disease Prevention and Control (ECDC) under specific contract no. 3 for implementation of the framework contract ECDC/2017/002 to C.B.E.M.R. and J.F.D.
FOOTNOTES
- Received 21 October 2020.
- Returned for modification 12 November 2020.
- Accepted 7 December 2020.
- Accepted manuscript posted online 9 December 2020.
Supplemental material is available online only.
- Copyright © 2021 Fischer et al.
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license.
This article is made available via the PMC Open Access Subset for unrestricted noncommercial re-use and secondary analysis in any form or by any means with acknowledgement of the original source. These permissions are granted for the duration of the World Health Organization (WHO) declaration of COVID-19 as a global pandemic.
