ABSTRACT
Nontuberculous mycobacteria (NTM) cause severe opportunistic infections and have a rising incidence in most settings. Rising diagnostic need must be met by national reference laboratories, which rely on Clinical and Laboratory Standards Institute (CLSI) guideline-approved manual readout of microtiter plates for antimicrobial susceptibility testing (AST) to determine antibiotic minimum inhibitory concentrations (MICs). Interpretation of these plates leads to different outcomes between laboratories. The SensiTitre Vizion digital MIC viewing system (Vizion) offers a more streamlined approach using semiautomated reading. Here, we conducted a blinded trial comparing the outcome of AST between manual readout and Vizion readout for 132 NTM isolates, amounting to 727 individual tests for antibiotic susceptibility ranging across 13 individual antibiotics with established CLSI breakpoints. From this, we calculated specificity, sensitivity, positive predictive value (PPV), negative predictive value (NPV) and the F1 value, as well as assessing major error (ME) and very major error (VME) rates. We find that Vizion-assisted AST produces significantly lower MICs (paired Wilcox signed rank test; P < 0.0001). The Vizion had an accuracy of 89,40%, producing 61 MEs (8.39%) and 16 VMEs (2.20%). The calculated specificity was 0.8370, the sensitivity was 0.9550, the PPV was 0.8460, the NPV was 0.9520, and the F1 score was 0.8970. We show that discrepant readings mostly stem from CLSI guideline breakpoints being close to, or overlapping, the MIC50 values, leading to small discrepancies crossing the breakpoint, contributing to VMEs and MEs. Using the Vizion in standard clinical diagnostics for NTM might lead to an overestimation of antibiotic susceptibility.
INTRODUCTION
Nontuberculous mycobacteria (NTM) cause severe opportunistic infections, most frequently of the respiratory tract (1). The prevalence of these infections increases (2). The most common causative species are Mycobacterium avium complex (MAC) and Mycobacterium abscessus (3). Treatment of these infections involves a lengthy, multidrug chemotherapy with moderate to poor outcome, depending on the species (3).
The technique of antimicrobial susceptibility testing (AST) to optimize NTM treatment regimens was long a subject of debate, but it is now well established for key antibiotics (4). This has led to increased demand for AST of NTM in mycobacteriology laboratories. The currently recommended method for AST of NTM is broth microdilution according to Clinical and Laboratory Standards Institute (CLSI) standards (5). However, manual reading of these tests assisted by mirror is subjective (6) and time-consuming, thus automation would be helpful, if accurate. The SensiTitre Vizion digital MIC viewing system (Thermo Fisher, Landsmeer, the Netherlands) is a semiautomated, software-supported AST recording instrument. Here, the plates are photographed by customizable lightning options to accommodate different microorganisms and settings, and MICs are determined on a desktop PC. In this study, we assessed the performance of using Vizion compared to manual reading in AST of NTM.
MATERIALS AND METHODS
Strains.The aim of this study was to assess the performance of the Vizion across all NTM isolates encountered in day-to-day clinical diagnostic practice. Therefore, all NTM isolates received for AST from 5 March to 30 April 2017 in our NTM reference laboratory were eligible for the study. We included M. avium ATCC 700898 and M. peregrinum ATCC 700686 for quality control purposes.
To culture strains for AST, tubes containing Middlebrook 7H9 medium (Becton, Dickinson, and Company [Becton Dickinson], Erembodegem, Belgium) plus 10% oleic acid-albumin-dextrose-catalase growth supplement (Becton Dickinson) were inoculated with patient strains. A 5% blood plate was used for purity control. Strains were cultured at 37°C (except rapidly growing mycobacteria [RGM] and M. marinum, which were cultured at 30°C) until a visible pellet was formed. Plate inoculate solution was set to a 0.5 McFarland turbidity, diluted 1:100 in cation-adjusted Mueller-Hinton broth, and brought on SensiTitre plates at 100 µl per well. SensiTitre plates inoculated with rapidly growing mycobacteria were kept for 3 days, and plates containing slowly growing mycobacteria (SGM) were kept for 7 days.
Susceptibility testing.We performed broth microdilution using commercially available plates for slow-growing and rapidly growing NTM, selected on basis of strain identification (SensiTitre SLOMYCOI and RAPMYCOI; Thermo Fisher) and according to CLSI standards (5). All plates were initially read manually by a trained technician and for this study, independently by a second technician; discrepant readings were resolved by consensus. After manual reading, plates were read using the Vizion (Thermo Fisher) semiautomated reading device, with data analysis using the SWIN software package version 3.3 (3.3.2.7 P/N SW515, 2011; Thermo Fisher Scientific, Waltham, MA), operated by a technician blinded to the manual reading results. All technicians were trained by a Thermo Fisher technician, and quality controls of Mycobacterium avium ATCC 700898 and M. peregrinum ATCC 700868 were used as positive and quality controls biweekly, as suggested by the CLSI guidelines. The MICs were interpreted as “susceptible,” “intermediate,” and “resistant” as defined by CLSI guidelines. Only results of manual readings were reported to requesting physicians.
For this study, the manual reading is defined as the reference method. We defined very major errors (VMEs), major errors (MEs), and minor errors (7), with a VME being an isolate that proved to be resistant by the reference method standard of manual reading but susceptible using Vizion and a ME being an isolate that proved susceptible by the reference standard manual reading but resistant using Vizion. All other discrepancies were considered minor errors.
Data analysis.Antibiotics with no defined breakpoint were excluded from all analyses. Amikacin has no defined breakpoint according to CLSI guidelines, but Brown-Elliott et al. proposed breakpoints identical to those for M. abscessus, which will be considered for this study (8). Only for antibiotics where breakpoints were available, initial paired MIC data were transformed into breakpoint steps. We defined the breakpoint for resistance reported in the CLSI guidelines as 0 and transformed the MIC on basis of the number of 2-fold dilution steps difference from the breakpoint, giving susceptible isolates negative values and more resistant isolates positive values. Whenever an isolate exceeded the testing range to either side, they were categorized as one higher or lower than the highest or lowest value of the range, respectively. Distributions of MICs for antibiotics reportable according to CLSI guidelines were tested for normality using he Kolmogorov-Smirnoff test. A paired Wilcoxon signed rank test was used to compare data obtained from manual reading and semiautomatic reading.
All individual manual readings and semiautomated reading comparisons were then plotted in a 2 × 2 confusion matrix, categorizing test outcomes as “susceptible” (S) and “nonsusceptible” (non-S) pooling the outcomes “intermediate” (I) and “resistant” (R). For this, we ignored differences in found MICs between manual reading and Vizion, focusing solely on discrepancies in the breakpoint interpretations defined by CLSI guidelines. From this matrix, we determined the overall testing accuracy of Vizion, as well as the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and the F1 score indicating a test’s accuracy considering the specificity and sensitivity (9, 10).
All statistical tests were performed in Prism, version 6.00 (GraphPad Software, Inc., La Jolla, CA). We defined significance as a P value of <0.01. Other calculations and data compilations were performed using Microsoft Office Excel 2013 software (Microsoft, Inc., Redmond, WA).
RESULTS
We included 132 NTM isolates; 107 were slow growers, and 25 were rapid growers. The species distribution is recorded in Table 1.
Species distribution
Transformed pooled MIC distributions for all performed tests are presented in Fig. 1A for the manual readings and Fig. 1B for the Vizion semiautomated readings. The MICs were not normally distributed (Kolmogorov-Smirnoff, P < 0.01). We found a significant difference between the average MICs of the Vizion semiautomated analysis and the reference method, with the Vizion measuring lower MICs (−1.818 ± 2.445 versus −1.979 ± 2.390, respectively; n = 727; paired Wilcox signed rank, P < 0.0001; Fig. 1C).
Overall distributions of found MICs normalized on the breakpoint to resistance. (A) Distribution of MICs found with the manual readout method, regarded as the reference method according to CLSI guidelines. (B) Distribution of MICs found with the automated Vizion readout method. (C) Average MIC found in manual and automated readout. MICs found with the automated Vizion readout are significantly lower than the MICs found with the reference method (−1.818 ± 2.445 versus −1.979 ± 2.390, respectively; n = 727; paired Wilcox signed rank, P < 0.0001).
The calculated confusion matrix is shown in Fig. 2. In 727 total tests (for a complete overview, see Table S1 in the supplemental material), the MIC determined by Vizion led to the same interpretation (S/I/R) as the reference method for 650 tests performing with an overall accuracy of 89.40%. We observed discrepant readings leading to MEs in 61 tests (8.39%) and leading to VMEs in 16 tests (2.20%). A tabular version of this matrix broken down per antibiotic is shown in Table S1.
Confusion matrix for the Vizion automated readout (left) versus reference method (top). Clinical breakpoints (susceptible [S], intermediate [I], and resistant [R]) were determined according to CLSI guidelines. I and R were pooled. In total, we performed 727 tests on 132 individual isolates.
The results of most individual antibiotics mirror the overall accuracy of Vizion semiautomated reading, i.e., around 90%. However, the two antibiotics, clarithromycin and moxifloxacin, disproportionally contribute errors and have different error patterns for SGM and RGM (Fig. 3; Table S1). Clarithromycin testing against SGM performed well with an overall accuracy of 97.20% (Fig. 3A), producing only MEs; against rapidly growing mycobacteria (RGM), automated reading of clarithromycin MICs produced more MEs and only had 84.00% accuracy (Fig. 3D). Moxifloxacin testing against SGM produced most VMEs of all tested antibiotics, relatively and in absolute numbers, with an accuracy of 81.31% (Fig. 3C). In contrast, for RGM, moxifloxacin testing using Vizion showed 100% accuracy (Fig. 3B).
MIC distribution of antibiotics with notable error patterns. The MIC50 and MIC90 were calculated from all isolates. The distribution of all isolates tested for the antibiotics is plotted, either from Vizion or obtained by manual readout. (A and B) Moxifloxacin has an accuracy of 81.31% when testing on SGM (A; producing mainly VMEs) but 100% accuracy on RGM (B; no errors). (C and D) Clarithromycin has an accuracy of 97.20% (producing exclusively MEs) when tested against SGM but 84.00% accuracy when tested on RGM (producing only MEs).
From the pooled data set, we calculated the specificity, sensitivity, PPV, NPV, and F1 value for each breakpoint, as shown in Table 2. The automated Vizion readout reaches a sensitivity of 95.50% and a specificity of 83.70% when determining the drug susceptibility of NTM. From this, we calculated a PPV of 84.60% and an NPV of 95.20%. The F1 score is 89.70%. The full data set is provided as Table S2 in the supplemental material.
Classification functions as calculated from the Vizion confusion matrixa
DISCUSSION
With the rising incidence of NTM disease, a fast and reliable AST method becomes more desirable. We could show that the mycobacterial landscape is comparable to earlier data and, as such, reflects the clinical reality in the Netherlands (11). The SensiTitre Vizion digital MIC viewing system offers such a fast method but suffers from significantly lower MIC readouts than the CLSI-recommended reference method. In 10.08% of the tests, these readouts lead to MEs or VMEs, which might lead in turn to inappropriate treatment decisions.
Although other studies have assessed the clinical performance of the Vizion before, this study is the first using it to assess the MICs of NTM. Jones et al. tested the Vizion’s performance with ceftaroline on multiple microorganisms, but not NTM, and found 99.2% essential agreement with manual reading of MICs (12). Jayol et al. recently tested the accuracy of three different (semi)automated MIC readout systems, including the Vizion, to determine whether it could correctly detect colistin resistance in Gram-negative bacteria. These researchers found a categorical agreement of 97.3% between Vizion semiautomated readout and visual readout (13). Although the authors used a different analytic framework, their approach is highly comparable to ours. Both studies report higher accuracy than our 89.92%. Rancoita et al. assessed the intralaboratory reproducibility of Vizion for M. tuberculosis and found around 95% agreement, as well as 90% agreement between duplicated strains (14), although they did not quantify the accuracy of Vizion versus mirror-assisted readout. Notably, these authors only tested 31 predefined M. tuberculosis strains and did not assess Vizion’s performance during clinical laboratory practice. The difference found between NTM and other bacterial strains may be due to the variable phenotypes of NTM in the SensiTitre plates, which we find to be distinct in morphology, including aspects such as rough and smooth phenotypes (15). NTM tend to not form milky blankets in the SensiTitre wells, as, for example, Gram-negative bacteria or M. tuberculosis often tend to do, but rather grow in broken, small colonies, which might be easily missed by reading the plates on a monitor but might be picked up by the technician’s eye when reading the plate directly.
Tests for some antibiotics seem to contribute more errors than others (the full MIC data are shown in Table S2). Moxifloxacin produces most VMEs (n = 20), followed by linezolid (n = 9), but only against SGM, and amikacin for both SGM and RGM (n = 9 for SGM and n = 3 for RGM). Often, VMEs and MEs result from MIC distributions being very close to, or partially overlapping, the current CLSI breakpoints (4, 5, 16). This is true for both moxifloxacin (MIC50 against MAC = 4 mg/liter; S/I/R = ≤1/2/≥4 mg/liter) and linezolid (MIC50 against MAC = 16 mg/liter; S/I/R = ≤8/16/≥32 mg/liter). Because RGM isolates tend to be either susceptible or completely resistant, this pattern is less pronounced here, increasing Vizion’s accuracy for moxifloxacin on RGM. Thus, if automated readings tend to place MICs one dilution lower than manual reading, this easily leads to a change in interpretation from “nonsusceptible” to “susceptible,” producing VMEs. The calculated Vizion specificity of 83.70% supports this notion. A systematically lower MIC, as found with a Vizion reading, favors the classification of an isolate as “susceptible.” This leads to false-positive samples producing type 1 errors. Higher false-negative rates will lower a test’s specificity. Notably, RGM tests did not deviate much from the overall accuracy. An exception is, again, moxifloxacin, which produced no errors.
One of the most important considerations of this study is the significance of the test’s outcome in clinical diagnostics. Most M. abscessus isolates, except for M. abscessus subsp. massiliense, have a functional erm(41) gene (17), which rapidly induces high-level macrolide resistance. Because of this, one might argue that the Vizion’s low accuracy of 84% in measuring clarithromycin MICs at day 3 against RGM is clinically irrelevant, since the reported MIC90 values for both clarithromycin and azithromycin for M. abscessus isolates with a functional erm(41) gene were consistently >16 mg/liter after 7 days of incubation, irrespective of the MIC values found on day 3 (18). It is also suggested that erm(41) sequencing might be an option to assess M. abscessus macrolide susceptibility, which might confirm phenotypes found in AST and serve as an extra layer of control for isolates with intermediate susceptibility (19).
There are some limitations to this study. First, the reference method, i.e., the visual interpretation of the plates, is subjective, with different laboratories finding different MICs for the same isolate (6). In addition, the current CLSI breakpoints have a very limited evidence base, and small changes in MICs at or near these breakpoints may not be clinically relevant (14). Although complicated by the fact that NTM disease is treated by multidrug regimens, the ultimate test for AST performance is to compare outcomes in controlled trials to MICs measured by different methods. Indeed, a direct correlation between AST and clinical outcome has only been found for clarithromycin and amikacin against both MAC and M. abscessus (4, 20, 21). Furthermore, our statistical method and data report differ slightly from other studies that work with categorical agreement (12, 13). Since no studies exist that discuss the clinical relevance of small deviations in MIC, we felt that for our study, deviations in breakpoint classifications might give a clearer picture of Vizion’s diagnostic power.
In summary, automated reading of MICs of NTM with the Vizion system yields MICs that are lower than those measured by visual reading. Because of the proximity—or even overlap—of clinical isolate MIC distributions to current breakpoints, small discrepancies in terms of MICs lead to large deviations in AST interpretation. A conclusive statement about the clinical impact of both reading methods can only be based clinical trials that follow either recommendations of manual SensiTitre plate reading or interpretation supported by Vizion.
ACKNOWLEDGMENTS
J.V.I. is supported by a grant from the Netherlands Organization for Scientific Research (NWO/ZonMW grant Veni 016.176.024).
The authors have no conflicts of interest to declare.
FOOTNOTES
- Received 8 November 2018.
- Returned for modification 5 December 2018.
- Accepted 1 February 2019.
- Accepted manuscript posted online 13 February 2019.
Supplemental material for this article may be found at https://doi.org/10.1128/JCM.01756-18.
- Copyright © 2019 American Society for Microbiology.
