INTRODUCTION

Top Abstract Introduction Materials and Methods Results Discussion References

Resistance to a variety of antimicrobial agents is emerging in bacterial pathogens throughout the world (8, 21, 50). Increases in the prevalence of penicillin resistance in Streptococcus pneumoniae (13, 36, 39, 47), methicillin resistance in Staphylococcus aureus (1, 3, 38), vancomycin resistance in enterococci (5, 9, 16, 28), extended-spectrum -lactamase-production in enteric gram-negative bacilli (2, 18, 37), and fluoroquinolone resistance in Neisseria gonorrhoeae (17) are just a few examples of the rising problem of resistance documented by both national and international surveillance systems in the past few years. The antimicrobial susceptibility testing data collected by the various surveillance systems are generated in several different fashions. In some systems, bacterial isolates are sent to a central laboratory for testing using a standardized reference method (12, 43), and the data are compiled and maintained by the central laboratory. In other systems, the Etest method is used by participating laboratories to test isolates locally, and the data are forwarded to a central database (10, 35). A third approach to surveillance is to collect the antimicrobial susceptibility testing data directly from microbiology laboratories by either a direct computer link (40) or through diskettes sent to a central laboratory in which standardized computer database software such as the WHONET is used (49). These systems usually employ a series of data checks and quality control filters before the data are entered into the larger database. The final surveillance system consists of the microbiology laboratories throughout the world that perform routine antimicrobial susceptibility testing of bacterial isolates but that do not belong to one of the above systems (40). This vast network, which generates tens of thousands of susceptibility results daily, is a significant source of resistance data. It is often through this system of astute clinical microbiologists that new resistant organisms are detected (30).

The proliferation of surveillance systems internationally has raised the issue of the accuracy of the data collected, particularly for systems in which testing occurs outside of a central site. The need to obtain and review the quality control data from noncentralized laboratories or to assess the competence of laboratories both before data collection begins and during testing has resulted in the call for wider proficiency testing of laboratories.

In 1995, the World Health Organization (WHO) initiated a program of quality control and proficiency testing focused on antimicrobial susceptibility testing to assist laboratories in both developed and developing countries to assess the accuracy of their antimicrobial susceptibility testing data. The goals of the program were (i) to assist laboratories, particularly those that were WHONET software users, in evaluating the quality of their antimicrobial susceptibility testing data, (ii) to validate the susceptibility testing data submitted to WHO and the WHO Collaborating Center for Surveillance of Antimicrobial Resistance in Boston, (iii) to provide guidance for laboratories wishing to develop quality assurance programs, and (iv) to provide laboratories with organisms that manifest novel antimicrobial resistance so that they could confirm that their susceptibility testing methods were capable of detecting the emerging resistance patterns. The program, which began with 17 laboratories, all of which were located outside of the United States, has evolved into the WHO External Quality Assurance System for Antimicrobial Susceptibility Testing (WHO-EQAS), which now includes approximately 130 laboratories in 42 countries (including one laboratory in the United States). Several countries have set up national quality assurance systems further distributing the proficiency testing strains to an additional 200 laboratories. Herein, we describe the results reported by approximately 130 laboratories for the first six proficiency testing challenges.

	MATERIALS AND METHODS

Top Abstract Introduction Materials and Methods Results Discussion References

Organism challenges. Three sets of two organisms each were sent to approximately 130 laboratories in 42 countries over a 3-year period (1996 through 1999). Most of the laboratories were WHONET software users or laboratories that had participated in WHO laboratory training programs. Some laboratories participated in only one or two of the challenges, and additional laboratories were continually added to the list of participants throughout the study period. New regulations disallowing the importation of infectious substances into various countries also limited the participation of several laboratories. Thus, the number of results and the antimicrobial agents tested for each organism in this report vary. The list of countries represented among the participating laboratories is shown in Table 1 by WHO region. The organisms were tested multiple times at the Centers for Disease Control and Prevention (CDC) by broth microdilution and disk diffusion using National Committee for Clinical Laboratory Standards (NCCLS) methods (32-34) to establish the reference MIC and disk diffusion values. In some cases (e.g., for methicillin-resistant S. aureus, vancomycin-resistant Enterococcus faecium, and erythromycin-resistant S. pneumoniae), PCR assays for mecA, vanA, and mefE, respectively, were used to confirm the resistance mechanisms (54). A data collection sheet was provided with each organism, requesting information on the antimicrobial susceptibility testing method used (MIC or disk diffusion), source of media and reagents, interpretive criteria used (e.g., NCCLS [32-34], the Comité de l'Antibiogramme de la Société de Française Microbiologie [CA-SFM] [22], or the British Society for Antimicrobial Chemotherapy [BSAC] [57]), and disk potency. The data sheets included a suggested set of antimicrobial agents to test and provided space to fill in additional drugs that were tested in the laboratory. Laboratories were asked to provide both quantitative results (MICs or zone diameters) and the qualitative interpretations (i.e., susceptible, intermediate, or resistant [S, I, or R, respectively]) for each antimicrobial agent tested. Data sheets were returned to CDC for analysis. The data were entered into a SAS data set (SAS, Cary, N.C.). Because NCCLS, CA-SFM, and BSAC often use conflicting interpretive criteria, only laboratories using NCCLS methods and interpretive criteria were considered for this manuscript, to make comparison of data feasible. If the laboratory provided an incorrect interpretation (i.e., an interpretive error) for an MIC or disk diffusion result, the correct interpretation (S, I, or R) was entered into Tables 3, 5, 6, 7, 8, or 9 (see below), and the error was noted in a footnote to these tables. The interpretive errors from laboratories using disk diffusion are included in Table 10 below, and interpretive errors from laboratories using MIC methods are listed in Table 11 below.

	RESULTS

Top Abstract Introduction Materials and Methods Results Discussion References

Organism challenges. The antimicrobial susceptibility testing methods used by participating laboratories for each of the challenge organisms are shown in Table 2.

Challenge WHO-1. The first challenge strain (WHO-1) was an isolate of Klebsiella pneumoniae that produced a TEM-3 extended-spectrum -lactamase (ESBL) (48) (Table 3). One hundred and thirty laboratories reported results; 63.9% (83 of 130) used a disk diffusion method, and 36.1% (47 of 130) used an MIC method (Table 2). The extended-spectrum cephalosporins and monobactams tested most frequently by participating laboratories are shown in Table 4. Six laboratories (four disk diffusion users and two MIC users) failed to test any extended-spectrum cephalosporins or aztreonam. Overall, 5.4% (7 of 130) laboratories (all disk diffusion users) reported WHO-1 to be susceptible to all extended-spectrum cephalosporins. For individual drugs, 91.4% of 105 laboratories reported an intermediate or resistant result for ceftazidime, 86.9% of 92 laboratories reported an intermediate or resistant result for cefotaxime, and 81.2% of 16 reported an intermediate or resistant result for ceftriaxone. On the other hand, 11.1% (5 of 45) of laboratories testing cephalothin, a narrow-spectrum cephalosporin, reported the organism incorrectly as susceptible to this agent. Of the nine laboratories reporting ceftazidime-susceptible results, six also reported susceptible results for cefotaxime or ceftriaxone. Only 2 of the 130 laboratories specifically reported the isolate as an "ESBL-producing strain." None changed the results for extended-spectrum cephalosporins or aztreonam from susceptible to resistant based on detection of resistance to another extended-spectrum cephalosporin, as suggested by NCCLS (34). Failure to test extended-spectrum cephalosporins (n = 6) and reporting of susceptible results for all extended-spectrum cephalosporins (n = 7) (this includes the five laboratories that reported susceptible results for cephalothin) were considered unacceptable results.

Challenge WHO-2. The second challenge strain (WHO-2) was a methicillin (oxacillin)-resistant, mecA-positive S. aureus (MRSA) (Table 5). One hundred twenty-seven laboratories reported results; 59.1% (75 of 127) reported disk diffusion results, and 40.9% reported MIC results. The MRSA strain demonstrated high-level resistance to oxacillin, erythromycin, clindamycin, and tetracycline. Approximately 91% of laboratories recognized the strain as an MRSA. Only one laboratory reported both penicillin and oxacillin as susceptible results, but eight other laboratories reported the strain as oxacillin or methicillin susceptible. These were considered unacceptable results. Three additional challenge strains of S. aureus were sent to the nine laboratories that reported the strain as oxacillin susceptible. Of the five laboratories returning results, four were successful in differentiating the two MRSA strains from the susceptible S. aureus strain.

Challenge WHO-3. The third challenge (WHO-3) was a vancomycin-resistant strain of E. faecium (VRE; previously referred to as NJ-1 [55]) that was also borderline resistant to penicillin but (using NCCLS breakpoints) susceptible to ampicillin. Of the 122 laboratories that tested this organism, 59.0% used disk diffusion and 41.0% used an MIC method. Only 2.6% (3 of 117) of laboratories testing vancomycin failed to recognize this strain, which contained the vanA resistance gene, as highly resistant to vancomycin (Table 6). While 80.3% of laboratories correctly reported the strain as penicillin resistant, 21.0% incorrectly reported it as ampicillin resistant using NCCLS interpretive criteria. Three of 39 (7.7%) laboratories reported the isolate incorrectly as beta-lactamase positive. As the beta-lactam results were on the borderline and represented an unusual phenotype, for this challenge, reporting vancomycin-susceptible results (n = 3) was considered unacceptable performance.

Challenge WHO-4. The fourth challenge (WHO-4) was an erythromycin-resistant (mefE-positive) strain of S. pneumoniae with reduced susceptibility to penicillin. The penicillin MIC varied from 0.06 µg/ml (susceptible) to 0.12 µg/ml (intermediate), but it invariably produced a zone diameter of 14 to 17 mm around a 1-µg oxacillin disk, which, according to NCCLS guidelines, indicates that an MIC test should be performed to determine if the isolate is susceptible or resistant to penicillin. Of the 89 laboratories that performed an oxacillin screen test, 35.9% reported a value of 20 mm, which indicates susceptibility to -lactam agents (Table 7). Only 59.4% (76 of 128) of laboratories reported an MIC result for penicillin. Although NCCLS does not recommend penicillin or cephalosporin disk diffusion tests for pneumococci, two laboratories reported results for a 1-U penicillin disk, and six reported results for a 10-U penicillin disk. In addition, 21 laboratories reported values for 30-µg cefotaxime disks, and 15 reported values for 30-µg ceftriaxone disks. Thirteen laboratories (10.2%) incorrectly reported the organism as susceptible to erythromycin, and 2 reported it incorrectly as intermediate or resistant to clindamycin. Reporting oxacillin zone diameters of 20 mm, a penicillin MIC of 2 µg/ml or reporting erythromycin-susceptible results was considered unacceptable performance for this organism (n = 40).

Challenge WHO-5. The fifth challenge (WHO-5) was an Enterobacter cloacae strain that was sent for identification and antimicrobial susceptibility testing. It was resistant to all penicillins, cephamycins, and cephalosporins, including extended-spectrum cephalosporins, but was susceptible to the aminoglycosides, trimethoprim-sulfamethoxazole, ciprofloxacin, imipenem and meropenem (4, 42). Few laboratories had problems in recognizing the -lactam resistance in this strain (Table 8), although 6.1% of laboratories (8 of 130) failed to identify this organism as an Enterobacter species. Susceptible results for ampicillin or extended-spectrum cephalosporins were considered to be unacceptable performance (n = 3). Because the focus of the survey was on susceptibility testing results and not identification, misidentifications were not considered as unacceptable performance.

Challenge WHO-6. The sixth challenge (WHO-6) was a glycopeptide-intermediate strain of Staphylococcus epidermidis (20). The reference MIC of vancomycin for this organism was 8 µg/ml, and the MIC of teicoplanin was 16 µg/ml. Of the 130 laboratories testing this organism, 105 (80.8%) used disk diffusion, and 19.2% used an MIC method. Ninety-seven (74.6%) of 130 laboratories testing vancomycin reported this isolate as vancomycin susceptible (Table 9), which was considered unacceptable performance. This included a laboratory that reported a teicoplanin zone diameter of 6 mm, which also would be considered incorrect (since it is an unusually small zone diameter). Four laboratories that tested vancomycin by disk diffusion reported zone diameters of 6 to 14 mm, which are unusually small for this strain (52). The two reports of vancomycin zone diameters of 6 mm were also considered unacceptable performance.

Overall performance and interpretation and reporting errors. Of the 74 laboratories that sent results on at least five of the six challenge organisms, 17 had fully acceptable results, 33 reported unacceptable results for a single challenge, 20 reported unacceptable results for 2 challenges, and 4 reported unacceptable results for 3 challenges. None reported unacceptable results for four or more of the challenges.

Quality control data. Each of the laboratories participating in the WHO-EQAS system was asked to submit 10 consecutive quality control results for Escherichia coli ATCC 25922, S. aureus ATCC 25923, and S. pneumoniae ATCC 49619 for analysis. Data were available from approximately 35 laboratories using NCCLS quality control ranges. Figure 1 demonstrates that many of the quality control results from oxacillin disk diffusion tests for S. aureus ATCC 25923 fell outside the specified control range. Of the 654 data points submitted by 35 laboratories, 16 values (2.5%) (from 3 different laboratories) were below the range and 162 values (24.8%) (from 22 laboratories) were above the range. By comparison, 130 of 705 disk diffusion results (18.4%) reported by 38 laboratories were out of range when testing vancomycin against the S. aureus control (Fig. 2). Twenty-eight laboratories reported values for vancomycin that were below the NCCLS quality control range, while 11 laboratories reported values beyond the upper end of the range. For oxacillin testing of S. pneumoniae ATCC 49619, 30 laboratories reported 167 of 606 of their results (27.6%) above the designated range, and 15 laboratories reported 92 results (15.2%) below the range (Fig. 3). For erythromycin testing of S. pneumoniae ATCC 49619, 41 of 677 values (6.1%) from 16 laboratories were below the range, and 149 values (22.0%) were above the range (Fig. 4). For E. coli ATCC 25922, ceftazidime results were more often in control. Of 500 reported values, only 19 (3.8%) were below the designated range, and 45 (9.0%) were above the range (Fig. 5).

View larger version (19K): [in this window] [in a new window]   FIG. 1.   Histogram showing quality control results for disk diffusion testing of S. aureus ATCC 25923 against oxacillin. The NCCLS quality control range against which results were compared was 18 to 24 mm. Data are all results received as of 30 August 1999 and tested by NCCLS methods.

View larger version (16K): [in this window] [in a new window]   FIG. 2.   Histogram showing quality control results for disk diffusion testing of S. aureus ATCC 25923 against vancomycin. The NCCLS quality control range against which results were compared was 17 to 21 mm. Data are all results received as of 25 August 1999 and tested by NCCLS methods.

View larger version (23K): [in this window] [in a new window]   FIG. 3.   Histogram showing quality control results for disk diffusion testing of S. pneumoniae ATCC 49619 against oxacillin. The NCCLS quality control range against which results were compared was 8 to 12 mm. Data are all results received as of 24 August 1999 and tested by NCCLS methods.

View larger version (18K): [in this window] [in a new window]   FIG. 4.   Histogram showing quality control results for disk diffusion testing of S. pneumoniae ATCC 49619 against erythromycin. The NCCLS quality control range against which results were compared was 25 to 30 mm. Data are all results received as of 24 August 1999 and tested by NCCLS methods.

View larger version (16K): [in this window] [in a new window]   FIG. 5.   Histogram showing quality control results for disk diffusion testing of E. coli ATCC 25922 against ceftazidime. The NCCLS quality control range against which results were compared was 25 to 32 mm. Data are all results received as of 30 August 1999 and tested by NCCLS methods.

	DISCUSSION

Top Abstract Introduction Materials and Methods Results Discussion References

The WHO-EQAS was designed to enhance the ability of laboratories to detect organisms with emerging antimicrobial resistance patterns and to ensure that the laboratories' reporting strategies for antimicrobial resistance were accurate. Overall, approximately 20% of laboratories reported fully acceptable results, while the number reporting unacceptable results for three or more of the challenges was low (<5%). Most laboratories had problems with the S. epidermidis and S. pneumoniae isolates (see below).

Because of ongoing changes in the interpretive criteria for antimicrobial susceptibility testing (e.g., NCCLS, BSAC, or CA-SFM), it is possible to perform a test correctly but report inaccurate results, particularly if old guidelines or criteria are used. There were 74 instances in this study in which the results of the susceptibility tests were interpreted incorrectly, which, in a clinical setting, would have resulted in inaccurate information being transmitted to the patient's chart. The goal of quality assurance is to detect and correct problems such as these in a continuous fashion to improve the accuracy of testing, record keeping, and reporting (41, 44). The WHO-EQAS proficiency testing program was designed specifically to help laboratories determine whether all aspects of their current antimicrobial susceptibility testing methods are accurate. To this end, feedback letters detailing the reporting errors were sent to each participating laboratory as part of the quality assurance aspect of the study.

For the proficiency testing portion of the WHO-EQAS, it was gratifying to observe that most laboratories were able to detect methicillin resistance in S. aureus and high-level vancomycin resistance in E. faecium. Only a few laboratories had problems detecting resistance to the extended-spectrum cephalosporins mediated by the chromosomal AmpC -lactamase in the E. cloacae isolate; additional laboratories also had problems identifying this isolate biochemically, but the two groups of laboratories did not completely overlap. Assessing the accuracy of bacterial identification was not a major goal of this program, but since NCCLS recommends testing for ESBL production among E. coli, K. pneumoniae, and Klebsiella oxytoca isolates, we wanted to determine if most laboratories could differentiate E. cloacae from K. pneumoniae.

The ESBL-producing K. pneumoniae was one of the first isolates sent to most participating laboratories and preceded publication of the NCCLS guidelines for ESBL detection (34). While approximately 88% of laboratories reported that the K. pneumoniae isolate was resistant to at least one extended-spectrum cephalosporin, only 1.5% reported that it was specifically an extended-spectrum -lactamase producer. None of the laboratories modified the interpretations of the other cephalosporins to "resistant" as currently suggested by NCCLS (34). Based on feedback from participating laboratories, the issue of ESBL reporting is a source of confusion for many laboratories and may take additional educational efforts before improvement will be seen (11, 53).

The key problem areas identified in our surveys were detection of reduced susceptibility to penicillin in pneumococci and detection of reduced susceptibility to glycopeptides in staphylococci. Approximately 36% of the laboratories reported oxacillin zone diameters of 20 mm for the S. pneumoniae strain WHO-4, indicating a failure to detect reduced susceptibility to penicillin. Of those laboratories that did report a zone diameter of 19 mm, only 68.2% reported an MIC result. Previous reports have noted that the oxacillin screening test for pneumococci is very sensitive and rarely fails to detect penicillin-resistant pneumococci. However, the test lacks specificity, often yielding zone diameters of 19 mm for strains of pneumococci for which the penicillin MICs are 0.03 to 0.06 µg/ml (15, 27). For this reason, the NCCLS has recommended that all pneumococci yielding oxacillin zone diameters of 19 mm should be tested for penicillin resistance using an MIC method (32). The range of penicillin MICs reported for this isolate was also remarkably wide, from 0.007 to 2.0 µg/ml. The modal penicillin MIC reported at CDC was 0.06 µg/ml; thus, several laboratories significantly overcalled the resistance level of this organism. Of the 76 laboratories reporting a penicillin MIC result, 40 used Etest and the other 36 used a variety of methods (Table 2). Proficiency testing results from pneumococcal challenges conducted in United States laboratories as collected in recent surveys by the College of American Pathologists (CAP) showed similar testing problems with -lactam drugs, indicating that testing of pneumococci for susceptibility to penicillin and cephalosporins in the United States is far from optimal (14). One of the key problems recognized in both the WHO-EQAS and CAP studies is the continued use of penicillin, cefotaxime, and ceftriaxone disks for pneumococcal testing. Although NCCLS has not approved interpretive criteria for these disks, laboratories continue to use them. The zone diameters and interpretations sent to the WHO-EQAS project indicate that a combination of gram-negative enteric, staphylococcal, and occasionally enterococcal breakpoints is being used to interpret results, which frequently leads to under reporting of resistance. Clearly, a portion of the problem relates to the expense of performing MIC versus disk diffusion tests in many developing countries. While disk diffusion methods for -lactam drugs were proposed many years ago (26), they have not been embraced by NCCLS because of their poor predictive values in multicenter studies. More recent attempts to optimize disk testing using multidisk systems have been encouraging (25), but the accuracy of the method has yet to be independently confirmed. Given the wide intermediate range for penicillin (0.1 to 1.0 µg/ml), it is very difficult to devise a disk diffusion test using a 2-U or 10-U penicillin disk that accurately differentiates susceptible and resistant isolates. Because the breakpoint of 0.1 µg/ml is so important for optimal therapy of pneumococcal meningitis (19, 24, 31, 56) and because high-level ceftriaxone- and cefotaxime-resistant pneumococci have emerged (45), the importance of using accurate MIC methods for penicillin and extended-spectrum cephalosporins cannot be overstated. Although the Etest method has been shown to be an accurate predictor of penicillin and cephalosporin resistance (29), it is often too expensive for widespread use in developing countries.

The emergence of reduced susceptibility to vancomycin in S. aureus is a serious public health issue (23, 46). CDC and its advisory committees have published recommendations on preventing the spread of vancomycin-resistant organisms (7) and separate guidelines on preventing the development and spread of staphylococci with reduced susceptibility to vancomycin (6). In the WHO-EQAS survey, few laboratories were successful in detecting reduced susceptibility to vancomycin in the S. epidermidis isolate. This is primarily because such organisms cannot readily be detected by disk diffusion testing (52). Testing of staphylococci, particularly for vancomycin resistance, in laboratories where disk diffusion is used will require the use of an alternate method, such as brain heart infusion agar plates containing 6 µg of vancomycin per ml (developed for enterococci), which have been shown to detect most of the glycopeptide-intermediate staphylococci to date (52). This test, which is inexpensive to prepare, should allow laboratories to detect strains of staphylococci with emerging glycopeptide resistance. Many laboratories also had problems with oxacillin testing of this isolate. Recently, NCCLS changed the oxacillin breakpoints for coagulase-negative staphylococci (34, 51). Based on the results submitted to WHO-EQAS, it appears that many laboratories are unaware of this change.

Quality control testing ensures that the reagents and media and the technologist conducting the test are performing in a predictable and reliable fashion. Unfortunately, adherence to a quality control program is often lax in both developed and developing countries, where it is considered to be either too time consuming or too expensive to perform or both. This is probably why only 30 to 40 laboratories have sent quality control data to WHO-EQAS to date. The quality control data received from the participating laboratories showed many failures. However, it is encouraging that the laboratories continue to generate and use such data to optimize their testing methods. The data from the WHO-EQAS studies have been reviewed by NCCLS to determine whether some of the narrow ranges, such as that for oxacillin and S. pneumoniae ATCC 49619, require revision. Quality control testing is at the heart of good surveillance data and should be reviewed and verified prior to accepting data from sentinel laboratories participating in surveillance systems, particularly if such data are used to guide changes in empiric antimicrobial chemotherapy.

Overall, the results of the WHO-EQAS challenges have been encouraging. However, there is a clear need for educational programs that emphasize proper laboratory testing methods, the importance of quality control, and the basic concepts of quality assurance. The majority of participants in the WHO-EQAS used disk diffusion as their testing method. While this technique is suitable for routine surveillance for resistance in most nonfastidious pathogens, it does have its drawbacks, particularly for testing beta-lactam agents against pneumococci and glycopeptides against staphylococci. Through the use of alternate screening methods and a strict program of quality control to ensure the quality of the media, the emergence of resistance to these classes of drugs in these organisms should be detected. In conclusion, surveillance for antimicrobial resistance is critical, but the data must be validated prior to use.