ABSTRACT
The diagnosis of Lyme borreliosis (LB) is commonly made by serologic testing with Western blot (WB) analysis serving as an important supplemental assay. Although specific, the interpretation of WBs for diagnosis of LB (i.e., Lyme WBs) is subjective, with considerable variability in results. In addition, the processing, reading, and interpretation of Lyme WBs are laborious and time-consuming procedures. With the need for rapid processing and more objective interpretation of Lyme WBs, we evaluated the performances of two automated interpretive systems, TrinBlot/BLOTrix (Trinity Biotech, Carlsbad, CA) and BeeBlot/ViraScan (Viramed Biotech AG, Munich, Germany), using 518 serum specimens submitted to our laboratory for Lyme WB analysis. The results of routine testing with visual interpretation were compared to those obtained by BLOTrix analysis of MarBlot immunoglobulin M (IgM) and IgG and by ViraScan analysis of ViraBlot and ViraStripe IgM and IgG assays. BLOTrix analysis demonstrated an agreement of 84.7% for IgM and 87.3% for IgG compared to visual reading and interpretation. ViraScan analysis of the ViraBlot assays demonstrated agreements of 85.7% for IgM and 94.2% for IgG, while ViraScan analysis of the ViraStripe IgM and IgG assays showed agreements of 87.1 and 93.1%, respectively. Testing by the automated systems yielded an average time savings of 64 min/run compared to processing, reading, and interpretation by our current procedure. Our findings demonstrated that automated processing and interpretive systems yield results comparable to those of visual interpretation, while reducing the subjectivity and time required for Lyme WB analysis.
Lyme disease is a multisystem, tick-borne disease caused by the spirochete Borrelia burgdorferi. In 2006, the Centers for Disease Control and Prevention (CDC) reported 19,931 cases of Lyme disease in the United States (7), confirming that the disease continues to represent a significant public health threat. The clinical manifestations of early localized disease range from nonspecific sequelae, including malaise, myalgia, and lymphadenopathy, to more characteristic findings, such as erythema migrans (EM). In the absence of appropriate therapy, disease progression may lead to significant complications, including rheumatologic, neurologic, or cardiac manifestations (15, 16).
The diagnosis of Lyme borreliosis (LB) can be made clinically when patients from regions where the disease is endemic present with EM (5, 8, 18). However, in patients without EM but with objective clinical findings suggestive of disseminated LB, serologic testing is an important diagnostic approach. Appropriate serologic testing should follow the two-tier algorithm recommended by the CDC (6), consisting of initial testing with a sensitive screening assay (e.g., enzyme immunoassay) with positive or equivocal specimens to be tested by Western blot (WB) analysis. Current CDC criteria for WB interpretation recommend that ≥2 bands on the immunoglobulin M (IgM) WB or ≥5 bands on the IgG WB be present for the immunoblot to be considered positive (6, 9). Although WB is considered to be highly specific, current testing protocols in most clinical laboratories rely on visual reading and interpretation of WB strips. These procedures require the laboratory technologist to visually compare band intensities on the patient strip to those of a weakly reactive control. This approach is labor-intensive, time-consuming, and subjective, allowing for potential intra- and interlaboratory variation in WB reading and interpretation. Previous studies analyzing the performance of LB serologic tests among testing laboratories have demonstrated significant variation in results, even for more objective methods, such as enzyme immunoassay (2, 3, 10). Therefore, given the inherent subjectivity in reading and interpreting WBs for diagnosis of LB (i.e., Lyme WBs), one would expect to observe significant variation in WB results, with potentially adverse effects on the laboratory diagnosis of Lyme disease and subsequent patient management decisions. Due to the need for more objective and consistent interpretation of Lyme WBs, we undertook a study to evaluate and compare two systems (TrinBlot/BLOTrix [Trinity BioTech, Carlsbad, CA] and BeeBlot/ViraScan [Viramed Biotech AG, Munich, Germany]) designed for automated processing, reading and interpretation of Lyme WBs. The goal of the present study was to determine whether automated systems yield comparable results to visual reading and interpretation while reducing the subjectivity and time required for Lyme WB analysis.
MATERIALS AND METHODS
Serum specimens.A total of 518 consecutive, unique serum specimens submitted to our reference laboratory for routine LB serologic testing between June and September 2007 were included in the study. The specimens were submitted without accompanying clinical information. The study protocol was reviewed and approved by the institutional review board of the Mayo Clinic.
In addition, two Lyme WB performance panels consisting of 55 clinically characterized and laboratory-characterized serum specimens were purchased from the CDC and Boston Biomedica, Inc. (BBI; West Bridgewater, MA).
Study design.Each specimen was processed by our current procedure, performed according to the manufacturer's instructions for processing MarBlot IgG and IgM strips (MarDx Diagnostics, Carlsbad, CA) using the Autoblot 6000 instrument (MedTec, Inc., Hillsborough, NC). The strips were visually interpreted, and the results were recorded manually. Each specimen was also tested by MarBlot IgG and IgM strips using the automated TrinBlot processor (Bee Robotics, Caernarfon Gwynedd, United Kingdom) with subsequent scanning and analysis by the BLOTrix interpretive software. In addition, each specimen was tested by the ViraBlot and ViraStripe IgG and IgM strips (Viramed Biotech AG) on the automated BeeBlot processor (Bee Robotics) with subsequent analysis by the ViraScan interpretive software. A laboratory technologist blinded to the results of visual interpretation in the laboratory then reviewed the BLOTrix and ViraScan software interpretive results for each specimen (Fig. 1). The laboratory technologist, when reviewing the software interpretive results for each strip, (i) ensured that the software had analyzed only bands demonstrating uniform intensity across the entire width of the strip, (ii) checked that any background intensity (non-band intensity) had been accounted for, (iii) verified that the software had properly aligned test bands on the patient strip with bands on the serum band locator control strip, and (iv) ensured that the densitometric read was focused on the center of each test band.
Study design.
WB assays.MarDx MarBlot IgM and IgG assays (Fig. 2) utilize antigens of B. burgdorferi strain B31 for Western blot analysis. The antigens are separated by electrophoresis through a sodium dodecyl sulfate-polyacrylamide gel. The resolved antigens are then transferred to a nitrocellulose membrane. Similarly, the ViraBlot and ViraStripe IgM and IgG assays use strain B. burgdorferi B31 as the source of antigen for Western blot analysis. ViraBlot IgM and IgG assays (Fig. 2) are manufactured by separation of antigens by gel electrophoresis, with subsequent transfer to nitrocellulose. In contrast, the ViraStripe IgM and IgG assays (Fig. 2) are generated by “printing” highly purified antigens at a defined location with standardized concentrations on a nitrocellulose membrane. The ViraStripe IgM and IgG assays are currently prototype Lyme WB strips, while the MarBlot and ViraBlot assays are both Food and Drug Administration cleared.
Comparison of WB strips for a single patient specimen. The same patient specimen was tested by the MarBlot, ViraBlot, and ViraStripe assays. Strips were scanned by their respective systems, and the images were captured in tag image file format (TIFF). The migration positions of bands used in the CDC interpretation criteria are indicated by molecular mass (in kilodaltons). PC, positive control; T, test (patient) sample.
WB strip processing, reading, and interpretation.For each specimen tested by our current Lyme WB procedure, 80 μl of the serum band locator, 20 μl of the weakly reactive and negative controls, and 20 μl of patient serum were added to the appropriate channels of the Autoblot 6000 instrument. After being incubated and washed, strips were air dried and mounted for visual reading and interpretation. Each test band was visually compared to the 41-kDa band on the weakly reactive control strip and was considered present if the intensity was equal to or greater than that of the weakly reactive control. The IgM assay was considered positive if two of the following three bands were present: 23, 39, and 41 kDa. The IgG assay was considered positive if five of the following ten bands were present: 18, 23, 28, 30, 39, 41, 45, 58, 66, and 93 kDa.
Each specimen was also tested by the ViraBlot and ViraStripe IgM and IgG assays, performed according to the manufacturer's instructions on the BeeBlot automated processor. In brief, strips were added to the incubation tray and allowed to presoak in 1.5 ml of wash buffer for 5 min. Next, 20 μl of patient serum or 100 μl of control was added to the appropriate channel of the incubation tray. After being incubated and washed, strips were air dried in the incubation tray, scanned by using the ViraCam scanner (Viramed Biotech, AG), and analyzed by the ViraScan analysis software version 2.01. Based on densitometric analysis as described by Nishizuka et al. (12), 8-bit, grayscale images at a resolution of 220 dots per inch were stored in bitmap format for subsequent analysis by ViraScan. With the aid of predefined band-locator images, ViraScan locates and measures the immunospecific banding patterns on each patient strip. Each band is then analyzed for band location and maximum intensity. After a subtraction of background intensity (the average non-band intensity), the software assigns a numerical value to each band. The software then divides the numeric value of each test band by the numeric value assigned to a separate calibrator control band to determine a relative intensity. For the present study, a test band was considered present if the relative intensity met or exceeded the following manufacturer's recommended cutoff settings: ViraBlot IgG, 75%; ViraBlot IgM, 70%; ViraStripe IgG, 85%; and ViraStripe IgM, 60%.
Each specimen was also tested by the MarDx MarBlot IgM and IgG assays, performed according to the manufacturer's instructions using a TrinBlot automated processor. After automated processing, the strips were air dried in the incubation tray, scanned, and analyzed by the BLOTrix analysis software version 2.6. Similar to ViraScan, BLOTrix software utilizes densitometric analysis to compare the intensity of each test band to that of a separate calibrator control band and calculate a relative percent intensity. For the present study, a test band was considered present if the calculated relative intensity met or exceeded the following manufacturer's recommended cutoff settings: TrinBlot IgG, 90%; and TrinBlot IgM, 90%.
Statistics.Statistical analyses were performed by using statistical analysis software (SAS Institute Inc, Cary, NC). In addition to the percent agreement, kappa coefficients were determined as a secondary measure of agreement. Result agreements by kappa values are categorized as near perfect (0.81 to 1.0), substantial (0.61 to 0.80), moderate (0.41 to 0.60), fair (0.21 to 0.40), slight (0 to 0.20), or poor (<0) (11).
Workflow analysis.The average assay time for testing by our current procedure was calculated by timing three separate runs (40 specimens/run) from the addition of strips to the incubation tray through the manual recording of results by the technologist. The average assay time for testing by the automated systems was calculated by timing three separate runs (40 specimens/run) from the addition of strips to the incubation tray through the review of software results by the laboratory technologist.
RESULTS
Agreement between automated and visual interpretation.To assess agreement, the qualitative results (positive or negative based on CDC criteria) were compared after the testing of 518 consecutive serum specimens. Among the 518 specimens, 74 (14.3%) were positive for IgG, and 191 (36.9%) were positive for IgM by our current procedure. BLOTrix analysis of the MarBlot assays demonstrated an agreement of 84.7% (439/518) for IgM and 87.3% (452/518) for IgG compared to results obtained by routine testing (Table 1). ViraScan analysis of the ViraBlot IgM and IgG assays showed 85.7% (444/518) and 94.2% (488/518) agreement, respectively, while analysis of the ViraStripe assays demonstrated 87.1% agreement (451/518) for IgM and 93.1% agreement (482/518) for IgG (Table 1). A technologist blinded to the results of routine testing then reviewed the BLOTrix and ViraScan interpretive results for each specimen. The technologist-adjusted BLOTrix and ViraScan results showed substantial agreement (0.61 < κ < 0.80) with results obtained by routine testing with visual interpretation (Table 1).
Agreement of Lyme WB strip results obtained by visual interpretation, automated software, and technologist-adjusted software interpretation among prospective serum specimens (n = 518)a
In addition to the analysis of 518 consecutive serum specimens, two performance panels (BBI and CDC) consisting of 55 serum specimens were tested by the automated systems. BBI serum specimens were also tested by our current Lyme WB procedure. Agreement was assessed by comparing the results of testing to the reference WB results provided with the performance panels (Table 2). Interestingly, the technologist-adjusted ViraScan results of the ViraBlot and ViraStripe IgM assays showed closer agreement (93.3 and 80%, respectively) with the reference BBI WB results than did the results obtained by routine testing (73.3%) (Table 2).
Agreement of Lyme WB strip results obtained by visual interpretation and technologist-adjusted software interpretation among the BBI and CDC serum performance panels
Sensitivity and specificity of automated systems.After the testing of 518 prospective specimens, the results were analyzed by using the visual interpretation of the MarDx assays as the “gold standard.” Although the automated systems demonstrated comparable performances overall, BLOTrix analysis showed higher sensitivities for IgM and IgG (93.3 and 81.1%, respectively) than did ViraScan analysis of the ViraBlot (86.1 and 74.3%, respectively) or ViraStripe (78.9 and 77.0%, respectively) strips (Table 3). In contrast, ViraScan analysis of the ViraStripe IgM and IgG strips showed the highest specificity (92.0 and 95.7%, respectively) compared to visual interpretation of the MarDx MarBlot assays (Table 3).
Sensitivity and specificity of automated software or technologist-adjusted software interpretation in prospective serum specimens (n = 518)a
Adjusted sensitivities and specificities were then calculated after visual review of the automated software results by a laboratory technologist (Table 3). A total of 37/518 (7.1%) MarBlot IgM and 40/518 (7.7%) MarBlot IgG results were adjusted after visual review of the BLOTrix analyses. The majority of the changes (56/77 = 72.7%) made to the BLOTrix interpretations were from positive to negative. These adjustments yielded a marginal increase in specificity for the MarBlot assays (Table 3). In contrast, a total of 8/518 (1.5%) ViraBlot IgM, 12/518 (2.3%) ViraBlot IgG, 8/518 (1.5%) ViraStripe IgM, and 15/518 (2.9%) ViraStripe IgG results were adjusted after review of the ViraScan analyses. The majority of changes (35/43 [81.4%]) made to the ViraScan interpretations were from negative to positive, resulting in the adjusted sensitivities and specificities outlined in Table 3.
The clinical sensitivity and specificity of the automated systems were further evaluated by using the 40-member CDC serum performance panel. Specimens were categorized by clinical diagnosis according to detailed histories included with the panel, and WB results were analyzed by comparison to the clinical findings. Reference CDC WB results showed a sensitivity of 51.4% (18/35) for IgM and 48.6% (17/35) for IgG in patients with confirmed or probable Lyme disease (Table 4). The technologist-adjusted BLOTrix results showed a sensitivity of 40.0% (14/35) for IgM and a sensitivity of 37.1% (13/35) for IgG. In contrast, ViraScan analysis of the ViraBlot IgM and IgG strips demonstrated sensitivities of 85.7% (30/35) and 40.0% (14/35), respectively, while the ViraStripe assays showed sensitivities of 31.4% (11/35) for IgM and 48.6% (17/35) for IgG. Each of the systems demonstrated 100% specificity for IgM and IgG with the exception of the ViraBlot IgM assay, which showed a specificity of 80% (4/5) (Table 4).
Clinical sensitivity and specificity of technologist-adjusted software interpretation among the CDC serum performance panel (n = 40)
DISCUSSION
It is estimated that over 2.5 million LB serology tests are performed annually in the United States (1, 18). In 2006, our laboratory at the Mayo Clinic performed 75,478 LB serology tests, with 37,338 (49.5%) of these being done by Lyme WBs. These numbers indicate that LB serology continues to play an important role in the diagnosis of the disease. Furthermore, these data emphasize the need to improve the efficiency of Lyme WB processing, reading, and interpretation due to the significant time and effort required by laboratory personnel.
A significant limitation of current Lyme WB testing is the subjectivity involved in the visual reading and interpretation of test strips. Preliminary studies in our laboratory have demonstrated considerable variation in Lyme WB results when strips are visually read by different laboratory technologists (M. Binnicker, unpublished data). This variation in WB results may contribute to inaccurate diagnoses, resulting in various consequences to patients, as described in past studies (4, 13, 14, 17). Therefore, an important need of clinical laboratories is to enhance the objectivity and consistency of Lyme WB interpretation.
Our findings show substantial agreement between the results of automated and visual interpretation of Lyme WBs. Both systems we evaluated demonstrated comparable results, excellent reproducibility (data not shown), and similar features and total average assay times (Table 5). We should emphasize that both systems are designed to aid in band identification and result interpretation and yet require a laboratory technologist to review and verify results prior to reporting. In our experience, the ViraScan software application was more intuitive to operate and required fewer result modifications by the reviewing laboratory technologist (Table 3). This difference may be due, in part, to the specific manufacturer's recommended cutoff settings used in our evaluation. Clinical laboratories should perform their own thorough evaluation prior to implementing an automated system, since the appropriate cutoff settings may differ between regions where Lyme disease is and is not endemic.
Comparison of features between automated systems
The present study has several additional limitations. First, the number of clinically characterized specimens tested was limited and, therefore, no firm conclusions can be made regarding the accuracy of the automated interpretive systems. Future studies should test a large panel of clinically defined and laboratory-defined specimens in order to more accurately determine the clinical sensitivity and specificity. Second, the data presented here compare the qualitative interpretations (positive versus negative) using the current CDC criteria and do not focus on a direct comparison of individual bands. However, we observed very good correlation, overall, for the detection of specific bands between automated and visual interpretation. Interestingly, correlation seemed to be lowest for the 41- and 58-kDa bands (data not shown), and this may be due, in part, to their migration proximity to the 39- and 60-kDa antigens, respectively. These antigens are often difficult to distinguish by visual interpretation and may also require a more detailed verification following the automated software interpretation. A third limitation of the present study is that the same laboratory technologist reviewed and verified the software results for each specimen. Although this approach was used for consistency, the decision to manually adjust software results may vary between technologists. In order to minimize subjectivity and result variability, it will be essential for testing laboratories to establish specific criteria to guide and regulate the modification of software results. A point of interest for future studies will be to determine whether automated systems decrease inter- and intralaboratory result variability in comparison to visual reading and interpretation.
In summary, automated systems showed results comparable to those obtained by routine testing, while demonstrating several advantages. First, the average turnaround time was reduced by 64 min/run, translating into a time savings of approximately 1,000 h/year for clinical laboratories testing 37,000 to 40,000 specimens by WBs. Second, automated systems yield an approximate savings of 0.3 full-time equivalent in comparison to routine testing with visual interpretation. Additional benefits include the ability to electronically store data, share results with clinicians, and interface results with the Laboratory Information System. Finally, automated systems allow for a more objective interpretation of test strips, which may prove to significantly enhance the consistency of Lyme WB results.
ACKNOWLEDGMENTS
We thank the laboratory technologists and assistants in the Infectious Diseases Serology laboratory at the Mayo Clinic, who provided excellent laboratory and technical support during this study. We also thank Joseph Yao for critical review of the manuscript. The reagents and kits used in this study were provided by Trinity Biotech and Viramed Biotech AG.
FOOTNOTES
- Received 1 February 2008.
- Returned for modification 27 March 2008.
- Accepted 29 April 2008.
↵▿ Published ahead of print on 7 May 2008.
- American Society for Microbiology
