Previous Article | Next Article ![]()
Journal of Clinical Microbiology, May 2008, p. 1692-1697, Vol. 46, No. 5
0095-1137/08/$08.00+0 doi:10.1128/JCM.01761-07
Copyright © 2008, American Society for Microbiology. All Rights Reserved.

National Serology Reference Laboratory, Australia, St. Vincent's Institute, Fitzroy, Victoria, Australia,1 Department of Medicine at St. Vincent's Health, University of Melbourne, Fitzroy, Victoria, Australia,2 Department of Microbiology and Immunology, University of Melbourne, Parkville, Victoria, Australia3
Received 4 September 2007/ Returned for modification 4 January 2008/ Accepted 6 March 2008
|
|
|---|
|
|
|---|
Immunoassays that are subjectively read and interpreted, including Western blot analyses, agglutination assays, and rapid immunoassays, are open to variability in interpretation both by the individual reader and between readers (10). The implementation of the essentials of quality management and monitoring outcomes through quality assurance programs will reduce variability and increase the accuracy of interpretation, therefore providing confidence in the testing process (6, 7, 10, 17). Laboratory quality assurance monitors all aspects of the testing process (preanalytic, analytic, and postanalytic) and employs two major methods (5, 7). Quality control (QC) is used to check that an analytical phase (test precision) is optimal (9, 13, 16). External quality assessment (EQA), also called proficiency testing, is the tool used to assess the testing process independently. Importantly, EQA can not only be used to monitor technical performances but also identifies training opportunities based on proficiency results (5, 7). Thus, adequate initial and ongoing training needs may be identified through the use of carefully constructed EQA schemes (EQAS) and are an important measure to ensure both proficiency and accuracy (5). The quality assurance of rapid HIV assays (single-use devices) might best include on-site performance monitoring and the retesting of samples as well as EQAS (17). The initial increased cost of EQAS means that testing programs in developing countries often are not deemed to have sufficient resources to allocate to quality assurance (17). Consequently, there is a need for alternative, less expensive methods that can be used in resource-constrained settings. An EQAS specifically for rapid HIV testing in nonlaboratory, resource-limited settings, therefore, must take all of this into account.
The training or, more specifically, the lack of adequate and ongoing training of test operators still represents a severe limitation to achieving accurate diagnostic testing (11, 12, 15, 33). The ability of untrained or nonlaboratory personnel to perform rapid HIV assays has been characterized in a number of different studies (12). Trained nonlaboratory personnel may have difficulties in performing, interpreting, and communicating the results of rapid HIV assays if the initial training is not appropriate or efficacious (5, 15, 17). Deviations from established training methods should be approached with caution and emphasize the importance of standardized, ongoing performance monitoring and training programs that should be a requirement for all operators (30). Photographed rapid HIV test results may be useful as a cost-effective training tool that could generate ongoing training interaction with the EQAS administrator or other central training body.
The aim of the present study was to investigate the feasibility of using difficult-to-read, photographed rapid HIV test results to assess the proficiency of operator interpretation and, secondly, the potential for using the method as a training tool. It was established that photographed results of rapid HIV assays could be interpreted with relative accuracy and that the prior experience of operators resulted in the more accurate interpretation of photographed results. From this initial study, we propose that assessing operator interpretation proficiency using photographed rapid HIV test results could be used as a complement to a quality assurance and training program or as an alternative to EQAS in resource-limited settings in which conventional EQAS is not practical. This would improve the outcomes of nonlaboratory-based testing where rapid HIV tests are employed, especially in resource-limited settings.
|
|
|---|
|
View this table: [in a new window] |
TABLE 1. Origins of laboratories participating in an external quality assessment exercise assessing photographed results of rapid HIV assaysa
|
Sample panels. Each assay had a defined sample panel (n = 7) consisting of a characterized HIV type 1 (HIV-1) antibody-negative control, an undiluted HIV-1-positive control, and five dilutions of this positive sample prepared so as to provide strong, weak, and negative results (i.e., beyond the detection limits for that assay) (10). Five dilutions or panel members plus two controls were selected on the basis of previous recommendations (8). Base matrix (defibrinated human plasma negative for anti-HIV-1/HIV-2, HBsAg, anti-HBs, anti-hepatitis C virus, and anti-human T-cell leukemia type 1/anti-human T-cell leukemia type 2; BBI Diagnostics) was used as the diluent. An experienced NRL scientist tested all panels with their respective assays by following each manufacturer's instructions.
Worksheets. For each assay type, the individual test cartridges with reactive/nonreactive bands for each panel member were photographed (with a Nikon D50 SLR digital camera) and formatted into a worksheet. The worksheets also included each manufacturer's interpretation instructions, a table in which scores could be recorded, and questions to assess the level of experience of each participant. Experience was defined by the assessment of any previous training with an estimate of how often an assay was performed by the participant (i.e., rarely, occasionally, or frequently). Each participant received a colored hardcopy of the worksheet so that the quality of the photographs supplied for interpretation purposes was uniform between participants and the NRL scientists.
Analysis of results. The NRL's simple/rapid HIV testing strategy dictates that two trained scientists interpret a test result, and a third is used for consensus before the result is reported. Three NRL laboratory scientists experienced with reading subjective HIV diagnostic assays (including rapid assays) interpreted the photographed results, and this was taken as the correct interpretation of any given result (24, 29). Participants were asked to score a sample on the basis of the intensity of reactivity as one of strong positive (++), weak positive (+), negative (–), or inconclusive (?) and then give their final interpretations of the results as positive (++/+), negative (–), or inconclusive (?) depending on the observed pattern of reactivity. Participant interpretations were scored against the correct interpretation as correct (0), inconclusive (1), false positive (2), or false negative (3). The sum of the scores for the five samples produced the cumulative error score for each participant and overall for each assay.
The error scores were analyzed in three ways. (i) First, the scores were analyzed as one group by the Kruskal-Wallis, nonparametric, one-way analysis of variance by ranks test on SPSS V14.0 (SPSS Inc., Chicago, IL) to analyze differences between the error scores of the five assays (the null hypothesis Ho:MDetermine = MMultispot = MUni-Gold = MCapillus = MSerodia; n = 740 and
= 0.05, where M is the median and
denotes the level of significance). Dunn's multiple comparison test was used for pair-wise comparisons to determine where any differences occurred. (ii) The second method was to analyze specific experience for each of the five assays by grouping error scores according to participants who indicated having previous experience or those that did not for each individual assay. The results were analyzed by Pearson chi-square tests (Ho:MIndicated Experience – MNo experience = 0; n = 148 and
= 0.05). (iii) Owing to the number of participants indicating specific experience, a second measure of experience was used. Participant interpretations were grouped according to Australian/New Zealand participants and participants from all other countries (international) as a general measure of experience. Location was considered a surrogate indicator for experience with rapid HIV assays. Only 17% of the participants in Australia and New Zealand, compared to 84% of the international participants, reported having had experience with at least one of the five rapid assays. An examination of the national testing strategies of participating laboratories and a record of their use of rapid HIV tests in the NRL's quality assurance programs supported the grouping by experience according to location. The error scores were analyzed by the Kruskal-Wallis test (Ho:MAustralian/New Zealand – MInternational = 0;
= 0.05) for each assay and for the cumulative error scores across the five assays (n = 148). In summary, the data were analyzed in three ways: (i) all interpretations, (ii) specific experience, and (iii) general laboratory experience.
It is anticipated that, in a program utilizing this method, the accuracy of interpretation as measured by the error scores will determine the proficiency of operator interpretation. This is turn will enable the level of improvement to be measured upon the training of the operator.
|
|
|---|
![]() View larger version (15K): [in a new window] |
FIG. 1. Interpretations of photographed results from five rapid HIV assays in a quality assurance exercise. Laboratory technicians (n = 148) from 121 diagnostic laboratories interpreted the photographed results of the five rapid HIV assays (Abbott Determine HIV-1/2, Bio-Rad Multispot HIV-1/HIV-2, Trinity Biotech Plc Uni-Gold HIV, Trinity Biotech Plc Capillus HIV-1/HIV-2, and Fujirebio Inc Serodia-HIV). Interpretations were graded as correct ( ), inconclusive ( ) compared to the consensus interpretation by three experienced readers from the NRL. The relative frequency of interpretation is shown for each assay.
|
2 = 292.96; P < 0.0001), except between Multispot and Capillus on a pair-wise comparison.
![]() View larger version (27K): [in a new window] |
FIG. 2. Participants' (n = 148) error scores of photographed assay results for five rapid HIV assays: (a) Abbott Determine HIV-1/2, (b) Bio-Rad Multispot HIV-1/HIV-2, (c) Trinity Biotech Plc Uni-Gold HIV, (d) Trinity Biotech Plc Capillus HIV-1/HIV-2, and (e) Fujirebio Inc. Serodia-HIV. (f) The total error scores for each assay. Reference interpretations (shown as capital letters in the upper portions of each panel) were determined by three experienced and trained NRL staff (P, positive; N, negative; I, inconclusive). Participant interpretations were compared to the reference and graded as correct (0), inconclusive (1), false positive (2), and false negative (3) and summed to give an error score.
|
2 = 12.08; P = 0.03), Capillus (
2 = 13.16; P = 0.02), and Serodia (
2 = 13.84; P = 0.02). There was a trend for experienced participants to have lower cumulative error scores for four of the five assays. Interpretations of results with the Capillus assay were the most accurate, regardless of experience.
![]() View larger version (20K): [in a new window] |
FIG. 3. Influence of experience on interpretation scores from 148 participants in a quality assurance exercise assessing the interpretation of photographed rapid HIV assay results. Two measures of experience were considered, and participants' error scores were grouped according to specific experience on the selected rapid HIV assay (a) and general laboratory experience with rapid HIV assays based on location (international and experienced as well as Australia/New Zealand and less experienced) (b). The mean error scores for specific experience (c) and general experience (d) are shown. The differences in error scores (*) between experienced and inexperienced participants for each assay were observed by the Pearson chi-square test for the Abbott Determine HIV-1/2 (P = 0.034), Trinity Biotech Plc Capillus HIV-1/HIV-2 (P = 0.022), and Fujirebio Inc. Serodia-HIV (P = 0.017) assays (a) and by the Kruskal-Wallis test for the Trinity Biotech Plc Uni-Gold HIV (P = 0.001) (P = 0.032 for all five assays [Total]) (b); the significance was set at P < 0.05.
|
2 = 11.143; P = 0.001) as well as when the error scores for all five assays were combined (
2 = 4.624; P = 0.032). The mean error scores of participants grouped for both specific assay experience (Fig. 3c) and general laboratory experience (Fig. 3d) with rapid HIV assays were similar. The average error score for the Capillus assay was close to 0, while that for Serodia was close to 4, emphasizing the difference in interpretation accuracy for these two assays. The other three assays performed similarly regardless of whether the grouping was specific or general (mean error score between 1 and 2).
These results suggested that in the present study, both specific assay experience and general laboratory experience in performing rapid assays contributed to the accurate interpretation of the photographed results, but that the mean error scores were similar for both experienced and inexperienced participants.
|
|
|---|
Testing strategies for HIV must include an algorithm for the second-round testing of all reactive or inconclusive samples from the initial screening assay to determine whether the initial reactivities were true or false (1, 24, 31). Samples interpreted as inconclusive or reactive should be retested in a second assay that uses a different technology and antigens. Samples that test negative in the primary assay are not retested, so false-negative interpretations are not defined in a standard clinical testing strategy. In the present study, inconclusive was the most common discrepant interpretation, suggesting that participants were able to identify correctly when a sample should be retested for confirmation. However, the number of false-negative interpretations seen for the Serodia assay in this study suggested that the ability of participants to interpret these results should be investigated further and the potential need for further training. This was reflected in the greater difficulty in the subjective interpretation of this assay, as evidenced by the 67% correct interpretation score for the Serodia assay versus 97% for the Capillus assay (Fig. 1). Field evaluations of rapid HIV serological tests have revealed that while most have high sensitivity and specificity, the Serodia assay did not perform as well (2, 3, 14, 19), confirming our results here regarding relative accuracy. These results suggest that certain assay formats were easier to read than others, although a more extensive analysis would be required to show this definitively.
Photographs have been used to assess eye health, investigate the accuracy of registered general nurses to grade pressure ulcers, assess medical students in emergency medicine certification exams, and assess proficiency in cytogenetic testing (27), and they currently are recommended for training personnel in the use of rapid HIV tests (4). The use of electronic images for quality assessment and training purposes also has been investigated in hematology (10). Thus, there is precedence for incorporating photographed rapid HIV assay results into an EQA program as an inexpensive method of assessing operator interpretation proficiency and, additionally, as a training tool.
The investigation of the present study into the impact that experience had on interpretation accuracy found that participants with specific assay or general laboratory experience with rapid HIV assays had a lower error score and were more accurate in their interpretations of the photographed results than were inexperienced participants. Mean error scores were consistent between groups and across the assays (error score below 2), except for the Serodia assay (error score of 4). From the sample-by-sample analysis of the individual dilutions, we identified difficulty in the interpretation of the samples mimicking weak reactivity. These dilutions were the least accurately interpreted photographs for all five assays, confirming previous studies (10, 18). These samples mimicked weak or nonspecific reactivity, similarly to that occurring during seroconversion or biological false reactivity, but most importantly do not offer information about the sensitivity and specificity of an assay (16). The accurate interpretation of rapid HIV assays in weakly reactive samples by inexperienced operators, especially when experienced laboratory participants had difficulty, implies that inexperience impacts significantly on the accuracy of results with these samples. This emphasizes the importance of training, as has been suggested previously (7, 20-23, 32). Photographed rapid HIV assay results as cost-effective training tools therefore will have importance in both initial and ongoing training programs. Assessing the degree to which trained, nonlaboratory personnel can accurately interpret photographed results is the next step in developing this method as a validated EQA for HIV testing in nonlaboratory or point-of-care settings. The assessment and training of nonlaboratory personnel may then be readily extended through the use of this method for subjectively read rapid HIV tests.
Quality assurance programs monitor performance, thereby maintaining the integrity of the testing process. When diagnostic assays that have not been adequately evaluated find their way into developing countries, the reliability of testing programs is at considerable risk (17). Further, if these assays are not quality assured on a regular basis, then errors resulting from poor performance may not be discovered (29). Therefore, it is important that all phases of diagnostic testing processes, whether performed inside or outside a laboratory, are monitored routinely through an EQA program (21). Currently there is concern over the lack of, and the standard of, diagnostics for other diseases of the developing world, such as tuberculosis and malaria. The method described in this paper has the potential to be used for other types of rapid assays and represents an effective means to help ensure the accuracy of interpreting diagnostic assays when more conventional quality assurance methods are not available.
To summarize, laboratory personnel who participated in this study, both experienced and inexperienced with rapid HIV assays, were able to interpret the photographed rapid HIV assay results with relative accuracy. Further, experience was shown to play an important role in achieving greater accuracy of interpretation. We propose that this method can be used in providing a novel, cost-effective approach to EQA for nonlaboratory rapid HIV testing. Once established, programs could be used for training and monitoring purposes, facilitating the more accurate interpretation of rapid HIV assays and thereby assisting HIV prevention efforts, especially in resource-limited countries. Most importantly, photographed test results will bring a practical and cost-effective approach to quality assurance for the scale-up of nonlaboratory HIV testing that is needed before 2010.
Published ahead of print on 19 March 2008. ![]()
|
|
|---|
This article has been cited by other articles:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Copyright © 2009 by the American Society for Microbiology. For an alternate route to Journals.ASM.org, visit: http://intl-journals.asm.org | More Info»