INTRODUCTION

Top Abstract Introduction Materials and Methods Results Discussion References

Prevalence studies indicate that around 5 to 15% of the exposed health care workforce is sensitized to natural rubber latex (NRL). The general population exhibits a much lower prevalence of NRL sensitization (around 6 to 7%) (1, 3, 4, 11, 12, 16, 17, 18). These prevalence estimates are based on seroprevalence with a variety of assays. The marked discrepancies in seroprevalence rates and risk estimates among studies were thought to be due to the reduced sensitivity of these assays compared to puncture skin tests (PST) (7) or overestimation of the seroprevalence where the true seroprevalence is low (20). PST has been regarded as a primary confirmatory test for the assessment of patients for immunoglobulin E (IgE)-mediated latex disease, although the absence of a Food and Drug Administration (FDA)-licensed Hevea brasiliensis latex extract in the United States has restricted its use in the diagnosis of latex hypersensitivity. Because of this, serological tests have become critically important in diagnosis. We have shown marked differences in the diagnostic performances of these serological tests compared to either clinical history or results of PST with a well-characterized skin test reagent (7). In that study, the current FDA-cleared latex IgE assays produced a substantial number (25 to 28%) of false-negative and false-positive IgE antibody results. In order to investigate whether a partial explanation of the poor association between serological assays and PST for the diagnosis of latex hypersensitivity was due to systematic biases within the assays themselves, we undertook a comprehensive analysis of their performance. Clinical accuracy and positive threshold cutoffs for latex-specific IgE using the three presently FDA-cleared diagnostic tests, CAP System RAST FEIA (CAP) (Pharmacia-UpJohn Corporation, Uppsala, Sweden), the AlaSTAT Microplate Assay (Diagnostic Products Corporation, Los Angeles, Calif.), and the HY-TEC EIA System (HYCOR Biomedical, Irvine, Calif.), were compared. We did this by using the results of nonammoniated latex PST as the diagnostic discriminator and preparing receiver operating characteristics (ROC) curves. The ROC plots graphically display the entire spectrum of a test's performance for a particular sample group by demonstrating the ability of a test to discriminate between alternative states of health. The points along the ROC curve represent the sensitivity-specificity pairs corresponding to all possible decision thresholds for defining a positive test result. On the y axis, sensitivity, or the true-positive fraction, is plotted. On the x axis, the false-positive fraction (or 1 specificity) is plotted. This is the fraction of truly negative subjects who nevertheless have positive test results; therefore, it is a measure of specificity (13). The area under the ROC curve (AUC) is an overall index of diagnostic accuracy that is not dependent on a decision threshold. An AUC of 0.5 indicates that the discriminatory ability of the test is no better than chance. An AUC of 1.0 indicates perfect discriminatory ability.

	MATERIALS AND METHODS

Top Abstract Introduction Materials and Methods Results Discussion References

Human sera. The Human Subjects Review Board of the National Institute for Occupational Safety and Health (NIOSH), Centers for Disease Control and Prevention, approved the study design. Subjects (n = 311) were recruited from across the United States as part of an FDA-reviewed multicenter latex skin-testing study protocol (8). Following informed consent, the subject's latex hypersensitivity history status was determined based on an extensively critiqued clinical history questionnaire (6). Whole blood was collected by venipuncture, clotted for 30 min, and centrifuged, and the serum was aliquoted, coded, and frozen until the time of analysis. All subjects underwent PST with a bifurcated needle with saline, histamine (1.8 mg/ml; Allermed Laboratories, San Diego, Calif.), and nonammoniated latex (Greer Laboratories, Lenoir, N.C.) at 1, 100, and 1,000 mg/ml as described elsewhere (12, 15). Sera were collected from 311 (131 latex PST-positive and 180 PST-negative) subjects. Separate quality control sera containing 1 to 3 IU of latex-specific IgE/ml were analyzed in multiple runs of each assay to assess between-assay variation. Inclusion and exclusion criteria, as well as a more detailed description of the FDA-approved study which was the basis of the samples obtained for the present study, are given elsewhere (6, 8).

Serological analyses. NRL-specific IgE antibody was measured in the three FDA-cleared immunoassays briefly described below using coded sera in a masked mode. After submission of the data, an independent investigator at NIOSH broke the latex hypersensitivity history and PST codes.

Statistical analyses: Curves were prepared and analyzed using GraphRoc for Windows (version 2.0; downloaded from http://members.tripod.com/refstat/grdownload.htm). Data on 311 samples for which we had data from all three analyses were analyzed. A two-tailed type I error level of 0.05 was considered significant. Statistical analyses were performed using SPSS (version 9; SPSS, Inc., Chicago, Ill.) and SAS (SAS Institute Inc., Cary, N.C.). Assay performance was computed using the following definitions (where FN is a false-negative diagnostic test result, FP is a false-positive diagnostic test result, TP is a true-positive diagnostic test result, and TN is a true-negative diagnostic test result). Sensitivity [TP/(TP + FN) · 100] was defined as the percentage of positive tests in subjects with a positive latex PST. Specificity [TN/(FP + TN) · 100] was defined as the percentage of negative tests in subjects with a negative PST. Predictive value for a positive test [TP/(TP + FP) · 100] describes the percentage of subjects with a positive test that have a positive PST. Predictive value for a negative test [TN/(TN + FN) · 100] is the percentage of subjects with a negative test that have a negative PST. Efficiency [(TP + TN)/(TP + FP + TN + FN) · 100] describes the percentage of subjects correctly classified as having a positive and negative latex PST. McNemar's test was used to assess statistical differences among the sensitivities, specificities, efficiencies, and positive and negative predictive values of the different methods. An averaging method (10) was employed to minimize bias in cases where sera had results below the limit of detection of the assay. For testing of the significance of differences between AUCs for multiple ROC curves, the method of Hanley and McNeil (9) was used. Confidence intervals (95%) were calculated from a normal distribution as previously described by Galen and Peters (5). The thresholds that maximized diagnostic efficiency for our cohort were chosen as the optimal cutoffs. If the optimal decision threshold obtained in the ROC analysis was different from the manufacturer's recommended threshold, a Bayesian analysis was conducted to obtain predictive values for a positive test, predictive values for a negative test, and efficiency of the test for both decision thresholds using the following formulas: PPV = Se(P)/[Se(P) + (1  Sp)(1  P)] and NPV = Sp(1  P)/[(1  Se)P + Sp(1  P)], where PPV is the predictive value of a positive test, NPV is the predictive value of a negative test, Se is sensitivity, Sp is specificity, and P is prevalence.

	RESULTS

Top Abstract Introduction Materials and Methods Results Discussion References

The reproducibilities of the three assays have already been reported (7). Briefly, intra-assay agreement was evidenced by 96% concordance of positivity among results of the 22 coded duplicate (split) specimens in all three assays using the respective manufacturers' recommended cutoffs. Intra-assay coefficients of variation for the interpolated kU/liter results obtained with the 17 split sera from PST-positive subjects were 24.9 (CAP), 16.1 (AlaSTAT), and 15.2% (HY-TEC). Between-assay variation for the three assays was 10.5% (n = 36; mean level, 0.78 IU/ml), 12.4% (n = 12; mean level, 1.61 kU/liter), and 20.3% (n = 69; mean level, 2.41 kU/ml) for the CAP, ALASTAT, and HY-TEC assays, respectively. The ROC AUCs ± standard error based on PST (Fig. 1) for the three diagnostic tests were 0.858 ± 0.024, 0.869 ± 0.024, and 0.924 ± 0.017, respectively, for AlaSTAT, CAP, and HY-TEC. The HY-TEC system had a significantly greater AUC based on PST than those observed for both AlaSTAT (P < 0.05) and CAP (P < 0.05) assays. When the diagnostic tests were probed as to the cutoffs giving maximal diagnostic efficiency for PST, the CAP and AlaSTAT assays yielded values of <0.35 kU_A/liter and <0.35 kU/liter, while the HY-TEC assay yielded 0.11 kU/liter. The diagnostic efficiencies based on PST at these cutoffs were 87.1, 88.1, and 88.7% for AlaSTAT, CAP, and HY-TEC, respectively. The diagnostic sensitivity (P < 0.001) and negative predictive value (P < 0.05) of the HY-TEC assay, based on PST, were significantly greater than those displayed by the CAP and AlaSTAT systems, while the specificity (P < 0.05) was significantly lower. These results are shown in Table 1. Table 2 indicates the positive and negative predictive values and efficiency of the HY-TEC assay using the manufacturer's recommended decision threshold and the optimal decision threshold obtained in the ROC analysis for various prevalence situations.

View larger version (23K): [in this window] [in a new window]   FIG. 1.   ROC curves based on PST obtained in the analysis of 311 sera in CAP, Diagnostic Products Corporation AlaSTAT, and HYCOR HY-TEC assays.

	DISCUSSION

Top Abstract Introduction Materials and Methods Results Discussion References

Clinical accuracy is the basic ability to discriminate between two subclasses of subjects where there is some clinically relevant reason to do so. This concept of clinical accuracy refers to the quality of the initial classification of the subjects based on a diagnostic discriminator. The accuracy of the probing provided by the discriminator is the basis of any comparisons of the usefulness of diagnostic testing. ROC curves yield a simple graphical method to evaluate the trade-offs obtained between sensitivity and specificity across all test cutoffs. In the present work, we use a discriminator (PST) which has been rigorously validated (6, 8), yielding the possibility to determine, with some confidence, the accuracy of the diagnostic tests used to dichotomize subjects.

Choosing the optimal decision is a trade-off between optimizing sensitivity and specificity. The optimal decision thresholds obtained in this analysis were selected assuming that the cost of a false-positive result and the cost of a false-negative result were equal, but this may not be the case in some clinical applications. The optimal decision threshold for a specific clinical application involves a number of factors that are not properties of the testing system; rather they are properties of the clinical application. These include prevalence, the outcomes and the relative values of those outcomes, the costs to the patient and others of incorrect classification (false-positive and false-negative classifications), and the costs and benefits of various interventions. These characteristics interact with test results to affect usefulness. Methods have been developed for determining the optimal decision threshold based on the prevalence and the costs of incorrect classification (14, 21). In general, a higher decision threshold is preferred if the prevalence is low or if the cost of a false-positive result is greater than the cost of a false-negative result. A lower decision threshold is preferred if the prevalence is high or if the cost of a false-negative result is greater than the cost of a false-positive result. We presented data for the HY-TEC assay demonstrating the effects of using the manufacturer's recommended decision threshold and the optimal decision threshold we obtained from the ROC analysis. These data assume that sensitivity and specificity are inherent properties of the test and thus independent of prevalence. Although this is generally assumed to be the case, sensitivity and specificity may vary among different subpopulations and thus are dependent on the composition of the population under study (2). Unlike sensitivity and specificity, diagnostic efficiency is dependent on disease prevalence, and the prevalence in the study sample may not be representative of the prevalence in the target population in some clinical applications; thus, the diagnostic efficiencies reported for the assays cannot be generalized to other clinical applications. Another disadvantage of comparing diagnostic efficiencies of different tests is that two tests may have the same diagnostic efficiency but perform quite differently. For example, one test may result in many false positives and few false negatives whereas another test may result in many false negatives but few false positives.

ROC analyses also provide support for the proposed hypothesis (15) that IgE antibody assays can detect different subsets of IgE antibody of a given specificity, possibly as a result of differential specificities of their allergen-containing reagents. There are several possible reasons for this: different batches of source latex are known to vary up to 25-fold in total allergen content (19); sensitized individuals produce specific IgE antibodies to at least 8, and possibly as many as 11, Hevea allergens, Hev b1 to Hev b11 (http://dmd.nihs.go.jp/latex/allergen-e.html); and all of these allergens differ in structure, size, net charge (pI), relative allergenicity, and abundance in NRL, and many have been sequenced (http://www.iit.edu/~sgendel/nonfdall.htm). Moreover, aqueous latex extracts vary widely in their relative contents of rubber particle-associated proteins (Hev b1, or rubber elongation factor, 14.6- or 58-kDa tetramer, and Hev b3, or prenyltransferase or small rubber particle protein, 23 to 27 kDa). The relative contents and ratios of Hevs in the final allergen preparation most probably could affect the diagnostic accuracy of a specific test. Other potential causes of allergen-containing-reagent heterogeneity include variable stability during storage and variable binding of allergen to labels (e.g., biotinylated copolymer in AlaSTAT) or solid supports (sponge in CAP; cellulose disk in HY-TEC) (15).

In the present study, we have examined the diagnostic accuracies of three FDA-cleared latex-specific IgE antibody immunoassays using 311 sera that were collected from subjects participating in a multicenter latex skin-testing study (6, 8). The diagnostic performances of the three assays using the manufacturers' recommended decision thresholds have already been described (7). We extend those findings by comparing the diagnostic accuracies of the three FDA-cleared anti-latex IgE tests by the use of ROC curve analyses. The results of these analyses indicate that the HY-TEC system yields a significantly greater AUC than CAP or AlaSTAT when PST is used as a diagnostic discriminator. At this cutoff, the HY-TEC system has an increased sensitivity of as much as 9.9% over CAP and AlaSTAT (at their respective maximally efficient cutoffs of <0.35 kU_A/liter and <0.35 kU/liter) with a reduction in specificity of only 4.4%, clearly indicating that it is the most sensitive of the three tests at their optimal thresholds in a simultaneous comparison. It should be kept in mind that the HY-TEC assay using the 0.11-kU/liter cutoff misclassified 16.8% of PST-positive individuals as negative and 7.2% of PST-negative individuals positive. For comparison, the HY-TEC test using a 0.05-kU/liter cutoff yields a diagnostic sensitivity of 91.6% with a diagnostic specificity of 73.3%, while at 0.11 kU/liter, the diagnostic sensitivity is 83.2% with a diagnostic specificity of 92.8%. Comparing the two HY-TEC cutoffs (0.05 and 0.11 kU/liter) indicates a loss of sensitivity at the higher cutoff of 8.4% with a gain in specificity of 19.5%. The positive and negative predictive values and efficiency of the HY-TEC assay using the manufacturer's recommended decision threshold and the optimal decision threshold obtained in the ROC analysis will change (Table 2) depending on prior probability (prevalence).

ROC analysis is uniquely suited to situations where multiple unknown complex multivariate responses are being examined simultaneously. In this case, PST with NRL (which is a complex mixture of numerous [240] proteins) was evaluated using FDA-cleared serum tests which potentially contain multiple similar or modified latex antigens. Although precise chemical correlations between the skin test and serum antibody specificities are difficult to know with certainty, the presence of a validated discriminator (PST) allows the mathematical interpretation of diagnostic-test responses at increasing positive-negative thresholds and their comparison to the presence or absence of disease. Finally, care should be exercised when interpreting negative IgE antibody results from the CAP and AlaSTAT assays, even at their manufacturers' recommended positive cutoffs, since these assays misclassify approximately 25% of subjects who are skin test positive as IgE antibody negative (false negative). Care should also be exercised when interpreting positive IgE antibody results from the HY-TECH assay, even at the manufacturer's recommended positive cutoffs, since this assay misclassifies approximately 25% of PST-negative subjects as IgE antibody positive (false positives).