INTRODUCTION

Top Abstract Introduction Materials and Methods Results Discussion References

The appearance of new diseases, therapies, or technologies often leads to new clinical laboratory measurements. This is often accompanied by novel instrumentation and reagents. Under these circumstances and until laboratory procedures and reagents are standardized and laboratory leaders and technologists have been trained, laboratories may differ substantially in analytic results with comparable samples, with resultant confusion and possible misinterpretation of clinical status. Thus, external performance evaluation programs are usually introduced to achieve better performance and comparability in laboratory testing.

A good example occurred early in the spread of human immunodeficiency virus (HIV) infection and AIDS. Measurements of CD4 T-cell levels were found to be central to evaluation of disease course and therapeutic decisions. However, no national proficiency testing procedures were in place. AIDS patients who were tested for CD4 levels at different locations often reported that laboratories differed substantially in their analytic results obtained by flow cytometry. Thus, investigator-initiated programs such as the Multicenter AIDS Cohort Study (MACS) (7) and the AIDS Clinical Trials Group (ACTG) (15) under National Institute of Allergy and Infectious Diseases (NIAID) auspices instituted successful external proficiency testing programs that achieved satisfactory comparability of peripheral blood lymphocyte subset measurements by flow cytometry. Subsequently, the Centers for Disease Control and Prevention and the College of American Pathologists introduced flow cytometry proficiency testing programs. More recently, an international program for quality assurance and standardization of CD4, CD8, and CD3 measurements (the QASI program) has been instituted (12). The ACTG has also instituted a performance evaluation program for quantitative HIV assays (11).

Immune system activation is increasingly recognized as a significant component of many diseases. Autoimmune disorders with activation include rheumatoid arthritis (10), inflammatory bowel disease (17), and multiple sclerosis (8). Immune activation has also been shown to be characteristic of aging, depression, and possibly some forms of chronic fatigue syndrome and fibromyalgia (9).

The role of immune activation in the pathogenesis of HIV and AIDS is receiving increasing attention. Cytokine levels in body fluids are elevated, as are soluble products of cytokine activity such as neopterin, ₂-microglobulin (₂M), and cytokine receptors (6). Furthermore, elevated levels of activation markers in plasma have been shown to be excellent prognostic factors in HIV infection, providing data comparable to but distinct from those provided by CD4 T-cell measurements or by viral load assays (5).

An external proficiency testing program for measurement of neopterin and ₂M, two important markers of immune activation, was instituted among the four centers participating in the Multicenter AIDS Cohort Study. Over a period of several years, shipments were distributed to participating laboratories, with general agreement among three laboratories for both ₂M and neopterin assays. One laboratory, however, was not able to master one of the assays or to obtain results that were consistent with those from the other three sites. Such discrepancies adversely affected the prognostic usefulness of such laboratory measurements (5). This prior experience emphasized the need for external proficiency testing programs to verify laboratory performance and to assess the suitability of various reagent sources to meet the needs of patients and physicians dealing with immune disorders.

In a separate series of earlier studies at the University of California at Los Angeles (UCLA) (1, 2) many factors were found to influence the outcome of assays for tumor necrosis factor alpha (TNF-), gamma interferon (IFN-), neopterin, soluble TNF receptor II (sTNF-RII), soluble interleukin-2 receptor alpha (sIL-2R), and ₂M. Substantial differences in apparent levels of analytes were frequently found when ELISA kits from different manufacturers were used (2). Furthermore, the analytic results from different lots of ELISA kits supplied by a single manufacturer occasionally differed by as much as 50%. In some cases, differences were found in the standards provided by separate manufacturers (2). In addition, it was demonstrated that many cytokines and products/markers of immune activation were stable on frozen storage and could be shipped to participating laboratories. Thus, batch testing of frozen stored samples is feasible. The findings indicated that for longitudinal studies, the levels of cytokines and immune activation markers in plasma or serum should be measured using preverified reagents from one manufacturer. Furthermore, proficiency testing and external quality assurance programs can help to develop a needed consensus.

The need for uniformity in the standards for quantitative assays is clearly apparent. International reference standards are available for many cytokines (13, 14, 16) but are not available for soluble cytokine receptors or soluble activation markers. However, a 1995 report (4) noted substantial differences in terms of sensitivity and results in 11 laboratories using a variety of assays for TNF-. Also, an earlier study (3) described substantial differences in commercial reagents and standards provided in ELISA kits for IL-2, IL-6, and TNF- in 1992. That report (3) ended with a plea for "real standardization of immune assays for cytokine quantitation." Progress and problems in this area were reviewed in 1997 (19).

By 1996, the Adult and Pediatric ACTG programs had established more than 15 immunology laboratories to evaluate immunologic parameters relevant to HIV infection and its therapy. Recommendations were made to the ACTG and the Division of AIDS (DAIDS), NIAID, that an external proficiency testing program be tried as a quality assurance procedure for laboratory performance of cytokine and activation marker measurements. Such a program was initiated in January 1997 and terminated in January 1999. Three separate shipments of biological fluids were carried out. Plasma as well as supernatant fluids from stimulated immune cells were evaluated. Replicate samples were included to evaluate intralaboratory variability. Assays for cytokines included TNF-, IFN-, IL-1, IL-2, IL-4, IL-6, IL-10, and the chemokines Rantes, MIP-I, and MIP-1. Assays for levels of immune activation markers in plasma included neopterin, ₂M, sTNF-RII, and sIL-2R. A total of 11 laboratories participated. Quite remarkable differences between laboratories became apparent in the analyses of the first shipment. Significant problems were uncovered. When these were addressed, more-uniform results were obtained. The value of an external proficiency testing program was documented.

	MATERIALS AND METHODS

Top Abstract Introduction Materials and Methods Results Discussion References

Participation. All of the Advance Technology Laboratories for the Adult ACTG and the Immunology Research Laboratories of the Pediatric ACTG were repeatedly invited to participate. Those that elected to participate and reported results from the Adult ACTG program are listed in Table 1. The same laboratory at UCLA (the Clinical Immunology Research Laboratory, Center for Interdisciplinary Research in Immunology and Disease) participated in both the Adult ACTG and Pediatric ACTG programs. The University of Miami School of Medicine and Albert Einstein Medical Center, New York, N.Y., participated briefly at the invitation of DAIDS, NIAID. Participation was defined as contributing analytic data on one or more of the sample shipments. Conference calls were held regularly, starting in March 1997, by the ACTG Cytokine and Soluble Marker (CSM) Focus Group, which constituted an advisory group for this program.

Implementation. Large volumes of plasma were obtained by FAST Systems from a number of seropositive and seronegative individuals. Levels of selected cytokines and soluble markers were determined by the procedures and with the reagents available at UCLA so that a range of levels could be selected. The results were reported to FAST Systems, where a new coding system (known only to them and to John Spritzler, SDAC) was introduced. John Spritzler (SDAC) and Myron Waxdal (FAST Systems) determined the composition of the shipments, and 1-ml aliquots were prepared for shipment.

Quantitation of plasma levels of cytokines and soluble activation markers. ₂-Microglobulin was quantified using microparticle enzyme immunoassay (microparticle EIA) (Abbott Laboratories, Abbott Park, Ill.) and enzyme immunoassay (EIA) kit (Coulter, Miami, Fla.). Neopterin was measured with a competitive EIA kit (ELI test; BRAHMS, Berlin, Germany). sIL-2R was determined with EIA kits (Endogen Inc., Cambridge, Mass., and Immunotech, Marseilles, France). sTNF-RII was quantitated in plasma at a 1:20 dilution by using EIA kits (HyCult, Uden, The Netherlands; R&D Systems, Minneapolis, Minn.; and Medgenix, Fleurus, Belgium). TNF- was measured with EIA kits (Medgenix; Innogenetics, Zwijndecht, Belgium; Endogen; Biosource International; and Genzyme). IL-10 was measured with EIA kits (from Immunotech, from Endogen and from Biosource International). IFN- was determined by using EIA kits with and without the CIRID at UCLA modification of the manufacturer's protocol (Immunotech, Endogen, Biosource International, Genzyme, T-Cell Diagnostics, and R&D Systems). IL-2, IL-4, IL-6, and RANTES were each measured with EIA kits from Endogen and from Biosource International. MIP-1 and MIP-1 were measured with R&D System's Quantikine EIA kits. All assays were performed according to the manufacturer's instructions.

Shipments. (i) On 30 September 1997, six aliquots of plasma were sent to participating laboratories. There were three aliquots (replicates) of a single normal plasma and one sample each from three HIV-seropositive individuals. Reports were collected in October, November, and December 1997. Analyses included TNF-, IFN-, IL-2, IL-10, ₂M, neopterin, sTNF-RII, and sIL-2R. The data were analyzed at SDAC (Harvard School of Public Health) and shared with participating laboratories in the following months.

Statistical procedures. Analytic data from participating sites were collected by FAST Systems in the months after each shipment and transferred to SDAC, Harvard School of Public Health. Results that were below the limit of detection were indicated as zero on the plots but were excluded from calculations of means and coefficients of variation.

	RESULTS

Top Abstract Introduction Materials and Methods Results Discussion References

Study I: HIV-seronegative and -seropositive plasma samples. Results were received from nine laboratories (Table 1). Three laboratories tested six analytes, four laboratories tested three analytes, one laboratory reported two assays, and one laboratory reported only one assay. Replicate samples (three) of an HIV-seronegative plasma were included in the shipment. These results are plotted in Fig. 1 as the first three samples, but they were in fact not distributed in sequence among the six samples. Mean levels of tested analytes at different laboratories are presented in Fig. 2A, and calculated coefficients of variation are shown in Table 2. Three HIV-seropositive plasma samples with various degrees of abnormalities are included in Fig. 1 as well.

View larger version (23K): [in this window] [in a new window]   FIG. 1.   Analytic data from study I for neopterin (A), sTNF-RII (B), and TNF- (C) measured in several laboratories (designated by capital letters). Three aliquots (triplicates) of plasma (samples 1, 4, and 5) from a single seronegative donor were included, and the data are grouped on the left. Results for three plasma samples (samples 2, 3, and 6) from different HIV-seropositive donors are presented on the right.

View larger version (35K): [in this window] [in a new window]   FIG. 2.   Mean levels of cytokines, chemokines, and soluble products of immune activation reported by individual laboratories (designated by capital letters). Mean levels from triplicate samples of normal plasma in study I (A) and supernatant fluid from PHA-stimulated PBMC in study III (B) are presented for the laboratories that reported data. Assay values for all analytes are in picograms per milliliter.

Study II: supernatant fluids. A total of 10 samples were shipped to seven sites. These included three replicates of a 72-h PHA-stimulated PBMC supernatant (samples 1, 4, and 8) and four nonstimulated samples (samples 2, 3, 6, and 9). One PHA-stimulated whole blood cell supernatant (sample 7) and two LPS-stimulated cell supernatants (sample 5 from PBMC and sample 10 from whole blood) are included. A total of nine markers (IFN-, TNF-, IL-2, IL-4, IL-6, IL-10, MIP-1, MIP-1, and Rantes) were measured using reagents from five sources. Representative results are presented in Fig. 3.

View larger version (31K): [in this window] [in a new window]   FIG. 3.   Analytic data from study II for IFN- (A), TNF- (B), IL-2 (C), and Rantes (D) measured in several laboratories (designated by capital letters). Three replicate sample (triplicate samples 1, 4, and 8) obtained from a PHA-stimulated 72-h supernatant are grouped on the left. Four samples (samples 2, 3, 6, and 9) were not stimulated. One PHA-stimulated whole blood supernatant (sample 7) and two LPS-stimulated cell supernatants, one from PBMC (sample 5) and one from whole blood (sample 10), are included. K and K' designate results from laboratory K using two different reagent sources.

Study III. Six supernatant samples (1 to 6), including three replicates and six different plasma samples, were included in this study. Two or more laboratories reported analyses for TNF-, IFN-, IL-2, IL-4, IL-10, and Rantes. Values for IL-6, MIP-I, MIP-I, sIL-2R, sTNF-RII, neopterin, and ₂M were reported from individual laboratories.

View larger version (26K): [in this window] [in a new window]   FIG. 4.   Analytic data from study III for TNF- (A), IFN- (B), and Rantes (C) measured in several laboratories (designated by capital letters). Three replicates of a stimulated PBMC supernatant (samples 1, 3, and 5) are grouped on the left. Three other supernatants (stimulated [samples 2 and 6] and nonstimulated [sample 4]) are presented. C and C' designate results from laboratory C using two different reagent sources.

	DISCUSSION

Top Abstract Introduction Materials and Methods Results Discussion References

An external evaluation program to assure quality performance of many clinically important assays is required of laboratories participating in the College of American Pathologists accreditation program. New assays, however, may not be included in these programs until a need is established, usually by extensive use in clinical practice. Proficiency testing programs for plasma cytokines, chemokines, and the soluble markers of immune activation are not in place.

The findings presented here emphasize the importance of having a well designed and critically evaluated external performance evaluation program when measurements are clinical relevant and are to be conducted at multiple sites. Levels of cytokines, chemokines and soluble products of immune activation are increased in many autoimmune and inflammatory disorders and in HIV infection and are altered by aging (18; J. L. Fahey, J. F. Schnelle, J. K. Thomas, M. E. Gorre, N. Aziz, and P. Nishanian, submitted for publication). Increasingly, immune-based therapies are designed to alter cytokine activities.

Initially, in the first study, laboratory performance was uneven as was evident with the results for replicate samples at several sites. This proved to be due to inexperience, inadequate supervision, misunderstanding of procedures, and other reasons. However, after discussions led by experienced laboratory personnel, difficulties were addressed, and replicate values were better by the third study. However, there were laboratories where the experienced technician left to go to graduate school or other employment and the process of education and training began again.

Differences in reagents and standards between suppliers was a more resistant problem. Because international standards are available for almost all cytokines, it was surprising to find marked differences within the cytokine measurements. On the other hand, the capacity for errors in laboratory performance is almost limitless. However, there was a group of laboratories, using the same supplier, which usually had comparable results.

International standards are not yet available for the soluble receptors and other products of immune activation. Thus, it is probably advisable to identify a single source of reagents and standards for each assay after their usefulness has been verified.

We did not find one reagent supplier superior to all the others, but the study was not designed to test all suppliers or reagents kits. In general when several were compared, one or two sources appeared to be more useful. However, we have had the experience of major changes in the quality of a reagent or in a reference standard from a single reagent manufacturer (2).

The distinction between an external proficiency testing program for performance evaluation and the provision of international reference standards, such as for cytokines, is important. International standards are used for calibration of reference materials. In contrast, external proficiency testing programs assess actual assay performance and allow laboratory evaluation. Many labs assume that they are doing good work, but participation in an external proficiency testing program is a means of proving it. Also, participation in well-run external proficiency testing programs for performance evaluations can help new, inexperienced, or otherwise disadvantaged laboratories to achieve high performance levels by consultation with more experienced personnel.

The premise in undertaking this external proficiency testing program was that laboratories which were funded to support multicenter clinical therapeutic trials might not provide similar values for tests of cytokines and soluble markers of immune activation (CSM). At the time that the studies were begun, CSM testing was still evolving rapidly, with many putative suppliers of reagents and standards and a variety of techniques. No proficiency testing programs for performance evaluations were in place, and some laboratories had not established methods or quality assurance procedures. A corollary assumption was that laboratories that participated would benefit from the experience. An obverse corollary would be that laboratories that did not participate might well have unrecognized problems. Deficient laboratory performance is documented in other studies (3, 4).

The experience reported here indicates the potential value of an external proficiency testing program for CSM on a larger, conceivably national, scale. The need reaches across many areas of adult and pediatric medicine. Physicians can use immunologic tests for cytokines and for the products of cytokine activity in the evaluation of many autoimmune diseases, of infectious diseases affecting the immune system (such as HIV infection), and of other disorders characterized by changing balances within the immune system. Furthermore, quality evaluations of relevant immune activities are essential for monitoring therapies designed to activate or suppress immune functions.