Volume 130, Issue S1 p. S22-S26
Laboratory evaluation
Open Access

Laboratory evaluation of three dual rapid diagnostic tests for HIV and syphilis in China and Nigeria

Yue-Ping Yin

Yue-Ping Yin

National Center for STD Control and Institute of Dermatology, Chinese Academy of Medical Sciences and Peking Union Medical College, Nanjing, China

Search for more papers by this author
Evelyn Ngige

Evelyn Ngige

National AIDS and STIs Control Program, Federal Ministry of Health, Abuja, Nigeria

Search for more papers by this author
Chukwuma Anyaike

Chukwuma Anyaike

National AIDS and STIs Control Program, Federal Ministry of Health, Abuja, Nigeria

Search for more papers by this author
Gbenga Ijaodola

Gbenga Ijaodola

National AIDS and STIs Control Program, Federal Ministry of Health, Abuja, Nigeria

Search for more papers by this author
Taiwo A. Oyelade

Taiwo A. Oyelade

World Health Organization, Abuja, Nigeria

Search for more papers by this author
Rui Gama Vaz

Rui Gama Vaz

World Health Organization, Abuja, Nigeria

Search for more papers by this author
Lori M. Newman

Lori M. Newman

World Health Organization, Geneva, Switzerland

Search for more papers by this author
Xiang-Sheng Chen

Corresponding Author

Xiang-Sheng Chen

National Center for STD Control and Institute of Dermatology, Chinese Academy of Medical Sciences and Peking Union Medical College, Nanjing, China

Corresponding author at: National Center for STD Control and Institute of Dermatology, Chinese Academy of Medical Sciences and Peking Union Medical College, 12 Jiangwangmiao Street Nanjing 210042, China. Tel.: + 86 25 8547 8901.Search for more papers by this author
First published: 29 April 2015
Citations: 22

Abstract

Objective

To determine the laboratory-based performance and operational characteristics of three dual rapid diagnostic tests (RDTs) for testing HIV and syphilis.

Methods

Three dual RDTs (SD Bioline, Chembio, and MedMira) were evaluated using 1514 serum specimens archived at laboratories or collected from clinics in China and Nigeria to determine sensitivity and specificity, with 95% confidence intervals. Concordance of testing results read by two technicians, stability of testing results read at two time points, and test operation characteristics were also assessed.

Results

All three of the evaluated RDTs gave excellent performance with a combined sensitivity ranging from 99.0%–99.6% for HIV and 98.3%–99.0% for syphilis, and a combined specificity ranging from 97.9%–99.0% for HIV and 97.2%–99.6% for syphilis. Concordance of testing results between two technicians and stability of testing results read within and one hour past the recommended reading period showed excellent agreement, with Kappa greater than or equal to 0.98.

Conclusions

All the tests were found to be very or fairly easy to use and easy to interpret the results. Further evaluations of these dual RDTs with whole blood in field settings, and more studies on the implication of introduction of these tests in HIV and syphilis control programs are needed.

1 Background

In 2008, 1.4 million pregnant women were estimated to be infected with syphilis worldwide, resulting in more than 520 000 adverse pregnancy outcomes, particularly in low-resource countries [1]. In 2013, there were more than 1.4 million pregnant women with HIV in low- and middle-income countries, with an estimated 240 000 children newly infected with HIV [2]. Thus, the elimination of mother-to-child transmission (EMTCT) of HIV and syphilis would directly contribute toward the attainment of Millennium Development Goals 4, 5, and 6, as well as the post-2015 goals. To achieve these goals, initiatives for the dual EMTCT of HIV and syphilis have been launched at global and regional levels [3–5]. As one of the important components of these initiatives, early diagnosis and timely intervention of pregnant women infected with HIV and/or syphilis is critical to reduce maternal morbidity and onward transmission from mother to fetus [6,7].

Serological tests are the diagnostic tests of choice for syphilis and HIV. With regards to syphilis, both treponemal (e.g., Treponema pallidum particle agglutination assay [TPPA] or Treponema pallidum hemagglutination assay [TPHA]) and non-treponemal (e.g., rapid plasma reagin [RPR]) tests can be used for laboratory diagnosis; traditionally, a non-treponemal assay is used for screening and a treponemal assay is used for confirmation. Enzyme immunoassays (EIAs) are used for HIV screening and, in high-resource settings, Western blotting is used for confirmation. All of the above tests are performed on serum or plasma derived from venous whole blood for screening of symptomatic and asymptomatic patients.

Screening of pregnant women for syphilis is recommended policy in most countries [8]; despite this, more than 520 000 adverse outcomes are recorded annually following maternal syphilis [1]. In some countries, despite the majority of pregnant women having access to antenatal care (ANC), only a small proportion of them undergo HIV and/or syphilis screening. For example, in 2012, more than 60% of pregnant women in Nigeria received ANC, but only 33% were screened for syphilis, and in some North East areas of the country only less than 10% of pregnant women were screened [9]. The infrastructure and resources required by the traditional assays, namely well-equipped laboratories with trained personnel, present a major barrier to testing uptake [10]. In addition, specimen referral and/or delivery of results can be delayed leading to loss to follow-up or lack of patient information.

To improve the screening and treatment coverage of pregnant women, a number of single rapid diagnostic tests (RDTs) for HIV or syphilis have been developed and scaled up [11,12], with several countries having recently adopted RDTs into their national antenatal HIV and syphilis screening policies [13]. However, use of two single RDTs usually requires two sets of procedures for specimen collection, test performance, and result reading, potentially leading to a significant increase in workload in busy clinic settings. In addition, the separate logistics, suppliers, and procurements of the two single RDTs could lead to stockouts. Recently, dual RDTs that can be used at point-of-care for simultaneously detecting antibodies to HIV and syphilis (dual HIV and syphilis treponemal RDTs) have been developed for use with venous whole blood, serum/plasma, and/or finger-stick whole blood. At least three dual RDTs (Chembio DPP HIV-Syphilis Assay, Chembio Diagnostic Systems Inc., USA; Multiplo Rapid TP/HIV Antibody Test, MedMira Inc., Canada; and SD HIV/Syphilis Duo, Standard Diagnostics Inc., Korea) are now available. Despite these tests having shown encouraging performance in terms of sensitivities and specificities as compared with reference tests in studies supported by the manufacturers [14], data from independent evaluations from laboratories in low-resource countries are limited. The present study aims to determine the laboratory-based performance of the three dual HIV/syphilis RDTs mentioned above compared with current reference standard assays and to assess the operational characteristics of these tests in China and Nigeria.

2 Materials and methods

Laboratory-based evaluation studies were conducted at the National STD Reference Laboratory of the National Center for STD Control (NCSTD) at the Center for Disease Control and Prevention, China, and the laboratories of the University College Hospital (UCH) in Ibadan, Southern Nigeria, and the Ahmadu Bello University Teaching Hospital (ABUTH) in Zaria, Northern Nigeria. These are the leading national laboratories qualified to conduct serological tests for HIV and syphilis in their respective countries and have participated in HIV and syphilis proficiency programs organized by WHO, the US Centers for Disease Control and Prevention, or their national HIV/AIDS and sexually transmitted infection programs.

A common protocol for the evaluation studies was prepared by WHO with inputs from the two study countries. The current laboratory performance studies were reviewed and approved by the WHO Ethics Review Committee, the Ethics Committee of the Hospital for Skin Diseases of the Chinese Academy of Medical Sciences, and the National Health Research Ethics Committee of Nigeria. In China, serum specimens archived from previous projects conducted by the NCSTD in multiple sites within the country were used. At the ABUTH site in Nigeria, specimens came from routine clinical services and, at the UCH Ibadan site in Nigeria, specimens were collected from its antenatal clinic and special treatment clinic as well as other hospitals and primary healthcare centers in Oyo State. The sample size estimation was based on selection of 50% of specimens with EIA-based HIV or TPHA/TPPA-based syphilis positivity for the evaluation and an expected point sensitivity of 99% with precision of 1% for the HIV or syphilis part of the evaluation kit. A total of 760 (380 positive and 380 negative for HIV, half of each positive or negative for syphilis) were needed for the evaluation.

The dual RDTs under evaluation were voluntarily donated by the manufacturers, but all other external funding for the studies came from WHO, the Program for Appropriate Technology in Health, and the Bill and Melinda Gates Foundation. However, the manufacturers were not involved in study design, data collection, data analysis, data interpretation, or writing of the report. The reference tests used as gold standard were selected according to the assays used in the evaluation laboratories and could be considered as evaluation reference tests. At NCSTD, the anti-HIV/P24 (HIV Ag/Ab) assay (fourth-generation HIV-EIA assay, Beijing Chemclin Biotech Co. Ltd, China) and the Serodia TPPA assay (Fujirebio Diagnostics Inc., Japan) were used as reference tests for HIV and syphilis, respectively. At UCH, the ELISA kit (Genscreener Plus HIV Ag-Ab, BIO-RAD, France) and the TPHA assay (Alington Scientific Inc., USA) were used for HIV and syphilis, respectively. Finally, at ABUTH, the HIV-1/2 Ag/Ab fourth-generation ELISA kit (Accu Diagnostics, CA, USA) and the Labkit TPHA assay (Chemelex SA, Spain) were used for HIV and syphilis, respectively.

A convenience sampling strategy was used to prepare the specimens for evaluation. All specimens with a volume of at least 60 μL were included. Each of the three RDTs, at all sites, was performed according to the manufacturer's instructions by Technician 1; test results were read within the reading time-frame as recommended by the protocol and recorded in the data recording form. Simultaneously, the same procedure to read the results was performed independently by Technician 2, who recorded the results in a separate data recording form. At NCSTD, the two technicians read and recorded the results again, independently, one hour later to evaluate test stability.

At all three sites, technicians conducting the evaluation were asked to assign, for each test, a score for the individual operational characteristics (clarity of kit instruction, ease of use, ease of interpretation of results, rapidity of test results, hands-on time, and training time required), as previously described [15].

The performance of the RDTs in terms of sensitivity and specificity, along with their 95% confidence intervals (CIs), was determined in comparison with the reference tests. The inter-reader consistency and the reading time stability were evaluated according to agreement between the two technicians and between the two reading times (those within the reading time-frame and one hour later), using the Kappa value. In addition, operational characteristics were scored following consensus by the two technicians. The χ2 test was used for statistical comparison. Analyses were performed with IBM SPSS version 19.0 (IBM, Armonk, NY, USA) and MedCalc software package version 12.2.1 (MedCalc Software, Mariakerke, Belgium).

3 Results

A total of 1514 specimens (754 archived at NCSTD, 380 clinically collected at UCH, and 380 archived at ABUTH) were included in the study, among which 728 (48.1%) were positive for HIV, 735 (48.5%) positive for syphilis, and 496 (32.8%) positive for both HIV and syphilis as identified by the corresponding reference tests (Table 1).

Table 1. Serum specimens used for laboratory-based evaluation in three study sites
Evaluation site Specimen source Specimen categorya Total
HIV+/TP+ HIV+/TP HIV/TP+ HIV/TP
National center for STD Control, China Archived at laboratory 216 132 139 267 754
University College Hospital, Nigeria Collected in clinics 140 50 50 140 380
Ahmadu Bello University Teaching Hospital, Nigeria Archived at laboratory 140 50 50 140 380
Total 496 232 239 547 1514
  • a HIV+ and HIV were defined as positivity and negativity for EIA-based detection of antibodies against HIV 1/2, respectively; TP+ and TP were defined as positivity and negativity for TPPA- or TPHA-based detection of treponemal antibodies, respectively.

Invalid tests occurred in 1 (0.07%), 5 (0.33%), and 2 (0.13%) tests conducted with Chembio, MedMira, and SD Bioline RDTs for HIV or syphilis detection, respectively. The performance results of the three RDTs against the reference tests for HIV and syphilis, in terms of sensitivity and specificity within 95% CIs, are shown in Table 2. All three RDTs showed encouraging laboratory performance for detection of HIV antibodies with a combined sensitivity of 99.6% (95% CI, 98.8%–99.9%) for Chembio, 99.5% (95% CI, 99.4%–99.8%) for MedMira, and 99.0% (95% CI, 98.0%–99.5) for SD Bioline, and a combined specificity of 97.9% (95% CI, 96.7%–98.7%), 98.3% (95% CI, 97.2%–99.0%), and 99.0% (95.% CI, 98.0%–99.5%), respectively. The combined sensitivity and specificity for identifying treponemal antibodies were 97.0% (95% CI, 95.5%–98.0%) and 99.6% (95% CI, 98.9%–99.9%) for Chembio, 94.2% (95% CI, 92.3%–95.7%) and 97.2% (95% CI, 95.8%–98.1%) for MedMira, and 96.6% (95% CI, 95.0%–97.7%) and 99.1% (95% CI, 98.2%–99.6%) for SD Bioline. No significant differences in the combined sensitivity or combined specificity for HIV were found among the three dual RDTs, although the MedMira test had a lower combined sensitivity (94.1%) and specificity (97.3%; P < 0.05).

Table 2. Performance of three dual rapid diagnostic tests for HIV/syphilis compared with respective reference tests
Rapid diagnostic test by evaluation site Comparison with HIV ELISA Comparison with TPPA/TPHA
Sensitivity (%) Specificity (%) Sensitivity (%) Specificity (%)
National Center for STD Control, China
  Chembio 99.2 97.2 95.7a 99.8
  MedMira 99.2 98.7 90.5a 96.1a
  SD Bioline 98.9 99.0 95.7 99.5
University College Hospital, Nigeria
  Chembio 100.0 100.0 98.9 100.0
  MedMira 99.5 100.0 98.9 98.9
  SD Bioline 98.4 100.0 97.4 97.4a
Ahmadu Bello University Teaching Hospital, Nigeria
  Chembio 100.0 97.4 97.4 98.9
  MedMira 100.0 95.8a 96.3 97.9
  SD Bioline 100.0 97.9a 97.4 100.0
Combined total (95% confidence interval)
  Chembio 99.6
(98.8–99.9)
97.9
(96.7–98.7)
97.0
(95.5–98.0)
99.6
(98.9–99.9)
  MedMira 99.5
(99.4–99.8)
98.3
(97.2–99.0)
94.2
(92.3–95.7)a
97.2
(95.8–98.1)a
  SD Bioline 99.0
(98.0–99.5)
99.0
(98.0–99.5)
96.6
(95.0–97.7)
99.1
(98.2–99.6)
  • Abbreviations : ELISA, enzyme-linked immunosorbent assay; TPHA, Treponema pallidum hemagglutination assay; TPPA, Treponema pallidum particle agglutination assay.
  • a Statistical significance at P < 0.05 by χ2 test compared with other study sites or other rapid diagnostic tests.

The sensitivity for HIV was comparable across the three sites (P > 0.05), but its specificity was lower at ABUTH for SD Bioline (97.9%) and MedMira (95.8%; P < 0.05). A lower sensitivity was observed at NCSTD for syphilis detection using Chembio (95.7%) or MedMira (90.5%; P < 0.05), as well as a lower specificity for syphilis using MedMira (96.1%). A lower specificity was also observed at UCH for syphilis using SD Bioline (97.4%; P < 0.05).

As shown in Table 3, agreement in testing results between the two technicians (Kappa ≥ 0.99) or between the two reading time-frames (Kappa ≥ 0.98) was excellent, indicating that all three of the RDTs had an almost perfect concordance (or repeatability) of results regardless of test conductor or time taken to read. The scores for operational characteristics are summarized in Table 4. The SD Bioline test obtained the highest score (15/16) with a significant superiority on clarity of kit instruction and ease of use. All RDTs provided results promptly; however, SD Bioline had relatively clearer instructions, a simpler testing procedure, and a short training time.

Table 3. Agreement in testing results read by two conductors and over two time points
Rapid diagnostic test by evaluation site HIV component in RDT Syphilis component in RDT
No. Agreement, % Kappa (95% CI) No. Agreement, % Kappa (95% CI)
Concordance of testing resultsa
  National Center for STD Control, China
    Chembio 754 99.9 1.00 (0.99–1.00) 754 100.0 1.00 (1.00–1.00)
    MedMira 754 100.0 1.00 (1.00–1.00) 754 100.0 1.00 (1.00–1.00)
    SD Bioline 754 99.5 0.99 (0.98–1.00) 754 99.9 1.00 (0.99–1.00)
  University College Hospital/Ahmadu Bello University Teaching Hospital, Nigeria
    Chembio 760 99.5 1.00 (1.00–1.00) 760 99.7 1.00 (1.00–1.00)
    MedMira 760 99.6 1.00 (1.00–1.00) 760 99.6 1.00 (1.00–1.00)
    SD Bioline 760 100.0 1.00 (1.00–1.00) 760 100 0.99 (0.98–1.00)
Stability of testing resultsb
  National Center for STD Control, China
    Chembio 754 99.1 0.98 (0.97–0.99) 754 98.9 0.98 (0.96–0.99)
    MedMira 754 99.6 0.99 (0.98–1.00) 754 99.3 0.99 (0.98–1.00)
    SD Bioline 754 99.7 0.99 (0.99–1.00) 754 99.2 0.98 (0.97–1.00)
  • Abbreviation: RDT, rapid diagnostic test; 95% CI, 95% confidence interval.
  • a Concordance was assessed using agreement of testing results read by two technicians independently.
  • b Stability was assessed using agreement of testing results read by a technician within the time specified in the manufacturer's protocol and 1 hour later.
Table 4. Operational characteristics of three dual rapid diagnostic tests for HIV/syphilis
Operational characteristics Top score Mean score
Chembio MedMira SD Bioline
Clarity of kit instruction 3 1.7 1.5 3.0
Ease of use 3 2.0 1.7 3.0
Ease of interpretation of results 3 2.3 2.3 3.0
Rapidity of test results 2 2.0 2.0 2.0
Hands-on time 2 1.0 1.0 1.0
Training time required 3 2.0 2.0 3.0
Total 16 11.0 10.5 15.0

4 Discussion

In addition to behavioral interventions, biomedical interventions are increasingly important for the successful control and EMTCT of HIV and syphilis. Early diagnosis and timely treatment of infected pregnant women are key elements in achieving these targets [16,17]. Routine screening for HIV and syphilis is recommended by WHO for women attending ANC since its effective implementation can reduce adverse outcomes during pregnancy [3,18]. Dual RDTs are an efficient tool for the creation of an integrated and functional platform to simultaneously detect HIV and syphilis infections during routine ANC visits and offer a unique opportunity to achieve the dual EMTCT of HIV and syphilis. The current study has a larger number of positive HIV or syphilis specimens compared with previous studies [14,19].

The above results indicate that all three of the dual RDTs have an acceptable sensitivity and specificity to detect HIV or syphilis, although the sensitivity to detect HIV antibodies (99.0%–99.6%) is generally higher than that for syphilis (94.2%–97.0%). The HIV testing sensitivity reported herein is similar to that reported by the individual manufacturers for the three dual RDTs as well as to that recently reported following a multisite evaluation study for SD Bioline in Africa [14]. However, the sensitivity for syphilis testing was generally lower than in previous reports [20]. Nevertheless, the sensitivities and specificities of the three dual RDTs met the minimal or optimal criteria for performance characteristics described in the target product profile [21], despite there being differences in sensitivity and/or specificity between study sites or RDTs. Of note, however, the target product profile requires whole blood finger-prick specimens to determine performance, whereas serum specimens were used herein. Therefore, further evaluations using finger-prick blood specimens are required.

The low variability between the three RDTs with respect to inter-reader consistency or the two time-frames suggests a high coherence and stability over time, which will be of immense benefit in busy or under-staffed ANC clinic settings in low-resource countries. These data may be useful for RDT manufactures to consider extending or simplifying the reading window. The clarity of operational instructions provided by the manufacturers was well understood and the simplicity of testing processes was easily handled by the operators, in particular for SD Bioline. In addition, evaluation of the operational characteristics suggests that all three tests show ease of use and result interpretation.

The introduction of tests in primary care settings can pose considerable challenges and difficulties, even for tests with acceptable performance in laboratory evaluations [22]. Further research on dual RDTs is needed to evaluate their performance with whole blood samples in primary care settings and to define the economic and programmatic implications of their use. This laboratory evaluation in China and Nigeria is part of a comprehensive package of evaluations proposed by WHO and the study team. In addition, a field-based evaluation study using finger-prick whole blood from women attending ANC clinics to validate the performance is underway in Zambia, as are introduction studies to determine the testing uptake, feasibility, and acceptability of introducing dual RDTs in ANC services in China, Colombia, and Nigeria. It is hoped that this research package will provide data to guide other countries as to the value of dual HIV/syphilis RDTs and inform future development of WHO guidelines.

5 Conclusions

All three dual RDTs evaluated in the present study showed acceptable sensitivity and specificity in testing for HIV-1/2 and treponemal antibodies from archived or clinical serum specimens in China and Nigeria. Owing to the ease/simplicity of procedures and interpretation of test results, these dual RDTs can serve as alternatives to the more cumbersome, demanding, and expensive conventional screening methods or single RDTs for HIV and syphilis, especially in resource-limited countries. These dual tests would not only simplify HIV and syphilis testing, but would also be more cost-effective and patient-friendly because of the need for a single blood sample [19]. Dual RDTs may thus be a key tool to accelerate the dual EMTCT of HIV and syphilis.

Acknowledgments

The authors thank Ms Wan-Hui Wei, Mr Hong-Chun Wang, and Dr Yan Han from NCSTD, China; Prof. Bakare and Mrs Ogunleye from UCH Ibadan, Nigeria; and Dr Olayinka and Miss Bakare from ABUTH Zaria, Nigeria, for conducting the evaluation with the RDTs. We also appreciate the staff who worked at the study sites collecting and preparing specimens. This work was jointly supported by a grant from the Bill and Melinda Gates Foundation through the Program for Appropriate Technology in Health/WHO Dual Testing for the Elimination of Congenital Syphilis (DTECS) Project and the UNDP/UNFPA/WHO/World Bank Special Programme of Research Development and Research Training in Human Reproduction. The National Center for STD Control, China, provided some financial support to the study. The test kits under evaluation were voluntarily donated by the companies; however, the companies were not involved in any study procedures, analysis, or interpretation of study results.

Conflict of interest

The authors declare that they have no conflict of interests.