Assessing the effect of the Expanding Maternal and Neonatal Survival program on improving stabilization and referral for maternal and newborn complications in Indonesia

To determine if the Expanding Maternal and Neonatal Survival (EMAS) program was associated with improved effectiveness of the referral system in Indonesia to facilitate timely and effective management of complications experienced by women and newborns.

including health, among the most developed provinces of Java and Bali compared with those on islands further away from the capital. 8 The two most common causes of maternal deaths in Indonesia are hypertensive disorders (pre-eclampsia and eclampsia) and postpartum hemorrhage (PPH), followed by sepsis, obstructed/prolonged labor, and complications of unsafe abortions. 9 Most births in Indonesia are attended by a skilled birth attendant (83%) and the majority (63%) of these births occur in a health facility. 10 Nevertheless, an assessment in five regions in Indonesia found that 41.9% of maternal deaths occurred at public hospitals and that Eastern Indonesia and Sulawesi provinces reported the second highest maternal mortality rates and lower levels of service coverage, compared with Java and Bali. 11 Persistent high maternal morbidity and mortality are likely linked to the lack of health workers' capacity to provide high-quality care, potentially due to insufficient training, limited experience in managing complications, inadequate infrastructure in which care is provided, and geography. 7,9,12,13 Accordingly, efforts have been focused on improving timely access to care, 14,15 including effective management of complications during labor at community health facilities (puskesmas), and, if necessary, timely and effective referral to a hospital for specialist care. 16 A well-functioning referral system should act as an early warning system for comprehensive emergency obstetric and newborn care (CEmONC) facilities; ideally, ensuring that the necessary staff, supplies, and equipment are fully prepared to provide emergency obstetric and newborn care 24 hours a day, 7 days a week. 16 Previous international studies have identified a number of facility-based factors that slow response times, including ineffective management of complications, a lack of supportive supervision, and unreliable standards of care and monitoring in referral facilities. [17][18][19][20] Overall, a lack of readiness to respond to emergency complications at referring facilities contributes to delays in the provision of care. These factors include inadequate transport, poor communication infrastructure, lack of basic equipment, and inadequate skills and knowledge on the part of providers. 17,18,21 In Indonesia, public health care within districts is provided through a network of puskesmas and hospitals. Across the country, PONED (Pelayanan Obstetri dan Neonatal Esensial Dasar) puskesmas are staffed with a physician and several midwives and nurses who are responsible for providing basic emergency obstetric and newborn care (BEmONC), including making detailed plans for how referrals are to be arranged. 22 While the puskesmas network covers a population of 30 000-50 000, many are located in relatively isolated areas with limited infrastructure, variable quality of services, and a lack of PONED capacity. 22,23 In the event of complications, puskesmas staff should refer clients to hospitals, which should be equipped to provide CEmONC. 22 Previous studies in Indonesia 24,25 identified transport as a major barrier to efficient referral; emergency transport was often unavailable and private transport was unreliable and incurred costs. One study identified patients' and providers' uncertainty about where to seek emergency care, which resulted in patients traveling to several hospitals before receiving treatment, as a source of delay within an ineffective referral system. 12 A reluctance to admit poorer women covered by social health insurance was also found to be the case in some facilities. 13,24 Other barriers include a lack of proper documentation for health insurance registration, the distance to health facilities, shortage of qualified health workers, overcrowded health facilities, and suboptimal health facility accreditation. 26 Indonesia's highly decentralized health system and a growing private sector, which includes private hospitals and midwives, may also contribute to continuing high maternal morbidity and mortality rates. These may include difficulties in coordination between public and private systems 7,9,27 and a lack of standardized protocols for managing emergency complication and referrals between puskesmas and hospitals. 28 Changes to the health insurance scheme for pregnant mothers may have also influenced service provision and client care-seeking practices. Between 2011 and 2013, Indonesia provided universal maternal health coverage through several different plans. 22,29 After 2014, a single payer system, Jaminan Kesehatan Nasional (JKN), was launched to facilitate achievement of universal health care. Beneficiaries of JKN are entitled to comprehensive maternity benefits, including institutional childbirth coverage for normal births at the puskesmas level, however, pregnant women are not allowed to directly seek treatment at a hospital or specialist clinic, except in an emergency situation or with a referral letter. 22,26 The United States Agency for International Development (USAID) funded the Expanding Maternal and Neonatal Survival (EMAS) program (September 2011 to March 2017) to support the government of Indonesia in reducing maternal and newborn mortality by strengthening the quality of care 30 provided in puskesmas and hospitals and in strengthening district-level referral pathways. The EMAS program is fully described in the overview paper published in this Supplement. 31 The purpose of the present study was to determine whether the EMAS program was associated with improved effectiveness of the referral system to facilitate timely and effective management of pregnant women and newborns with complications. We present an analysis of available monitoring data alongside evaluation data to assess outcomes and impact of the EMAS referral interventions, including indicators assessing the effectiveness, efficiency, and timeliness of the referral system to facilitate the management of women and newborns with complications.  32 A phased approach presented an opportunity to assess progress and identify lessons to improve implementation strategies and approaches as the program expanded into additional districts. A new facility-based monitoring system was developed and implemented in 2013 to track key maternal and newborn evidence-based practices and a formal impact evaluation study was conducted in the last 2 years of the program (2015 and 2016) to assess the two main program objectives related to strengthening the referral system and improving quality of care. 30,31 The overall objective of the EMAS referral component was to increase the efficiency and effectiveness of referral pathways for emergency complications. The EMAS program employed a districtwide approach to improving the referral system. All facilities (puskesmas and hospitals) in a given district were formally networked to all of the surrounding facilities, including private hospitals. Interventions to maximize the effectiveness of the referral network included:

| Description of the EMAS program and the referral interventions
1. Referral network memoranda of understanding (MOUs) that strengthened linkages and formalized referral networks between BEmONC (puskesmas) and CEmONC (hospital) facilities within a given district. The MOUs also formally integrated private facilities and providers into the district referral system for the first time. Performance standards were regularly monitored both at the facility level and at the district level to assess facility-and district-level readiness to manage obstetric and newborn emergencies, identify health system weaknesses, and craft action plans to help address the weaknesses.
3. An automated electronic referral exchange system, SijariEMAS, enabled puskesmas and private midwives to send a two-way message (via phone/call center, SMS, or mobile or web application) to a central server prior to referral that would automatically route that referral to the most appropriate hospital using a pre-specified algorithm per the agreed MOU referral flow. The selected hospital could then choose to accept or reject the referral case depending on their capacity and capabilities at the time. In addition, puskesmas staff could receive messages through the system from the referral hospital about how to stabilize and prepare the patient prior to referral.

| Assessment design
Designing interventions and evaluations that aim to address a multitude of factors that may impact on something as complex as maternal and newborn mortality is challenging. 33 This study aims to assess the outcomes and impact of the EMAS referral interventions utilizing available monitoring and evaluation data to present two separate assessments: (1) A longitudinal analysis to assess the impact of the overall EMAS referral system on improving stabilization practices for emergency complications prior to referral, and identifying facility and geographical characteristics that are associated with improved stabilization practices; and (2) a quasi-experimental analysis to compare improvements in the referral system following the introduction of the electronic automated referral system (SijariEMAS), the primary referral intervention, using evaluation data. These methods are described separately below.

| Routine maternal and newborn health monitoring system-assessment of improvements in stabilization practices for emergency complications prior to referral
Procedures Under the EMAS program, a facility-based maternal and newborn health (MNH) monitoring system was implemented in early 2013 to routinely compile data on facility-level MNH-related process and impact indicators. Aggregate data were collected monthly from health facilities using a routine data collection form. Data were initially col-

Instruments and measures
The MNH monitoring system utilized a standardized set of registers in EMAS facilities, four designed for use in hospitals and three designed for use in puskesmas. Staff at both hospitals and puskesmas routinely entered data into the facility register. The registers were in turn used to complete a summary aggregate form for reporting to the Ministry of Health every month. Data quality and use of these registers was monitored by EMAS program staff, who conducted routine data assessments for completeness and timeliness across all districts and a random quality assurance check each month.
Two indicators from the hospital register that relate to the effectiveness of the referral system were selected for this analysis: 1. Proportion of mothers who were referred to hospital owing to severe pre-eclampsia/eclampsia who received magnesium sulphate (MgSO 4 ) to stabilize their condition before referral.

Proportion of newborns with suspected severe infection who
received an antibiotic before referral.
Information about these two indicators was verified through an accompanying referral letter or information from the midwife who accompanied the mother when referred to a hospital.

Statistical analyses
To estimate the rate of change in the two referral indicators between the start date of EMAS monitoring data (T=0) and end date of EMAS monitoring data (T=1), we used random-effects Poisson regression models where "EMAS exposure period" was the key independent variable and each facility served as its own intercept in the model. As the performance of the facilities was heterogeneous across the six provinces and hospital types by size of hospital beds, delivery volumes, and administration (private and public), the model calculated the difference between T=0 and T=1 by facility, adjusted by hospital facility type (public vs private) and province (West Java, Central Java, East Java, Banten, North Sumatera, and South Sulawesi). Data were analyzed using Stata version 14 (StataCorp LLC, College Station, TX, USA). Results are presented stratified by the three EMAS implementation phases. Significance was set at P<0.05.
One key advantage of this model specification was that the exponentiated beta exp(β) associated with "EMAS exposure period" directly estimated the incidence rate ratio (IRR), which was essentially the rate of change of the indicators (MgSO 4 and antibiotic use).
Overall, this analysis included data from 45 months of exposure for Phase 1, 33 months of exposure for Phase 2, and 21 months for Phase 3; however, the first few months of data collection at the start and at the end of each phase were excluded owing to instability of key maternal mortality indicators (e.g. some health facilities reported unusually high case fatality rates).

| EMAS evaluation study-assessment of the referral system following the introduction of an electronic automated referral system (SijariEMAS)
Procedures An evaluation study of EMAS was conducted in the last 2 years of the EMAS program (Phase 3 of the program), which employed a quasiexperimental pre-post control trial design. This study was conducted in six Phase 3 intervention and six comparison districts selected from the six EMAS-focus provinces. Data were collected at two time periods (2015 and 2016) from a total of 13 hospitals and 24 puskesmas, and included 2100 clinical observations of obstetric and newborn patients. The study sites, selection, and data collection processes are described in detail in companion articles. 30,31 Sample A total of 1609 clinical observations, excluding low birth weight cases, were conducted. Referral cases were included in this study if they were considered an "eligible" referral case, defined as any referral that was received at a participating EMAS evaluation study hospital from a puskesmas within the same district and that could be tracked during a specified 4-week observation period. A total of 180 referral cases were included in this analysis.

Statistical analyses
The evaluation study was designed as a quasi-experimental pre-post assessment, allowing for a difference-in-difference (DID) analysis, comparing changes between EMAS intervention and comparison facilities at baseline (2015) compared with endline (2016). However, the limited number of referral cases captured over the observation period, particular at endline, precluded a DID analysis (Table 1). In addition, SijariEMAS was not implemented universally across all EMAS interventions sites by the endline period, which would have significantly biased the DID analysis. Instead, a modified analysis was conducted that collapsed data from 13 hospitals and 24 puskesmas, combining the two data collection periods (2015 and 2016) to allow for comparison of referral cases that utilized SijariEMAS with referral cases that did not utilize the SijariEMAS system.
The impact of the EMAS program was subsequently assessed by comparing differences in each outcome indicator, comparing referral cases that utilized SijariEMAS with referral cases that did not use SijariEMAS. We used a χ 2 analysis for dichotomous variables; for continuous time variables, we compared the mean differences in response times of referral cases that used SijariEMAS with those that did not use SijariEMAS.

| Routine MNH monitoring system-assessment of improvements in stabilization practices for emergency complications prior to referral
A total of 28 340 maternal referral cases were registered over the study period, with an average of 12.35 pre-eclampsia/eclampsia referral cases per phase. Use of MgSO 4 for the treatment of preeclampsia/eclampsia cases at puskesmas prior to referral was lower (24%) at the start of monitoring, than at the end (61%) (Fig. 1)

| EMAS evaluation study
Among all cases of obstetric complications (n=434) whose management was directly observed in hospitals, a total of 180 eligible pregnant women who were referred to hospitals were included in this analysis ( Table 2  were referred using SijariEMAS (Table 3). All cases were referred to public hospitals. PONED puskesmas had significantly higher proportions of referral cases using SijariEMAS, but there was no significant difference for non-PONED puskesmas. No differences were seen by province in the proportion of referral cases both using or not using SijariEMAS, except in East Java where no referral cases were reported in intervention districts. The most common reasons for a puskesmas to refer a case included "service required was not appropriate to be delivered at the puskesmas (e.g. cesarean delivery)," "no equipment to manage complication," "no staff with the clinical skills/capacity to manage complication," and "not enough staff to manage complication." These were similar across cases that did and did not use SijariEMAS, except for a higher proportion of cases that reported "not enough staff to manage complication" used SijariEMAS (P<0.01) and a higher proportion of cases that reported "no staff with the clinical skills or capacity to manage complication" did not use SijariEMAS. Referral outcomes were similar across cases that did and did not use SijariEMAS.

| Referral effectiveness
When pregnant women were referred using SijariEMAS, there were consistently higher levels of effective communication (P<0.01) and advanced notification (P<0.01) between the puskesmas and hospital compared with pregnant women who were referred without using SijariEMAS (

| Referral efficiency
The time that puskesmas staff took between when the decision was made to refer a pregnant woman to when she actually departed the puskesmas was similar among cases that used SijariEMAS compared with those that did not use SijariEMAS (approximately 30 minutes each, P=0.32) (Table 5). However, the elapsed time between the woman leaving the puskesmas and arriving at hospital was slightly shorter among cases that used SijariEMAS compared with those that did not (75 minutes vs 60 minutes), although this was not significant (P=0.36).  the EMAS program to monitor and track progress, which did not include any control sites, our ability to attribute changes in stabilization practices directly to the EMAS program is limited. While it is possible that improvements in stabilization practices seen in this study may have been due to other possible factors outside of EMAS interventions, such as changes in the health insurance scheme JKN, consistency in the effects seen in stabilization practices for both obstetric and newborn complications during all three phases, from T A B L E 3 Characteristics of referral cases that used SijariEMAS compared with those that did not use SijariEMAS (n=180).   There was limited evidence for the impact of SijariEMAS on improving the timeliness of the referral, with significant variations in response times from when the decision was made to refer the pregnant women in need to time of departure from the puskesmas, and the time between leaving the puskesmas and arriving at the hospital. Reducing delays in pregnant women receiving CEmONC is critical in reducing risks, but ensuring effective stabilization of complications prior to referral is also key. 22 One potential explanation for these findings is that facilities that used SijariEMAS were also more likely to appropriately stabilize pregnant mothers with pre-eclampsia/eclampsia with MgS0 4 or provide antibiotics to newborns with suspected severe infection prior to referral compared with cases that did not use SijariEMAS, which may have subsequently increased their response times. These additional processes for preparing a patient for referral may take longer, but could arguably result in better quality of care and appropriate management of emergency complications overall. Response times between the pregnant women departing the puskesmas and arriving at the hospital may also be dependent on factors not captured in this evaluation, including variation in distances between puskesmas and hospitals.

| DISCUSSION
A systematic review of emergency obstetric referral interventions in low-resource settings identified similar challenges in addressing the delays in referral pathways for obstetric and immediate newborn care. 25 The review highlighted that while transport and communication interventions were becoming more popular and evidence is building for their effectiveness, it is difficult to measure the impact of referral interventions, which often included multiple interventions designed to reduce delays at several stages of the referral process. These factors make it hard to disentangle the contribution of each component of the intervention.
A recent study in Bangladesh, 34 also highlighted that important factors in a successful referral included early detection of complications experienced by mothers, quality support by a midwife, and quick transfer to the referral center. Our study findings were consistent with these findings and provide further evidence for the utility of an automated elec-

| Limitations
We recognize that our study has a few limitations. We utilized data from the MNH data monitoring system that was set up by the EMAS project to assess the overall effect of the EMAS program, thus limiting the ability to compare data from EMAS-focused districts with nonprogram comparison districts. In addition, the EMAS program began in July 2012, but the monitoring system was not implemented until 2013; therefore, it was difficult to directly assess the overall improvement in MNH care indicators between the baseline and endline periods of the EMAS program. The system did not collect additional data regarding other potential interventions that may have impacted MNH outcomes over the course of the program, thus attributions for any changes in outcomes presented in this analysis cannot be conferred. The ability to triangulate data from 101 hospitals across the three different phases and the consistency of effects on the two indicators we examined during all three phases provides some confidence in the impact assessment of the EMAS program. The limited number of referral cases observed in the comparison districts at endline (n=16) limited our ability to perform a DID analysis, comparing intervention and comparison districts as planned, and prevented any analysis of associates with referral outcomes. However, the modified analysis, which utilized SijariEMAS as a proxy measure for "exposure," did enable additional analysis providing evidence for the impact of SijariEMAS, and potentially demonstrated the broader strengthening of the referral system that the EMAS program provided through introduction of a referral MOU and the referral performance standards. While it is unclear why some EMAS intervention districts did not consistently use the SijariEMAS for all referral cases, a source of potential bias, the limited sample size prohibited a DID analysis that could control for such factors. It was noted that in South Sulawesi and North Sumatra internet reception can be poor, which may have prevented providers from using the SijariEMAS system, and it was noted by project staff that providers preferred to use cell phones to communicate about referrals cases.
T A B L E 5 Referral efficiency outcomes comparing referral cases that used SijariEMAS with those that did not use SijariEMAS. The timing of the evaluation study allowed us only to include data from Phase 3 intervention districts in this analysis. This is likely to attenuate the EMAS effectiveness, which shows the EMAS effect for 1 year only, rather for 5-years-the total program period. The small number of overall referral cases tracked within the evaluation study prevented complication-specific referral and case-management analyses.

| CONCLUSION
Notwithstanding the limitations, we believe that the findings from this study indicate that the EMAS program contributed to improvements in the overall management of emergency obstetric and newborn com-

ACKNOWLEDGMENTS
We would like to acknowledge the many women, children, and providers who participated in this study and enabled the collection of such important and sensitive information. We would like to thank the large data collection team for all their hard work and diligence, especially Andang Maulana Syamsuri for overseeing and driving much of the data collection and management of the evaluation data. We would also like to acknowledge Holly Blanchard and Gayane Yenokyan who were involved in the study design, data analysis, and manuscript writing at various stages throughout.

CONFLICTS OF INTEREST
The authors have no conflicts of interest to declare.