Improving health services to displaced persons in Aceh, Indonesia: a balanced scorecard
Grace J Chan a, Kristin B Parco b, Melva E Sihombing b, Susan P Tredwell b & Edward J O'Rourke a
a. John Hopkins School of Public Health, 615 N Wolfe Street, Baltimore, MD, 21205, United States of America.
b. International Organization for Migration, Geneva, Switzerland.
Correspondence to Grace J Chan (e-mail: firstname.lastname@example.org).
(Submitted: 09 March 2009 – Revised version received: 01 December 2009 – Accepted: 04 December 2009.)
Bulletin of the World Health Organization 2010;88:709-712. doi: 10.2471/BLT.09.064618
After the Indian Ocean tsunami in December 2004, there were 530 000 internally displaced persons in Aceh province, Indonesia. Local health care was often unavailable, since health workers were missing and facilities destroyed. The International Organization for Migration (IOM) worked with the Ministry of Health and local district health offices (DHOs) to construct temporary clinics for displaced populations.
As of November 2006, 34 000 persons still lived in temporary accommodation centres in the Aceh Barat and Nagan Raya districts of Aceh.1 Local DHOs were occupied rebuilding a permanent health infrastructure, so temporary clinics had little support.
DHOs asked IOM to assist in monitoring temporary clinic operations. Through site visits, we learnt that the clinics lacked health promotion activities and maintained inconsistent stocks of supplies. Often health workers were newly trained, isolated and unsupervised. Unreliable financial arrangements between DHOs and the government agency for tsunami rehabilitation and reconstruction disrupted planning and contributed to high turnover of clinic staff. The absence of a health information system made it difficult to monitor service delivery and outcomes.
IOM proposed a balanced scorecard to document quality issues for each clinic and to serve as a management tool for DHOs and the Ministry of Health in this setting. A balanced scorecard is a set of simple measures used to describe and improve overall performance.2 Afghanistan and the Netherlands recently adopted balanced scorecards to improve their national health systems, largely because of the tool’s ability to quickly identify problems and guide action. In these countries, measures for the scorecard were derived from large national databases.3,4 Such large-scale databases do not exist in Aceh. Instead, we conducted regular onsite surveys to generate information for the balanced scorecard, which then guided rapid cycle improvement efforts.5
In September 2006, IOM staff conducted initial assessments of temporary health clinics, concentrating on four domains: (i) health worker training; (ii) facility resources; (iii) community satisfaction and outreach and; (iv) service provision. In each area, we developed simple measures. Through discussions with district health officials and clinic staff, each measure was selected based on its significance, potential for improvement and ease of data collection (Fig. 1). We hoped to develop a sustainable model to be used by DHOs and local staff.
Fig. 1. Balanced scorecard measuring health services to displaced persons in Aceh, Indonesia, August 2007a,b
Two local IOM staff members, a nurse and a public health specialist, collected data through interviews, observations and basic assessments such as counting supplies and determining water availability. In the accommodation centres, the team sampled every fourth household. Data were combined to create composite measures using STATA, version 9 (StataCorp, College Station, United States of America).
The scorecard was presented in an easy-to-read table with each measure in a row and each clinic in a column. Measures were highlighted on the scorecard by colour to signify whether target levels were met: green (met target), yellow (partially met target) and red (below target). For example, the “drug” measure was green for clinics with more than 90% of essential drugs in stock, yellow for 90–50% and red for < 50% (Fig. 1). Target levels were developed from focus group meetings with provincial and district health officials using national standards.
The balanced scorecard was piloted in February 2007. Beginning in March 2007, it was implemented every 1–2 months in seven clinics and their nine associated temporary accommodation centres. The project was completed in December 2007, coinciding with the relocation of displaced residents to permanent housing and the closure of centres and clinics. Over the 10-month implementation period, a total of seven scorecards were created. During each cycle, data collectors from IOM spent 4 hours per clinic on clinical assessments and affiliated resident interviews.
Several improvements in health services were observed after the scorecard was implemented. Clinic staff received training on disposal of used needles, thermometer use and equipment sterilization. To focus on preventive medicine, clinic staff initiated events such as health promotion talks and fairs. Clinic staff also revitalized the community child health programme by conducting monthly visits to temporary accommodation centres, during which they gave immunizations and tracked the growth curves of children aged less than 5 years. Clinic staff began to actively manage an inventory of drugs. Communication improved between management, staff and accommodation centre residents.
Over the course of 10 months, there was a steady decline in the number of clinics reusing needles. After training, the score for proper thermometer use was excellent across all clinics. Community satisfaction increased from 68% to 84%. The percentage of residents receiving outreach activities increased from 14% to 85%. The number of sampled children aged less than 5 years with up-to-date vaccinations increased from 6% to 40%. The availability of essential drugs increased slightly from 49% to 59%. Equally important were the unmeasured benefits from increased monitoring and improved communications.
The balanced scorecard was an effective management tool used every 2 months for rapid improvement in a disaster rehabilitation setting. Each time the scorecard was used, we expected to see improvements in areas that were previously below standard.
In selecting the indicators, we balanced their importance with ease of measurement and implementation using existing resources. For example, while disposal of used needles is arguably not the most important indicator, it acted as a proxy for systems indicators such as staff knowledge and training. As clinic staff became more focused on the scorecard’s output, the reported data became less reliable. For example, after training on thermometer use, all patients at one clinic recorded normal temperatures of exactly 37.0 °C. The scorecard could not directly make up for deficiencies in staff training nor did it quickly change established practice patterns, such as over-prescription of antibiotics and steroid medications. It had minimal effect on external factors such as the drug supply chain.
Lessons and accomplishments
By measuring progress frequently and presenting results graphically, the scorecard provided a collaborative, simple, evidence-based tool to focus efforts on improving health services in a low-resource environment. The implementation in Aceh highlighted the following principles:
A data set was produced and constantly updated to provide up-to-date information on clinic activities and outcomes. As data accumulated, the scorecards gave clinic coordinators greater capacity to direct changes. At follow-up meetings, based on the most current set of measures, clinic coordinators discussed problems with DHOs and IOM and developed targeted interventions. Clinic staff could view graphs showing data on supplies and staffing levels. IOM could support its recommendations to DHOs with data. DHOs became more aware of clinic-level activities and needs, and responded with more supervisory visits.
The scorecard benefited from efforts to reduce the number of measures, so as not to overwhelm users with the volume and frequency of new information. Initially, monthly data collection occurred faster than the pace of interventions, so we adjusted the frequency of data collection to every 2 months to better align with reasonable expectations for change. Clinic staff and district health officials found it helpful to review a manageably small set of critical indicators. The number of essential measures could be further pared down. Rather than using STATA, the creation of measures could be done through a simpler programme.
Clinic coordinators and district level supervisors used the scorecard to assess quality, prioritize resources and direct change by identifying action points with limited additional investment in resources and data infrastructure. There was a shift of resources to target problem areas and redefine staff roles and accountability. Interviews were conducted during quiet times in the clinic so as not to interfere with clinical operations. The DHO reallocated previously-budgeted but unused time for monitoring clinical operations to this new management tool. This effort triggered discussions on quality in Aceh as there was a large variation between clinics. The scorecard can be used to monitor trends across facilities and highlight excellent clinics as role models to share their knowledge with other clinics.
Basic skills improved among the clinic staff. The data collection process provided a structured forum for IOM to interface with the community and clinic staff, facilitating opportunities for feedback and training beyond the scope of the study. For example, after evaluating thermometer use among clinic staff, IOM workers would immediately discuss basic thermometer use. Reviews of medical records revealed an over-prescription of antibiotics with no correlation with clinical signs. This led to on-site teaching regarding appropriate antibiotic use.
The scorecard enhanced communications between accommodation centre residents, clinic staff and DHO and IOM management staff. Through the process of data collection, IOM had the opportunity to interact with and teach clinic staff and accommodation centre residents. Regular meetings to discuss the findings of the scorecard brought clinics together and promoted the dissemination of ideas. By identifying priorities, clinic staff could better focus their attention and resources.
Leadership was a large, unmeasured factor affecting clinic performance. Clinics with enthusiastic leaders were the ones initiating and advocating for quality improvement. More emphasis could be placed on developing the role of supervisors at the district level and on training clinic leaders.
A balanced scorecard would be relevant to primary health-care facilities attempting to implement improvements with limited resources. Although beyond the scope of this project, the lessons learnt from the period of rehabilitation may contribute to the ongoing reconstruction of the permanent health infrastructure in Aceh (Box 1). The use of a balanced scorecard may be adapted to the permanent subdistrict health clinics. These lessons may apply to other primary health-care settings where there is a need for a simple data-driven method to focus priorities and a mechanism to improve communications between management, staff and beneficiaries.
Box 1. Summary of main lessons learnt
- Reduce the number of indicators measured by the scorecard so as not to overwhelm users with the volume and frequency of new information.
- Collect data every 2 months to keep pace with interventions.
- Hold regular meetings to discuss findings and determine priorities with clinic staff.
We thank the health officials at the ministry, provincial, district and community levels, especially Hasbi Quraisy, head of DHO Nagan Raya, and Amir Hamzah, head of DHO Aceh Barat. Special thanks to the clinic staff and shelter residents, Azrul Azwar, University of Indonesia, and to our colleagues at Harvard Medical School, especially Judith Palfrey, Myron Belfer and Kate Powis.
Grace J Chan was at Harvard Medical School when this work was carried out.
Funding was provided by AmeriCares, Harvard Medical School and the Lovejoy Research Fund.
- Temporary settlement monitoring project: findings in the field [Report no. 4]. Jakarta: Consortium for Assistance and Recovery toward Development in Indonesia; 2006.
- Kaplan RS, Norton DP. The balanced scorecard – measures that drive performance. Harv Bus Rev 1992; 70: 71-9 pmid: 10119714.
- Peters DH, Noor AA, Singh LP, Kakar FK, Hansen PM, Burnham G. A balanced scorecard for health services in Afghanistan. Bull World Health Organ 2007; 85: 146-51 doi: 10.2471/BLT.06.033746 pmid: 17308736.
- ten Asbroek AH, Arah OA, Geelhoed J, Custers T, Delnoij DM, Klazinga NS. Developing a national performance indicator framework for the Dutch health system. Int J Qual Health Care 2004; 16: i65-71 doi: 10.1093/intqhc/mzh020 pmid: 15059989.
- Brassard M, Ritter D. Problem-solving/process-improvement model. The memory jogger II, 1st ed. Salem, NH: Goal/QPC; 1994:115.