Bulletin of the World Health Organization

Retrospective comparative evaluation of the lasting impact of a community-based primary health care programme on under-5 mortality in villages around Jamkhed, India

Vera Mann a, Alex Eble b, Chris Frost a, Ramaswamy Premkumar c & Peter Boone b

a. Department of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, Keppel Street, London, WC1E 7HT, England.
b. Centre for Economic Performance, London School of Economics, London, England.
c. Schieffelin Institute of Health, Research and Leprosy Centre, Karigiri, India.

Correspondence to Vera Mann (e-mail: vera.mann@lshtm.ac.uk).

(Submitted: 24 February 2009 – Revised version received: 20 January 2010 – Accepted: 02 February 2010.)

Bulletin of the World Health Organization 2010;88:727-736. doi: 10.2471/BLT.09.064469

Introduction

The Bellagio Child Survival Study Group brought the issue of child survival to the forefront of the global health care agenda in 2003 when it reported that 10 million children were dying every year.1 Reducing child mortality by two-thirds before 2015 is one of the United Nations Millennium Development Goals.2 As the Bellagio series and the Lancet neonatal survival series3 and Alma-Ata series4 all point out, most of these deaths are preventable. In India, as in many developing countries, providing even basic health care in rural areas is a major challenge for the government. Analyses of India’s health system have suggested that rural health care has been neglected by the government and that increasing privatization may further reduce health care in remote areas.5,6

Community-based primary health care provided by trained community residents has been shown to improve child survival in areas with high child mortality. The Warmi project in Bolivia and the Society for Education, Action and Research in Community Health (SEARCH) in Maharashtra, India, have demonstrated significant reductions in perinatal and neonatal mortality.7,8 Recent trials in Nepal and in Uttar Pradesh, India, have reported reductions in neonatal mortality of 30% and 52%, respectively.9,10

Many aid agencies strive to address the lack of health care and improve child survival in developing countries, but they rarely conduct rigorous independent evaluations of their work. Such evaluations are expensive, carry the risk of showing negative results and are under-appreciated by donors. Consequently, there are few credible data on the impact of ongoing aid work.1115 Though randomized controlled interventions can generate such data,15 a study with prospectively assigned controls is seldom an option and retrospective approaches must therefore be used to assess the impact of aid efforts.

The Comprehensive Rural Health Project

The Comprehensive Rural Health Project (CRHP), one of the inspirations for the 1978 International Conference on Primary Health Care at Alma-Ata, has been working for the past 40 years to address the issues highlighted in the aforementioned Lancet series.1,3,4 It has had a major influence in health policy circles, including the World Health Organization (WHO) and schools of public health,16 and is a rare example of a long-term community-based primary health care project. The project was highlighted in the Alma-Ata Rebirth and Revival series as a model for delivering primary health care to poor rural regions,4 and – if found to be effective upon rigorous evaluation – could prove important for achieving the Millennium Development Goals on child mortality reduction elsewhere in India and in other vulnerable areas.

CRHP was founded in 1970 by physicians Mabelle and Rajnikant Arole, who envisioned a system that delivered both curative and preventative care to India’s most vulnerable people. Based in the town of Jamkhed, CRHP also serves surrounding areas in the central part of Maharashtra state. The area is predominantly rural, poor and drought-prone. Public health care and education are notoriously poor. CRHP has gradually expanded from a single hospital in Jamkhed and currently covers approximately 300 communities with a total population of over 500 000 people.17

Activities in project villages

The CRHP model, which focuses on community-centred primary health care, is described in detail elsewhere.17,18 Activities in the project villages are carried out at three levels: village health workers, mobile health teams and the secondary-care hospital in Jamkhed.

Village health workers, who are the cornerstone of the model, are local women selected by their communities, often from the lowest caste. They receive intensive training from CRHP trainers in primary health care and health promotion, including family planning, women’s and children’s health and home birth delivery. Training is also provided in community development, organization, communication skills and personal development. The primary role of these workers is to disseminate health knowledge in their respective communities through discussion groups and household visits, administer basic remedies and medications, perform safe deliveries and detect and refer high-risk pregnancies and deliveries to more qualified health care providers. During participatory discussions with community members, village health workers address issues such as care-seeking, family planning, adequate birth spacing, nutrition, hygiene, sanitation and safe drinking water.

Mobile health teams, comprising a nurse, a physician, a social worker and paramedics, visit project villages monthly to support and mentor village health workers and to refer complicated cases to the hospital. The secondary-care hospital in Jamkhed is a 40-bed low-cost facility operated by CRHP that provides quality emergency, medical, surgical and outpatient care to residents from the surrounding catchment area.

Research aims

Although many publications have praised CRHP’s work,4,1823 to date no rigorous impact study has been conducted. Overall child survival has improved in Maharashtra over the past few decades,2426 yet infant and child mortality rates in remote areas of Maharashtra remain much higher than for rural India overall.27,28 Rigorously evaluating the effect of CRHP will provide insight into the impact of community health interventions in such areas. Our primary aim was to compare mortality among children under 5 in CRHP villages of a typical size and in nearby control villages of similar size. Secondary aims were to compare sanitation; health knowledge; number of children per woman; place and type of delivery and type of birth attendant; indicators of antenatal, delivery and postnatal care; and child morbidity.

Methods

Study design

For statistical efficiency, we sampled equal numbers of villages in CRHP and control sites and approximately equal numbers of households, women and children within each village.

Surveys

Two surveys were conducted after a specially-hired team identified and numbered every household in all study villages. A household survey collected information on household-level indicators such as wealth, sanitation and water supply. Women in the surveyed households were interviewed and a full birth history was obtained, along with information on the woman’s background, history of pregnancy-related care, health care expenditure and morbidity of children under 5 years of age. Information on health knowledge was also collected for each eligible woman present in the household. These surveys were modelled on questionnaires from the third National Family Health Survey (NFHS-3)29 and were conducted by women hired locally from a women’s college near Jamkhed. They were trained for 10 days by a National Family Health Survey trainer and supervisor and supervised by locally-hired managers and a project manager. Training manuals, created by the authors, were based on the manuals for NFHS-3.30 Data collection took place between September 2007 and January 2008.

Sampling CRHP villages

The CRHP worked in four small administrative regions known as blocks or talukas: three in the core area near the hospital (Jamkhed, Ashti and Karjat) and one (Akole) approximately 220 km from the core area. We focused only on villages in the core area because Akole is distant and consists largely of tribal villages with characteristics different from those found in the core-area villages. We restricted eligibility to villages with a population of between 400 and 3000 in which a CRHP village health worker had been working for at least 5 years.

In the core area, CRHP implemented its comprehensive health intervention in 153 villages. Of these, 69 were excluded because village health workers had not worked for at least five years in the village. A further nine were excluded because their populations were outside the specified range. A total of 75 villages, all of them non-tribal, in the core area satisfied all criteria and were included in the study.

Sampling control villages

To select control villages with characteristics similar to those of CRHP villages while minimizing the risk of contamination bias owing to proximity to the CRHP intervention area, a buffer zone was drawn around the three core blocks of the project. Since the core intervention region is approximately elliptical in shape, we drew an elliptical buffer line that was never less than 5 km from any intervention village. A second ellipse, with the same foci, was drawn 25 km outside the first (Fig. 1). Control villages were selected from the area between the two ellipses. The 25 km width ensured that the number of eligible control villages in the area between the two ellipses was greater than the total number of eligible intervention villages. From the 135 non-tribal villages in this area with a population between 400 and 3000 according to the 2001 national census, 75 were selected at random for the study. CRHP was the only nongovernmental organization delivering home-based care, health education and clinical services in the study villages. Government health services were available in both intervention and control villages.

Fig. 1. Map of control and intervention areas in retrospective study of the impact of the Comprehensive Rural Health Project (CRHP), Maharashtra state, India

a This area comprises the 153 CRHP villages, of which 75 satisfied all inclusion criteria.

b This area comprises the 135 eligible control villages, of which 75 were randomly selected to be surveyed. The inner ellipse is at a distance of at least 5 km from any CRHP village.

Sampling households and women

In each village, 30 households were randomly selected from the mapping list of all households. An additional 60 households were randomly selected as back-ups. Our survey teams attempted to interview all potentially eligible women in the 30 selected households in each village. They returned twice to the village to reach as many selected households as possible. If it was impossible to identify and interview at least one eligible woman after two visits to a selected household, the team approached the next household on the back-up list, continuing until at least 30 households were interviewed. As six interviewers were simultaneously conducting interviews, in some villages the team interviewed more than 30 households.

Eligible women and births

Women were eligible if they were between 15 and 59 years of age, were currently or previously married, had lived in the village for at least one year before the survey and had at least one child born alive in the village in the past 15 years. Some women who lived in intervention or control villages at the time of the survey had given birth and raised children in other villages. To minimize recall bias, we included only women who had given birth to a liveborn child between 1 September 1992 and 31 December 2007 in the same village in which they were interviewed. To allow sufficient time for CRHP to have had an impact, analyses in intervention villages were further restricted to women who had given birth five years after the project began operating in their villages.

Sample size

The sample size was calculated to attain 80% statistical power to detect a 25% reduction in under-5 mortality at the 5% significance level using a conventional 2-sided test. On the basis of data from the 1998–1999 National Family Health Survey for rural India,25 we assumed a 10% under-5 mortality rate in the control villages and hence 7.5% under-5 mortality in the CRHP villages, with an intra-cluster correlation coefficient of 0.0175, estimated from the National Family Health Survey data for the period 1985–1995 for the 81 non-tribal rural villages in Maharashtra state included in this data set.25

Under these assumptions, 54 eligible births were needed from each of the 75 sampled villages in both the CRHP and the control area during the 15 years before the interviews. At an assumed average of two eligible births per household, at least 27 households per village were required.

Data analysis

All data obtained for all eligible villages, women and children were analysed. Demographic and other characteristics at the village, household and mother level were summarized by type of village (CRHP or control). These included characteristics considered as confounders and/or secondary outcomes, such as the educational level of the woman, treatment of drinking water, toilet facilities, number of women reporting miscarriage and/or stillbirth, number of Caesarean deliveries, number of children per woman, number of antenatal and postnatal care visits, mean health knowledge score and its standard deviation, and health care-seeking activities.

The overall health knowledge score was derived by summing the total number of correct responses given to 10 health knowledge questions. Questions with more than one possible correct answer were weighted by the inverse of the total number of correct answers so that no single question contributed more than one point out of 10 to the score.

The percentages of villages with irrigation in the CRHP and control groups were compared using Fisher’s exact test. Mean ages and health knowledge scores were compared using linear regression models with robust standard errors (Huber-White sandwich estimator) to allow for clustering within villages. Categorical characteristics and secondary outcomes at household, woman and child level were compared using logistic regression models. Robust standard errors were used, and Wald tests for joint significance were performed. Village-specific crude neonatal, infant and under-5 mortality rates were estimated by the Kaplan–Meier method and are presented separately for control and CRHP villages. Crude mortality rates for the entire set of surveyed villages are given in 5-yearly intervals.

Cox regression models were used to compare hazard rates between CRHP and control villages using robust standard errors (Lin-Wei estimator) to account for clustering of women in villages and for the non-independent outcomes for children born to the same woman. As CRHP attempted to work in the most disadvantaged villages, most of which lacked irrigation and had relatively large proportions of low-caste inhabitants, we decided a priori to adjust for irrigation status of the villages and for women’s caste and religion. Interaction terms between the intervention and different age bands (neonatal, post-neonatal to 5 years of age) were included in the models to test the proportionality assumption – i.e. whether the intervention had a differential effect on mortality at different ages. When a statistically significant interaction was found, age-specific effects of the intervention are also reported.

Ethical approval

Ethical approval was obtained from the Institutional Ethics Committee of the Maharashtra Association of Anthropological Sciences in India and from the Ethics Committee of the London School of Hygiene and Tropical Medicine.

Results

Data on the study population are presented in Table 1. More women in CRHP villages belonged to less-advantaged castes, their families were less likely to own land and they were less likely to have irrigation available (Table 2). Factors that may have been influenced by the CRHP intervention are summarized in Table 3. In 2007, more households in CRHP villages had toilet facilities and treated drinking water. Women in the intervention villages achieved, on average, a health knowledge score that was 0.15 standard deviations higher (0.26 points higher in absolute score) than the score obtained by women in the control villages. The number of children per woman and the percentage of women who reported having a stillbirth or miscarriage were similar in the two groups. Indicators of antenatal and postnatal care and of location and type of delivery of the last child were also similar. Among women reporting that in the three months before the survey their child under 5 years of age had experienced symptoms of one of the main diseases that cause child death, there was no significant difference in the use of medical centres when seeking treatment.

In control villages, 42.3% of women had no education at all compared to 39.3% of women in the CRHP villages. The mean (median) number of years of education in CRHP villages was 4.4 (4) for women. In control villages the figure was 4.3 (4).

Out of 10 896 eligible live births to 4940 eligible women during the evaluation period, 619 under-5 child deaths were reported. Of those deaths, 37 in control villages and 48 in intervention villages were reported to have occurred on the day of delivery. For 2 of the children reported dead, age at death was missing. Those children were excluded from the survival analyses. For an additional 11 children, the caste of the mother was not reported. Five-yearly estimates of age-specific mortality rates for the entire study area (CRHP and control villages) over the evaluation period are shown in Table 4.

Fig. 2 shows that age-specific mortality rates in the CRHP and control groups run closely together in the neonatal period and then diverge, with consistently lower mortality in the CRHP villages from approximately 1 to 5 years of age. This pattern is also apparent when crude estimates of age-specific childhood mortality rates are compared (Table 5) and is confirmed by formal analysis using Cox regression models (Table 6).

Fig. 2. Kaplan–Meier estimates of age-specific childhood mortality in control and intervention villages in retrospective evaluation of the impact of the Comprehensive Rural Health Project (CRHP), Maharashtra state, India, September 1992–December 2007a

a There were 5515 eligible live births in the control villages and 5379 eligible live births in the intervention villages.

Overall, the hazard of death in CRHP villages was reduced by 10% for the period from birth to 5 years of age when the data were controlled for irrigation status of villages and the caste and religion of mothers. This effect was not significant at the 5% level, with the 95% confidence interval (CI) extending from a 25% reduction to a 9% increase. However, there was evidence (from the result of a test of the proportionality assumption implicit in the Cox model) that the ratio of the hazard in the CRHP villages to that in the control villages varied according to children’s age. For this reason we applied an interaction model in which the hazard ratio comparing the CRHP villages with the control villages differed in the neonatal and post-neonatal periods. This interaction was statistically significant (P < 0.04 for the model, adjusted for irrigation, caste and religion). The model provided no evidence that neonatal mortality was lower in CRHP villages (hazard ratio = 1.03; 95% CI: 0.82 to 1.29), but showed a statistically significant reduction (P < 0.02) in post-neonatal (up to 5 years) mortality (hazard ratio = 0.70; 95% CI: 0.52 to 0.94). Although childhood mortality rates declined over the birth periods covered by the study (Table 4, heterogeneity test for the effect of time period: P < 0.003 in Table 6), the hazard ratio for CRHP did not materially change when birth period was included as an additional covariate in the models. Further, there was no evidence that the hazard ratio for CRHP villages varied between birth periods (interaction test P > 0.1).

Discussion

In our evaluation of the effect of CRHP on childhood mortality over the period September 1992 to December 2007 we found a 30% reduction in the hazard of child death after the neonatal period for CRHP villages in comparison with villages in the control area; the reduction was significant at the 5% level. We did not, however, find a similar reduction in the hazard of neonatal death.

There are several possible explanations for this inconsistency. Causes of neonatal deaths in rural India – primarily birth asphyxiation, preterm birth and neonatal sepsis – are different from causes of post-neonatal child deaths – primarily pneumonia, diarrhoea and malaria.1,3,31,32 WHO-recommended first-line treatments for malaria, pneumonia and diarrhoea can be administered in the community.3,32 CRHP may have been more effective at ensuring prevention, early recognition and treatment of these diseases than it was at delivering clinical services, as suggested by superior levels of sanitation and water treatment in CRHP villages (Table 3).

There may also have been differential reporting of live and still births. A previous study found substantial underreporting of neonatal deaths in Maharashtra.33 In the CRHP villages, a higher number of deaths occurring on the day of delivery (48 versus 37 among controls) and a lower number of women with stillbirths (315 versus 342 among controls, Table 3) were reported. Women in control villages may have been more likely to classify early postpartum death as a stillbirth than those in intervention villages because the CRHP had sensitized women to the notion of neonatal death and its prevention.

Our childhood mortality estimates for the entire study area are in line with other survey estimates of mortality for the region, although they are somewhat higher than official estimates for Maharashtra from the National Family Health Surveys for 1994–199825 and 2001–2005.26 However, these surveys sample from both urban and rural areas and, since urban regions have lower mortality, they underestimate mortality for rural areas. Estimates based on the 1994–1998 National Family Health Survey data for rural Maharashtra only show much closer agreement with our results: neonatal, infant and under-5 mortality rates of 38.4, 52.3 and 67.6 deaths per 1000 live births, respectively.34

The main strength of this study is that it provides the most carefully collected evidence to date that the hazard of child death was lower in CRHP recipient villages than in nearby non-intervention villages several years after the project was implemented. Our findings suggest that a community-based programme that trains village health workers to improve community knowledge can have long-lasting impacts on child mortality. We designed the study carefully to minimize the risk of bias when controls are selected retrospectively. We believe the methods used here could contribute to future evaluations of similar long-term projects.

Our study had several limitations. Reported mortality may have been impacted by migration. Movement of women from intervention to control villages may have reduced the mortality rate in their new residence and the inverse may also have occurred when women moved from control to intervention villages. We were only able to reduce this dilution partially by restricting our analyses to children whose mothers were resident in the same village at the time of both delivery and survey.

Despite careful selection of the controls, women from CRHP villages may have answered some of the questions in the survey differently from women in control villages because of their improved knowledge. As noted above, this differential reporting could have hindered our ability to show a reduction in neonatal mortality in CRHP villages, if such a reduction did indeed exist.

CRHP may also have been more effective when the disease burden was higher. When the project began in 1970, the child mortality rate in India was 192 deaths per 1000 live births. By 2000 the rate had dropped by more than half,35 partly owing to a reduced incidence of malaria, improved treatment of diarrhoea and pneumonia and increased access to clinical services, including safer birthing practices, as a result of programmes introduced by the Government of India and by Maharashtra state.36 Although these programmes and the clinical and health services provided by the government affected both our intervention and control villages equally, the large improvement in child mortality might have reduced the scope for measuring CRHP-generated improvements in recent years.

Funding cuts have forced CRHP to stop its work in 44 of our 75 study villages at some point in the past 25 years. Thus, at the time of our surveys only 31 (i.e. less than half) of the study villages had CRHP-led activities. If CRHP had run intensively in all study villages from its inception to the end of our survey period, we might have seen greater impact. Nevertheless, we have found that the project area continues to have better child survival outcomes and health knowledge among mothers than surrounding areas. Our results contrast with the reported failure of voluntary village health worker programmes in other studies.37 Since CRHP village workers received only minor monetary compensation, remuneration is unlikely to account for the difference. Future studies should look specifically at the effect of compensation (versus, for example, empowerment of village health workers to give them a sense of “ownership” of the programme) on the success of community-based primary health care programmes. We believe that appropriate retrospective evaluation of the lasting impact of such programmes will make an important contribution to the evidence for short-term effects from randomized control trials.


Acknowledgements

We thank the following individuals for their contributions: Raj, Ravi and Shobha Arole from CRHP for allowing us to conduct our field work in the CRHP villages; the mapping team for mapping the villages; our interviewers and supervisors for their hard work conducting the surveys; Prakash Fulpagare for his excellent training of interviewers and advice; Mark Fisher for designing the database; SITES India for entering the vast amount of data; the Ethics Committee of the Maharashtra Association of Anthropological Sciences for excellent advice, encouragement and approval; and the panel of experts convened in Pune in November 2006, who helped conceive the study: Hemant Apte, Ravi DeSouza, BS Garg, Michelle Kermode, RK Mutatkar, Chandrakant S Pandav, Henry Perry and Brahma Reddy.

Funding:

This study was funded by Effective Intervention (located in the Centre for Economic Performance in the London School of Economics).

Competing interests:

Ramaswamy Premkumar taught in CRHP training programmes and conducted research for CRHP between 1999 and 2001. During this study he had no involvement in any CRHP activity in the project villages.

References

Share