Increasing access to health workers in underserved areas: a conceptual framework for measuring results
Luis Huicho a, Marjolein Dieleman b, James Campbell c, Laurence Codjia d, Dina Balabanova e, Gilles Dussault f & Carmen Dolea g
a. Department of Paediatrics, Universidad Peruana Cayetano Heredia, Lima, Peru.
b. KIT Development, Policy and Practice, Royal Tropical Institute, Amsterdam, Netherlands.
c. Instituto de Cooperación Social, Integrare, Barcelona, Spain.
d. Global Health Workforce Alliance, Geneva, Switzerland.
e. London School of Hygiene and Tropical Medicine, London, England.
f. Intituto de Higiene e Medecina Tropical, Universidade Nova de Lisboa, Lisboa, Portugal.
g. Department of Human Resources for Health, World Health Organization, 20 avenue Appia, 1211 Geneva 27, Switzerland.
Correspondence to Carmen Dolea (e-mail: email@example.com).
(Submitted: 05 October 2009 – Revised version received: 19 November 2009 – Accepted: 05 January 2010.)
Bulletin of the World Health Organization 2010;88:357-363. doi: 10.2471/BLT.09.070920
Health workers’ willingness to practise in underserved areas, such as rural, remote or poor areas, is a recognized challenge in achieving equitable access to health services. Many countries have developed strategies to attract and retain qualified health workers in these areas. But the evidence on the successes or failures of such interventions is scarce and weak, and it is difficult to compare lessons and measure results from the few evaluations that are available.1–3
There has been significant progress in generating common understanding and debate on ways to evaluate the impact of development interventions, and on using consistent monitoring and evaluation terminology, such as “outputs”, “outcomes” and “impact”.4–6 However, this has yet to be applied to the evaluation of human resources for health interventions and, specifically, to those seeking to increase access to health workers in underserved areas. There is an urgent need to achieve an agreement and strengthen the evidence that would underpin sound policy recommendations in this area.
Seeking to address this gap, this paper proposes a conceptual framework to guide managers, policy-makers and evaluators in the assessment of interventions to increase access to health workers in underserved areas. The framework aims to support all stages of policy development. It suggests a logical sequence that can be followed when deciding on any intervention that addresses attraction and retention issues and assists in formulating key questions to be answered when designing, monitoring and evaluating such interventions. It includes examples of indicators to inform the process, which can be adapted to a specific context. A comprehensive consideration of indicators to monitor the health workforce in general is offered elsewhere.7
Challenges in evaluation
Policy- and decision-makers need to know whether interventions work or not, why they work and in which context. Therefore it is important to have information about the effects of the interventions, but also about the factors that made the intervention succeed or fail (the questions dealing with “when, why, how, and in what circumstances such interventions work well or fail to work”).8
Evaluating interventions to improve human resources for health is complex for different reasons. First, relating these interventions to health status is very difficult due to broad socioeconomic, cultural, political and health systems factors that influence health. For instance, although improved health outcomes, such as reduction in maternal mortality, are directly correlated with increased availability of health workers, it is difficult to attribute the improvement directly to a certain health workforce intervention.9,10 Illustrative country case studies showed that Afghanistan and Ethiopia have implemented comprehensive health sector strategies with multiple co-existing interventions, like the recruitment of community health workers, which may have promoted access to life-saving services, thus improving health status.11
Another problem is that many evaluations lack a baseline against which to assess the results, particularly in countries with a major health worker deficit,3,10,12 as well as a specific intervention logic that clarifies the expectations of the intervention designers (S Kane, B Gerretsen, R Scherpbier, M Dal Poz, MA Dieleman, unpublished data, 2009).13 Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluation of human resource interventions.14,15
Thus, the main challenges which evaluators face are related to the multidimensional nature of interventions, and the difficulty of assessing the influence of contextual factors. To be able to assess such interventions in complex systems, there is a need for multimethod and multidisciplinary monitoring and evaluation approaches, inclusive of all relevant stakeholders.16 Another challenge is generating lessons across regions and countries to inform the global health agenda and promote a meaningful dialogue at both national and international level. There is clear need for a framework that offers a model for monitoring and evaluation of health worker retention interventions taking into account such challenges.
The proposed framework is based on a systems approach and differentiates between “inputs”, “outputs”, “outcomes” and “impact”, with regards to the results of interventions to attract and retain health workers in underserved areas. It builds on the proposed common framework for monitoring performance and evaluating progress in the scale-up for better health,17 and proposes indicators to measure progress in implementing various strategies, allowing users to determine what works or not, and to explore contextual factors influencing their success or failure. The framework has two aims: (i) to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its inception/design phase through to its results, by suggesting key questions about the relevance, efficacy, efficiency, effectiveness and sustainability of the intervention; and (ii) to guide the monitoring of interventions, through a focus on a routine collection of a set of indicators, applicable to the specific context.
We used the following definitions for evaluation and monitoring:4
We define “evaluation” as the systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability.
We define “monitoring” as a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.
Later in this paper, we examine the application of the framework to the stages of policy development: design, implementation, monitoring and evaluation (Fig. 1). The sequence in design and implementation phases does not necessarily imply a linear policy process, as monitoring and evaluation may help identify a need for new interventions. They are, in effect, part of an ongoing policy cycle.
Fig. 1. The conceptual framework for measuring efforts to increase access to health workers in underserved areas
The evaluation of the design stage should identify the extent to which the chosen intervention was relevant and adequate to the need, the expectations of the population and the health system context, and the reasons for selecting a particular option.18 By relevance, we mean the extent to which the objectives and the elements of the intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.4 The relevance issue can and should also be addressed retrospectively; and then the question is whether the objectives of an intervention or its design were and are still appropriate given changed circumstances.4
Questions that an evaluator should ask are: did the intervention respond to a documented need, was it appropriate for the given context and how was it selected? Is the choice of the intervention based on evidence or at least on robust arguments?
The assessment of the situation analysis should check if the problem of limited access to health workers has been clearly described. Is it due to an insufficient number of students willing to work in underserved areas, or an insufficient production of health workers, or were there other reasons? The evaluation needs to assess if a health labour market analysis has been conducted, to identify whether the lack of access to health workers in underserved areas was a matter of insufficient demand, insufficient supply, or insufficient remuneration and incentives, and unattractive working conditions.19
The situation analysis should also identify whether the elements in place to support the implementation of the intervention were assessed and if a feasibility analysis preceded the choice of the intervention. Strategies to increase access to health workers in underserved areas require participation of stakeholders from different sectors. Therefore, questions to be asked include: were mechanisms for multi-stakeholder engagement in place and used, were regulatory frameworks analysed before the intervention, were the intervention costs calculated and considered affordable (availability of funds), was the organizational capacity to support its implementation analysed? At this stage, the Human Resources for Health Action Framework is a recognized tool that can be used to guide the situation analysis (Fig. 2).21 It comprises health system dimensions such as policy, finance, education, partnership, leadership and human resources management systems.
Fig. 2. The adapted Human Resources for Health Action Framework20
At the end of this phase, a strategy can be proposed to address the perceived need of lack of access to health workers in underserved areas and of maldistribution of health workers. Such a strategy can be in one or more of the following thematic areas: education, regulatory, financial, and/or management and support, described elsewhere.3
When the implementation of an intervention is evaluated, questions might include: did the activities take place according to plan, did any changes occur in implementation capacity or in the context that influenced the implementation of the intervention components? Did any changes occur regarding the involvement and commitment of stakeholders? Which ones and why? Was any action undertaken to adapt the intervention to the changes? What was done and why? Or why was no action taken? Monitoring the progress in implementation can be informed by the systematic collection and analysis of a set of key indicators, which are further discussed in this paper.
Intervention results can be evaluated at different levels (Fig. 1): outputs, intermediate outcomes, outcomes and impact. Questions to be asked include: did the intervention achieve its intended results at the level of outputs, outcomes, or impact? Is the intervention sustainable? What were the contextual factors (political, economic, social, or organizational) that have influenced the results?
The level of “outputs” is the direct effect, attributable to the intervention, such as increased attractiveness of the areas/or the profession, improved recruitment and deployment, and increased retention of health workers in those areas.
The level of intermediate “outcomes” is that of improved health workforce performance, the dimensions of which have been described in The world health report 2006 as “availability, competence, responsiveness and productivity”.10 Some attraction and retention interventions can have a direct effect on these dimensions too. For example, changes in rural curricula can improve competencies to address rural health problems and performance-based financial incentives can improve productivity. However, often improved health workforce performance cannot be attributed to retention interventions alone, as there are many other factors that contribute, such as education strategies or health sector management reforms.
The level of “outcomes” is that of increased health systems performance, measured by increased coverage and utilization of services, and improved responsiveness. Improved health workforce performance could contribute to improved health system performance but a host of other factors can contribute as well, such as the availability of drugs, infrastructure and equipment.22
Finally, the level of “impact” is understood as the effect on health outcomes and an improved health status, and other unexpected effects on underserved communities (e.g. attraction/retention of other professionals or services). This cannot be directly attributed to retention interventions, as the simple availability of health workers cannot directly improve health outcomes in the absence of effective components such as drug supply, functioning facilities and good clinical practices.9 But a well performing health workforce is a significant component of a strong health system that, together with other social determinants, can lead to improved health status.22
Evaluations of any attraction and retention intervention need to consider the contextual factors that may explain why and how such interventions have achieved the intended outputs, outcomes and impact, i.e. the social, economic and political context. This includes a stakeholder analysis including their respective power bases and interests, as well as an analysis of the capacity of the health sector to implement. Evaluations should use both qualitative and quantitative methods to measure outputs and impacts, but also to analyse formal and informal policy processes accompanying the design and implementation of interventions. Ideally, to find out if an attraction and retention intervention works or not, a cluster randomized controlled trial with an adequate control group would be needed, but this may not be feasible or ethical in the “real life” policy scenario. Moreover, causal chains are often complex in public health interventions, making results of randomized controlled trials subject to effect modification in different populations.20 Alternative observational methods such as before-after and plausibility designs can also be considered, taking care to document and control the contextual factors appropriately.20,23 Studies with adequacy and plausibility designs can improve both the internal and external validity of randomized controlled trials, and plausibility studies are often the only feasible option for getting valid evidence of impact when attempting the evaluation of large-scale interventions.23
The framework depicted in Fig. 1 proposes a tool to guide the monitoring of interventions to increase access to health workers in underserved areas. At each stage of the implementation process, indicators are proposed to assess whether the implementation is on track.
Indicators to monitoring at the design stage should be those that will allow for a detailed description of the stocks and flows of health workers across sectors, age and sex categories, geographical areas and professions. Detailed description of these indicators, as well as data sources for health workforce monitoring, are described elsewhere.7
When monitoring the implementation of interventions, systematic data collection is required on the planned activities, by constantly reviewing the production capacity, as well as the organization and management capacity. Monitoring implementation also requires tracking changes in context and stakeholders, so as to be able to adapt the interventions when required.
Regarding the monitoring of results of the intervention, at the level of “outputs” it is important to monitor whether the dimensions of “attractiveness”, “engagement/recruitment”, “deployment” and “retention” are achieved. Indicators to monitor progress on these dimensions can include, for example, changes in the intentions of students or health workers to relocate, reside in or leave underserved areas (attractiveness). The available funded positions in underserved areas can be an indicator of effective engagement, whereas the number of health workers recruited and changes in vacancy rates can be indicators of effective deployment and recruitment. Finally, the effects of an intervention on retention can be measured by duration in post, turnover, absenteeism, stability index or even survival rates.24,25
At the level of “outcomes”, it is proposed to monitor the progress on improved health workforce performance and health systems performance, through dimensions and indicators proposed by The world health report 2006 (Table 1).10 As said before, assessing the longer-term impact of human resource interventions on health outcomes should be done within the broader context of factors influencing health status.
The effectiveness of attraction and retention interventions can only be measured at the level of outputs and, to a certain extent, at the level of outcomes. In addition, implementing human resource retention interventions can take many years before any results can be seen, for example in the case of creating new schools in rural areas or rolling out financial incentive schemes. Strong human resource information systems, linked to national health information systems, as well as clearly defined and agreed metrics or indicators are required to monitor the progress of such interventions in terms of increased attractiveness, improved recruitment and duration of stay, increased availability, improved competence, productivity and responsiveness. Table 2 makes an attempt towards advancing a set of key questions and indicators to be used when evaluating and monitoring retention interventions for human resources for health.
Table 2. Proposed questions and indicators for the evaluation and monitoring of interventions to increase access to health workers in underserved areas
The conceptual framework presented here needs to be extensively piloted and will be refined through further inputs from different sources. There are ongoing plans for applying it in several countries that are in the process of designing and implementing attraction and retention strategies in underserved areas. Efforts will also be made to make sure the evaluation tool is user-friendly. For the sake of argument, a hypothetical example of its potential use is presented in Box 1.
Box 1. A hypothetical use of the conceptual framework
The framework can be applied to a hypothetical scenario of country X. The evaluation will explore: (i) Does the intervention respond to the problem identified and has the intervention achieved the intended results? (ii) What improvements in intervention and policy design are needed, or are alternative strategies required? (iii) What indicators should be used to measure results? Step 1: situation analysis For example, in country X, a situation analysis has identified a lack of health workers in rural areas. A study of intentions of young graduates to practise identified the unwillingness of health workers to go to those areas due to a lack of opportunities for professional development and low salaries. Thus, the main bottlenecks seem to be in initially attracting staff to rural areas. Step 2: choice of interventions The intervention that has been implemented sought to “create more schools in rural areas” (education category) and to implement a “rural allowance scheme” (financial category) for some categories of health workers, and improve support and supervision (management). The intended result of this multi-level intervention was to increase attractiveness of rural practice for health professions and improve access to professional training to students with rural backgrounds, as well as an increased duration of stay of the categories of health workers covered by the scheme. Step 3: measuring results using a selection of country-appropriate indicators The indicators chosen for monitoring progress and measuring the direct effects (“outputs”) of the intervention would be: “intentions to go to rural areas after graduation”, “number of applicants from rural background to the newly developed schools compared to urban”, “number of graduates choosing to work in rural areas”, “duration of stay in post of the health workers covered by the scheme”. The indicators to measure the indirect effects (“outcomes”) of the intervention would be: “changes in staff ratio within 3 years”, “increase in outpatient visits” before and after, “changes in staff satisfaction”, “user satisfaction” perceptions of community representatives of service responsiveness to need. The outputs and outcomes can be related statistically to take-up of training programmes, to the percentage of staff covered by financial incentive schemes and supported by an experienced manager, to establish correlation and formulate plausible explanations for the effect of the different types of incentives on the way the intervention has performed. Eventually, the final impact of the intervention can be estimated by improved “health outcomes”, bearing in mind that these changes cannot entirely be attributed to the intervention itself. Step 4: monitoring Analysis of longitudinal data from indicators to examine how the situation has changed, especially seeking to identify unintended consequences of implementing the intervention. For example, attracting staff in rural areas may have resulted in their recruitment by nongovernmental organizations and private facilities rather than working in the public sector.
The way forward
This framework proposes a common approach to facilitate the evaluation and monitoring of interventions to increase access to health workers in underserved areas. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of such interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It is hoped that it will promote the use of a common understanding/logic on how attraction and retention interventions work, using a systems perspective. It starts with a common set of indicators, which enable comparison between various cases, and facilitates reviews of published studies. As such it can be used to monitor and evaluate interventions, either using a method-based or a theory-based evaluation approach with a specific set of indicators.
This work was funded by the World Health Organization, Health Systems and Services, Department of Human Resources for Health, Health Workforce Migration and Retention Unit, Geneva, Switzerland. Additional support was received from the WHO Collaborating Centre on Human Resources for Health at the Royal Tropical Institute (KIT), The Netherlands.
- Grobler L, Marais BJ, Mabunda SA, Marindi PN, Reuter H, Volmink J. Interventions for increasing the proportion of health professionals practising in rural and other underserved areas. Cochrane Database Syst Rev 2009; 1: CD005314- pmid: 19160251.
- Wilson NW, Couper ID, De Vries E, Reid S, Fish T, Marais BJ. A critical review of interventions to redress the inequitable distribution of healthcare professionals to rural and remote areas. Rural Remote Health 2009; 9: 1060- pmid: 19530891.
- Dolea C, Stormont L, Braichet J-M. Evaluated strategies to increase attraction and retention of health workers in remote and rural areas. Bull World Health Organ 2010; 88: 379-385.
- Development Assistance Committee. Glossary of key terms in evaluation and results-based management. Paris: Organisation for Economic Co-operation and Development; 2002. Available from: http://www.oecd.org/dataoecd/29/21/2754804.pdf [accessed on 15 February 2010].
- Leeuw F, Vaessen J. Impact evaluations and development: NONIE guidance on impact evaluation. Washington, DC: The Network of Networks on Impact Evaluation; 2009. Available from: http://siteresources.worldbank.org/EXTOED/Resources/nonie_guidance.pdf [accessed 15 February 2010].
- White H. Theory-based impact evaluation: principles and practice. New Delhi: International Initiative for Impact Evaluation (3ie); 2009.
- Dal Poz M, Gupta N, Quain E, Soucat ALB. Handbook on monitoring and evaluation of human resources for health – with special applications for low- and middle-income countries. Geneva: World Health Organization; 2009.
- Mills A, Gilson L, Hanson K, Palmer N, Lagarde M. What do we mean by rigorous health-systems research? Lancet 2008; 372: 1527-9 doi: 10.1016/S0140-6736(08)61633-5 pmid: 18984174.
- Anand S, Bärnighausen T. Human resources and health outcomes: cross-country econometric study. Lancet 2004; 364: 1603-9 doi: 10.1016/S0140-6736(04)17313-3 pmid: 15519630.
- The world health report 2006: working together for health. Geneva: World Health Organization; 2006.
- Peters DH, El-Saharty S, Siadat B, Janovsky K, Vujicic M, editors. Improving health service delivery in developing countries: from evidence to action. Washington, DC: The World Bank; 2009.
- Terms of reference for an independent evaluation of the impact of Malawi's Emergency Human Resources Programme. Lilongwe: Department for International Development (UK); 2009.
- Dieleman M, Gerretsen B, van der Wilt GJ. Human resource management interventions to improve health workers’ performance in low and middle income countries: a realist review. Health Res Policy Syst 2009; 7: 7- doi: 10.1186/1478-4505-7-7 pmid: 19374734.
- Dieleman MA, Kane S, Zwanikken P, Gerretsen B. Identifying patterns in retention intervention studies: revisiting the evidence from a realist perspective. Geneva: World Health Organization; 2009.
- Fogarty L, de Vries D, Reavely E, Bunch E. Indicators for evaluating human resources for health capacity-building. In: Evaluation 2009: context and evaluation, American Evaluation Association, Orlando, 11-14 November 2009.
- Bristow G, Farrington J, Shaw J, Richardson T. Developing an evaluation framework for crosscutting policy goals: the Accessibility Policy Assessment Tool. Environ Plan A 2009; 41: 48-62 doi: 10.1068/a4092.
- Monitoring and Evaluation Working Group of the International Health Partnership and related initiatives (IHP+). Monitoring performance and evaluating progress in the scale-up for better health: a proposed common framework. Geneva: World Health Organization; 2009. Available from: http://www.internationalhealthpartnership.net//CMS_files/documents/a_proposed_common_framework_EN.pdf [accessed 15 February 2010].
- Codjia L, Jabot F, Dubois H. Evaluation du programme d’appui à la médicalisation des aires de santé rurales au Mali. Geneva: World Health Organization; 2008. In French.
- Lemiere C, Herbst C, Jahanshani N, Smith A, Soucat A. Reducing geographical imbalances of health worker distribution: what works, what does not and why? A labour market perspective of urban-rural health disparities, policy solution and experiences in sub-Saharan Africa. Washington, DC: The World Bank; 2009.
- Habicht J-P, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999; 28: 10-8 doi: 10.1093/ije/28.1.10 pmid: 10195658.
- Human Resources for Health (HRH) Action Framework [internet site]. Available from: http://www.capacityproject.org/framework/ [accessed 15 February 2010].
- Everybody's business: strengthening health systems to improve health outcomes. Geneva: World Health Organization; 2007. Available from: http://www.who.int/healthsystems/strategy/everybodys_business.pdf [accessed 15 February 2010].
- Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health 2004; 94: 400-5 doi: 10.2105/AJPH.94.3.400 pmid: 14998803.
- Buchan J. The benefits of AHP workforce stability. Edinburgh: Scottish Executive Health Department; 2005.
- Rabinowitz HK, Diamond JJ, Markham FW, Rabinowitz C. Long-term retention of graduates from a program to increase the supply of rural family physicians. Acad Med 2005; 80: 728-32 doi: 10.1097/00001888-200508000-00004 pmid: 16043525.