Use of evidence to support healthy public policy: a policy effectiveness–feasibility loop
Sarah Bowman a, Nigel Unwin b, Julia Critchley c, Simon Capewell d, Abdullatif Husseini e, Wasim Maziak f, Shahaduz Zaman a, Habiba Ben Romdhane g, Fouad Fouad h, Peter Phillimore a, Belgin Unal i, Rana Khatib j, Azza Shoaibi j & Balsam Ahmad a
a. Institute of Health and Society, Newcastle University, Newcastle, England.
b. Faculty of Medical Sciences, University of the West Indies, Cave Hill Campus, PO Box 64, Bridgetown, BB11000 Barbados.
c. St George’s Hospital Medical School, University of London, London, England.
d. School of Population, Community and Behavioural Sciences, University of Liverpool, Liverpool, England.
e. Institute of Community and Public Health, Birzeit University, Birzeit, West Bank and Gaza Strip.
f. Robert Stempel College of Public Health and Social Work, Florida International University, Miami, United States of America.
g. CVD Epidemiology and Prevention Research Laboratory, National Public Health Institute, Tunis, Tunisia.
h. Syrian Center for Tobacco Studies, Aleppo, Syrian Arab Republic.
i. School of Medicine, Dokuz Eylül University, Konak, Turkey.
j. Institute of Community and Public Health, Birzeit University, Birzeit, West Bank and Gaza Strip.
Correspondence to Nigel Unwin (e-mail: email@example.com).
(Submitted: 18 March 2012 – Revised version received: 12 August 2012 – Accepted: 20 August 2012 – Published online: 14 September 2012.)
Bulletin of the World Health Organization 2012;90:847-853. doi: 10.2471/BLT.12.104968
Healthy public policy and its potential
Public policies, defined here as policies formulated at any level of government,1 have been key in bringing about some of the great public health achievements of the 19th and 20th centuries, including clean water and sanitation, immunization, safe working conditions and fluoridation of water.2 These examples represent “healthy public policies”, that is, public policies that have or are intended to have a positive impact on population health. Thus, public policies provide one of the primary means for a society to organize its efforts to protect and improve population health.3 It is increasingly recognized that public policy can also play a key role in the prevention and control of chronic, non-communicable diseases (NCDs). Indeed, the United Nations High-Level Meeting on Non-Communicable Diseases strongly emphasized the need for population-wide interventions involving education, legislation and regulatory and fiscal measures.4 Public policy measures are likely to be as important in promoting healthier diets and physical activity as they have been in reducing exposure to tobacco. However, the evidence on which public policies are effective in promoting healthier diets and physical activity is much scantier than the evidence surrounding tobacco control policies, some of which are known to be highly effective.5,6
Healthy public policy should be directed by evidence.7,8 However, an evidence-based approach to formulating healthy public policy has proved difficult to develop and implement.9 Evidence is required in three broad areas: policy content, policy implementation (i.e. translating content into effective policy) and policy outcomes (i.e. achievement of the desired effect).9 The aim of this paper is to present a framework designed to facilitate the development and implementation of evidence-based healthy public policy for the prevention and control of NCDs. The framework, known as the “policy effectiveness–feasibility loop” (PEFL), combines epidemiological modelling, local situation analysis and policy option appraisal and is designed to explicitly involve policy-makers. Below we describe the PEFL framework and its application within a project in four parts of the eastern Mediterranean.
Policy-making and research
In developing the PEFL framework, we were guided by the literature on the factors that influence public policy development and on ways of encouraging the use of research evidence to formulate healthy public policy (Box 1). The framework highlights the importance of the “interface” between researchers and policy-makers, particularly of promoting personal contact and dialogue between them. Policy-makers need to receive clear, persuasive information on disease burden and on interventions designed to mitigate the burden. Offering policy-makers choice and flexibility, such as a series of policy options with different costs and benefits, is more likely to result in action than providing them with a single solution. It is important that the options reflect the political, social and economic realities within which a given policy will be implemented. Finally, ongoing policy surveillance and outcomes assessment should be conducted to guide future policy development and implementation.
Box 1. Elements encouraging the use of research evidence in health policy, as addressed in the policy effectiveness–feasibility loop framework
- Demonstrate public health burden and benefit or harm from an intervention9
- Use analytical tools and prepare and communicate data effectively, simply and persuasively.9 Summarize with clear recommendations;8 personalized through a story9
- Promote personal contact between researchers and policy-makers7,9 e.g. through networks and “intermediary groups” (e.g. the media)10
- Invest in providing accessible, timely and relevant research for policy-makers3,11,12
- Clear political leadership, coordination and guidance7
- Estimate intervention cost9
- Consider “interface issues” when setting priorities, commissioning research and communicating findings7
- Conduct policy surveillance and track outcomes with different types of evidence9
- Provide policy-makers with costed incremental policy options, enabling a stepwise approach to policy implementation9
The PEFL framework
The PEFL framework has three main components: epidemiological modelling, situation analysis and option appraisal (Fig. 1). The epidemiological modelling and situation analysis components are conducted simultaneously, and the results of both are then integrated at the option appraisal stage. The framework is designed to foster dialogue and collaboration between researchers and policy-makers, especially during situation analysis and option appraisal. During the former, policy-makers act as participants who provide data and insights on a particular situation; in the latter, policy-makers play the key role, which consists of identifying the options that should be worked up in detail and subsequently implemented.
Fig. 1. The policy effectiveness–feasibility loop
Each component of the PEFL framework is briefly described below. A critical description of its application in four parts of the eastern Mediterranean follows for illustrative purposes.
In considering policy options for the prevention of chronic NCDs, a natural starting point is to try to understand their epidemiological distribution and, in particular, what drives trends in their incidence and mortality over time. The aim of the epidemiological modelling is to gain this understanding. The resulting model can then be used to explore the potential impact on incidence and mortality of different policy options intended to modify their determinants.
Our approach to epidemiological modelling is based on the coronary heart disease (CHD) IMPACT model, with the addition of a “new” diabetes policy model.13 The CHD IMPACT model, designed to estimate the relative contribution of changes in risk factors and treatment coverage to trends in mortality from CHD, has been successfully used in several high- and middle-income countries.14–16 The model requires data on trends in CHD mortality, major risk factors (e.g. smoking, arterial hypertension and diabetes) and coverage of effective treatments for the primary and secondary prevention of CHD (e.g. thrombolytic therapy for acute myocardial infarction and statins for hypercholesterolaemia). Identifying locally available data and assessing its quality are therefore important initial steps in the PEFL framework.
The goal of the situation analysis is to identify policy gaps as well as opportunities for implementing new policies designed to improve health outcomes of interest. Our approach to the situation analysis has been developed over many years. At first it built on rapid appraisal methods17,18 that were later adapted and tested during the performance of situation analyses of governmental and health system responses to NCDs in Africa, particularly Cameroon and the United Republic of Tanzania.19 The approach was further developed as part of the project described in this paper.
The situation analysis is designed to investigate existing policy and its implementation at three levels (Fig. 2). Level 1, or document analysis, involves identifying, collating and systematically reviewing all relevant government materials on health policies and health services provision. Documents are identified with the help of policy-makers and other local stakeholders and are reviewed using standard proformas. This analysis is conducted to gain an understanding of how the policy response is conceptualized (i.e. how it is meant to work in theory). In Level 2, key informants are interviewed to assess their awareness of and adherence to the theory and to obtain their insights into gaps and shortcomings within the system. Key informants include policy-makers within ministries of health as well as other important stakeholders. We developed semistructured interview schedules for these interviews. Finally, Level 3 involves investigating the extent to which the policies are influencing practice. In our work this has entailed direct participant observation (e.g. observing the delivery of care in health facilities) and individual interviews with patients and family members. Triangulating the data from the three levels provides a picture of policy as conceived theoretically versus actual practice and makes it possible to identify opportunities for implementing new policies and improving the implementation of existing ones.
Fig. 2. Situation analysis: overview of methods
Option appraisal and selection
Option appraisal and selection begins once the results of the epidemiological modelling and of the situation analysis have been obtained. The modelling identifies the major drivers of trends in disease incidence and mortality and enables “what if” analyses to explore the impact of policies intended to reduce risk factor levels or increase the coverage of health-care interventions. The situation analysis provides insight into existing policy gaps and the feasibility and acceptability (political, social and economic) of different approaches to filling those gaps.
The results of the epidemiological modelling and situation analysis are used to generate a list of policy options. Items on the list are then prioritized with input from policy-makers and other stakeholders, and some of the options given higher priority are further investigated by the research team, who examine in detail their estimated costs, potential health benefits and cost–effectiveness ratios. The options worked up in this fashion are presented back to the policy-makers, who then choose the policies they wish to implement.
Using the PEFL framework
The PEFL framework was developed as part of a multinational project known as MedCHAMPS (MEDiterranean studies of cardiovascular disease and hyperglycaemia: analytical modelling of populations’ socioeconomic transition).20 The overall aim of the project was to inform policy for the prevention and control of cardiovascular diseases and diabetes in four parts of the eastern Mediterranean: the Syrian Arab Republic, Tunisia, Turkey and the West Bank and Gaza Strip.20 The project, which began in 2009, was set up in a way intended to foster close collaboration between researchers and policy-makers. Its advisory committees, composed of policy-makers and researchers established within each study area at the start of the project, provided a forum for discussing the project’s progress and results as these became available. Seven senior policy-makers attended many meetings throughout the project. Although it would be premature to comment on the long-term success of the PEFL framework as applied in MedCHAMPS, which will not end until early 2013, some conclusions can be drawn from the challenges faced and lessons learnt while implementing this approach.
The main modelling activity undertaken in MedCHAMPS was to build an IMPACT model in each of the four project areas to explain trends in CHD mortality over the past 10 to 15 years. The biggest challenge was finding the data needed to populate the epidemiological models. The first few months of the project were spent in identifying available data sources and appraising their quality and completeness. Among the sources identified and used were routine mortality statistics, national and local risk factor surveys and statistics on health facility activity and drug use. Nonetheless, a shortage of data remained a major challenge and this was accounted for in the model by making explicit assumptions based on expert opinion and conducting sensitivity analyses to test the effect of changing the assumptions. The results of the CHD modelling will be described in detail in another paper, but here it is worth noting that trends in age-adjusted CHD mortality differ between project areas. Mortality is increasing in Tunisia and the Syrian Arab Republic but is declining in Turkey and the West Bank and Gaza Strip. The modelling was able to show that these differences are partly due to different trends in the prevalence of smoking, mean blood pressure and mean blood cholesterol levels. Despite the shortage of data, however, the models were able to account for 75–100% of the trends in CHD mortality and from them it was possible to examine the likely impact of interventions on future epidemiological trends.
Because diabetes is a major, growing public health problem in the eastern Mediterranean, the need to model trends in diabetes rates was identified. A new diabetes model was built as part of MedCHAMPS.13 The model is designed to require minimum data while providing policy-makers with useful insights into the potential impact of different policy options on rates of diabetes and its complications.
Fig. 2 illustrates the three levels involved in the situation analysis, as described earlier. The biggest obstacles to achieving this ambitious design were the relative novelty of and lack of familiarity with qualitative research on the health system in project areas. There were differences between areas. Qualitative health system research was most familiar in Turkey and the West Bank and Gaza Strip, where there is literature on the organization of the health sector, and it was least familiar in the Syrian Arab Republic, where no such literature exists and where critical scrutiny of a sector of government activity was inherently problematic. These differences were important, moreover, because we needed to obtain ethical consent and official approval for similar timetables to gain access to clinics and key informants and perform the situation analyses in all four settings simultaneously. We were fortunate in that the standing of our research partners in each study area assured such access.
To examine “practice” we undertook direct observations in four clinics in each study area. We purposively chose the clinics to reflect different socioeconomic characteristics, both public and private facilities and a mix of urban and rural settings around the cities where project partners were based (Ramallah, Aleppo, Tunis and Izmir). At each clinic we conducted structured interviews with staff to investigate approaches and challenges in caring for patients with cardiovascular diseases and diabetes. We also interviewed patients and family members, also purposively chosen, to explore their health beliefs and treatment experiences and identify any differences by age, sex or socioeconomic status. The “practice” component of the situation analysis is not intended to be representative in an epidemiological sense, but rather, to complement and be triangulated with the information gained from document reviews and key informant interviews so as to highlight recurring issues, problems or dilemmas.
We fortunately completed data collection for the situation analysis a few months before the events that marked the beginning of the “Arab awakening” in Tunisia in December 2010. Therefore, in two of the four project areas – the Syrian Arab Republic and Tunisia – our data reflect the situation that existed on the cusp of the events that have been unfolding since late 2010.
As a starting point for the option appraisal, recommended and potentially effective policy measures for the prevention and control of diabetes and cardiovascular diseases were considered. Two frameworks were used for this purpose: stepwise policy options for NCD prevention and control from the World Health Organization21 and the framework for public health interventions developed by the United Kingdom’s National Institute of Health and Clinical Excellence.22 In the workshops, policy-makers and researchers considered the recommended options in light of the findings of the epidemiological modelling and situation analyses. The following questions were posed: (i) In light of the results of the epidemiological modelling, what interventions are particularly relevant given the known burden of disease and its determinants? (ii) In light of the results of the situation analyses, what interventions are likely to be acceptable and feasible? A list of potential policy options resulted from this exercise. To further prioritize the items on the list, policy-makers and other stakeholders were asked to score each potential policy option on a small set of criteria, including feasibility and likely public health impact, in the same manner in which WHO prioritizes the research agenda for NCDs.23 All stakeholders complied with the request.
The prioritized options underwent further workup. This included cost estimates, cost-effectiveness ratios and a close look at their potential health benefits and implementation requirements, such as resources needed, roles and responsibilities involved and plans for monitoring and evaluation. Using this overall approach, five policy options were worked up in detail for each study area.
This paper presents a pragmatic framework for developing and prioritizing policy interventions tailored to local epidemiological, political and social conditions. The PEFL framework was developed as part of a multinational project aiming to inform policy for the prevention and control of cardiovascular diseases and diabetes in four middle-income territories. Central to the proposed approach is the iterative involvement of policy-makers in the collection of evidence and its appraisal.
The PEFL framework resembles the “equity effectiveness loop” proposed by Tugwell,24 which is intended to estimate the impact of interventions to reduce socioeconomic inequalities. The major difference is that our approach includes assessing the local context and the feasibility of potential interventions. Furthermore, our focus is on policy-level interventions, and hence the situation analysis involves local policy-makers. The situation analysis and option appraisal stages of our framework are analogous to policy dialogue,25 as they facilitate discussion between stakeholders and researchers on policies and how to implement them. The outputs resulting from application of the framework can be used to prepare policy briefs26 for informing stakeholder discussions on policy options.
Studies have shown that evidence rarely leads to policy changes directly.10 The relationship between evidence and policy implementation has been described as “complex, multifactorial, nonlinear and highly context-specific”.11 Research evidence may be outweighed by other factors, such as political pressure from powerful interest groups. However, the adoption of evidence-based policy can be facilitated by involving policy-makers in extended communication and interaction with researchers9 and by using conceptual frameworks such as the PEFL. Conceptual frameworks can also be useful in assessing research utilization and addressing the increasing demand for accountability in research expenditure.8,27 Policy-makers and planners are often enthusiastic about decision-support frameworks, but these have seldom been used in practice. Some policy-makers may feel that frameworks oversimplify complex situations or may not understand how frameworks operate work.12 Frameworks should be easy to understand and their assumptions should be explicit, particularly when planning and funding cycles are short and reorganization is frequent.27 The PEFL framework is conceptually simple but does not oversimplify the epidemiological modelling, situation analysis and option appraisal components. Our approach is consistent with the “interfaces and receptors” model proposed by Hanney et al.8 insofar as it aims to create interfaces between policy-makers and researchers at various stages. Explicitly involving policy-makers in the epidemiological modelling stage is particularly valuable in promoting dialogue between researchers and policy-makers on what is currently known and where further data are required
As shown in Box 2, one potential limitation of the PEFL framework is the methodological expertise required to conduct the epidemiological modelling, the situation analysis and the option appraisal. The MedCHAMPS project has been building capacity in these three areas within the project areas. To facilitate the implementation of the PEFL framework in other settings, researchers and policy-makers should jointly investigate the validity and utility of ways of conducting epidemiological modelling, situation analysis and option appraisals that require fewer resources. In conclusion, our early results show that implementing the PEFL framework within the MedCHAMPS project has successfully enabled researchers and policy-makers to work together on identifying evidence-based, cost-effective and feasible policy options for NCD prevention and control.
Box 2. Strengths and limitations of the policy effectiveness–feasibility loop (PEFL) framework
- Draws on previous frameworks, providing comprehensive consideration of socio-political and economic context
- Explicitly incorporates policy-makers’ views
- Includes economic feasibility assessment
- Applies to a variety of settings
- Incorporates sensitivity analysis, highlighting elements of the framework where uncertainty exists
- Allows policy-makers and implementers to draw their own conclusions and actions from results
- Need for further examples of practical application
- As with other frameworks, not all data readily available for input
- Involving policy-makers throughout may bias research used
- high degree of coordination and communication needed, with possible resource implications
- Expertise required to estimate costs, perform epidemiological modelling and provide contextual information
MedCHAMPS Collaboration: Ireland, the Syrian Arab Republic, Tunisia, Turkey, the United Kingdom of Great Britain and Northern Ireland, and the West Bank and Gaza Strip. Scientific leads from each of the partner organizations are as follows: Abdullatif Husseini (Birzeit University, West Bank and Gaza Strip); Wasim Maziak, Fouad M Fouad (Syrian Center for Tobacco Studies, Syrian Arab Republic); Habiba Ben Romdhane (National Public Health Institute, Tunisia); Belgin Unal (DokuzEylul University School of Medicine, Turkey); Julia Critchley (Project Principal Investigator, Newcastle University, United Kingdom); Simon Capewell (University of Liverpool, United Kingdom); Kathleen Bennett, Trinity College, Dublin, Ireland).
The authors received no specific funding for this article. The MedCHAMPS project is funded from the European Community’s Seventh Framework Programme (FP7/2007–2013) under grant agreement n°223075 – the MedCHAMPS project.
- Milio N. Glossary: healthy public policy. J Epidemiol Community Health 2001; 55: 622-3 doi: 10.1136/jech.55.9.622 pmid: 11511639.
- Centers for Disease Control and Prevention. Ten great public health achievements – United States, 1900–1999. MMWR Morb Mortal Wkly Rep 1999; 48: 241-3 pmid: 10220250.
- Powles J. Public health policy in developed countries. In: Detels R, Beaglehole R, Lansang MA, Guilliford M, editors. Oxford textbook of public health. 5th ed. Oxford: Oxford University Press; 2009.
- Resolution A/RES/66/2. Political declaration of the High-level Meeting of the General Assembly on the Prevention and Control of Non-communicable Diseases. In: Resolutions adopted by the General Assembly at its 66th session. New York: United Nations; 2012. Available from: http://www.un.org/Depts/dhl/resguide/r66.shtml [accessed 31 August 2012].
- Asaria P, Chisholm D, Mathers C, Ezzati M, Beaglehole R. Chronic disease prevention: health effects and financial costs of strategies to reduce salt intake and control tobacco use. Lancet 2007; 370: 2044-53 doi: 10.1016/S0140-6736(07)61698-5 pmid: 18063027.
- Critchley J, Unwin N. The role of public policy. In: Herman WH, Kinmonth AL, Wareham NJ, Williams R, editors. The evidence base for diabetes care. 2nd ed. Chichester: John Wiley & Sons; 2010.
- Tang JL, Griffiths S. Review paper: epidemiology, evidence-based medicine, and public health. Asia Pac J Public Health 2009; 21: 244-51 doi: 10.1177/1010539509335516 pmid: 19443874.
- Hanney SR, Gonzalez-Block M, Buxton M, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 2003; 1: 2- doi: 10.1186/1478-4505-1-2 pmid: 12646071.
- Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health 2009; 99: 1576-83 doi: 10.2105/AJPH.2008.156224 pmid: 19608941.
- Macintyre S. Evidence in the development of health policy. Public Health 2012; 126: 217-9 doi: 10.1016/j.puhe.2012.01.026 pmid: 22341770.
- Young J, Mendizabal E. Helping researchers become policy entrepreneurs: how to develop engagement strategies for evidence-based policy-making. London: Overseas Development Institute; 2009.
- Taylor-Robinson D, Milton B, Lloyd-Williams F, O'Flaherty M, Capewell S. Policy-makers' attitudes to decision support models for coronary heart disease: a qualitative study. J Health Serv Res Policy 2008; 13: 209-14 doi: 10.1258/jhsrp.2008.008045 pmid: 18806178.
- O’Flaherty M, Critchley J, Wild S, Unwin N, Capewell S, for the MedCHAMPS Project. Forecasting diabetes prevalence using a simple model: England and Wales 1993–2006. J Epidemiol Community Health 2010; 64: A7- doi: 10.1136/jech.2010.120956.18.
- Unal B, Critchley JA, Capewell S. Modelling the decline in coronary heart disease deaths in England and Wales, 1981-2000: comparing contributions from primary prevention and secondary prevention. BMJ 2005; 331: 614- doi: 10.1136/bmj.38561.633345.8F pmid: 16107431.
- Ford ES, Ajani UA, Croft JB, Critchley JA, Labarthe DR, Kottke TE, et al., et al. Explaining the decrease in U.S. deaths from coronary disease, 1980-2000. N Engl J Med 2007; 356: 2388-98 doi: 10.1056/NEJMsa053935 pmid: 17554120.
- Cheng J, Zhao D, Zeng Z, Critchley JA, Liu J, Wang W, et al., et al. The impact of demographic and risk factor changes on coronary heart disease deaths in Beijing, 1999-2010. BMC Public Health 2009; 9: 30- doi: 10.1186/1471-2458-9-30 pmid: 19159492.
- Scrimshaw NS, Gleason GR, editors. Rapid assessment procedures: qualitative methodologies for planning and evaluation of health related programmes. Boston: International Nutrition Foundation for Developing Countries; 1992.
- Anker M, Guidotti RJ, Orzeszyna S, Saprie SA, Thuriaux MC. Rapid evaluation methods (REM) of health services performance: methodological observations. Bull World Health Organ 1993; 71: 15-21 pmid: 8440033.
- Unwin N, Mugusi F, Aspray T, Whiting D, Edwards R, Mbanya JC, et al., et al. Tackling the emerging pandemic of non-communicable diseases in sub-Saharan Africa: the essential NCD health intervention project. Public Health 1999; 113: 141-6 doi: 10.1016/S0033-3506(99)00139-0 pmid: 10910412.
- Mediterranean studies of cardiovascular disease and hyperglycaemia: analytical modelling of population socio-economic transitions (MedCHAMPS). Newcastle: Newcastle University; 2011.
- Preventing chronic diseases: a vital investment: WHO global report. Geneva: World Health Organization; 2005.
- Kelly MP, Stewart E, Morgan A, Killoran A, Fischer A, Threlfall A, et al., et al. A conceptual framework for public health: NICE's emerging approach. Public Health 2009; 123: e14-20 doi: 10.1016/j.puhe.2008.10.031 pmid: 19100588.
- Prioritized research agenda for prevention and control of noncommunicable diseases. Geneva: World Health Organization; 2011.
- Tugwell P, de Savigny D, Hawker G, Robinson V. Applying clinical epidemiological methods to health equity: the equity effectiveness loop. BMJ 2006; 332: 358-61 doi: 10.1136/bmj.332.7537.358 pmid: 16470062.
- Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health policymaking (STP) 14: organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst 2009; 7: S14- doi: 10.1186/1478-4505-7-S1-S14 pmid: 20018104.
- Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP) 13: preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst 2009; 7: S13- doi: 10.1186/1478-4505-7-S1-S13 pmid: 20018103.
- Taylor-Robinson DC, Milton B, Lloyd-Williams F, O'Flaherty M, Capewell S. Planning ahead in public health? A qualitative study of the time horizons used in public health decision-making. BMC Public Health 2008; 8: 415- doi: 10.1186/1471-2458-8-415 pmid: 19094194.