Bulletin of the World Health Organization

Improving the use of evidence in health impact assessment

Jennifer Mindell a, Jane Biddulph b, Lorraine Taylor c, Karen Lock d, Annette Boaz e, Michael Joffe f & Sarah Curtis g

a. Research Department of Epidemiology and Public Health, University College London, 1-19 Torrington Place, London, WC1E 6BT, England.
b. Research Department of Infection and Population Health, University College London, London, England.
c. National Institute for Health and Clinical Excellence, London, England.
d. London School of Hygiene and Tropical Medicine, London, England.
e. Division of Health and Social Care Research, King’s College London, London, England.
f. Department of Epidemiology and Public Health, Imperial College London, London, England.
g. Department of Geography, Durham University, England.

Correspondence to Jennifer Mindell (e-mail: j.mindell@ucl.ac.uk).

(Submitted: 08 June 2009 – Revised version received: 29 October 2009 – Accepted: 18 November 2009 – Published online: 23 December 2009.)

Bulletin of the World Health Organization 2010;88:543-550. doi: 10.2471/BLT.09.068510

Introduction

The health of an individual is influenced by a range of factors amenable to public policy on, for example, housing, education and transport. Consequently, multidisciplinary policies outside the jurisdiction of health services or health ministries have the potential to influence health.1 While policy-makers increasingly focus on obtaining better information, they often receive little support with decision-making. Health impact assessment (HIA) has been proposed as one mechanism that can support decision-making;2 it focuses primarily on policy outside the medical sector and on intersectoral actions.3 Recently, the World Health Organization (WHO) Commission on Social Determinants of Health recommended that the impact of all policies on health inequality should be assessed.1 In practice, HIA also includes an assessment of impact on health inequality.4

Consideration of the health impact of policies has been encouraged across the world. Major centres for HIA include Australia,5 Thailand6 and several countries in Europe.79 The Finnish government made HIA and “health for all policies” central strands of its 2006 presidency of the European Union.10 In addition, HIA is being introduced in the United States of America.11 The comparatively high rates of morbidity and mortality experienced in middle- and low-income countries can only partly be addressed by improving health-care provision, so the need for HIA is even greater in these countries than in the developed world.

Various definitions of HIA have been proposed over time.12 Ratner et al. (1997) defined HIA as “any combination of procedures or methods by which a proposed policy or program may be judged as to the effect(s) it may have on the health of a population”.13 In 1999, the WHO Regional Office for Europe added “and the distribution of those effects within the population”4 to include consideration of health inequalities; this concept is central to the Jakarta declaration.14 Further, HIA has also been described as “the use of the best available evidence to assess the likely effect of a specific policy in a specific situation”,15,16 leading to comparisons with evidence-based medicine. Current HIA methodology has been criticized for a lack of rigour in collecting and analysing evidence.1719 Thus, despite the policy drive to encourage its use, HIA will be discredited if it fails to be both rigorous and well founded.20

It is generally agreed that three types of knowledge are combined in HIA: that provided by stakeholders based on their experience; local data; and publicly available evidence, including past HIAs (Fig. 1). It is important that better frameworks are developed for integrating different types of research evidence so that they can be used in decision-making21 and, consequently, so that HIAs can make better use of publicly available evidence, such as scientific assessments, research studies published in peer-reviewed journals and previous HIAs. For simplicity, the term “evidence” is used throughout this paper to cover all publicly available evidence.

Fig. 1. Flowchart showing the roles of publicly available evidence, local data and stakeholders’ experience in the process of health impact assessment (HIA)
Fig. 1. Flowchart showing the roles of publicly available evidence, local data and stakeholders’ experience in the process of health impact assessment (HIA)

Several specific difficulties are encountered when reviewing evidence for use in HIA.22 These include: tight timescales; a diverse evidence base (created, in part, by the diversity of health impacts and complex causal pathways); the wide range of stakeholders; the need to make recommendations to decision-makers; the need to consider the reversibility of adverse factors that damage health and have an unequal impact on different population subgroups; and the need to review relevant qualitative as well as statistical evidence.21,22 Another important problem is a lack of skills and training, particularly in searching the literature, undertaking critical appraisals and synthesizing findings. In medium- and low-income countries, where specific expertise is likely to be scarce, there is a particular need for guidance on how to conduct high-quality HIA. Guidance for those commissioning HIAs is also necessary to enable them to evaluate the evidence obtained.

This paper describes the formulation, development and testing of a simple guide to reviewing evidence for use in HIA. The guide was designed to improve the way evidence is used by providing user-friendly and accessible assistance for those responsible for commissioning, conducting or appraising evidence reviews. The intention was to make the guide authoritative and practical. In addition, it was designed to support those carrying out both brief and comprehensive reviews of evidence in HIA, thus reflecting the flexibility required by the application of HIA itself. Finally, we investigated whether the guide enabled users to distinguish evidence reviews of acceptable quality from those of poor quality.

Methods

A steering group, which comprised the authors of this report plus the London Health Observatory’s HIA facilitator while she was in post, was set up to draft and amend the guide, to organize consultations and to disseminate the guide. Fig. 2 shows the various stages involved in the formulation (phase 1), development (phase 2) and testing (phase 3) of a guide to reviewing evidence for use in HIA. The entire process involved substantial consultation with individuals who carry out evidence reviews for use in HIAs, such as HIA academics, HIA practitioners and public health professionals (Table 1). A modified Delphi technique involving iteration and feedback was used at each stage during the many consultations on, and revisions to, the guide.

Fig. 2. Flowchart showing the different stages in formulating, developing and testing a guide to reviewing evidence for use in health impact assessment (HIA), 2003–2006
Fig. 2. Flowchart showing the different stages in formulating, developing and testing a guide to reviewing evidence for use in health impact assessment (HIA), 2003–2006

Members of an advisory group were asked to comment on the contents of the developing guide throughout the process. The advisory group and the individuals interviewed in the qualitative research study in phase 1, the invitees and volunteers who piloted and conducted a peer-review of the developing guide in phase 2, and individuals who tested the final guide in phase 3 were all independent of each other, and no individual was involved in more than one phase.

Wider consultation, with feedback from numerous additional individuals, was implemented by members of the steering group who presented various preliminary versions of the guide at HIA training courses, at workshops at international HIA and national public health conferences, and via email networks.

Phase 1: formulation

A scoping review of the literature and of existing guidelines on reviewing different types of research (i.e. quantitative and qualitative) was undertaken to identify criteria for the principles underpinning the assessment of research evidence likely to be relevant to HIA. Key themes and criteria were collated and used to structure the content of an outline of the guide that was presented to the advisory group for comment (Fig. 2).

The aim of the qualitative research study was to determine the most practical, relevant and accessible format for the guide. A series of in-depth, face-to-face interviews lasting about 75 minutes each was conducted with 10 representatives of the guide’s target audience. Participants were recruited from three cities in England: London, Birmingham and Cambridge. To achieve a cross-section of the different sectors, one individual involved in each of commissioning, conducting and appraising HIA evidence reviews was recruited from each of the three cities and at least one representative from each of local government, a public health authority and the independent (commercial) sector was recruited from each city.

The interviews were semi-structured and included open-ended questions. An outline of the guide was used towards the end of the interviews to elicit the interviewees’ attitudes to and views on format and presentation. Full details of the qualitative research study are provided elsewhere.23

Phase 2: development

The outline of the guide was modified on the basis of feedback from the advisory group, the results of the qualitative research study and feedback following wider dissemination of the outline (Fig. 2), and an initial version of the guide was produced. This was the starting point (stage 1) for phase 2, which also included wider dissemination, feedback and revision (stage 2), a pilot study (stage 3) and peer review (stage 4). Stage 2 was an iterative process in which initial and developing versions of the guide were sent alternately to the advisory group and to a wider range of individuals, as described above. The feedback received was used to amend the guide and produce the next version.

In the pilot study, a copy of the almost-final version of the guide was presented with an evaluation tool, which comprised a set of questions about the content, format and usability or usefulness of the guide.24 No instructions were provided with the guide to mimic real life in which HIA practitioners would use it as a stand-alone document for a range of purposes, including commissioning evidence reviews, conducting literature reviews and appraising an existing review for use in a specific HIA. Minor changes were made to the guide following feedback from the pilot study and, subsequently, peer reviewers were sent a modified copy for comment.

Phase 3: testing

To test the guide’s use in practice (stage 5), appraisers were sent one of five selected, brief or comprehensive evidence reviews that had been produced to support an HIA and asked to judge the review against the essential steps for a brief evidence review listed in the guide plus, if relevant, additional elements recommended for a “more comprehensive” review.24 In particular, appraisers were asked whether the review was “of sufficient quality” to be made widely available. Where possible, the author or authors of the evidence review and the appraisers were blinded to each other’s identities.

In stage 6, the final version of the guide was produced in a hard-copy form25 and made publicly available in September 2006 as a free downloadable Adobe Acrobat document on a nationally recognized public health web site in the United Kingdom of Great Britain and Northern Ireland,26 along with various supporting resources. The guide was also made available on a United Kingdom HIA web site.27 As a proxy estimate of the guide’s use in the field, the number of downloads from the guide’s web sites26,27 was ascertained.

Results

It took 2 years to progress from the outline of the guide used in phase 1 in 2004 to the final version in phase 3.

Phase 1: formulation

The criteria specifying the nature of the guide were developed following a scoping review of the literature and of existing guidelines on reviewing different types of evidence, such as systematic reviews,28 qualitative studies29 and non-randomized studies. The criteria included: the clarity of purpose and provenance of the review; the rationale for the review process; the processes employed by the review, including search strategies and inclusion and exclusion criteria; the quality of the research and the “strength” of the evidence; acknowledgement of likely biases; a summary of the conclusions drawn; and the limitations of the review. Consultations during this phase led to pivotal decisions being made about the purpose, presentation, format and content of the guide.

The qualitative, in-depth, face-to-face interviews revealed a range of divergent views about the practical purpose of a guide to reviewing evidence for use in HIA. Less experienced HIA practitioners wanted a guide that would provide advice on how to review evidence for an HIA. By contrast, more experienced practitioners perceived the value of the guide to be in establishing quality standards for reviewing evidence.

Both groups judged the most practical, and motivating, format to be a good practice approach rather than a checklist of procedures, a step-by-step toolkit or a set of rules and instructions. This viewpoint felt relevant and appealed to all practitioners, as the same document could both provide guidance on the process of reviewing evidence for HIA and establish procedural standards.

Other findings identified by the qualitative research study were that the guide should:

  • be written in plain, simple English;
  • avoid academic and subject-specific terms to maximize accessibility and user-friendliness;
  • provide a glossary of specialist terms whose use was unavoidable;
  • be as simple and short as possible to provide quick and easy access to practical information;
  • not be extended by being combined with practical examples or background literature;
  • offer guidance separately for “brief” and “more comprehensive” reviews, presented side-by-side, rather than consecutively;
  • employ an improved layout to aid comprehension; and
  • use different terms to prevent misinterpretation.

Phase 2: development

The guide was developed in accordance with the qualitative research findings. A three-column format was used to present guidance on “brief” and “more comprehensive” evidence reviews side-by-side. The first column detailed “essential steps in a brief evidence review” and the second, “additional elements for a more comprehensive evidence review”. A third column listed “tips and resources”. For details, see the final version of the guide (available at: http://www.lho.org.uk/viewResource.aspx?id=10846).

During development, stage 1 centred on enhancing information on “tips and resources”, introducing additional steps into the literature search and review processes, and making the guide easier to use and understand. With each stage during development, the comments made on the guide became progressively less substantial.24

Of the 32 participants directly involved in developing the guide, 26 were based in the United Kingdom, 5 came from elsewhere in Europe and 1 came from outside Europe (Table 1). Many other individuals from outside Europe actively involved in HIA were consulted through wider dissemination of the developing versions of the guide.

The most substantive modifications to the guide that arose in stages 3 and 4 were amendments to the introductory pages, the inclusion of examples of questions that a literature review for HIA might address, and the development of a glossary, as attempts to avoid jargon had not been entirely successful.

Phase 3: testing

The appraisers involved in testing the guide’s use in practice (stage 5) found it to be useable and accessible. Upon using the guide, one of the selected evidence reviews appraised was considered not to meet the required quality standards. Reviews often lacked information on how various steps in the review process had been conducted. For example, details of the criteria used in searching for relevant studies or for assessing the quality of studies included may have been missing, making it difficult to assess the representativeness of the evidence presented or the validity of the conclusions.

Appraisers generally made very positive comments about the guide, even those who said that they were initially sceptical. For example, one HIA practitioner commented: “My initial thoughts were that the guidance was too rigid for rapid HIAs but the more I worked through it and thought about it the more I agreed with it.” Another stated that although the criteria, “set a high standard for the quality of evidence reviews, they are reasonable and well judged in the context of HIAs. I would and will use them as a benchmark when undertaking general evidence reviews for HIAs …” Full details of the appraisals are contained in a published report.24

The final version of the guide produced in stage 6 is a 12-page A4 booklet26,27 that takes the reader through nine steps (steps A–I; Box 1) in reviewing evidence common to “brief” and “more comprehensive” evidence reviews. Supporting information for the guide has also been made available on the web, included a glossary,30 details of published sources of quality criteria31 and a document on assessing causality.32

Box 1. Steps in reviewing the evidence contained in a guide developed for reviewing evidence for use in health impact assessment (HIA)26,27

Step A: framing the question or questions

Step B: determining whether a literature review is required, and its scope

Step C: purpose, organization and structure

Step D: setting inclusion and exclusion criteria

Step E: literature search

Step F: critical appraisal

Step G: interpretation

Step H: conclusions

Step I: reporting

By mid-October 2009, the guide had been downloaded almost 2500 times: 2050 from the London Health Observatory web site (http://www.lho.org.uk)26 and 438 from the HIA Gateway27 at the Association of Public Health Observatories web site (http://www.apho.org.uk), both in the United Kingdom. Between November 2008 and April 2009, monthly downloads from the London Health Observatory web site averaged 113 (range 56–237). The average for the period from May to October 2009 was higher, at 170.

Requests for feedback elicited responses from a range of professionals from around the world, including a medical consultant in Nigeria, a research student in Israel and a policy officer and coordinator for a Healthy Built Environments programme in Australia. They were generally enthusiastic and found the guide to be useful in their locations despite its origins in the United Kingdom. Although the legislative and environmental contexts of HIA can differ, it was appreciated that the underlying principles of good practice in reviewing published evidence apply across all jurisdictions.

Discussion

We undertook a substantive, systematic, practitioner-led consultation with the aim of producing a guide to reviewing evidence for use in HIA that is simple, systematic and accessible to a cross-section of people involved in HIA. A glossary was produced explaining any jargon or words with different lay and technical meanings whose use could not be avoided in reviewing evidence for HIA. The guide stipulates the minimum criteria that must be satisfied by any HIA evidence review, however brief it is or however limited the resources for producing it. It also suggests additional elements that a review should include when circumstances permit to make its conclusions more robust.

Although a comprehensive review might be considered the gold standard for reviewing evidence for use in HIA, the guide also sets minimum quality standards for brief reviews, which can be conducted in days or sometimes weeks. In practice, anyone commissioning an HIA can specify that the guide is used when preparing an evidence review and, for example, require that all steps essential for a brief review should be included plus a specified list of additional elements for a more comprehensive review. When the guide’s use was tested in practice, it enabled users to distinguish between satisfactory and unsatisfactory evidence reviews.

Although expensive and time-consuming, the substantive consultations contributed considerably to the development and refinement of the guide. The guide was considered fit for purpose by those involved in the peer review and in testing the guide’s use in practice. With each successive consultation, fewer changes were required. The guide may still be 12 pages long, but this is the shortest document that is able to cover all the elements considered essential during the development phase.

Reviewing evidence for an HIA is complicated by differences in collating evidence between different contexts.22 The challenge of framing questions for a broad evidence review for HIA was dealt with by including brief examples in the guide. In addition, the guide makes it clear that study design must be taken into consideration when addressing questions relevant for HIA, which is different from the traditional hierarchical approach adopted for evaluating clinical evidence.33

If the guide is to have a positive impact on the process of HIA, it must be seen to be useful and user-friendly. The substantive consultations with HIA practitioner, HIA academics and public health professionals ensured that a wide range of stakeholders were involved throughout the entire development process and that the guide is suitable for both conducting and appraising evidence reviews for HIA. Moreover, the involvement of individuals in the United Kingdom, in other parts of Europe and in other continents was invaluable for making the guide applicable outside Europe.

Dora34 notes that HIA “brings transparency to the use of evidence in decision-making”. However, rigour is also needed to sustain the use of HIA. Use of the guide could improve the quality of evidence used in HIA and, therefore, increase the credibility of HIA. During development of the guide, it was noted that existing guidelines and toolkits, despite being theoretically sound and well regarded by experts, were often inappropriate for users’ needs, as has been observed in other public health contexts.35 The development process described in this paper was unusual in focusing specifically on the difficulties HIA practitioners encountered when trying to adopt an evidence-based approach in their daily work. The consultative process used here is one way of introducing scientific knowledge into public health practice.

During development of the guide, we also note that, within the HIA field, there was: (i) limited capacity for reviewing and synthesizing research findings; (ii) a lack of familiarity with the principles of critical appraisal and with research methods; (iii) more broadly, barriers to knowledge transfer and to the implementation of evidence-based practice; and (iv) a need to increase the capacity for reviewing evidence. While the guide’s use was being tested in practice, it was suggested that training based on the guide could be carried out.

The guide described here is unique. An existing Canadian decision-making tool on the use of evidence 36 is intended for use by managers and planners when making decisions on priorities for starting or ending specific interventions. Consequently, it differs from the tools needed to assess the potential impact of other policy proposals.37

The final published version of the guide25 has been widely disseminated and made available to networks of individuals actively involved or likely to become involved in HIA. Around 1700 printed copies of the final version have been distributed via national and international conferences and HIA training courses, mostly to international participants who do not have English as a first language (H Dreaves, personal communication, 2009) as well as to public health professionals and HIA practitioners. The high total and monthly download rates from web sites indicate that existing users are finding the document useful and are recommending it to others. The guide has also been made available at international conferences and several hundred were distributed in Thailand.

Finally, local adaptations of the guide can incorporate information on country-specific legislation and regulations, links to language-specific web-based resources and “local ownership” (i.e. the professional groups involved in the local process that resulted in the document accept it and are committed to its use), all of which are important for the successful implementation of guidelines.38,39 For example, an Australian version is being prepared with permission from the authors and funders of the original United Kingdom version. It includes minor changes to the text and alternative suggestions for information resources. Therefore, despite its origins in the United Kingdom, the guide is being used around the world and is bringing the underlying principles of reviewing evidence to the notice of HIA practitioners.


Acknowledgements

We would like to thank: the Department of Health for funding this project; Francesca Taylor for conducting, analysing and reporting the qualitative research; Raj Mehta at the London Health Observatory for creating the web pages; and Anne Sweetmore for designing the guide and its many revisions.

We would also like to thank the following individuals for their support for the project, for participating in the qualitative research, for commenting on draft principles or draft guidelines, for taking part in the pilot study or peer review of the guide, for offering an evidence review for appraisal, or for appraising one of the evidence reviews: Kukuwa Abba, Helen Atkinson, Muna Aziz, Lea den Broeder, Anna Boltong, Caron Bowen, Ceri Breeze, Antonella Cardone, Ben Cave, Rowena Clayton, Andrew Cook, Anthea Cooke, Adam Coutts, Margaret Douglas, Debbie Fox, Cheryl France, Angela Harden, Claire Higgins, Anna Hillier, Bobbie Jacobson, Stephen James, Caroline Keir, Mike Kelly, John Kemm, Carolyn Lester, Anita Linell, Mary Mahoney, Marco Martuzzi, Peter Molyneux, Jayne Parry, Mark Petticrew, Jenny Popay, Andrew Pratt, Francesca Racciopi, Deborah Richardson, Simon Sanderson, Amanda Sowden, Francesca Viliani, Salim Vohra, Rhiannon Walters and Colleen Williams.

Funding:

This work was undertaken by authors who were funded by the Department of Health through its Policy Research Programme (grant reference number 030 0072).

Competing interests:

None declared.

References

Share