Chapters 11-15
Answer each of the following questions:
- Develop a working plan for how you would involve stakeholders in the development of the intervention evaluation plan. How does your plan address timing and budget constraints? In what ways does your plan include attention to interactions and group processes? Which strategies will be used to address the four criteria for having a good evaluation question?
- Participation rates for three programs in Layetteville and Bowe County are given in Exhibit 9-3. Review the information for the violence prevention program delivered in schools to individual students and the adult immunization program. Given that information, which type of effect evaluation samples would you suggest for each of those programs, and why?
- It is common for stakeholders to essentially ask, “How can you prove that our program had an effect?” What would be your standard, simple, “sound bite” response to this question?
- List the four criteria for assessing the rigor of qualitative studies. Imagine that you are planning to conduct in-depth interviews of program participants to assess program effects. For each criterion, give one example of how you would address that criterion in your plan for the outcome evaluation.
L. Michele Issel, PhD, RN Professor of PhD Program
University of North Carolina at Charlotte College of Health and Human Services
Charlotte, North Carolina
Rebecca Wells, PhD, MHSA Professor
The University of Texas School of Public Health
Houston, Texas
Health Program Planning and Evaluation A Practical, Systematic Approach for Community Health
FOURTH EDITION
World Headquarters Jones & Bartlett Learning 5 Wall Street Burlington, MA 01803 978-443-5000 [email protected] www.jblearning.com
Jones & Bartlett Learning books and products are available through most bookstores and online booksellers. To contact Jones & Bartlett Learning directly, call 800-832-0034, fax 978-443-8000, or visit our website, www.jblearning.com.
Substantial discounts on bulk quantities of Jones & Bartlett Learning publications are available to corporations, professional associations, and other qualified organizations. For details and specific discount information, contact the special sales department at Jones & Bartlett Learning via the above contact information or send an email to [email protected]
Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning Company
All rights reserved. No part of the material protected by this copyright may be reproduced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the copyright owner.
The content, statements, views, and opinions herein are the sole expression of the respective authors and not that of Jones & Bartlett Learning, LLC. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not constitute or imply its endorsement or recommendation by Jones & Bartlett Learning, LLC and such reference shall not be used for advertising or product endorsement purposes. All trademarks displayed are the trademarks of the parties noted herein. Health Program Planning and Evaluation: A Practical, Systematic Approach for Community Health, Fourth Edition is an independent publication and has not been authorized, sponsored, or otherwise approved by the owners of the trademarks or service marks referenced in this product.
There may be images in this book that feature models; these models do not necessarily endorse, represent, or participate in the activities represented in the images. Any screenshots in this product are for educational and instructive purposes only. Any individuals and scenarios featured in the case studies throughout this product may be real or fictitious, but are used for instructional purposes only.
This publication is designed to provide accurate and authoritative information in regard to the Subject Matter covered. It is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional service. If legal advice or other expert assistance is required, the service of a competent professional person should be sought.
15842-7
Production Credits VP, Executive Publisher: David D. Cella Publisher: Michael Brown Associate Editor: Danielle Bessette Vendor Manager: Nora Menzi Senior Marketing Manager: Sophie Fleck Teague Manufacturing and Inventory Control Supervisor: Amy Bacus Composition and Project Management: S4Carlisle Publishing
Services
Cover Design: Scott Moden Director of Rights & Media: Joanna Gallant Rights & Media Specialist: Merideth Tumasz Media Development Editor: Shannon Sheehan Cover Image: © Lynne Nicholson/Shutterstock Printing and Binding: Edwards Brothers Malloy Cover Printing: Edwards Brothers Malloy
Library of Congress Cataloging-in-Publication Data Names: Issel, L. Michele, author. | Wells, Rebecca, 1966- author. Title: Health program planning and evaluation: a practical, systematic approach for community health/L. Michele Issel and Rebecca Wells. Description: Fourth edition. | Burlington, MA: Jones & Bartlett Learning, [2018] | Includes bibliographical references and index. Identifiers: LCCN 2017010386 | ISBN 9781284112115 (pbk.) Subjects: | MESH: Community Health Services—organization & administration | Program Development—methods | Health Planning—methods | Program Evaluation—methods | United States Classification: LCC RA394.9 | NLM WA 546 AA1 | DDC 362.12068—dc23 LC record available at https://lccn.loc.gov/2017010386
6048
Printed in the United States of America 21 20 19 18 17 10 9 8 7 6 5 4 3 2 1
iii
© Lynne Nicholson/Shutterstock
Contents List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
List of Exhibits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvii
Preface to the Fourth Edition . . . . . . . . . . . . . . . . . . xix
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv
List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxvii
SECTION I The Context of Health Program Development 1
Chapter 1 Context of Health Program Development and Evaluation . . . . . . . . . . . . . . . 3
History and Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Concept of Health . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Health Programs, Projects, and Services . . . . . . 4
History of Health Program Planning and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Evaluation as a Profession . . . . . . . . . . . . . . . . . . . . . . . . . 8
Who Does Planning and Evaluations? . . . . . . .10
Roles of Evaluators . . . . . . . . . . . . . . . . . . . . . . . . . .10
Planning and Evaluation Cycle . . . . . . . . . . . . . . . . . . .11
Interdependent and Cyclic Nature of Planning and Evaluation . . . . . . . . . . . . . . .11
Using Evaluation Results as the Cyclical Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
Program Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . . .13
The Fuzzy Aspects of Planning . . . . . . . . . . . . . . . . . . .14
Paradoxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Uncertainty, Ambiguity, Risk, and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Introduction to the Types of Evaluation . . . . . . . . . .19
Mandated and Voluntary Evaluations . . . . . . .20
When Not to Evaluate . . . . . . . . . . . . . . . . . . . . . .21
The Public Health Pyramid . . . . . . . . . . . . . . . . . . . . . . .21
Use of the Public Health Pyramid in Program Planning and Evaluation . . . . . . . .23
The Public Health Pyramid as an Ecological Model . . . . . . . . . . . . . . . . . . . . .23
The Town of Layetteville in Bowe County . . . . . . . . .25
Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25
Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Chapter 2 Relevance of Diversity and Disparities to Health Programs . . . . . . . . . . . . . . . . . . 29
Health Disparities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
Diversity and Health Disparities . . . . . . . . . . . . .32
Diversity and Health Programs . . . . . . . . . . . . . .33
Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38
Influences of Sociocultural Diversity on Interventions . . . . . . . . . . . . . . . . . . . . . . . . .38
Influences of Biological Diversity on Interventions . . . . . . . . . . . . . . . . . . . . . . . . .39
Approaches to Developing Programs . . . . . . .39
Profession and Provider Diversity . . . . . . . . . . . .40
The Three Health Provider Sectors . . . . . . . . . .43
Diversity Within Healthcare Organizations and Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43
Organizational Culture . . . . . . . . . . . . . . . . . . . . . .44
Cultural Competency Continuum . . . . . . . . . . .44
Enhancing Cultural Competency . . . . . . . . . . .48
iv Contents
Types of Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . .75
Organizational Assessment . . . . . . . . . . . . . . . . .75
Marketing Assessment . . . . . . . . . . . . . . . . . . . . . .76
Needs Assessment . . . . . . . . . . . . . . . . . . . . . . . . .76
Community Health Assessment . . . . . . . . . . . . .77
Workforce Assessment . . . . . . . . . . . . . . . . . . . . . .77
Steps in Planning and Conducting the Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77
Form and Develop the Team . . . . . . . . . . . . . . . .78
Create a Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79
Involve Community Members . . . . . . . . . . . . . .79
Define the Population . . . . . . . . . . . . . . . . . . . . . .80
Define the Problem to Be Assessed . . . . . . . . .81
Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81
Prioritize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82
Make a Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . .82
Implement and Continue . . . . . . . . . . . . . . . . . . .83
Anticipate Data-Related and Methodological Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83
Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85
Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .85
Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87
Chapter 4 Characterizing and Defining the Health Problem . . . . . . . . . . . . . . . . . . . 91
Collecting Data From Multiple Sources . . . . . . . . . . .91
Public Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91
Primary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92
Observational Data . . . . . . . . . . . . . . . . . . . . . . . . .92
Archival Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93
Proprietary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . .93
Published Literature . . . . . . . . . . . . . . . . . . . . . . . .93
Data Beyond Street Lamp . . . . . . . . . . . . . . . . . . .93
Collecting Descriptive Data . . . . . . . . . . . . . . . . . . . . . .94
Magnitude of the Problem . . . . . . . . . . . . . . . . . .94
Dynamics Leading to the Problem . . . . . . . . . .94
Population Characteristics . . . . . . . . . . . . . . . . . .96
Attitudes and Behaviors . . . . . . . . . . . . . . . . . . . .96
Years of Life and Quality of Life . . . . . . . . . . . . . .96
Stakeholders and Coalitions . . . . . . . . . . . . . . . . . . . . .50
Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51
Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .53
Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54
SECTION II Defining the Health Problem 57
Chapter 3 Community Health Assessment for Program Planning . . . . . . . . 59
Defining Community . . . . . . . . . . . . . . . . . . . . . . . . . . . .59
Community as Context and Intended Recipient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60
Defining Terms: Based, Focused, and Driven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61
Types of Needs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62
Types of Strengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
Approaches to Planning . . . . . . . . . . . . . . . . . . . . . . . . .64
Incremental Approach . . . . . . . . . . . . . . . . . . . . . .64
Apolitical Approach . . . . . . . . . . . . . . . . . . . . . . . .66
Advocacy Approach . . . . . . . . . . . . . . . . . . . . . . . .66
Communication Action Approach . . . . . . . . . .67
Comprehensive Rational Approach . . . . . . . . .67
Strategic Planning Approach . . . . . . . . . . . . . . .68
Summary of Approaches . . . . . . . . . . . . . . . . . . .69
Models for Planning Public Health Programs . . . . .69
Mobilizing for Action through Planning and Partnership (MAPP) . . . . . . . . . . . . . . . . . .70
Community Health Improvement Process (CHIP) . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
Protocol for Assessing Community Excellence in Environmental Health (PACE-EH) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
In Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
Perspectives on Assessment . . . . . . . . . . . . . . . . . . . . .71
Epidemiological Perspective . . . . . . . . . . . . . . . .72
Public Health Perspective . . . . . . . . . . . . . . . . . . .74
Social Perspective . . . . . . . . . . . . . . . . . . . . . . . . . .74
Asset Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . .74
Rapid Perspective . . . . . . . . . . . . . . . . . . . . . . . . . .75
v Contents
Path to Program Outcomes and Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Components of the Effect Theory . . . . . . . . . 135
Matching Levels: Audience, Cause, Intervention, and Effects . . . . . . . . . . . . . . . 137
Generating the Effect Theory . . . . . . . . . . . . . . . . . . 138
Involve Key Stakeholders . . . . . . . . . . . . . . . . . 138
Draw Upon the Scientific Literature . . . . . . . 138
Diagram the Causal Chain of Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Check Against Assumptions . . . . . . . . . . . . . . 141
Functions of Program Theory . . . . . . . . . . . . . . . . . . 141
Provide Guidance . . . . . . . . . . . . . . . . . . . . . . . . . 141
Enable Explanations . . . . . . . . . . . . . . . . . . . . . . 142
Form a Basis for Communication . . . . . . . . . . 142
Make a Scientific Contribution . . . . . . . . . . . . 143
Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Discussion Questions and Activities . . . . . . . . . . . . 144
Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Chapter 6 Program Objectives and Setting Targets . . . . . . . . 147
Program Goals and Objectives . . . . . . . . . . . . . . . . . 147
Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Foci of Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 148
Objectives and Indicators . . . . . . . . . . . . . . . . . 151
Good Goals and Objectives . . . . . . . . . . . . . . . 154
Using Data to Set Target Values . . . . . . . . . . . . . . . . 156
Decisional Framework for Setting Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Stratification and Object Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Use of Logic Statements to Develop Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Options for Calculating Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Caveats to the Goal-Oriented Approach . . . . . . . 170
Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Discussion Questions and Activities . . . . . . . . . . . . 171
Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Statistics for Describing Health Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99
Descriptive Statistics . . . . . . . . . . . . . . . . . . . . . . 100
Geographic Information Systems: Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Small Numbers and Small Areas . . . . . . . . . . 101
Epidemiology Rates . . . . . . . . . . . . . . . . . . . . . . 102
Stating the Health Problem . . . . . . . . . . . . . . . . . . . . 102
Diagramming the Health Problem . . . . . . . . 102
Writing a Causal Theory of the Health Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Prioritizing Health Problems . . . . . . . . . . . . . . . . . . . 110
Nominal Group Technique . . . . . . . . . . . . . . . . 111
Basic Priority Rating System . . . . . . . . . . . . . . . 111
Propriety, Economics, Acceptability, Resources, and Legality (PEARL) Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Prioritizing Based on Importance and Changeability . . . . . . . . . . . . . . . . . . . . . 114
Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Discussion Questions and Activities . . . . . . . . . . . . 117
Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
SECTION III Health Program Development and Planning 121
Chapter 5 Program Theory and Interventions Revealed . . . . . . . . . . . . . . . . . . 123
Program Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Process Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Effect Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Finding and Identifying Interventions . . . . . 126
Types of Interventions . . . . . . . . . . . . . . . . . . . . 127
Specifying Intervention Administration and Dosage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Interventions and Program Components . . . . 130
Characteristics of Good Interventions . . . . . 131
vi Contents
Budgeting as Part of Planning . . . . . . . . . . . . . . . . . . 204
Monetize and Compute Program Costs . . . . . 204
Budget for Start-Up and Evaluation Costs . . . 205
Break-Even Analysis . . . . . . . . . . . . . . . . . . . . . . . 205
Budget Justification . . . . . . . . . . . . . . . . . . . . . . 207
Budget as a Monitoring Tool . . . . . . . . . . . . . . . . . . . 209
Budget Variance . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Types of Cost Analyses . . . . . . . . . . . . . . . . . . . . 209
Information Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Health Informatics Terminology . . . . . . . . . . . 214
Information Systems Considerations . . . . . . 214
Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Discussion Questions and Activities . . . . . . . . . . . . 217
Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs . . . . . . . . . . . . . . . 219
Assessing the Implementation . . . . . . . . . . . . . . . . . 219
Implementation Documentation . . . . . . . . . 220
Implementation Assessment . . . . . . . . . . . . . 221
Implementation Evaluation . . . . . . . . . . . . . . . 221
Efficacy, Effectiveness, and Efficiency . . . . . . . . . . . 222
Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . 223
Quantifying Inputs to the Organizational Plan . . . . . . . . . . . . . . . . . . . 223
Human Resources . . . . . . . . . . . . . . . . . . . . . . . . 228
Physical Resources . . . . . . . . . . . . . . . . . . . . . . . . 229
Quantifying Outputs of the Organizational Plan . . . . . . . . . . . . . . . . . . . 230
Information Systems . . . . . . . . . . . . . . . . . . . . . . 230
Monetary Resources . . . . . . . . . . . . . . . . . . . . . . 230
Quantifying Inputs to the Services Utilization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Participants and Recipients . . . . . . . . . . . . . . . 230
Intervention Delivery and Fidelity . . . . . . . . . 231
Quantifying Outputs of the Services Utilization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
Coverage as Program Reach . . . . . . . . . . . . . . 234
Participant-Related Issues . . . . . . . . . . . . . . . . . 238
Program Logistics . . . . . . . . . . . . . . . . . . . . . . . . . 240
SECTION IV Implementing and Monitoring the Health Program 173
Chapter 7 Process Theory for Program Implementation . . . . . . . . . . . 175
Organizational Plan Inputs . . . . . . . . . . . . . . . . . . . . . 175
Human Resources . . . . . . . . . . . . . . . . . . . . . . . . 177
Physical Resources . . . . . . . . . . . . . . . . . . . . . . . . 179
Transportation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Informational Resources . . . . . . . . . . . . . . . . . . 180
Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Managerial Resources . . . . . . . . . . . . . . . . . . . . 180
Fiscal Resources . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Organizational Plan Outputs . . . . . . . . . . . . . . . . . . . 182
Time Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Operations Manual . . . . . . . . . . . . . . . . . . . . . . . 182
Organizational Chart . . . . . . . . . . . . . . .