Section
2
Program Evaluation Overview
What is program evaluation?
Program evaluation has been defined as “the systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding.”4
Program evaluation serves as a means to learn about the impact of your program and how that impact can inform sustainability. All of us are committed to having an impact. That means being attentive to what we do, monitoring how it’s going, and then making appropriate changes to improve our efforts. This continuous quality improvement (CQI) approach helps you think about evaluation as a continuous and systematic cycle of collecting data and responding to that data to improve program processes.5,6 The cycle’s components include identifying, planning, and improving. In the context of CHWP’s, you may want to ask questions such as:
- What actions will help our program best reach our goals and objectives?
- What can our program do more efficiently?
- What can our program do more effectively?
Why do I need to evaluate my CHWP?
Undertaking program evaluation is no easy feat, especially when many CHWPs are under-resourced. The idea of adding one more thing to your plate may seem daunting, but we know you are reading this toolkit because you recognize the value of program evaluation. When it comes to your CHWP, the value of program evaluation is extensive. Evaluation can help you do the following:
- Make data-driven decisions about your program
- Identify objectives and goals achieved as a result of your program (among participants as well as providers and the larger system in which the program is being delivered)
- Ensure the time, money, and other resources that enable program delivery and attendance are being spent efficiently and appropriately
- Ensure that your program and its activities are being delivered as intended
- Establish and sustain program practices that work
- Make the case for your program to interested parties
- Improve your program if the delivery or resources are not working as intended
- Determine if your program is meeting national guidelines
The University of Kansas’s Community Tool Box offers additional examples of the ways in which program evaluation can be used.
The value of program evaluation extends beyond single programs. When CHWPs across the country begin to evaluate their programs using similar evaluation frameworks and measures, comparisons can be made across programs to learn about the overall effectiveness of CHWPs. This can allow programs to learn from one another and share best practices. It also allows for the pooling of data across programs to show larger impact.
What is important to know regarding program evaluation?
When you are ready to conduct a program evaluation, there are several questions that you will need to address. The Community Tool Box highlights these questions:
- What will you evaluate?
- What criteria will you use to assess program performance?
- What performance standards do you want a program to attain for it to be considered successful?
- What conclusions about program performance can you draw based on the available evidence?
- What is important to your institution, organization, or community partner(s)?
You can learn more about program evaluation, including how to frame your evaluation from the Community Tool Box’s chapters on Evaluating Community Programs and Initiatives.
How do I conduct a program evaluation?
Now that you know what program evaluation is and why it is important, you may be wondering how to get started. Here we share an example of a framework you can use as a guide to develop your evaluation. A framework is a tool that can help you organize, link, or order key evaluation components to help you develop an evaluation plan. Often, these tools display relationships graphically. Frameworks create a way to understand:
- Evaluation questions
- Evaluation components and their logical order
- Evaluation data sources and data collection methods
One of the most commonly used program evaluation frameworks is the CDC’s Framework for Program Evaluation. This framework consists of six connected steps (listed below) to help tailor an evaluation of your CHWP. For more information on this framework, refer to the Overview of the Framework.
- Engage stakeholders
- Describe the program
- Focus the evaluation design
- Gather credible evidence
- Justify conclusions
- Ensure use and share lessons learned7
Each step in this framework has multiple options. There is no single way to conduct an evaluation; rather, it’s about selecting the best options at each step that maximize the following factors:
- Utility: Who needs the information from this evaluation? What specific information do they need?
- Feasibility: How much money, time, and effort can we put into evaluation?
- Propriety: Who needs to be involved in the evaluation for it to be relevant, engaging, and ethically sound?
- Accuracy: What evaluation design will lead to gathering valid and reliable information?8
Cultural competence in evaluation is an area you want to consider incorporating in evaluation because it can help your evaluation efforts be culturally relevant and provide meaningful findings to interested parties.9 CDC’s Framework for Program Evaluation can be used with the Evaluation Guide: Practical Strategies for Culturally Competent Evaluation. This guide provides cultural context for each of the six steps listed above in the framework.
*Adapted from the Centers for Disease Control and Prevention. Practical Strategies for Culturally Competent Evaluation. Atlanta, GA: US Dept of Health and Human Services; 2014.
What is a logic model, and do I need one?
A logic model is a tool that can help you plan, describe, manage, and evaluate your program. It graphically presents the shared relationships between your program’s activities, its intended effects, and assumptions of how your program works.10 Developing a logic model is a straightforward endeavor, and its basic level depicts how what you put into your program (i.e., resources and activities) results in what you get out of it. Logic models can positively impact your program because they may help you:
- Communicate your program’s purpose and expected results to staff and interested parties.
- Describe how your program will achieve its desired results.
- Serve as a valuable reference point for everyone involved in your program.
- Identify facilitators and barriers to program implementation.10
However, some programs may not require a formal logic model. In such cases, your program may have a logic model that staff and interested parties are implicitly aware of and understand, instead of one that is formally described. For many programs this works out fine, but if you find yourself in a scenario where an interested party is requesting a logic model, you can refer to Appendix 1, which provides a printer-friendly logic model template for your use.
Now that you have learned a little bit about program evaluation and the tools that can facilitate preparing you for a program evaluation, let’s take a closer look at what is required to be ready to evaluate.