Section
3
Evaluation Readiness
What is evaluation readiness?
For an organization to undertake an evaluation of a CHWP it is currently offering, the organization must first be ready for evaluation.
Evaluation readiness is determined by an organization’s ability to successfully assess how well a project was implemented, how successful that project was, and what were the determinants of the degree of success or failure.7 Evaluation readiness has several components, including leadership support; organizational culture and readiness; and the evaluation expertise, skills sets, and available resources. Because a key challenge for CHWPs is funding, considerations on the sustainability of the program are critical.
Organizational readiness reflects how well an organization is prepared to undertake an evaluation by way of operations, resources, and work culture and attitudes towards evaluation.11 The extent to which all individuals or groups in an organization are psychologically and behaviorally prepared to implement a CHWP evaluation affects organizational readiness. This includes ensuring that all interested parties are on the same page about the purpose and goals of evaluation, including program sustainability.
An organization starts the process of becoming evaluation-ready with mission and vision statements for the CHWP that include the need for evaluation. These statements must align with the purpose of the organization. Mission statements define why a program exists while helping to guide decisions about priorities, actions, and responsibilities. A vision statement is a clear and aspirational statement of what your program wants to accomplish.
Community-based CHWPs may face unique challenges such as the lack of access to resources to undertake evaluation in their efforts to develop an evaluation plan. However, clearly defining these needs and how they align with the mission and vision for the CHWP and working with the organization’s leadership and interested parties may help in overcoming these challenges. With the right tools and training, existing staff may be able to develop and implement an evaluation. Furthermore, continued development of trust-based relationships with interested parties and community partners will help develop a successful evaluation effort.
What are evaluability assessments?
An evaluability assessment is a methodical process to decide whether a program has the necessary information, engagement with interested parties, and organizational structure to be evaluated successfully.12 The evaluability assessment will help you determine if it is be feasible to conduct a full evaluation. It helps identify whether program evaluation is justified, feasible, and likely to provide valuable information. If a program already has an evaluation plan, conducting an evaluability assessment can help gauge the effectiveness of that plan. A final advantage of an evaluability assessment is that it is likely to be less expensive than a full evaluation. However, if a full evaluation can be carried out, that is the preferred course of action. You can find more detailed information about evaluability assessments from BetterEvaluation which provides an overview that is based on a literature review of evaluability assessment.
What is important to know regarding evaluation readiness?
When determining your program’s evaluation readiness, it is important to fully understand the needs of your program. This may include staffing and training, financial and capital resources, technology, and infrastructure needs. At the same time, consider what aspects of your organization’s culture, policies, and procedures will help or hinder evaluation efforts. Use this information as you engage in discussions with others in your leadership that focus on the need for evaluation. You can use the same process of reflection to have similar discussions with other interested parties and community-based partners.
If the information is available, find out the prevalence of overweight and obesity in your program’s community. Information might be available through resources such as County Health Rankings and Roadmaps. Also try and understand the local drivers, including perceptions, which influence behavioral choices in your community. Communicate this information back to your organization, interested parties, and community partners. You will also want to determine how you will use data from the CHWP to support your evaluation efforts, including data relating to enrollment, attendance, and outcomes. Determining the evaluation readiness of your organization and program is an on-going process, and you can expect to make frequent improvements as both your needs and experience change. Similarly, the actual evaluation of the CHWP is also an on-going process that needs frequent adjustments.
The Primary Care Obesity Network
Nationwide Children’s Hospital (NCH) developed the Primary Care Obesity Network (PCON) as a community outreach program with primary care practices linked to NCH’s tertiary care obesity center.13,14 Early in the development of PCON, partners, champions, primary care practices and community members agreed on the importance of evaluation to the successful implementation and support of weight management programs and services in primary care centers and in the community. These members understood that the need for evaluation must be driven by the clarity of the program’s purpose, goals, and sustainability. As a result, all members were able to appreciate their roles, and the PCON model was borne.
PCON can be likened to a hub (the tertiary care center) and spokes (the primary care practices) model of care. The hub provides training, education and materials, and administrative support and serves as an integrator between the central clinic and community. Additionally, the hub serves as a referral source for the “spokes” (primary care practices); however, the hub has no operational responsibilities within each primary care practice. In this way the overall costs were lowered, and this allowed the hub to take on the responsibility of evaluation and data analysis.
The PCON evaluation plan consists of three levels of outcomes:
- Child and family outcomes.
- Process outcomes.
- Balancing outcomes.
These outcomes address whether the program affects a child’s and/or family’s knowledge, health, or well-being as it relates to their healthy weight.
These outcomes address how the program is implemented.
These outcomes help determine if the program has any negative outcomes that may affect the program’s acceptability or even outcomes.
PCON has several metrics that have a fixed and precise relationship to PCON’s goals. The variables are at the program, provider, and patient/family levels and include the following: program utilization, resource utilization, training and continuing education participation, anthropometrics (i.e., BMI, BMI z-score), and behavior change.
Healthcare-Community Partnerships for Child Obesity Treatment
Duke University developed a model for child obesity treatment that involves both the primary care provider and a community-based recreation center.15,16 The primary care provider screens for obesity and related co-morbidities and provides family-centered counseling on lifestyle modifications. The community-based recreation center can be a parks and recreation facility, a local YMCA, after-hours school facility, or other community settings. Families participate in evening and weekend cooking classes and physical activities tailored to children with obesity. The model is implemented using an evidence-based curriculum and executed through a shared-use agreement between the healthcare and community entities.
This healthcare-community model has evidence for engaging low-income and racially/ethnically diverse families over a 6- to 12-month period and has demonstrated effectiveness in improving physical activity and weight-related quality of life. Outcomes are sustained to at least 2 years.
The healthcare-community evaluation plan assesses the following:
- Proportion of patients with high BMI who are referred from the primary care physician to the community program.
- Proportion of referred patients who participate in the community program.
- Proportion of patients who meet the USPSTF benchmark of ≥ 26 hours over a 2- to 12-month period of intensive, behavioral, comprehensive weight management.
- Health outcomes attributed to participation, including BMI and related co-morbidities.
Ready for evaluation?
Tools to help you prepare for evaluation table
- Ready, Set, Change! is an online decision support framework for assessing organizational readiness. This evidence-based framework maps organizational readiness to four distinct constructs around perceptions, willingness and ability of staff and the organization as a whole to undertake change. This free, online tool is intuitive to use and provides helpful insight on organizational readiness.
- The National Implementation Research Hexagon Tool is both a qualitative and quantitative resource to assess evaluation readiness. It is structured around a team approach to three program indicators and three implementing site indicators. This is a free, multi-item printable tool that needs to be scored. Whereas this tool can be used at any stage in a program’s evaluation and implementation, it is best used in the formative stages of a program’s evaluation. The tool can also help users apply an equity lens to consider how their program could advance equitable outcomes for all individuals and families.
- The Wilder Collaboration Factors Inventory is a set of several online resources, including questionnaires and surveys, that can be used to quantitatively assess the degree and strength of collaboration within an organization and provide guidance on evaluating relevant work. This inventory can help an organization connect primary care practices with community partners when they have different organizational structures and policies.
Summary
When planning and considering the evaluation of a CHWP, it is important to understand both readiness of the parent organization and the evaluation readiness of the CHWP itself. Understanding both levels of readiness will help ensure that any evaluation plan incorporates organizational factors, culture, and perceptions around evaluation, as well as the structural and usability aspects inherent to the program that may help or hinder evaluation. These include expectation management on the part of the organization, partners, and community members as they relate to timelines, feasibility of the program’s approach to weight management, and definitions of success.