Section
4
Process Measures
What are process measures?
Process measures focus on different aspects of your program’s delivery and activities such as enrollment, setting, transportation, participation, readiness to change, and mastery of skills. For each of these aspects, it is important to consider related facilitators and barriers.
This will help you describe and examine how your CHWP activities are delivered.17,18 Process measures enhance your understanding of what is being done during the delivery of the program and how children and families are receiving your program. In this way, you will begin to understand what factors may help or hinder the uptake and effectiveness of your program. Furthermore, by looking at process measures you will be able to acquire a better understanding of the fidelity with which your program is being delivered, compared to how it was intended to be delivered. Finally, process measures will not only allow you to describe your program, but they will help you to modify your CHWP as you implement it and discover areas that may be improved or enhanced.
What types of process measures are appropriate for CHWPs?
Process measures may be categorized according to those that capture information on the program and participants and those that focus on providers, including the interventionist who delivers the program. Whereas the goals of your program are to engage children and their families, meet enrollment goals, and enhance completion of program activities, the reality is that these may be very difficult to accomplish and track. By identifying and collecting process measure data proactively, you will be able to enhance engagement and participation in your program. In fact, in some instances process measures can be used as a surrogate for outcomes.
Program-based measures include those that describe:
- Requests to enroll, including phone calls, online registrations, and referrals
- Enrollment
- Attendance
- Program activities
Assessing requests to enroll or referrals about the program and enrollment helps you to determine if you are finding and enrolling the children and families you originally intended to target for program participation. By tracking referral and enrollment, you will be able to determine if your outreach and enrollment goals are likely to be met. Program attendance has been shown to correlate with successful weight- and behavior-related outcomes in studies of weight management programs.19,20 Process measures relating to attendance include the number of sessions attended by each child and family and the proportion of enrolled children and families who completed all sessions in your program. Finally, process measures on program activities include in-session activities and between session activities, such as homework and at-home or outside activities.
Provider-based measures will help you understand the process that supports your program’s delivery. You can assess how providers deliver the program—specifically, the fidelity of training and delivery of the program. This includes understanding the extent to which training of providers and program delivery are done in relation to the program model or protocol. You can also assess how well the intended program is received, is culturally acceptable, and understood by the child and their family.
Provider-based measures are important because they can help you monitor the quality and fidelity of your program. You can gather process measures on provider training by tracking the number of training sessions provided and attendance at them as well as the proportion of providers who completed all the training sessions. If resources allow, you can consider conducting a pre-and post-test of provider-based measures to help ensure your program is being delivered as intended. For example, you can test knowledge of program components and delivery recommendations before and after the training. This pre-and post-test design can offer you insights into the effectiveness of your training and suggest revisions to its content and delivery.
There are two approaches to evaluate provider fidelity to program implementation. One approach uses an expert who rates real-time or transcribed intervention sessions using software such as the Roter Interaction Analysis System (RIAS) Software21,22 or Motivational Interviewing Treatment Integrity coding.23,24Another approach relies on a report, usually completed by the parent (in some instances, the adolescent), of what was discussed during a coaching or counseling interaction.25 Many of these measures capture information on general counseling skills and some focus on specific techniques, such motivational interviewing, which has been shown to be effective in pediatric obesity interventions.26 These types of evaluation can be expensive and require a skilled rater, so they are usually carried out only in research settings. However, if your program has the resources and capacity to evaluate fidelity in this way, it can offer insights on how to modify program delivery as needed. Alternatively, you may wish to track fidelity to motivational interviewing using a brief form of the Health Care Climate Questionnaire (HCCQ).27 The HCCQ has been validated and studied in assessing motivational interviewing for weight management. The brief form of the HCCQ consists of five to seven questions and captures a parent’s self-report of the provider’s autonomy support and empathy during counseling. Because it only takes 3–5 minutes to complete and does not require trained personnel, it is a very low cost and feasible questionnaire to use.
NET-WORKS
Now Everybody Together for Amazing and Healthy Kids, or NET-WORKS, was part of the Childhood Obesity Prevention and Treatment Research (COPTR) Consortium. Four NET-WORKS field centers were funded for seven years—two centers focused on obesity prevention and two on obesity treatment. Each center conducted different interventions.
The NET-WORKS Minnesota (NET-WORKS MN) site included three components: primary care, home visit (the main component), and community-based parenting classes. NET-WORKS MN used these three synergistic components to influence parents or primary caregivers to ultimately help children improve their BMI by changing home environments and parenting practices.
Two key process measures were documented in the study database across all the intervention components: the dose delivered (i.e., amount and length of intervention sessions offered) and dose received (i.e., participant attendance and completion of the offered sessions).
NET-WORKS MN put many systems in place to track what was happening, including:
- Tracking which families had no participation after certain time periods and implementing reengagement protocols to figure out how to ensure the intervention worked for these families
- Utilizing reengagement protocols included telephone calls and distributing packets to give families some content when they were not participating in order to bridge active participation in the program
An example of how these strategies can be used to reengage a family over time is shown below:
Click to enlarge
*Figure reproduced with permission by Nancy E. Sherwood, PhD, through work supported by cooperative agreement U01HD068890 from the Eunice Kennedy Shriver National Institute of Child Health and Human Development at the National Institutes of Health.
Consequently, NET-WORKS MN was able to reengage families and work with them for enough time over the course of the program to track their progress and outcomes. NET-WORKS MN learned that it is critical to be flexible and responsive to the unique needs of families. This could be accomplished with tracking systems and reengagement strategies that are tailored to specific contexts.
How should I select process measures?
You may want to consider how to involve partners and members from your community in determining which process measures to use in your CHWP. By engaging these members and helping them to understand the importance of identifying process measures, you will be able to better choose measures that are meaningful to these important groups. You will also be able to clarify the goals and expectations for your program to your partners and community members, which can improve your program’s sustainability of. Furthermore, you will be able to engage in discussions that highlight the value of your program in improving the health of the children you are caring for. In doing so you will be able to define the benefit and value of your program that satisfies your partners and community members. These discussions are ultimately linked to the evaluation readiness steps described in Section 2: Program Evaluation Overview.
If I had to choose, what process measures are ideal for all CHWPs to evaluate?
In summary, process measures that CHWPs may want to prioritize evaluating include:
- Number of referrals or requests to enroll
- Number or proportion of children and families who enrolled after a referral or request to enroll
- Number of sessions attended by each child and family
- Proportion of children and families who completed a prespecified portion, or all of the sessions in your program
- Assessment of provider skill and communication, such as a pre-and post-test skills assessment or the brief version of the Health Care Climate Questionnaire (HCCQ)
- Number of visits to the program web site, or time spent on program web sites
See Appendix 2 for a summary of these measures. This appendix presents Level 1 and Level 2 measures to facilitate your choice of measure, depending on your capacity and familiarity with evaluation.
Summary
Process measures are a critical step in understanding aspects of your CHWP that may help or hinder its effectiveness so that targeted improvements can be implemented. Carefully consider which process measures you wish to include and discuss these with your partners. Be sure to include process measures that track key components of your program, including enrollment, attendance, and activities as well as provider skill and communication.