Section

4

Process Measures

What are process measures?

Process measures focus on different aspects of your program’s delivery and activities such as enrollment, setting, transportation, participation, readiness to change, and mastery of skills. For each of these aspects, it is important to consider related facilitators and barriers.

This will help you describe and examine how your CHWP activities are delivered.^17,18 Process measures enhance your understanding of what is being done during the delivery of the program and how children and families are receiving your program. In this way, you will begin to understand what factors may help or hinder the uptake and effectiveness of your program. Furthermore, by looking at process measures you will be able to acquire a better understanding of the fidelity with which your program is being delivered, compared to how it was intended to be delivered. Finally, process measures will not only allow you to describe your program, but they will help you to modify your CHWP as you implement it and discover areas that may be improved or enhanced.

Back to top

What types of process measures are appropriate for CHWPs?

Process measures may be categorized according to those that capture information on the program and participants and those that focus on providers, including the interventionist who delivers the program. Whereas the goals of your program are to engage children and their families, meet enrollment goals, and enhance completion of program activities, the reality is that these may be very difficult to accomplish and track. By identifying and collecting process measure data proactively, you will be able to enhance engagement and participation in your program. In fact, in some instances process measures can be used as a surrogate for outcomes.

Program-based measures include those that describe:

• Requests to enroll, including phone calls, online registrations, and referrals
• Enrollment
• Attendance
• Program activities

Assessing requests to enroll or referrals about the program and enrollment helps you to determine if you are finding and enrolling the children and families you originally intended to target for program participation. By tracking referral and enrollment, you will be able to determine if your outreach and enrollment goals are likely to be met. Program attendance has been shown to correlate with successful weight- and behavior-related outcomes in studies of weight management programs.^19,20 Process measures relating to attendance include the number of sessions attended by each child and family and the proportion of enrolled children and families who completed all sessions in your program. Finally, process measures on program activities include in-session activities and between session activities, such as homework and at-home or outside activities.

Provider-based measures will help you understand the process that supports your program’s delivery. You can assess how providers deliver the program—specifically, the fidelity of training and delivery of the program. This includes understanding the extent to which training of providers and program delivery are done in relation to the program model or protocol. You can also assess how well the intended program is received, is culturally acceptable, and understood by the child and their family.

Provider-based measures are important because they can help you monitor the quality and fidelity of your program. You can gather process measures on provider training by tracking the number of training sessions provided and attendance at them as well as the proportion of providers who completed all the training sessions. If resources allow, you can consider conducting a pre-and post-test of provider-based measures to help ensure your program is being delivered as intended. For example, you can test knowledge of program components and delivery recommendations before and after the training. This pre-and post-test design can offer you insights into the effectiveness of your training and suggest revisions to its content and delivery.

There are two approaches to evaluate provider fidelity to program implementation. One approach uses an expert who rates real-time or transcribed intervention sessions using software such as the Roter Interaction Analysis System (RIAS) Software^21,22 or Motivational Interviewing Treatment Integrity coding.^23,24Another approach relies on a report, usually completed by the parent (in some instances, the adolescent), of what was discussed during a coaching or counseling interaction.²⁵ Many of these measures capture information on general counseling skills and some focus on specific techniques, such motivational interviewing, which has been shown to be effective in pediatric obesity interventions.²⁶ These types of evaluation can be expensive and require a skilled rater, so they are usually carried out only in research settings. However, if your program has the resources and capacity to evaluate fidelity in this way, it can offer insights on how to modify program delivery as needed. Alternatively, you may wish to track fidelity to motivational interviewing using a brief form of the Health Care Climate Questionnaire (HCCQ).²⁷ The HCCQ has been validated and studied in assessing motivational interviewing for weight management. The brief form of the HCCQ consists of five to seven questions and captures a parent’s self-report of the provider’s autonomy support and empathy during counseling. Because it only takes 3–5 minutes to complete and does not require trained personnel, it is a very low cost and feasible questionnaire to use.

Back to top

Back to top

How should I select process measures?

You may want to consider how to involve partners and members from your community in determining which process measures to use in your CHWP. By engaging these members and helping them to understand the importance of identifying process measures, you will be able to better choose measures that are meaningful to these important groups. You will also be able to clarify the goals and expectations for your program to your partners and community members, which can improve your program’s sustainability of. Furthermore, you will be able to engage in discussions that highlight the value of your program in improving the health of the children you are caring for. In doing so you will be able to define the benefit and value of your program that satisfies your partners and community members. These discussions are ultimately linked to the evaluation readiness steps described in Section 2: Program Evaluation Overview.

Back to top

If I had to choose, what process measures are ideal for all CHWPs to evaluate?

In summary, process measures that CHWPs may want to prioritize evaluating include:

• Number of referrals or requests to enroll
• Number or proportion of children and families who enrolled after a referral or request to enroll
• Number of sessions attended by each child and family
• Proportion of children and families who completed a prespecified portion, or all of the sessions in your program
• Assessment of provider skill and communication, such as a pre-and post-test skills assessment or the brief version of the Health Care Climate Questionnaire (HCCQ)
• Number of visits to the program web site, or time spent on program web sites

See Appendix 2 for a summary of these measures. This appendix presents Level 1 and Level 2 measures to facilitate your choice of measure, depending on your capacity and familiarity with evaluation.

Back to top

Summary

Process measures are a critical step in understanding aspects of your CHWP that may help or hinder its effectiveness so that targeted improvements can be implemented. Carefully consider which process measures you wish to include and discuss these with your partners. Be sure to include process measures that track key components of your program, including enrollment, attendance, and activities as well as provider skill and communication.

Back to top