Section

8

Remote Evaluation of CHWPs

Programs may be delivered in an in-person setting, remotely through apps, or through other virtual means.

Some programs may use combinations of all of these delivery methods. Regardless of how a program is delivered, the evaluation of a program may face pressures requiring the evaluation itself to be conducted using remote methods.

To continue to assess the impact of your program, evaluation must adapt, even when circumstances are beyond your control. This is particularly true of outcome evaluation. You may need to conduct your evaluation remotely, using the support of technologies that support or utilize data downloads (e.g., Bluetooth), videos, or photographs. You may also need to conduct telephone interviews or meet participants and program staff at off-site locations.

While this toolkit is not focused on the delivery modality of CHWPs, we do want to share some guidance for remote evaluation of programs being delivered by any means. As we continue to learn more from evaluation of CHWPs, we will be sure to update this toolkit. Please refer back regularly for more updates.

Back to top

What is important to know regarding remote evaluation?

Program evaluation is not without challenges, and the same can be said about remote evaluation of CHWPs. While the remote collection of process measures may be similar to current methods (i.e., the counts or proportions of enrollees or sessions attended or completed), collecting outcome measures remotely will likely look different. For example, your program may have had standardized methodology to collect outcomes such as height and weight and measures of fitness and strength which may not be possible to collect safely in-person. Adjusting to remote data collection needs may strain your limited resources. This also may limit the types of outcomes your program can evaluate.

While remote evaluation adds a layer of complexity, keep in mind that many outcome measures can be evaluated with monitoring devices/ apps or patient self‐reported data.⁵³ A good place to start when determining how to evaluate outcomes remotely is to assess your program’s goals and identify how you can continue to achieve those goals in your new context. This may require you to get creative when collecting some outcome measures, but you may find that some data collection may be easier. For example, lengthy surveys or questionnaires on psychosocial outcomes may be less costly and less interruptive when completed online. Previous barriers may transform into opportunities for more robust data collection that can inform different aspects of your program.

Ideally your program will try to use measurements taken at home as well measurements conducted by professionals outside of the home. The best way to gather and monitor data over time may depend on both individual child or family factors as well as contextual considerations.⁵³ Whatever approach you take, consider which methods will allow you to most easily and cost effectively collect the most valid and reliable measurements. Methods that allow you to collect objective data such as height and weight measured by a trained professional using standardized tools are more reliable even if they are conducted remotely, such as at an outdoor facility in your local community. Parent or participant self-reported weight and height using standardized equipment at participant’s homes may be an adequate proxy measure that may also be more acceptable to parents and participants.

Back to top

How can I conduct a virtual or remote program evaluation?

When conducting evaluation remotely, there is no one size fits all approach. Your program will have to consider the context in which it is working and what resources are available to determine the best approach. It is recommended that you review all the ideal process outcomes from Section 4 and outcome measures from Section 5 to identify which are feasible for your program to collect when conducting remote evaluation. This will help ensure consistency over time within your program from one cycle to the next and allow for comparisons across different programs. Preparing your staff as well as participants and their families for evaluation to occur remotely is critical. Sharing resources, whether paper or virtual, is important. Staff and participants need to know what to expect, but how you share this information will depend on the way your staff and program participants prefer to receive information. You will also need to identify the resources, staff, and materials needed to conducted evaluation remotely. Staff may need to be trained to take measurements outside of the program site using portable technology. You may need to purchase licenses for video sharing technology, not only to implement a program virtually, but also as a way to conduct evaluations. You also need to have a plan for the types of measures you can evaluate remotely.

Height and weight are the most common anthropometric outcome measures collected in CHWPs, and you may find yourself wondering how your program can systematically collect this information in a remote environment. Other measures of interest that may be assessed remotely include those relating to strength or fitness. Some programs have found resources to send equipment to families at home.

Each of the examples shared in the continuum has contextual limitations that limit the feasibility of obtaining an accurate measurement. Standard guidance should be provided to limit variability. For example, measurements taken at home differ if taken on carpet vs hardwood floors. It is important to aim for optimal reliability, balancing the limitations with the needs of your patients and your programs.

Back to top

Summary

Whatever the reason for conducting an evaluation of your program remotely, know that this type of evaluation data can be collected. While telehealth interventions are not new, remote evaluation of these programs is. As virtual or remote CHWPs programs grow, it will be critical to evaluate them to learn about their impact and effectiveness. This may allow more programs to increase their scalability, especially to more rural or remote areas of the country. The availability of remote evaluation may justify conducting these programs in this manner. As more data are collected, we also will be able to better assess the validity and accuracy of these data collection methods, which will support program evaluation. These efforts will be critical to the future of CHWPs.

Back to top