Approach to evaluation

In complex initiatives, “theories of change” – expectations for how an initiative will bring about change  - often shift as lessons are learned and events in the world (e.g. an important election, a natural disaster) shape the work. Initiatives that are at an early stage or that involve multiple stakeholders often move in directions not anticipated even by the best-designed plans. Developmental approaches build the capacity of PLIs to make use of evaluation data by facilitating conversations about data in real time. This enables PLIs to constantly test and refine their strategies and approaches.

Participatory approaches ensure that parent leaders and PLI staff are engaged in:

• defining the outcomes that matter to them, such as advancing racial justice;
• posing and refining evaluation questions;
• structuring evaluations to support program improvement and strategic decision-making;
• making decisions about what data to collect and in what form; and
• making sense of data.

We expect that participatory evaluation of PLIs will facilitate the development and testing of new ways to measure and document processes and outcomes that reflect the experiences and needs of parent leaders and PLI staff. In particular, PLIs desire more ways to capture community-level changes that traditional methods, like surveys, cannot capture. In our review of existing methods and measures, we noted particular gaps in evaluation of impacts on public officials, collective action, and civic climate. Our Methods page offers suggestions for methods that are participatory and capture these “outer layers” of the Theory of Change.

While unique and newer methods are indeed needed, PLIs can use and adapt existing resources such as published evaluations, scholarly literature on parent leadership, and research and evaluations from other fields such as psychology, sociology, community organizing, and civic engagement. Using existing measures saves the time and expense of drafting, piloting, and revising scales, and can enhance confidence that the scales measure what they intend to measure (validity) and that they work consistently (reliability). However, existing measures must be chosen with care. Some may have been developed and tested with communities very different from those where PLIs work or may come from research traditions that don’t aim to advance values central to PLIs, such as racial justice and empowerment.

Qualitative approaches can help advance the racial justice goals of PLIs, by elevating the stories, reflections and analysis of parents of color. These narratives can not only counter negative stereotypes and deficit-based assumptions but also better reflect the transformative changes that result from building relationships. An over-reliance on quantitative measures tends to elevate the influence of the program on participants, but overlook more broad community and civic outcomes. 

While prioritizing qualitative approaches, we can select quantitative methods that address specific questions and complement qualitative findings. Mixed methods approaches (using both qualitative and quantitative data) increase the rigor of evaluation findings. Quantitative methods have several advantages:

• They facilitate the collection of data from a large number of people.
• They lend themselves to measuring change over time.
• They allow for comparing similarities and differences between multiple cohorts or between multiple initiatives.
• They allow insight into links between participants’ experiences with the program (e.g. satisfaction, attendance) and program outcomes.
• They are often less work to administer.

PLIs need affordable access to expertise, scholarly knowledge, existing evaluations, and networks of their peers with whom they can reflect, learn, and problem-solve. They need infrastructure and funder investment in capacity building for data collection, analysis, and acting on findings. They need tools, strategies, and best practices so that evaluation strengthens their reach and impact, rather than distract from their core work. Enhancing evaluation skills and monetary resources for evaluation will ensure that PLIs are able to reflect thoughtfully on successes and challenges, hold themselves accountable, support empowered and effective parent leaders, and advocate with funders and officials for the kind of evaluation that best reflects their work.