Evaluating Advocacy: An Exploration of Evidence and Tools to Understand What Works and Why

"The advent of the Internet and social media has contributed to the growth of advocacy initiatives in terms of their size, frequency and scope. With this growth in formal advocacy programmes comes the challenge of evaluating them."

This International Initiative for Impact Evaluation (3ie) working paper describes growth in advocacy initiatives and analyses challenges in evaluating successful advocacy programmes. Factors associated with advocacy interventions, measured using impact evaluations, are selected from successful programmes. The paper summarises them in a toolkit section. 

Along with support from the Water Supply and Sanitation Collaborative Council, 3ie reviewed "56 impact evaluations and used positive deviance analysis to identify factors associated with successful advocacy programmes." The review was complimented by in-depth interviews with 14 advocacy and advocacy evaluation experts and a literature review "to interpret findings, assess challenges related to evaluating advocacy, and identify suitable methods."

The evidence base was drawn from 56 evaluations of advocacy interventions or campaigns from across the world. Each evaluation had to be an experimental or quasi-experimental impact evaluation. The campaigns included primarily information and media campaigns, among others. The campaign goals evaluated included:

• "Behaviour change of beneficiaries
• Opinion, attitude or knowledge change of beneficiaries
• Improved material situation of individuals
• Democratic change or channels for citizen involvement in political process
• Behaviour change of policymakers or public servants
• Civil society capacity building, organising or building of alliances
• Budget allocation or reallocation"

Key factors influencing successful advocacy programmes include: 

• Who advocates, for example: "Behaviour change programmes that involved a government official as an advocate were found to be 83 per cent more likely to be effective than those that did not. When research teams were advocates, interventions were 72 per cent more likely to be effective."
• Whether incentives are offered, for example: "Interventions that provided incentives to change the behaviour of beneficiaries or targets (individuals or households) had a success rate of 93 per cent. Monetary or in-kind support offered to participants was small (e.g. sweets, stickers or calendars during informational visits)."
• Whether the target group is offered comparison information about other groups, for example: "Three out of four advocacy programmes that provided the target group (e.g. households or communities) with comparison information about the intervention in another community were more successful than advocacy programmes that provided information only about the target group. For instance, telling people their neighbours are conserving energy is three times more effective in curtailing a household’s energy expenditure than simply telling them they can cut their bills by using certain methods."
• Who delivers messages, for example:  "Information provided directly by government representatives or non-governmental organisations yielded more positive results (73%) than information provided through any other method."  
• Which channels are used to disseminate information, for example "Information campaigns that used newspaper and other mass media channels were most successful (73%). Half of the campaigns that provided information to individuals via text message (50%) were reported as successful, while only one third (33%) of those that provided information via phone calls were successful." 

Methodological challenges in evaluating advocacy include:

• The complexity of factors influencing outcomes and the nature of advocacy that is often " factors influencing outcomes."
• "Theory-based impact evaluations work well in three scenarios: (1) when there is a critical constraint or a bottleneck in the theory of change that is not informed by past evidence; (2) when the intention is to test the efficacy of the intervention (e.g. does increased handwashing lead to reduced disease incidence?); and (3) when there is a need to show measurable change ...."
• Methods that are useful but have their limitation include: case study, process tracing, outcome mapping and qualitative comparative analysis.
• Essential factors include: understanding the nature and purpose of the initiative and the factors influencing it; constructing clear questions; mapping stakeholders and contributors and identifying critical nodes in the overall theory of change - all useful in indicating evaluation methods.
• Having a clear idea of what data to collect determines quantitative and qualitative methods to be used.
• Political economy analysis of contextual factors influencing the impact of advocacy programmes is indicated in examining issues, appropriateness of campaign messages, and factors affecting success.
• "Studies examining policy advocacy programmes and those looking at how information type, channel and provider influence outcomes would contribute to building the evidence base on effective advocacy."

The paper concludes that "evaluating advocacy actions requires a combination of methods, and that any single method is too limited to evaluate the dynamic and multi-level, multi-actor nature of advocacy initiatives."