Evaluation of Social Mobilization Network (SM Net): Inception Report

In response to the reduction in funding for the Social Mobilization Network (SM Net), and in order to investigate whether this system could be used to deliver other health interventions, the United Nations Children's Fund (UNICEF) office for Somalia has commissioned an evaluation of the SM Net programme. This report presents the work undertaken during the inception phase of the SM Net evaluation, which primarily focused on developing a comprehensive understanding of the programme through the review of reports and secondary data, as well as the evaluation of similar programmes through an extensive literature review. Together, these were used to define the theory of change (ToC), research questions, indicators, and data collection tools to be employed in evaluating the programme. The report also explains the timeline and action plan for delivery of research outputs. (See Related Summaries, below, for the final evaluation report.)

The report begins by providing some background on polio in Somalia and the SM Net activities. In 2013, Somalia was affected by an outbreak of poliovirus type 1 (WPV1), which resulted in 199 WPV1 cases in South Central zone and Puntland, including five cases in Jariban district of Mudug region in 2014 in the months of May and June. The outbreak was attributed to a large reservoir of children unvaccinated against poliovirus. Approximately one million children had not been immunised between 2009 and 2012 due to restricted access for humanitarian actors, especially in the South Central zone. In addition, the cases in 2014 were closely related to nomadic groups, whose movements often make it difficult to track and implement health practices.

A number of activities were implemented in response to the outbreak, including the Social Mobilization Network (SM Net), which was set up by UNICEF in all three zones of Somalia with the aim of raising awareness on polio and increasing coverage of the polio vaccine in access-compromised areas. It is anticipated that the SM Net system could be used to deliver other campaigns, including other immunisations (such as measles and tetanus), handwashing practices, and infant and young child feeding (IYCF).

Initially, SM Net services began with a limited human resource capacity in each zone, with the focus on advocacy activities to leverage support from the highest level of government, clan, religious and community leaders in all three zones. Mass communication campaigns were delivered through interactive short messaging service (SMS) and BBC Somalia, often in partnership with other ongoing nationwide health campaigns. As the programme has progressed, the forms of communication have become more targeted towards communication campaigns that are most effective at reaching hard-to-access groups. The programme then expanded to include the capacity building of community mobilisers (CMs) in the poliovirus and interpersonal communication (IPC) skills to provide peer-to-peer sensitisation prior to the arrival of vaccination teams in communities. (The report delves into the structure of SM Net in detail.) In 2014, the SM Net initiated the nomadic tracking strategy and completed the process in Puntland with 303 nomadic elders. The network now increasingly focuses on behavioural change and the "missed children".

For this inception report, the literature provided by UNICEF as part of the SM Net programme was reviewed. A desk review of the existing online literature covering reviews of the Global Polio Eradication Initiative (GPEI), routine and supplementary immunisation campaigns, communication for development (C4D), social mobilisation activities, and knowledge, attitudes, and practices (KAP) studies was also undertaken. The information collected provided contextual knowledge on the SM Net evaluation and direction on the formulation of research design, particularly in identifying the most appropriate indicators for programme evaluation. The next section of the report first looks at the role of the GPEI in global polio eradication and how the work of SM Net fits within this structure. It then summarises the findings of two SM Net evaluations, one in Somalia and one in India.

For example, in 2013, the Harvard School of Public Health was commissioned in partnership with UNICEF to deliver high-quality, standardised data on KAP of the SM Net intended groups to be reached in Somalia. Based on the Harvard Study (results of which are shared in the inception report), the SM Net team devised a new workplan to address some of the issues identified across the three zones. For instance, given the high levels of awareness identified in the study, the team decided to continue the communication campaign, but to use more appropriate communication channels and local community networks and actors (such as local government, health workers and religious leaders) to ensure that there is greater awareness of the campaigns prior to the arrival of vaccination teams. Whilst the behaviour of vaccinators and CMs was appreciated, there were several ways in which the relationship in both vaccinators and CMs could be further developed. The SM Net team decided to address this by putting together a set of minimum requirements needed to ensure that team members would be trusted by the community (for example, that they should be residents of the community they serve).

Finally, the report reviews the existing literature on immunisation (e.g., causes of low immunisation rates in Somalia), role of supplementary immunisation activities (SIAs), and C4D approaches (e.g., social mobilisation in Somalia, and communication methods and effective community campaigning in Somalia). The literature review points to some key contextual factors that affect impact as well as key lessons learnt and best practices in delivering impact through SIAs and C4D activities. In short, the review shows that barriers to accessing immunisation services in Somalia are complex, so GPEI's approach to addressing polio eradication is holistic and includes implementing SIAs in addition to routine immunisation activities, as well as C4D approaches. The review also showed the value of understanding the contextual factors that determine access to health services, particularly in hard-to-reach groups such as nomadic pastoralists. This includes ensuring greater community involvement in the process, and a clear understanding as to who are the key influencers (including religious and community leaders) and decision makers in child immunisation, as well as the main communication sources for these groups. There is a need to focus on adopting both mass communication campaigns (particularly radio), as well as house-to-house visits through social mobilisation in order to raise general awareness. Specific communities also need to be targeted with appropriate messaging. Finally, studies have also shown the economic value of combining vaccination efforts with other health care support, be it for livestock or health services for children or adults.

The report includes a selection of most relevant indicators extracted from the literature. These indicators are summarised in Table 6 (pages 26-28) together with the frequency they have been used. They were used as reference points in designing the themes to be covered in this evaluation of SM Net.

A ToC was developed for the evaluation to articulate how interventions by the SM Net programme may lead to its overall goals (see Figure 2 on page 29). The four types of intervention outlined in the ToC involve advocacy activities, community engagement, capacity building of CMs, and mass media messaging. The assumed outputs of these interventions are: enhanced knowledge and awareness; a communication campaign that reaches the intended population; increased engagement and collaboration between partners; and a functioning infrastructure for effective community mobilisation. Together these would result in :improved trust and positive attitude towards oral polio vaccine (OPV) and the frontline staff; increased coverage of OPV and reduced refusal rates; and increased prioritisation of polio vaccine at the regional and national level. These would all impact on the overall goal of the GPEI of eradicating polio and acute flaccid paralysis (AFP) in Somalia. An additional outcome is that the SM Net model is then applied to other health campaigns.

Several factors both internal and external to the programme have the potential to affect the delivery of outputs and outcomes within the defined ToC. The two most important factors in Somalia are related to the political and security situation.

Key research questions have been developed that are categorised according to the standard evaluation criteria of relevance, effectiveness, efficiency, sustainability, and impact (see Table 7 on page 34). The themes are developed to capture lessons learnt, a cross-cutting issue across all the research questions (see Table 12, pages 45-46).

The methodology will use a mixed methods approach, including qualitative and quantitative data, assimilated from both primary and secondary sources. It is described. Data collection tools are described and include, for example in-depth interviews (IDIs) and focus group discussions (FGDs). Certain topics will be covered by all the tools, such as how SM Net could be replicated to deliver on other child survival and development interventions and anticipated challenges of doing so, while others will only be explored with some of the research groups, such as the effectiveness of the SM Net communication model, which will be investigated with frontline workers and community members only.