Evaluation of Social Mobilization Network (SMNet)

A key finding of this independent assessment of India's Social Mobilization Network (SMNet) is that the organisation was effective in reducing polio vaccine refusal rates, reaching underserved communities with health messages, and helping these communities connect with frontline health workers. Established by the United Nations Children's Fund (UNICEF) in Uttar Pradesh (UP) in 2002 and expanded to Bihar in 2005 - the 2 Indian states with the highest incidence of polio cases - to generate community support for polio immunisation activities, SMNet deployed community mobilisers in areas identified as high risk with the main task to work with resistant communities and to encourage uptake of the oral polio vaccine (OPV) during Supplementary Immunization Activities (SIAs). The World Health Organization (WHO) declared India as no longer polio endemic in February 2012. However, it remains important to sustain OPV coverage for all children under five. To aid this effort, UNICEF engaged Deloitte Touché Tohmatsu India Private Limited (Deloitte) to undertake a detailed evaluation of the SMNet in UP and Bihar. The two main purposes of this evaluation were:

1. To assess if SMNet can be utilised to support other child health interventions like routine immunisation (RI); and
2. To document innovations, best practices, and lessons learned from India's polio programme to inform polio eradication programmes in endemic countries.

The report describes the SMNet structure and strategy in detail. Briefly, the SMNet follows a tiered personnel structure with mobilisers at community, block, district, and sub-regional levels. At the time of this evaluation, SMNet comprised approximately 5,400 members across 46 districts in UP and around 1,350 members across 19 districts in Bihar. SMNet envisaged the use of effective advocacy, behaviour change communication (BCC), and social mobilisation to: ensure continued political, administrative, and social support for immunisation; generate demand for immunisation; and create a broad-based, community-driven movement to increase awareness of the people, fieldworkers, and partners in service delivery. The SMNet approach also focused on interpersonal communication (IPC) to build trust and reduced resistance to vaccination amongst the community. An innovative strategy of the SMNet intervention was to build and maintain supportive local networks and linkages with community, religious, and cultural leaders. These networks of influential opinion leaders helped respond to local fears and misconceptions. In addition, the SMNet intervention has used data to understand the types and causes of resistance to immunisation and has responded to contextual changes. As a result, the strategies of the intervention have evolved over time in order to strengthen the polio eradication effort, build capacity, enhance quality, and promote local ownership.

Keeping in mind the complexity and scale of the SMNet, this evaluation used a mixed-method design that included a literature review and desk research, a secondary analysis, and primary qualitative research. A total of 8 districts (4 in UP and Bihar) were selected from within each of 5 regions. Each district was represented by a SMNet intervention block (i.e., 8 intervention blocks); within each of these blocks, 1 intervention area was sampled. In addition, 1 non-intervention area was selected from each of the 5 regions for primary data collection. The sampling plan therefore included 13 catchment areas at the sub-block level. A total of 317 stakeholders were met for qualitative data collection through in-depth interviews and focus group discussions (FGDs).

Main findings (see the full report for details and supporting data):

• Relevance:
  • The design and interventions of SMNet are aligned with community needs.
  • The SMNet approach has been relevant to achieve the results of the polio eradication programme by reducing resistance to vaccination and reaching the unreached.
  • SMNet has accounted for contextual realities in the programming environment and responded to evolving priorities over the years through relevant strategies.
• Effectiveness:
  • Knowledge, awareness, behaviour, and practices (KABP) related to OPV have improved in SMNet areas.
  • Reduction in refusal rates can be attributed to SMNet awareness raising and mobilisation measures.
  • There has been a net increase in number of children accessing the OPV at vaccination booths in UP.
  • The community has a high level of faith and trust in the Community Mobilization Coordinators (CMCs).
  • SMNet has worked in close collaboration with the public health system.
  • The SMNet interventions' original objective have been achieved to a large extent.
  • The SMNet focus has expanded over time from only polio immunisation to other child health issues through the Polio Plus initiative and increased focus on RI.
  • The intervention has responded to numerous challenges and constraints with innovative solutions.
  • By focusing on high-risk areas and high-risk groups/underserved community - the Muslim community, brick kiln and construction workers, nomadic groups, and slum dwellers - the intervention was designed to address issues of equity. This approach ensured that those who were worst affected were prioritised.
  • The intervention engaged women at decentralised levels as change agents. Preference was given to employing and training female CMCs in order to reach mothers/caregivers. As a secondary result of the intervention, women have been empowered and are active decision makers in household decision making. Through the SMNet, CMCs have access to a platform for growth and learning, and there is both a large degree of social recognition and an increase in mobility.
• Impact:
  • The SMNet intervention has led to a decline in refusal to OPV. Resistant households (XR) were reduced by approximately 61% in UP and by 42% in Bihar between 2007 and 2012.
  • The intervention has led to consistent and significant increase in KABP with the access of communities to frontline workers (FLWs) and CMC visits. (Meta-analysis of KABP studies showed a strong linear relationship (correlation coefficient (r)=0.51 for KA and r=0.90 for BP).
  • The intervention's underserved strategy was developed to cover those in hard-to-reach areas and chronically missed communities due to issues in access, reach, and health seeking behaviours of these communities. Through locally relevant information, education, and communication (IEC) and outreach activities, the intervention sought to reach the unreached.
• Efficiency: Among the findings of a comprehensive Value for Money Analysis (VfM): SMNet has utilised funds in an economical manner and has indicated allocative efficiencies.
• Sustainability:
  • CMCs are accepted by the community and by other stakeholders as FLWs and change agents.
  • SMNet is convergent with the public system and its FLWs.
  • Keeping sustainability concerns in mind, SMNet has started building a strategy on "Polio Plus" interventions. SMNet needs to develop sustainable plans for quality assurance, stringent monitoring/surveillance, and capacity strengthening of its personnel in the context of a broadened scope and ensure clear role definition to complement role of other FLWS (Accredited Social Health Activists (ASHAs) and Anganwadi Workers, or AWWs).
  • Effective strategies of SMNet like engaging local health workers, involving religious and local leaders to address resistance, and utilising area-specific and population-specific approaches can be considered for replication in other contexts.

Central conclusions and learnings from the SMNet programme, as excerpted from the report:

• The SMNet was evidence-based in its strategic planning and programmatic focus, which ensured that its objectives, spatial targeting, and interventions were efficacious and effective towards the main goal of eradicating polio.
• The strategic positioning of the programme along with convergence with the public system has been crucial in determining its impactful functioning and sustainability.
• The SMNet programme structure provided for effective linkages with the public health system at all levels - national, state, district, and blocks, with participatory relationships with Panchayati Raj Institutions (PRIs). These need to be institutionalised. In addition, the SMNet needs to maintain links with civil society.
• The SMNet has remained relevant to the needs of the community and to the changing national context. Many factors may independently, or in interaction with each other, contribute to a need for shifting the focus areas of a complex and large-scale programme like the SMNet such as: national policies, policy context in the state, development priorities, donor priorities, emergencies, etc. The programme has to be flexible and dynamic to respond to such changes. In doing so, the programme needs to ensure that it remains relevant by, for example, ensuring that it serves the need of the most vulnerable. The SMNet responded to the large outbreak of polio in Bihar in 2009 by introducing CMCs and strengthening field-level interventions. Since 2012, the intervention has also placed CMCs at border areas to control importation of the virus.
• Clear statement of goals and objectives and process/outcome/impact indicators of the SMNet have been lacking, and it is important to have these in place, along with rigorous systems and mechanisms to review these regularly against activities and changing policy needs. Relatedly: It is imperative that a programme like SMNet plans for a results monitoring/evaluation framework right at the outset.
• The SMNet has maintained lean but optimal staffing. Lean staffing at administrative/managerial levels with the right expertise is important to ensure quality of services and technical inputs. This contributes to efficiency in human resource management while focusing on building an optimal operational cadre, which is important for a decentralised community-based programme like SMNet.
• A rationalisation of human resources is important in the context of existing personnel at the ground level in the public system. The SMNet in Bihar undertook this in involving existing AWWs instead of introducing CMCs in most areas. This is important to avoid duplication of roles and responsibilities of personnel and maintain high resource efficiencies.
• Balancing expanding scope while maintaining quality and focus of the programme on polio eradication is essential to ensure effectiveness and sustainability. This assumes importance in the context of the Polio-Plus focus and SMNet's increasing involvement in RI and other child health interventions.
• Stringent systems and processes for financial management with rigorous records of budgets, allocations, and expenditures are crucial for a programme the size and scale of SMNet.
• Knowledge management is critical along the life cycle of the programme. SMNet had several innovations that were undertaken at decentralised levels as responses to local contextual realities. In order to learn, make course corrections, and create knowledge about these strategies and operational mechanisms, it is important to have official documentation of the evolution and rationale of change along the project life cycle.

Finally, recommendations based on the evaluation are put forth in four broad areas: institutional and policy level; structural level; operational level; and replication level.

Click here for the 121-page main section report.
Click here for the 78-page annexures.