Can Big Data Be Used for Evaluation? A UN Women Feasibility Study

This feasibility study from UN Women was compiled to support "the evaluation community’s better understanding of how to make use of big data sources by providing an analysis of the pros and cons of some potential data sources, initial step-by-step protocols for their use, and recommendations based on lessons learned about using big data sources in a meaningful way for evaluation."

The study investigates the use of Twitter, Facebook, and radio data to improve the evaluation of gender equality and women’s empowerment initiatives. Using two case studies, Pakistan and Mexico, which, as stated here, present challenges and barriers in access to data sources, the researchers document their process of accessing, analysing, and triangulating big data sources with traditional data. (Other sources discussed were Whatsapp and news data.)

The purpose was to determine how big data could help to:

• "...improve the evaluation of UN Women’s work using additional evidence streams from big data, mainly focusing on social networking/media and news platforms.
• Apply the new United Nations Development Group (UNDG) “Principles for Big Data and the Sustainable Development Goals” (SDGs) and the “Risks, Harms and Benefits Assessment Tool” to provide feedback for their use and refinement in regard to gender equality and women’s empowerment issues.
• Support understanding of how UN Women and its partners might effectively use big data to support future evaluation efforts on WPP and in other thematic areas."

For this social media analysis, Twitter data was selected for Mexico and Facebook data for Pakistan with the intention of:

• Testing a measurement model to select the best big data indicators " (e.g., keywords and/or hashtags) that correlate with well-established indicators from traditional data, meaning they are measuring the same construct."
• Describing "the universe of tweets [posts] relevant for UN Women campaigns...in terms of demographics and language and identify biases."
• Analysing results (engagement, topics, sentiment) "to derive insights about the contribution of UN Women interventions, disaggregating by gender."
• Triangulating with traditional methods.

Measuring relevancy of tweets/posts included counting engagement through likes, shares, and comments; recording demographics of users; and classifying the sentiments of the content. Comparison of engagement was measured regionally and longitudinal engagement was mapped, including demographics and sentiment. Pages 22 and 24 describe in detail the construction and analysis methodology.

Findings include, for example:

• Twitter
• "Twitter appears more appropriate for evaluating UN Women’s interventions aimed at fostering political participation and attitudes towards gender equality.
• Social network analysis can help to reveal the online network of users and their degree of influence within their network. ...
• Facebook
• "Private or semi-private discussions may pose ethical issues..."
• Because of self-selection bias on political and social issue discussions, "[o]ther sources hold more promise, such as radio data, responses to SMS campaigns, and responses to newspaper articles online."
• Radio data
• "Radio programmes can be designed to gather useful information for evaluation through voice or SMS... (e.g., documenting community conversations)." But it must be carefully constructed and coordinated to gather and analyse a large volume.

Recommendations for using big in evaluation include the following:

1. Understand the place of the big data source (i.e., Facebook or Twitter) in a country before considering it as an evaluation data source - for example, it may only include majority language users (e.g., Urdu or Punjabi in Pakistan), excluding users of other languages. It is recommended to discuss the results through key informant interviews (KII)or focus group discussions(FGD)with people who provided the data.
2. Incorporate big data in evaluation design from the outset so that, for example, traditional data can be collected to compensate for coverage problems, and so that control and intervention groups can be closely monitored. Also consider if new tools need to be developed for collection and analysis and what best ethical practices need to be incorporated.
3. "Big data should precede traditional data when sequencing and evaluating..." because all groups can then be provided with representation, big data and traditional methods can be aligned for triangulation, and scoping can effectively include new trends and surprising case studies.
4. Shape big data to enhance value - "[f]or example, launching social media campaigns, using hashtags and posts that trigger meaningful reactions and engagement from diverse groups.... Set up interactive radio shows that invite participation from audiences through SMS and social media, while gathering individual demographics....Big data platforms can be both the intervention tool (e.g., hashtag campaigns) and the source of evaluation data. This does not present any problem, but studies using big data analysis need to make this distinction explicit, and adjust methods and conclusions according to the objective of the evaluation."