Purposive Sampling in a Qualitative Evidence Synthesis: A Worked Example from a Synthesis on Parental Perceptions of Vaccination Communication

There has been little written on how best to limit the number of included studies in a qualitative evidence synthesis. In a qualitative evidence synthesis, too much data due to a large number of studies can undermine the ability to perform a thorough analysis. Purposive sampling of primary studies for inclusion in the synthesis is one way of achieving a manageable amount of data. This article describes the development and application of a sampling framework for a Cochrane qualitative evidence synthesis on parents' and informal caregivers' views and experiences of communication about routine childhood vaccination (see Related Summaries, below). The researchers suggest that their systematic 3-step approach to sampling may prove useful to other qualitative evidence synthesis authors.

Having been responsible for the Cochrane synthesis, these researchers outline their process and the lessons they learned. They explain that, to be eligible for inclusion in the synthesis, studies had to have: used qualitative methods of data collection and analysis; included parents or informal caregivers as participants; and focused on views and experiences of information about childhood vaccination. Seventy-nine studies met eligibility criteria. The authors believed that this number of studies would be too large to synthesise and considered ways to limit the number of included studies. For example, they concluded that narrowing the scope of the synthesis was not an acceptable option, as it would be difficult to narrow by vaccine type, since the majority of the studies did not state explicitly which vaccines the study explored.

Therefore, they decided to sample from the included studies, combining different sampling strategies to create their own purposive sampling framework. The framework was piloted on a group of 10 studies, and the researchers discussed challenges that arose. The final, 3-step sampling framework was as follows:

1. Sampling for maximum variation: Because the focus was to develop a global understanding of the phenomenon of interest, when it was found that the majority of the studies that met the inclusion criteria took place in high-income settings, the researchers' first step was to sample all studies from low- and middle-income countries (LMICs). This helped ensure a geographic spread and reasonable representation of findings from all income settings, which was also part of the 'Communicate to vaccinate' project in which the synthesis was embedded.
2. Sampling for data richness: The researchers created a simple 1-5 scale for assessing data richness (see Table 2 in the paper). After assessing the data richness of the remaining included studies, they sampled all studies that scored a 4 or higher for data richness.
3. Sampling for study scope / sampling for match of scope: The researchers examined the studies that remained and sampled studies where the study findings and objectives most closely matched synthesis objectives. That is, they looked for studies that had primarily focused on parental perceptions about vaccination information and communication but had not been sampled in the first 2 steps for various reasons. After applying this framework, 38 of the 79 eligible studies were included in Cochrane synthesis article.

During the process of writing the qualitative evidence synthesis, the researchers continued to discuss the strengths and weaknesses of the approach, also presenting the approach to other teams doing qualitative evidence syntheses. These presentations and ensuing discussions facilitated the identification of other strengths and weaknesses of the approach. For example:

• Sampling means possibly missing articles with information about particular populations, settings, or interventions. However, the researchers argue that this approach allowed them to achieve a good balance between the quality of the analysis and the range of settings and populations within the included studies.
• The framework meant that the researchers may have sampled studies with thinner data; the decision to focus on study location in step 1 may have meant sampling studies from LMICs that scored a 1 or 2 for data richness (a potential weakness), but all studies scoring a 4 or higher for data richness were sampled in step 2.
• The researchers adapted the data richness scale to combine steps 2 and 3 of the sampling framework, resulting in a table where the richness of data in an included study is not ranked by the total amount of data but by the amount of data that is relevant to the synthesis objectives.
• It has been suggested that studies with richer data, also described as conceptual clarity, may self-weight in the findings of qualitative evidence syntheses (contribute more data to the synthesis) and be found to be more methodologically sound. Nine studies from LMIC contexts were sampled in step 1; these contributed to, on average, the least number of synthesis findings. Twenty-four studies were sampled on the basis of data richness in step 2; these contributed to a large number of findings. The 5 studies sampled in step 3 because their findings most closely matched the synthesis objectives also contributed to a large number of findings.

The researchers assert that, despite having weaknesses as well as strengths, their sampling framework allowed them to limit the number of studies included in the synthesis in order to make analysis manageable, while still achieving the objectives of the synthesis. Based on their experience, they suggest that the framework could be narrowed to a 2-step approach with the combination of data richness and closeness to the synthesis objectives. Further steps could be added to address synthesis specific objectives such as population or intervention.

Going forward, they call for research into purposive sampling for qualitative evidence synthesis to test the robustness of different sampling frameworks. More research also needs to be undertaken on how best to rate data richness within qualitative primary studies. They include a discussion on working with the GRADE-CERQual approach to develop a sampling framework. The CERQual approach aims to transparently assess and describe how much confidence decision makers and other users can place in individual synthesis findings. The researchers propose studies comparing different sampling approaches for the same synthesis question and looking at whether these result in different findings. Finally, they say, it is important that better guidance is developed for review authors on how to apply different sampling approaches when conducting a qualitative evidence synthesis.

In conclusion: "Transparent and tested approaches to sampling for synthesis of qualitative evidence are important to ensure the reliability and trustworthiness of synthesis findings."