SMI Process Evaluation (SMIPE)

Purpose

The Salud Mesoamérica Initiative Process Evaluation (SMIPE) assessed the processes of the initiative from conception through implementation. This study was complementary to the ongoing quantitative evaluation conducted by IHME and was conducted to evaluate SMI’s relevance, effectiveness, and implementation, and to explore the following questions:

1. What components of SMI influenced whether outcomes were achieved or not according to stakeholders?
2. How did SMI contribute to the visualization and prioritization of the poor in the policy-dialogue agenda at the national and regional level?
3. What was the contribution of SMI in the performance of health systems in the region?

SMIPE aimed at uncovering topics to investigate in the upcoming quantitative evaluations, and explain findings from the previous ones. These questions were investigated both at the regional level, and specifically in Mexico, through the experience of the State of Chiapas as a case study at the local level. 

Evaluation Design

SMIPE used an explanatory and exploratory design in order to draw broadly applicable lessons from its successes and clarify the factors behind the failures in some aspects of the initiative. The explanatory investigation aimed to reveal the reasons that SMI interventions had mixed success reaching indicator targets. These data augment and explain existent quantitative results. The exploratory design of SMIPE shed light on the unique aspects of SMI, such as its regional focus, the collaborative nature of its partnerships, and the results-based aid structure underlying it. Given the novelty of SMI in the region, it is important to understand the relevance and effectiveness of its components.

Methods

The qualitative methods employed for the process evaluation were designed in conjunction with the quantitative surveys being conducted by IHME:

• Document review: an exhaustive review of SMI documents was conducted to inform and develop the evaluation. This includes revision of operational plans, proposals, budgets, expenditure reports, cost-benefit analyses, barrier studies, reports, memos, quantitative study results, and other operational documents.
• Key informant interviews: key informants were selected from a pool of SMI decision-makers and programmatic actors to be interviewed, including funders, the IDB, ministry of health officials, local partners, health care workers, and midwives.
• Focus group discussions: users of the health care system, both men and women in areas both with and without large indigenous populations, participated in focus group discussions to explore their interactions with the health care system, as well as their perceptions, behaviors, and knowledge around health, pregnancy, family planning, child care, and other topics related to SMI.
• Partnership surveys: in order to evaluate the dynamic horizontal relationships involved in SMI, a partnership survey was administered to visualize the network of actors involved in SMI. The tool was designed to demonstrate how member organizations are connected, show how resources are leveraged and exchanged, measure the levels of trust in a network, and link outcomes to the process of collaboration.