MERCY CORPS JORDAN – BY MEAGAN DOOLEY

 

How do we measure success?

As the development world is pushed to use limited resources more effectively and efficiently, how we define success and measure impact will become increasingly more important.

This summer I’m working at Mercy Corps Jordan office as a Program Management Intern. The Jordan office is looking to amplify the way it uses both qualitative and quantitative data to talk about program impact. Two of my intern colleagues are working on the quantitative end of this equation. Each of the 17 programs at Mercy Corps Jordan uses a different data collection system, so the M&E team is working to streamline approaches, standardize program indicators, mandate the collection of sex and age disaggregated data, and integrate data analytics into an iterative and adaptive program management process.

Out to dinner with colleagues after a long day of prepping for our upcoming workshop in downtown Amman.

Out to dinner with colleagues after a long day of prepping for our upcoming workshop in downtown Amman.

Another intern and I are working on the qualitative side of this puzzle. The systematic collection of qualitative program data has not been an office priority over the last few years. During the program design process, teams conduct market assessments, hold focus groups with target beneficiaries, and interview government leaders to identify key implementation barriers. This qualitative data is valued and key to the work we do. But after the design phase, the emphasis shifts to quantitative data. Program officers have a plethora of qualitative anecdotes in their head from daily fieldwork, but this data is only documented if a donor requires “success stories” in their quarterly or annual report. As I’ve gone on site visits and talked to program staff over the past few weeks, I’ve found that some teams think success story collection is a chore, simply a means of ticking a donor box, rather than a valuable means of program learning. Others see the value in collecting this data, but struggle to define what connotes “success” in a program that targets violence reduction or behavior change.

Syrian children living in the host community of Mafraq in northern Jordan. I was there visiting our cash assistance program beneficiaries, and the kids wanted to take pictures.

Syrian children living in the host community of Mafraq in northern Jordan. I was there visiting our cash assistance program beneficiaries, and the kids wanted to take pictures.

Qualitative data has the power to transform the way our program teams work.  Qualitative data can be used throughout the program lifecycle, not just in the design phase, to help our teams practice adaptive program management. Under the 2016 Jordan Compact, Jordan agreed to grant 200,000 Syrians work permits in exchange for favorable trade terms with the EU. Mercy Corps’ community strengthening programs work to address root causes of Syrian and Jordanian tensions, which often include competition over scare economic, education, health, and natural resources. A few weeks ago we were up in the north interviewing a mayor about the impact of the work permit legislation on tensions in his community. We assumed the work permit legislation would increase tensions, since unemployed Jordanians might feel they were being overlooked for Syrian workers. However, the mayor said the work permit law has actually decreased tensions in his community. Syrians are working in Jordan with or without work permits. When they work illegally, employers can underpay them, driving wages down for the entire community. When Syrians have permits, they can demand minimum wage, which actually increases wages for all workers. This qualitative data was surprising, but is now reshaping the way we implement economic livelihood programs and advocate with the Jordanian government around work permit issues.

One of our camp programs works to increase access to education for students with physical and developmental disabilities. The team just built a new accommodative equipment customization and repair workshop, and we were there with UNICEF staff for the grand opening. Here you see parallel bars that physical therapists use as a walking aid.

One of our camp programs works to increase access to education for students with physical and developmental disabilities. The team just built a new accommodative equipment customization and repair workshop, and we were there with UNICEF staff for the grand opening. Here you see parallel bars that physical therapists use as a walking aid.

Qualitative data helps us find cross-program synergies and share program learnings throughout the Jordan office. Mercy Corps Jordan has scaled up quickly in response to the Syrian crisis – with a staff of just 30 in 2011, our office now tops 200. With this rapid growth has come a large amount of siloing – most teams have only vague idea of what other programs are doing and how their work fits into the larger Mercy Corps approach in Jordan. Sharing success stories or anecdotal “lessons learned” is one way to try and bridge these gaps and create a culture of sharing and collaboration. One new initiative to address this need is a monthly program coordinators meeting, which provides a space for mid-level staff to come together and share successes and challenges in their individual programs. At our June meeting, one of the child protection program coordinators shared that their team was having trouble with tribal tensions in the camps. They recently discovered that all the Syrian volunteers at their child-friendly spaces in Za’atari Camp were from the same tribe in Syria and because of this, children from other tribes had stopped attending sessions. Though they fired that cohort of volunteers and hired a diverse group of new recruits, the team wondered how they could work to build bridges across different tribal elements in the camps. Coordinators from our conflict mitigation programs jumped in with ideas about community leader training and social cohesion activities. These two teams are now collaborating to adapt the conflict program resources for child and adolescent audiences. Without this venue for sharing, our teams might not have thought to connect with one another to find ways that disparate program activities could complement one another.

Meagan co-facilitating a "Storytelling" workshop with 25 of our program coordinators.

Meagan co-facilitating a “Storytelling” workshop with 25 of our program coordinators.

Qualitative data can be used to connect to the heart and call an audience to action at a time when facts are questioned. Last week, we hosted a CODEL visit for five congressional representatives and their staff. We took them to visit Syrian families living in host communities who have received cash-based assistance for documentation from Mercy Corps. The visit, as with most donor trips, was rushed, and after a 5 minute briefing from program staff, the members only had about 15 minutes to spend with two families. Our program staff is knowledgeable and passionate, and makes a compelling argument about why civil documentation and cash based assistance are key elements to building refugee resiliency in host communities. Yet hearing a Syrian family’s story, even in a rushed 15 minute visit, can have more lasting impact than three hours of analyzing cash effectiveness with clearly demonstrated statistical significance and gender breakdowns. Yet visiting a family alone is not enough – staff must prepare beneficiaries to share their stories succinctly, think through the types of questions visitors might ask, and arrange for visits with families whose cases clearly illustrate the impact of cash and the power of documentation. If we fail to adequately prepare our beneficiaries for these types of visits, Mercy Corps misses out on a chance to achieve real policy impact.

Week 1 of a 3 week workshop around gathering and sharing effective stories to illustrate program impact.

Week 1 of a 3 week workshop around gathering and sharing effective stories to illustrate program impact.

This summer, I’ve gone out with all 17 of our program teams to help them gather, write up and share program success stories as part of a larger organizational effort to embed storytelling and qualitative data in all the work we do. This effort culminates this month in a three-week storytelling workshop a colleague and I are leading with 25 program coordinators. One week into the training, I’m excited about the potential these coordinators have to reshape the way their teams talk and think about the impact of their work. It’s been an interesting shift to eat, sleep and breath qualitative data this summer after spending so much time building up my quantitative skills during my first year at GHD. While I am still a data nerd, I am learning organizations need both qualitative and quantitative data to gain a true snapshot of program impact. These data sets inform one another – quantitative data does exist in a vacuum, but comes alive when it connects to real beneficiary stories that illustrate significant change. As the development field puts greater emphasis on data and analytics, it’s important that we not lose the anecdotal power of qualitative stories. For these stories, if nothing else, remind us why we got into this field in the first place – to help create opportunity where there is crisis, to promote growth where there is need, and to empower and build capacity so our beneficiaries can lead productive, resilient, and dignified lives.

Leave a Reply

Your email address will not be published. Required fields are marked *