Hunger Blog

Different Levels of “Knowing” the Impact of Microfinance

Posted by Chris Dunford, Megan Gash and Bobbi Gray Kotara
Freedom from Hunger Research, Evaluation and Monitoring Team, Davis, CA

A discussion has occurred recently in the microfinance blogosphere about the statement Measuring the Impact of Microfinance: Our Perspective, written by leaders of ACCION, FINCA, Grameen Foundation, Opportunity International, Unitus and Women’s World Banking.  Bloggers such as David Roodman, Rich Rosenberg, Sushmita Meta and Tim Ogden have skillfully dissected the shortcomings in the statement’s arguments and tone.

As microfinance practitioners with a strong hand in research, we at Freedom from Hunger can sympathize with both sides of this debate.  We know the importance of scientifically valid research to make the case for sustained allocation of resources to a development intervention.  The philosophy and methods of science have been for centuries the globally accepted convention for knowing what we know with enough confidence to build the modern world.

Freedom from Hunger has led rigorous research-including randomized control trials- and partnered directly and indirectly with research institutions and individuals across the globe to measure the impact of Credit with Education and its variants. The results from these studies have validated some of our claims and failed to validate others.  We are challenged to embrace the revealed weaknesses and to reflect with our practitioner partners and take action collectively to make important improvements in our products and services.

However, as fundraisers (apologies to our ‘development’ colleagues who prefer other labels), we know all too well that the phrase “statistically significant difference” does not move the human spirit or open the pocket book, even pocket books controlled by executives of charitable foundations and development assistance agencies.  Human beings respond to stories about individual, recognizable human beings; this is part of our DNA.  We get excited when the dreary litany of global problems is interrupted by stories of human spirit, determination and progress.

The recent research studies in India and the Philippines seem to conform to best practices of credible impact research, so let’s accept the results for what they are, which are mostly positive and realistic.  Alex Kobishyn wrote recently on the India Development Blog (http://www.indiadevelopmentblog.com/) that the organization at the heart of the J-PAL study, Spandana, was pleased with the results of the study, being as positive as they could expect to measure impact over the time frame of the study.  Alex says Spandana has incorporated the findings internally and funded a second study to gauge medium-term impacts.  If the practitioner at the heart of the study is taking these results in stride, so should the rest of us.

Microfinance loans are a financial instrument to help manage finances, just as are savings accounts and insurance.  A person can potentially take one tool and use it to transform her or his life and that of a whole family, even a community.  We have stories to “prove” the possibility, but other stories (like the financial diaries presented by Portfolios of the Poor) show that such transformation is not the case for all.  Isn’t that to be expected?  To reiterate what the researchers from IPA and J-PAL themselves have said in response to the media (on the Nicholas Kristof blog http://kristof.blogs.nytimes.com/2009/12/28/the-role-of-microfinance) “microcredit seems to have delivered exactly what a successful new financial product is supposed deliver-allowing people to make large purchases that they would not have been able to otherwise.”

Scientific truth is the aggregate of claims left standing after repeated attempts to falsify them.  That’s what scientific research does, falsify claims-or not.  The claim that microfinance very often helps poor entrepreneurs and households to manage cash flow and smooth consumption has not been refuted by research to date.  But these impacts seem quite modest compared to the grander claims of microfinance enthusiasts.  The point of Our Perspective seems to be that stories about individual people and families tell us that these seemingly modest impacts really make a positive, even a major difference in the lives of the poor, at least some of them.  Stories are observations or data points like any other.  But what are their limitations?  We should ask some questions about them.

Are we hearing the whole story of this individual or that family?  What else is happening in her or his life?  An advantage of the financial diaries and similar “story” collection methods is that we are more likely to capture the whole complex story by more in-depth and comprehensive observation and questioning than we do through either client interviews for fundraising communications or quantitative surveys with highly structured questions and response choices.

Is this story typical or atypical?  We would know the answer if the people featured were randomly selected from the microfinance clients rather than cherry-picked.  This is not so hard to do.  Think how interesting it would be to have a fairly large number of stories from a representative sample of clients, households or enterprises.  If we could report that 70 percent of these cases showed a positive trend (however that may be objectively defined), 25 percent showed no meaningful change in their lives, and 5 percent have fallen backward, that would be a powerfully positive indication of the value of being a microfinance client.

But is the microfinance participation what causes the positive change?  Here it gets complicated.  The dreaded RCT (randomized control trial) enters the scene like an unwelcome auditor who takes too much of our time, costs too much money and even disrupts our operational plans.  But that is what is needed (in some form) to really answer this question.  Notice, however, that the evidence can still be in the form of a qualitative story analyzed for positive or negative trends.  The debate is not between quantitative vs. qualitative research.  It is over the value of evidence from studies where research “subjects” have not been randomly assigned to get the “treatment” (micro-loans, for example) or the “control” (no micro-loans).  It is this awkward random assignment of the “treatment” that allows us to know if we can “attribute” improvements in lives to microfinance participation (or whatever is the “treatment”).   Ugh!  What a pain.  But it’s got to be done, and done well, at least a few times in different situations before we can say our claims of positive trend are supported (more accurately, not refuted) according to the globally accepted convention.

Let’s be clear about something very important.  There are various levels of “knowing” the value of microfinance or any other development intervention.  We cannot ignore the personal experience of development practitioners who have decided that microfinance is worth their time and effort.  Not all of us were born thinking positive thoughts about microfinance or trained from professional adolescence to do nothing but microfinance, and therefore we do not have a vested interest in microfinance being the “silver bullet” against poverty.  Many of us have decades of experience in fieldwork with the poor.  Perhaps we’ve seen disheartened agriculture extension agents who could not persuade farmers to adopt new technologies that would assure enough food for the family throughout the year.  Perhaps we’ve noted that mothers bring their kids into primary health care clinics only when they are far-gone with preventable illnesses.  Perhaps we’ve trained budding entrepreneurs for months only to see their businesses flounder in the face of competition.  Perhaps we’ve looked into grateful but discouraged faces and wondered what, if anything, might engage the poor in their own development.  And then perhaps we saw a group of women come together in mutual support to borrow or just save for their own purposes, at first very uncertain and then over time getting more and more self-confident and evincing a palpable sense of hope, opening up to possibilities like flowers opening to the sun.

You will have to forgive us for knowing that something different was happening, something definitely worth investing time, money, even careers in.  We at Freedom from Hunger know very well how difficult it is to capture with the methods of science this transformational experience we can see with our own eyes-not in special cases picked out for fundraising collaterals, but over and over again as we visit and talk with more and more of these participants in microfinance.  This difficulty reflects an inadequacy of method rather than a lack of “evidence” in the broader sense of the word.  We researchers have to try harder and more often to capture this more complex and elusive reality.

Any serious practitioner of microfinance gets out to visit microfinance clients and would-be clients, and even people who can’t be bothered with microfinance, and thereby knows microfinance is no magic bullet.  Very poor women are not leaping up and out of poverty in droves.  We can see this is not happening with frequency.  They are just better off in small but significant ways-practically significant, even if not statistically significant.  We also know that microfinance practitioners are not equal in their methods and, therefore, the results for their clients.  And any journalist or marketer worth the name would get out there and see this, too.  It is just an act of naiveté to buy the hype of microfinance enthusiasts carried away by the relief of having found something, anything that makes a real difference for the poor, and then to get upset when awakened to the more modest but still encouraging facts on the ground.

Everyone, let’s get real here!  The research to date is not telling us to abandon microfinance but to learn from experience and improve products, systems and practice.  The recent studies both validate some claims and reveal weaknesses in others.  The results, and the various public responses to these results, remind us to keep on innovating and learning how to do better microfinance, better research, better marketing communications and better journalism.