Measuring the deeper impacts of your impact investments can taste both sweet and sour. Sweet because it informs us whether our investment is driving positive change, and sour when it comes to the realities of collecting it.
At SK2 Fund, impact measurement has always been a focus of our work with our programs providing pay-it-forward loans to small developing world businesses. A data management system was set up, and we collect various data regularly. The data are self-reported by the entrepreneurs to our local program managers, who can conduct random verification on business visits.
With the introduction of our next-level impact investments, we added a goal of influencing improved treatment of and benefits to employees. To verify this, we feel a need to hear directly from the employees, especially frontline workers like smallholder farmers to know if their lives are actually being improved due to our investments.
A Lean Data-style survey was designed to ask if the interviees received increased salaries and/or benefits in the last year, if they could afford any additional quality of life enhancing amenities, and if their overall quality of life had changed. We asked 5 simple questions to a sample of business owners, office employees, customers, and most importantly, smallholder farmers and frontline workers in Viet Nam and Cambodia.
Sweet
Of the 34 interviewees, 33 answered that their lives have been improved. Many bought new clothes, televisions, refrigerators, washing machines, fixed their bathroom or house, bought health insurance, new tools for farming, invested in the next season, or paid for their children’s studies. This should seem to be a good result to us, so why would we think it is also sour?
Sour
Only 1 person said that her life quality decreased, because her salary stayed the same while living expenses increased every day. Of 34 interviews, we could only meet 17 in person, and only 6 ethnic minority tea pickers who were randomly selected when they came to the factory to sell tea leaves. We want to avoid “self reported” information from the business owners about their employees and suppliers, but in most cases, we still rely on them to select the employees and farmers we can talk to.
Why? First, there is our limited bandwidth to do a large number of in-person interviews. We are not a well-staffed MFI or iNGO with lots of representatives in the field where we work. We considered an electronic survey, but as we found out in the field, even answering the simplest questions is difficult, as the responses often go around in circles, do not directly answer the questions, and sometimes, need interpretation in local and personal contexts.
Secondly, there is the problem of getting unfiltered answers from employees, suppliers, or customers not selected by the entrepreneurs (that is a conflict of interest by itself and might even pose danger to employees if it came back to the entrepreneurs that they did not give the desired answers).
Going Forward
We decided to discontinue our lean data experiment. However, we still think there are other ways to get more detailed information, even if too random to qualify as a sufficient sample size.
1. In due diligence and in follow up impact reporting, all entrepreneurs should be asked the average wages and benefits they pay both their office staff and frontline employees/suppliers. Then, in later reporting, we ask them to update those figures. When we make an in-person site visit, we simply ask a few random employees what they receive in benefits and compare to what the entrepreneur reported.
2. Just as important as lean data will be continuing to collect individual stakeholder impact stories for our investee reporting. So long as we continue making that central to our impact reporting, we will see tangible evidence regarding quality-of-life changes that follows the spirit and intent of capturing lean data.