Impact assessment

Measuring for success: What’s the right approach to impact assessment?

While robust systems of measurement form an integral part of a company’s business operations this does not always extend to CSR programs. Most companies spend a significant amount of time and effort to quantify, monitor and evaluate their business parameters allowing them to set targets, create benchmarks, track progress, undertake midcourse corrections and effectively cater to stakeholder needs.

When supporting a CSR program, a similar approach to assessment is needed. Companies invest resources such as funds, expertise, employee time, products etc. to generate specific social outcomes and it is important to measure and assess whether these resources have translated into desired outcomes.

There are multiple approaches to measuring impact which prioritize different objectives. As a company, you may be more interested in capturing quantitative data to define program impact whereas your implementation partner may be more oriented towards qualitative impact that provides deeper insights. Choosing the right approach becomes critical in this scenario.

Samhita has engaged with companies to develop CSR strategy, design and implement programs and has also conducted impact assessments across multiple geographies. This article draws upon Samhita’s experience from these projects, insights gained from the knowledge building events we have organized for companies and social organizations and our interactions with the larger base of companies that have a keen interest in CSR.

Efficiency versus effectiveness

A program evaluation should capture both efficiency of a program and its effectiveness.

Efficiency mainly involves an on-going and in-depth review of existing project monitoring data collected by the company or NGO. Targets, timelines and budgetary allocations estimated in the project proposal are compared to actuals. This is sometimes referred to as process evaluation.

Effectiveness essentially refers to the program’s success in achieving social outcomes and impact and is referred to as impact evaluation or assessment. An impact assessment involves designing research studies, which at the simplest level includes baseline-endline studies but can get more complex if exact attribution to the program is desired.

A related point here is the use of appropriate methods of evaluation. Currently, while randomized control trials (RCTs) are the gold standard of evaluations as they show direct attribution to the program, there are other methods such as difference-in-difference that are also rigorous and scientific.

Quantitative versus qualitative measurement

One of the most common dilemmas facing companies and NGOs is whether to measure efficacy using quantitative or qualitative data. The Bain Philanthropy Report for 2013 showed that donors and NGOs have serious differences when defining impact, in terms of what needs to be measured and how, with NGOs tending towards qualitative assessment and donors insisting on quantitative indicators.

However, both methods are required. While quantitative data provides information on the scale or breadth of impact (for e.g. 70% women reported an increase in income after a vocational training program), qualitative information provides insights into the depth of the change (for e.g. women felt more confident to participate in family decisions because of their enhanced earnings). It is important for NGOs and donors to mutually define the impact indicators for a program.

The process of evaluation versus the end product

The process of undertaking an impact evaluation is as important as the end product, i.e. the final report. Articulating a theory of change for any CSR program (how and why inputs for the program will lead to desired impact) helps the company or NGO understand the validity of the assumptions underlying the program before implementation begins.

Evaluation does not end with the assessment and final report. The program needs to be measured regularly and learnings/findings integrated into programs.

Internal assessment versus external assessment

While NGOs and companies can conduct internal assessments to measure change, it is recommended that these are supplemented by external evaluations that are conducted in close collaboration with the implementers. This allows for external validation of the program, minimizes the implementer’s bias and at the same time is integrated with the operational realities of the program. An external evaluation also helps bring in a fresh perspective and benchmark performance of the program with similar initiatives that the evaluators may have assessed.

Busting myths around assessments

Impact assessments are expensive: One of the most commonly cited barriers to impact evaluation is the assumption that this exercise has to be expensive. While there are certain methods that are more resource intensive than others, this does not mean that it is the norm. The use of technology can significantly bring down costs during data collection. Samhita has implemented a real-time monitoring system that helps companies and NGOs track the progress of their projects. Since it is a cloud-based system with data being entered into a web-portal, it delivers detailed reports and reduces overhead costs. Setting up a robust monitoring framework during project preparation and implementation can ensure less time and money spent in collecting data at a later period. An adequate but not excessive sample size will also reduce time and costs significantly.

Investing in impact assessment takes away focus from beneficiaries: Many NGOs and companies typically argue that given limited resources, they would rather invest money on programs directly targeted at beneficiaries than in ‘overheads’ such as evaluation. However, investing in impact assessment is akin to a company investing in marketing and consumer research – these yield results over a long period of time.

Foregoing evaluation may prevent the company from understanding the true outcomes of the program and factors that contributed to its success or failure. Investing in impact assessment will directly contribute to a program’s future design and hence enhance the impact for beneficiaries. Impact assessment often provides pointers to improve efficiency, which may help reduce cost per beneficiary or improve delivery process.

Furthermore, the findings from assessments, when shared in the public domain, become a ‘public good’ that other companies and NGOs can leverage without reinventing the wheel.

Impact assessments can be done at the end of a program: Often, companies and NGOs do not think about impact evaluations while designing or conceptualizing the program but try to insert it at the end. While impact can be measured only when the program has been live for a minimum period, in order to understand and attribute the changes to the program, it is crucial to start thinking about frameworks and metrics at the conception and design stage. This will ensure that the activity has been planned and budgeted and that baseline data can be collected. Most impact assessments become complex, lengthy and expensive because of lack of a robust baseline.

Current trends in CSR with regard to measuring impact are encouraging. We are seeing a definite change in the way companies and donors are approaching their CSR strategies and activities. A recent roundtable hosted by Samhita and JPAL with the CXOs of some of the largest companies in India indicated that with rising instances of professionals and business heads leading CSR programs, there is a shift to identify result-oriented programs and measure impact. It is important that the assessments are seen as collaborative exercises between donors and NGOs to ensure that CSR programs achieve their goals and contribute to social development.

– Poorvaja Prakash & Anushree Parekh,

Samhita Social Ventures