Staff in administration and data collection may participate, through collection and compilation of information or data. It’s a good idea if someone specific is put in charge of collecting data. Clients/users/community can often be helpful in determining end user impact. You can solicit input through low-cost methods such as interviews, surveys, etc. When appropriate, provide assurances of privacy when delicate information is shared. Be mindful of cultural sensitivities and potential cultural barriers when seeking to collect data.
Meaningful Evaluations: Not Just Data
How can you make your evaluation useful?
Why à What
The reasons why we evaluate ourselves shape what we learn from the process.
Why Evaluate?
1. To reveal proof of effectiveness
2. To create a ‘road map’ for the future
3. To learn to improve impact
If we only seek to document effectiveness, we may miss opportunities. Similarly, impact evaluation is a planning tool – but not a plan. It is important to use evaluation for learning purposes. This helps to uncover what really happened as a result of activities, and also helps to avoid staff fears about revealing failures. Failures represent vital learning opportunities. Create a supportive environment for learning from past mistakes.
On Evaluative Learning: “Whether you are a program officer in a foundation or an executive in a non-profit, don’t use evaluation to outsource your own judgment; use it to inform your judgment and then stand behind it… evaluation is a tool in the quest for impact and effectiveness… it is not a “magic 8 ball” that can tell you what to do.”
-Gara La Marche, President and CEO
The Atlantic Philanthropies
February 2008 BBB Charity Effectiveness Symposium
The Key Question to Ask:
What is useful and important to learn?
Basic Evaluation Questions to Ask:
1. Who are we?
- Beware of unexamined assumptions and implicit expectations
- Peter Drucker’s Who-What-What questions:
- Who is our customer?
- What does our customer value?
- What is our business?
- What is our charity’s distinct role in creating positive impact?
- The context in which we are operating – internal and external environment, needs, etc.
2. What are we doing?
- Activities
- Quality of implementation
- Barriers vs. opportunities
- Resources
- Evaluation plans
- Readiness
- Processes
3. What data do we need?
- What do our funders want to know?
- What data do we have, and what else can we get?
- How much data can we manage?
- Benchmarks: meaningful, realistic targets? Industry standards?
- When should we collect data? Periodically or ongoing?
- Whose job is it to collect the necessary data?
- Systems needed?
- Affordability?
4. What’s our ‘story’?
- How do end users see us?
- What’s our ‘area of excellence’?
- What did or didn’t work, and why?
- What have we learned?
- What’s our ‘results story’?
- How can we tell our story effectively?
- “No numbers without stories; no stories without numbers.”
5. Are we making progress against our mission?
- What is the baseline for our measurements?
- What are the core elements required to make programs work?
- Time frames: Assessment of short-term, immediate and direct impacts; and/or assessment of long-term, incremental, sustainable impact over time
6. How much progress is enough?
- What does realistic success look like?
- Modest impact can represent success, especially with growth over time.
- Is the bar for success set high enough?
- Discuss and agree on progress goals with funders in advance
Developing Your Report
Common evaluation terms
Logic Models: Outputs, Outcomes, and Indicators
- What outputs could you measure?
- What indicators might help you measure your impact?
- What outcomes do you desire?
Outputs are the level of goods or services that are produced, as measured by quantitative indicators. They can generally be controlled by the producer, depending on “inputs” or resources (labor, equipment, etc.) and “strategies.” Examples of outputs might be the number of seminars offered or the number of brochures printed.
Indicators are quantifiable results and anecdotal feedback that can be used as proxies for hard-to-measure outcomes. Examples include the number of attendees at a seminar, survey responses from program attendees, or stories and reports about the impact and value of programs. You can have multiple indicators for a complex, or difficult to determine, outcome, but not too many. It’s important to ask what the right indicators are in evaluating a particular outcome. Choosing to assess an indicator that is not actually a good evaluation yardstick for progress against your goal could waste considerable amounts of your time, money, and resources.
Outcomes are generally regarded as the social impact of an organization’s outputs. The term “outcomes” can mean different things, depending on who is using it: it’s a good idea to ask your funder how they view the term. Outcomes can be more difficult or costly to measure, as they depend on factors such as environmental conditions that are not under your control. Short-term direct impact outcomes are easier to measure with user feedback. Measurement becomes complicated when attempting to assess long-term change. Be careful when choosing outcomes, and be sure that your communications systems allow critical information to come forward. An example of a disastrous outcome goal from the past: launching the Challenger space shuttle mission on schedule (a better outcome goal would have been ensuring a safe launch of the Challenger).
How does the organization’s mission fit in? For example, if the mission is to help the children of recent immigrants succeed in school, and the organization runs a program to provide hot breakfasts for 122 students, 5 days a week, you might reasonably ask these types of questions: Are the students served actually the children of recent immigrants? Do the breakfasts help the students succeed in school? Are the provided breakfasts nutritionally sound? Is it possible to serve more students with programs? Are there other, more effective ways to achieve the mission? And so on.
For more information on logic models, visit innonet.org, where you can find a ‘logic model builder,’ and a host of tools and resources of use in developing logic models.
Do We Still have the Right Mission?
Acting on Evaluation Results: Informing the Future
It’s worth asking whether your mission needs tweaking. But a nonprofit’s mission should never be changed without serious consideration. Consult professional advisers (such as lawyers, consultants, accountants, etc.) about potential implications of such a change.
Plan for the Future
Your evaluation should help you check on past performance and plan future actions. In examining the results of your evaluation, first look to see if the organization’s goals have been met. If they were not, ask what might be done to improve performance in the future. If objectives were fully met, ask what new ones might be identified. In future evaluations, build on the old – address concerns from prior reports.
Evaluation Benefits
1. Generation of valuable data to guide strategic planning for the future
2. Generation of valuable data for documentation of results and effectiveness for potential funders
3. Possible value for PR uses
Questions to Ask at the End
1. Did we measure the right things?
2. Did we learn what we wanted to learn?
3. Could the collection of data have been easier/more efficient?
4. What systems can we implement that would allow assessment to be constant and ongoing?
Remember:
We learn by doing
Technology evolves, making new and different data available
The needs of our community change
We can always improve the process “for the next time”
Repeat!
Questions?
Contact us!
NYPAS
New York Philanthropic Advisory Service
Education and Research Foundation of the Better Business Bureau of Metropolitan New York
www.newyork.bbb.org
nypas@newyork.bbb.org
212-358-2815
BBB Charity Effectiveness Online Resource:
www.newyork.bbb.org/charityeffectiveness
We welcome your input on how to improve this informative piece.