Most nonprofits focus on creating meaningful change. To accomplish this, they invite supporters to make investments, aka donations to their organization, enabling them to deliver essential programs and services.
To demonstrate the effectiveness of their programs and services, organizations collect data on participants and other benefactors. They use this information to make important decisions, track, course-correct, and report successes.
Common Challenges
Collecting and analyzing data can present challenges for nonprofits, especially those lacking proper training and resources.
Here are three common challenges associated with the process:
Impact measurement: Quantifying the social impact of an organization’s work is not simple, especially when dealing with complex and long-term issues. Impact measurement can require the use of different indicators, methods, and instruments to capture the changes and benefits that programs are attempting to produce for individuals and communities.
Measuring long-term impact requires significant amounts of time and resources. Monitoring program recipients for many months or even years to collect data can be a daunting task—one that most smaller organizations are not equipped to do.
Evaluation quality: Conducting evaluation in a rigorous and ethical way and following the best research practices present challenges.1 The quality of the evaluation depends on various factors such as the design, implementation, analysis, and reporting, as well as the qualifications and experience of the individuals evaluating the findings.2
Evaluation use: Communicating and applying the evaluation findings in a meaningful and actionable way can be arduous. Evaluation use may involve sharing the results with internal and external audiences such as funders, board members, staff, or other organizations, and using the results to improve the nonprofit’s performance, accountability, and learning.3 Each audience may require personalized formats and methods of delivery.
Data Collection Errors
Knowledge and precision are needed to develop research tools. Common errors include:
Population specification error: This occurs when an organization defines its target population incorrectly or incompletely, resulting in wrong or incomplete sets of data.2 This can affect the validity of the data and results.
Sample frame error: This happens when an organization uses a list or source of potential participants that does not match or cover its target population, resulting in a sample that is not representative of the population it actually serves.
Selection error: This arises when an organization uses a biased or inappropriate method to select its sample, such as convenience sampling or voluntary response sampling, and ends up with a sample that is not random or diverse enough. For instance, there may be an inclination to select a sample group simply based on their accessibility and willingness to participate (friends, relatives, and co-workers).
Nonresponse error: This occurs when an organization has a poor response rate or a high dropout rate among participants and ends up with missing or incomplete data. This can affect the completeness and comparability of the data and results.
Measurement error: This happens when an organization has a flaw or mistake in its measurement processes, such as poorly worded questions, unclear instructions, faulty instruments, or human errors, and ends up with inaccurate or inconsistent data.
Compliance Considerations
In addition to design and evaluation issues, organizations that collect data on their program participants must know and comply with strict federal standards. HIPAA requires organizations to protect personal health information, which includes such things as name, date of birth, contact information, Social Security number, and other personal information. Compliance includes mandatory staff training and written policies and procedures for handling this information.
Many states have privacy laws that regulate how organizations collect and use data, e.g., California Consumer Privacy Act (CCPA). Be sure to check your state’s laws.
The possibility of making mistakes while conducting an evaluation should be taken seriously, considering the implications of flawed data. Organizations have a responsibility to use the highest ethical and professional standards when collecting and reporting information.
Errors such as those examined in this article can be avoided with staff training and development or by seeking relevant external assistance.
The Annie E. Casey Foundation’s Kids Count4 offers a wealth of information on this topic and can help users find survey questions, measures, and instruments that contribute to meaningful data-collection activities.
For more information on nonprofit data collection, see our related articles, “Counting What Counts – Nonprofit Data Collection Best Practices” and “Counting What Counts – Impact Measurement.” If you have any questions or need assistance, please reach out to a professional at FORVIS.