Academic Return on Investment
By John Gatta, Ph.D.
CEO, ECRA Group
Associate Professor, Northwestern University
School leaders make decisions every day about how to best allocate community resources to ensure all students receive a high-quality education. Documenting the impact of programmatic investments on student outcomes is the essence of Academic Return on Investment (AROI). AROI aims to answer a simple, but often elusive question: Is a district investment having a positive impact on student outcomes above and beyond what likely would have happened without the investment?
AROI quantifies the relationship between cost and quality, and is motivated by the strategic intentions of school boards and administrators to increase quality and optimize resource allocation. This is particularly important as school districts explore efficiencies in a post ESSER funding environment by recognizing that increasing educational quality requires the alignment of resources toward what is proven to accelerate student success. Ultimately, AROI analysis provides scientific evidence to support tough decisions related to which programs to keep and which programs to cut.
Asking the Right Question
Investments come in many forms such as programs, interventions or other district initiatives, but AROI discussions move beyond investment of resources. AROI analysis requires districts to articulate and document the student outcomes that an investment is designed to impact.
Consider an investment in a reading support program designed to accelerate reading growth. The question “How did students in the program grow?” is not the right question to evaluate AROI. The more meaningful AROI question is “How did students in the program grow compared to how they would have grown without the program?” Only by quantifying the effect of the reading support program on student growth beyond what would have likely occurred without the program can one determine whether the academic return justifies the cost.
The Process of AROI
Specifically, how does one determine if students benefitted from an investment more than they would have without the investment? School districts have struggled to answer this question because of the difficulty in establishing meaningful comparisons or control groups against which to benchmark results.
Rigorously documenting AROI requires analytic models to generate a unique comparison, or control, for each student based on his/her individual past performance. AROI analysis can then be used to examine a student’s actual achievement under the program against his/her projected achievement under the typical effect of the district. Such information can then be linked to district resource allocations so that informed decisions can be made regarding programs, student impact and other investments. Effectively implementing AROI requires that school leaders untangle the notion of investments from the return that each investment provides in terms of student growth.
AROI is a 4-step Process
AROI uses a disciplined method of inquiry comprised of four simple steps. The process is illustrated using a sample reading program.
1. Invest The first step in performing an AROI analysis is to document the investment being made, including the student outcomes the investment is intended to impact. For example, in the sample school district, the district recently invested $1.5 million into a reading program designed to accelerate the rates of reading growth for struggling readers. The program includes small group instruction, one-on-one tutoring and technology-based resources.
2. Question The second step is to articulate the question that defines the AROI analysis. The question that the district is trying to answer in this case is, “How did students in the program grow in reading compared to how they would have grown without the program?” This is the crucial question to ask as it recognizes that there is a degree of reading growth that will have existed without the investment.
3. Evaluate The third step is to analyze student and program data in a manner that answers the AROI question defined in step 2. In this example, create a personalized growth projection on Indicator of Academic Readiness reading for each student assuming the program did not exist. Then, compare observed growth for each student with the program to projected growth without the program.
Since the district has determined the criteria for success, in our example of reading scores, then the district will be able to answer the question through a projected versus actual analysis. This projection can be based on historical data, such as the student’s past performance on reading assessments, or on other factors that may impact reading growth, such as the student’s socioeconomic status or English language proficiency. Once the district has created these projections, they can then compare the observed growth for each student within the program to the projected growth without the program. The difference between projected and actual results is most commonly reported as a standardized effect size.
4. Act The final step is to validate program effectiveness, use results to improve the program or consider reallocating resources to more effective programming. This is done by linking the investment data defined in step one and evaluation results from step three. Once the district has evaluated the program, leaders can then act on the information they have gathered. If the program is found to be effective, the district can validate its effectiveness and use the results to improve it. This may include expanding the program to serve more students, increasing funding for the program or providing additional resources. On the other hand, if the program is not effective, the district should consider reallocating resources to more effective programming.
The process of AROI is not only beneficial for the district but also for the students, parents and community members. By using this process, the district can ensure that they are equitably providing the best possible education for their students. Additionally, parents and community members can have confidence that their tax dollars are being used effectively.
A New Way of Doing Business
It is important for school districts to regularly assess and evaluate programs to ensure that they are providing the best possible education for their students. By using the process of AROI, school districts can make data-driven decisions and ensure that they are providing effective programs for their students. It is also important for the district to communicate the results of their evaluations to the community and stakeholders, to ensure transparency and build trust with the community.
You can imagine applying this framework to every program every year within the district by listing each program, the number of students each program serves, the financial investment for each program and the AROI of those programs. Over time, you will improve programs through the continuous quality improvement process, and you will demonstrate to the board and the community that you are a good steward of their resources.
In summary, the process of AROI is a systematic approach to evaluating educational programs and making data-driven decisions. By using this process, school districts can ensure that they are providing effective programs for their students and using resources in the most efficient way possible. It is an ongoing process that requires regular assessment, evaluation and communication to the community.