As school districts deploy more educational software than ever, parents and educators are asking a fundamental question: does this technology actually help students learn? A recent self-evaluation by Khan Academy, aligned with federal guidelines, shows how independent research and structured classroom usage make or break these digital tools. To support students, we must look beyond marketing claims and examine the data behind the software.
What Happened
Educational technology is only as good as its proven impact on student achievement. In its self-evaluation, Khan Academy shared results from four independent randomized controlled trials (RCTs) assessing its math programs. In Tennessee, a study by the National Bureau of Economic Research followed middle school students using the software during daily interventions and found learning gains of 0.06 to 0.08 standard deviations (SD). Similarly, elementary students in Texas showed gains of 0.12 SD in a study published in the Journal of Human Resources.
However, these studies also revealed a key caveat: usage levels matter. In the same Texas study, middle schoolers who averaged only four minutes of practice per week saw no educational benefits. An RCT in Brazil published via the Munich Personal RePEc Archive found no math gains because the schools faced widespread internet outages and device shortages. As we previously reported, consistent, active practice is necessary. Tech tools cannot work if infrastructural issues block access or if usage is too low.
The Bigger Picture
To understand these numbers, we have to look at how researchers measure learning. According to an analysis of education studies on ResearchGate, the median effect size across more than 700 randomized trials of educational programs is 0.10 SD. While a gain of 0.12 SD might sound modest, it represents a meaningful improvement compared to standard educational interventions.
But as researcher Matthew Kraft explains in his analysis of the implementation feasibility trap, school leaders cannot judge edtech by statistical gains alone. They must also look at the return on investment and practical classroom support. For example, an NBER trial in Uttar Pradesh, India showed that schools given access to Khan Academy plus physical facilitators, who spent time troubleshooting and protecting practice hours, saw student scores jump by 0.44 to 0.47 SD.
This need for active human guidance is clear in the shift toward artificial intelligence. While AI tools promise personalized tutoring, they often suffer from "answer leakage," where the AI simply gives students the answer instead of teaching them. A study published in the Proceedings of the Joint Conference on Digital Libraries found that many specialized math AI models struggle to hold back answers. An evaluation reported by ETIH EdTech News showed that unprompted AI models gave students the final solution in 97% of simulated tutoring scenarios. To make these tools productive, research published by the National Bureau of Economic Research states that platforms must intentionally structure AI to enforce mastery-based practice instead of passive homework completion.
What This Means for Families
For parents and educators, these findings offer a reality check. Placing a child in front of educational software without a structured plan will not yield results. To see gains, students must meet specific dosages. For instance, research in the Proceedings of the National Academy of Sciences suggests that mastering roughly two skills per week (or 60 skills a year) translates to a 0.18 SD learning gain.
As schools continue navigating AI rollouts, families must oversee how kids use these programs. If an AI tool is acting as an "answer engine," it bypasses the productive struggle necessary for real comprehension.
What You Can Do
- Target regular weekly milestones. Ensure your child sets consistent, manageable goals, such as mastering two math skills per week, rather than cramming hours of study right before tests.
- Check for answer leakage by sitting with your child and watching how their digital tools respond. If an AI math helper or chatbot immediately outputs the final step-by-step solution without asking probing questions, it is doing the cognitive heavy lifting for them.
- Ask your school about implementation support to find out if your district provides dedicated practice periods, device troubleshooting, and educator training to support the learning software they purchase.