Effect of AI powered virtual agents on learning outcomes

By Winnie Tam, Centre for University and School Partnership, The Chinese University of Hong Kong

Decades of research and technological progress have facilitated more sophisticated natural language-based interactions and adaptive learning through artificial intelligence. A recent meta-analysis by Dai and colleagues examined the impact of AI-powered virtual agents in computer-based simulations on learning performance.

The meta-analysis included experimental studies comparing learning outcomes between groups using computer-based simulations featuring AI-powered versus non-AI virtual agents. Performance-based measures were analysed from 22 studies published mainly between 2018-2021 (90% of studies), yielding 49 effect sizes. Random effects modeling revealed a moderate overall effect (Hedges’ g = 0.43).

Moderation analysis uncovered several insights:

•            Module-based AI technology achieved the highest impact (g = 0.50), followed by natural language processing/machine learning (g = 0.42) and rule-based designs (g = 0.23).

•            Human-like with text agents produced the greatest gains (g = 0.78) relative to fictional characters (e.g. cartoon; g = 0.55) or human avatars (g = 0.35).

•            Interaction modality (text, voice, multi-modal) and agent role (e.g., guidance, feedback) had no significant differential impact.

•            While intervention duration significantly moderated outcomes, the findings were indeterminate as to whether longer and shorter intervention lengths would produce better results.

Though limited in number of studies, this work provides initial empirical support for AI-powered virtual characters to augment learning via computer simulations.

One-on-one high-impact tutoring and early literacy: A pilot study using current school staff

By Cynthia Lake, Johns Hopkins University

The National Student Support Accelerator released a report of a pilot study conducted by Stanford University researchers exploring the impact of a high-impact early literacy tutoring program called “Once” on kindergarten and first-grade students’ literacy outcomes. Once is designed to be implemented in 15-minute daily one-on-one literacy tutoring sessions during the school day, featuring structured lesson plans, weekly live coaching for tutors, and data to monitor student progress and adapt to their ongoing learning needs. The study took place during the 2022-2023 school year in a large, urban school district on the East Coast with over 300 students who were identified as scoring below grade-level benchmarks in early literacy skills in 13 schools. A randomized controlled trial was used to assign eligible students to either Once tutoring provided by non-teaching staff (n=105) or a business-as-usual comparison group (n=199).

The findings indicated that students who received Once tutoring generally made larger gains on end-of-year DIBELS assessments compared to those who did not receive tutoring (effect size =+0.05), with more pronounced results among males and those who performed well below benchmark. However, these improvements were not statistically significant, partially due to the small analytic sample sizes and partially due to factors related to implementation. Treatment students received significantly fewer sessions than expected; instead of the targeted 140 sessions between November and June, students received an average of 42 sessions. The study highlighted that students with additional needs, such as those designated for Special Education or classified as English Learners, received more tutoring sessions on average. Although the results point toward a positive impact of the Once program, the study suggests that larger-scale research is necessary to determine the statistical significance of these findings. It also offered insights into the benefits and challenges of using existing non-teaching staff to deliver high-impact tutoring.

How does school start time impact young students’ outcomes?

By Nathan Storey, Johns Hopkins University

Existing research has suggested that secondary students benefit from later school start times to accommodate adolescent sleep patterns and address chronic sleep loss, as well as other mental and physical health outcomes. Later starts can also positively impact attendance, grades, test scores, and disciplinary incident rates. However, less is known about how start times impact elementary school students. In a 2022 study, Bastian and Fuller provided interesting insight into this issue.

Based on two studies conducted  in North Carolina elementary schools, the researchers explored the impact of school start time on student engagement and achievement. In the first study, they found that moving school an hour earlier was associated with increased absences, particularly for White and rural students, but that the shift did not have consistent impact on academic achievement. In a state-wide analysis, earlier start times did predict significantly higher math scores, particularly for students of color, disadvantaged students, and rural students. In addition, an earlier start time seemed to lead to an initial increase in suspensions, but the suspension rates went back down in subsequent years.

In the second study, the researchers compared students at schools starting at 7:25 or 7:45 am (16 schools), 8:30 am (5 schools), and 9-9:15 am (7 schools). They found that earliest start times were associated with less sleep for 5^th graders (45 minutes less per school night), while students at middle start time (8:30 am) got 26 minutes less sleep. This study also found increases in absences in early-start schools (nearly one additional day per year), but that there was no association with suspensions. In addition, they found no significant impact on either reading or math achievement related to start time changes.

These findings suggest that school districts considering earlier start times for elementary students may see some benefits, but should consider the impact on student absences and explore approaches to maximize participation and engagement.

Teach for America: Analyzing effects across various timeframes

By Carmen Pannone, University of Cagliari (Italy)

Teach for America (TFA) recruits and trains teachers to address hard-to-staff vacancies in public schools, providing support throughout their 2-year commitment as corps members. Studies, including randomized controlled trials and analyses of administrative records, have shown that TFA has consistently yielded positive effects on student outcomes, particularly in math, with more modest results  in other subjects such as English Language Arts (ELA).

A recent study aimed to further explore the impact of Teach for America (TFA) by assessing its short- and long-term effects. The authors conducted this study by analyzing data from Miami-Dade County Public Schools (M-DCPS) spanning from 2010 to 2021. Using a fixed effects model to compare outcomes of TFA teachers with a control group of other teachers in the same schools, the authors found statistically significant positive effects of the TFA program on same-year math scores (ES = +0.09) and ELA scores (ES = +0.03). Additionally, they observed a slight decrease in chronic absenteeism (ES = -0.01, non-significant) and suspensions (ES = -0.01, p < .10), along with a slight increase in grade repetition (ES = +0.02). When considering these non-test factors collectively, students in TFA classrooms for a given year exhibited a non-significant overall increase of about +0.01 standard deviations.

Additional analyses exploring the impacts in the year following being in a Teach For America classroom revealed students exposed to TFA tended to demonstrate improved non-test outcomes in their subsequent year of school (ES = +0.03). On the contrary, the positive impact of TFA on math test scores did not appear to be persistent. Recognizing the high turnover rates of TFA corps members in high-need schools, the authors acknowledge that TFA’s presence in such contexts highlights the challenge of securing talented teachers for these schools. In the absence of viable alternative hiring programs, TFA seems to provide a vetted option with demonstrated benefits for students across various outcomes.