# ST Math Teacher Information Strategy: Preliminary Findings Proving Ground Pilot, 2016-2017 School Year In the 2016-17 school year, a group of Proving Ground partner districts and charter management organizations collectively piloted a strategy to regularly provide teachers with information about their students’ average ST Math usage (in minutes) compared to students in other classes in the same school. We found that teachers who received emails indicating their classes had below-average usage increased the time their students spent on the software by approximately 12%. We were unable to detect an impact of the information strategy on test scores. ## Understanding the Challenge ST Math is an instructional software program designed to improve mathematics learning and achievement in elementary and middle school students. In spring 2016, we conducted an analysis of the implementation and impact of ST Math in 13 education agencies in the Proving Ground network. Based on the following pieces of evidence, partners decided to test an information strategy with teachers: Increased usage of ST Math was associated with significant gains in math test scores. Students who increased their weekly usage of ST Math experienced gains of 0.04 standard deviations (SDs) on state accountability tests, which is equivalent to approximately 1.3 months of learning. There was large variation in student usage across schools and classrooms, and teachers were largely responsible for these differences. Teachers who used ST Math at a high level in one year also had high levels of usage in subsequent years. ## Identifying Potential Solutions Proving Ground partner sites hypothesized that teachers may not have time in their busy schedules to regularly check their ST Math dashboards, and therefore may be unaware of their students’ usage patterns. Furthermore, teachers in the same school often do not have sufficient opportunities to use data to collaborate and share best practices. Based on these hypotheses, partner sites wanted to implement a strategy that would provide information directly to teachers to increase their knowledge of student usage patterns, which in turn could impact the time students spent on the software and subsequently improve their math achievement. ## Testing the Information Strategy In the 2016-17 school year, four partner agencies participated in a pilot where selected teachers received regular emails informing them of their students’ average ST Math usage relative to other classes in the same grade range in their school. The email strategy had two goals: First, by providing teachers with easily accessible information on their students’ ST Math usage, the agencies could fill any information gaps caused by time constraints. Second, including a comparison of a teacher’s class to the school average could lead to more discussion about ST Math usage across classrooms and encourage teachers to share practices related to ST Math implementation. ## Methodological Approach We assessed the impact of this strategy by examining three different outcomes: minutes of ST Math usage, ST Math syllabus progress, and end-of-year test scores. We compared the outcomes of students whose teachers received emails with those of students whose teachers did not receive emails. Because we randomly assigned teachers to receive emails or not, the differences we observe in these outcomes can be considered an effect of the information strategy and not representative of other differences between the two groups of teachers or their students. ## Pilot Test Results ![first response to email graph](/sites/g/files/omnuum5021/files/styles/hwp_1_1__720x720_scale/public/provingground/files/screen_shot_2018-08-21_at_5.01.19_pm.png?itok=-3CNSUCR) The pilot started with four sites and decreased to two over the course of the pilot due to attrition. . After the first email went out, we saw no difference in the average ST Math usage of students whose teachers received an email and students whose teachers did not receive an email. However, teachers who received emails indicating their classes’ ST Math usage was lower than the school average increased their average student usage by about 5 minutes per week (a 12% change) relative to teachers with similar usage levels who did not receive an email. This increase mainly occurred in the first two weeks after the email was sent. Over the course of the school year, the partner agencies sent four informational emails to teachers. We found that teachers with below-average usage who received emails experienced an improvement of 5 minutes per week in their students’ ST Math usage over the course of the year, compared to teachers with below-average usage who received no emails. The average amount of progress per hour that students made through the ST Math curriculum was the same between groups. Thus, it appears the emails did not merely add additional time, but helped the students make progress. We also assessed the effect of the email on students’ math test scores. We were unable to detect a difference in math scores between the low-using teachers who received emails and the low-using teachers who did not receive emails. While the increase in ST Math usage was substantive, any corresponding impact on test scores was too small for us to accurately measure with the sample of students for whom we had complete information.## Adjusting Solutions Based on what we learned from this pilot, potential adjustments for the future to increase the impact of this lowcost strategy could include: - Including celebratory language for teachers whose students use the software more than the school average to encourage continued usage of the software. - In each email, identifying three students making slow progress per hour on the ST Math syllabus. The goal of this would be to help teachers provide targeted assistance to students who are struggling with the software. Proving Ground, housed at Harvard University’s Center for Education Policy Research (CEPR), works with school districts to help them identify and test solutions to specific challenges, such as chronic absenteeism, that are obstacles to student achievement. *Proving Ground creates a network of districts and charter management organizations to address each specific challenge, and provides partner districts with data analysis, strategic advice, hands-on assistance, and peer networking opportunities.* See also:- [ Case Studies ](/resource-type/case-studies) - [ Report ](/resource-type/reports) - [ Research Brief ](/resource-type/research-briefs)