A week after the pandemic forced Minneapolis students to attend classes online, the city school district’s top security chief got an urgent email, its subject line in all caps, alerting him to potential trouble.
Over the next six months, Matlock got nearly 1,300 similar emails from Gaggle, a surveillance company that monitors students’ school-issued Google and Microsoft accounts. Through artificial intelligence and a team of content moderators, Gaggle tracks the online behaviors of millions of students across the U.S. every day. The sheer volume of reports was overwhelming at first, Matlock acknowledged, and many incidents were utterly harmless. About 100 were related to animated pornography and, on one occasion, a member of Gaggle’s remote surveillance team flagged a fictional story that referenced “underwear.” Hundreds of others, however, suggested imminent danger.
In emails and chat messages, students discussed violent impulses, eating disorders, abuse at home, bouts of depression and, as one student put it, “ending my life.” At a moment of heightened social isolation and elevated concern over students’ mental health, references to self-harm stood out, accounting for nearly a third of incident reports over a six-month period. In a document titled “My Educational Autobiography,” students at Roosevelt High School on the south side of Minneapolis discussed bullying, drug overdoses and suicide. “Kill me,” one student wrote in a document titled “goodbye.”
Nearly a year after The 74 submitted public records requests to understand the Minneapolis district’s use of Gaggle during the pandemic, a trove of documents offer an unprecedented look into how one school system deploys a controversial security tool that grew rapidly during COVID-19, but carries significant civil rights and privacy implications.