The Generative AI Learning Penalty: Evidence from Chinese Secondary Education
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning. The data combine monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. We exploit staggered AI adoption in a difference-in-differences design. AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years. The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. The learning losses are concentrated among roughly 80% of AI users whose behavior is consistent with homework outsourcing, as indicated by exceptionally short homework completion time coupled with high homework scores. AI users who maintain similar homework completion time as non-AI users experience small learning losses.
Edit: moving my comment up here
Just in case as it’s formatted a bit weirdly
X-Axis: Homework scores
Y-Axis: Exam scores
I disagree that it’s no different to before AI. These days you can prompt for the whole paper to be written for you.
I definitely hyper-focused by analysis on homework and exams like math, engineering, science. The study itself separates these into different analysis. The use in writing is actually significantly less, I think because of how obvious the use of AI is in any type of creative writing.
AI is good for bull shitting an email at your work. It’s awful for writing of you’re trying to not get caught using AI. You can generate a paper with AI. But, if you want to actually write it like a human, check sources, verify claims, and remove hallucinations. It’s literally just awful and likely more time consuming than just writing a bull shit paper in your own words to start.
Have you ever read an article or paper of any reasonable length written by AI? It’s very obvious and horrible writing that no human would ever write. And the survey was done in China. Natural sounding writing in Chinese is significantly more difficult for AI to generate than in English. Partly due to the dataset bias, but also because the language is just more complex. The “floor” for writing in the Chinese language is much higher from what I know about it an AI writing.