**How does GenAI affect grades in college? It can help learn, but it can also substitute for learning ("substitution hypothesis"). And how do students *experience* courses that GenAI can easily ace?** We spent year+ trying to answer these questions, using some of the best data: - Intuition: If GenAI helps learn, grades should increase in courses regardless of what assessment method they use. If GenAI substitutes learning, grades should increase disproportionately in courses with "susceptible" assessments (i.e. take-home essays). - Data: admin data from one of largest U.S. universities - Method: Compare grades and student evals in susceptible vs. non-susceptible courses before and after ChatGPT Findings: - No disproportionate increase in grades in GenAI-susceptible courses - No disproportionate change in student evals Interpretation: - Not clear that GenAI results in learning substitution or reduced student satisfaction - This could be for lots of reasons: students use it to learn, faculty adjust courses, etc. Caveat: - Analyses like these use lots of assumptions and researcher-degrees-of-freedom. So results should always be taken as suggestive rather than conclusive - COVID is a big confounder, and results can be sensitive to how transient or permanent you assume its effects are