This event has passed.

IDEAS/STAT Optimization Seminar: “The Size of Teachers as a Measure of Data Complexity: PAC-Bayes Excess Risk Bounds and Scaling Laws”

Name: IDEAS/STAT Optimization Seminar: “The Size of Teachers as a Measure of Data Complexity: PAC-Bayes Excess Risk Bounds and Scaling Laws”
Start: 2025-03-27T12:00:00-04:00
End: 2025-03-27T13:15:00-04:00
Location: Amy Gutmann Hall, Room 414

March 27 at 12:00 PM - 1:15 PM

Zoom link: https://upenn.zoom.us/j/98220304722

Abstract:

We study the generalization properties of neural networks through the lens of data complexity. Recent work by Buzaglo et al. (2024) shows that random (nearly) interpolating networks generalize, provided there is a small “teacher” network that achieves small excess risk. We give a short single-sample PAC-Bayes proof of this result and an analogous “fast-rate” result for random samples from Gibbs posteriors. The resulting oracle inequality motivates a new notion of data complexity, based on the minimal size of a teacher network required to achieve any given level of excess risk. We show that polynomial data complexity gives rise to power laws connecting risk to the number of training samples, like in empirical neural scaling laws. By comparing the “scaling laws” resulting from our bounds to those observed in empirical studies, we provide evidence for lower bounds on the data complexity of standard benchmarks.

Joint work with G. K. Dziugaite.

Dan Roy

Canada CIFAR AI Chair at the Vector Institute and professor in the Departments of Statistical Sciences, Computer Science, Electrical and Computer Engineering, and Computer and Mathematical Sciences, University of Toronto

Daniel Roy is Canada CIFAR AI Chair, Founding Faculty, and Research Director of the Vector Institute, one of the three nationally funded AI laboratories in Canada, housed in Toronto, with over 700 research staff from over 12 member universities. He is also a Professor in the Department of Statistical Sciences at the University of Toronto, with a cross appointment in Computer Science. Roy’s research spans machine learning, mathematical statistics, and theoretical computer science. His work has received numerous awards, including a best paper award at the 2024 International Conference on Machine Learning. Roy is a recipient of an NSERC Discovery Accelerator, an Ontario Early Researcher Award, and a Google Faculty Research Award. Roy serves as an action editor for the Journal of Machine Learning Research and Transactions of Machine Learning Research, and on senior program committees of the leading ML conferences. Prior to joining Toronto, Roy was a Research Fellow of Emmanuel College and Newton International Fellow of the Royal Society and Royal Academy of Engineering, hosted by the University of Cambridge. Roy completed his doctorate in Computer Science at the Massachusetts Institute of Technology, where his dissertation was awarded the MIT EECS Sprowls Award.

Details

Date:: March 27
Time:: 12:00 PM - 1:15 PM
Event Categories:: Colloquium, Seminar
Event Tags:: AI, CIS, ESE, IDEAS, STAT
Website:: https://jasonaltschuler.github.io/opt-seminar/

Organizer

: IDEAS Center
: View Organizer Website

Venue

: Amy Gutmann Hall, Room 414
: 3333 Chestnut Street
Philadelphia, 19104 United States + Google Map