Theory & analysis papers

Critical Learning Periods in Deep Neural Networks arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Achille/Soatto — early training phases matter disproportionately (like critical periods).

Data-dependent Sample Complexity via Lipschitz Augmentation arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Generalization bound reading from Tengyu Ma’s lecture set.

Regularization Matters: Nets vs Induced Kernel arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Tengyu Ma lecture reading — when finite-width nets differ from their NTK.

Identifying and Attacking the Saddle Point Problem arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Dauphin et al. — saddle points as the hard part of nonconvex deep optimization.

Full Spectrum of Deepnet Hessians at Scale arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Papyan — Hessian spectrum under SGD; ties to the final guest lecture.

Exact Solutions to Nonlinear Dynamics of Deep Linear Nets arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Saxe et al. — closed-form learning dynamics; Andrew Saxe lecture core.

Implicit Bias of Gradient Descent on Separable Data arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Characterizes how GD converges to max-margin solutions.

On the Role of Implicit Regularization in Deep Learning arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Srebro lineage — seeking the real inductive bias of gradient methods.

Explaining and Harnessing Adversarial Examples (FGSM) arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Goodfellow et al. — simple linear explanation + fast gradient sign method.

Intriguing Properties of Neural Networks arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Early adversarial examples paper — started the robustness literature.

Robustness May Be at Odds with Accuracy arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Fundamental tradeoff between clean accuracy and adversarial robustness.

Towards Deep Learning Models Resistant to Adversarial Attacks arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Madry et al. PGD adversarial training — robustness baseline.

Deep Learning and the Information Bottleneck Principle arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Tishby & Zaslavsky — IB lens on what deep nets compress and preserve.

Emergence of Invariance and Disentanglement (Achille & Soatto) arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Information-theoretic view of representations — Soatto guest lecture core.

Mean Field View of Two-Layer Neural Networks arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Mean-field landscape analysis — Song Mei lecture lineage.

Wide Neural Networks Evolve as Linear Models (Lee et al.) arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Shows wide nets under GD behave like linear models in function space.

Deep Neural Networks as Gaussian Processes arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

NNGP correspondence — infinitely wide nets as GPs. Pair with Pennington lecture.

Neural Tangent Kernel (Jacot et al.) arxiv.org · Sep 24 arxiv.org · Sep 24

▲1 1 person saved or upvoted this

Defines the NTK; explains lazy-training convergence. Anchor paper for Lecture 6.