ECE 595 / STAT 598 Machine Learning

Spring 2022
Jan 10, 2022 - Apr 30, 2022

Week 1

• Jan 11, 2022. Lecture 0.
  • Topics: Course overview and mathematics review. (Hand-written Note.)
  • Reading:
    • Python Tutorial.
    • Python for Matrices. (Password is in BrightSpace.)
    • Python for Plotting. (Password is in BrightSpace.)
• Jan 13, 2022. Lecture 1.
  • Topics: Linear regression. (Hand-written Note.)
  • Reading:
    • Linear Algebra Review.
    • Probability Review.
    • Chapter 7.1.0-7.1.4 of SC.

Week 2

• Jan 18, 2022. Lecture 2.
  • Topics: Examples of linear regression. (Hand-written Note.)
  • Reading:
    • Chapter 7.1.5 of SC.
    • Linear Regression Examples.
    • Python Demo of Linear Regression with Outliers
• Jan 20, 2022. Lecture 3.
  • Topics: Ridge and LASSO Regression. (Hand-written Note.)
  • Slides from last year. (PDF)
  • Reading:
    • Chapter 7.4 of SC.
    • Python demo for Ridge and LASSO
    • Ridge Regression:
      • Stanford CS 229 Note on Linear Algebra
      • Lecture Note on Ridge Regression
      • Theobald, C. M. (1974). Generalizations of mean square error applied to ridge regression. Journal of the Royal Statistical Society. Series B (Methodological), 36(1), 103-106.
    • LASSO Regression:
      • ECE / STAT 695 (Lecture 1)
      • Statistical Learning with Sparsity (Chapter 2)
      • Elements of Statistical Learning (Chapter 3.4)

Week 3

• Jan 25, 2022. Lecture 4.
  • Topics: Optimization I. (Hand-written Note)
  • Slides from last year. (PDF)
  • Reading:
    • Tutorial on Optimization.
    • Python demo for the class
    • Python demo for CVX.
    • Unconstrained Optimality Conditions:
      • Nocedal-Wright, Numerical Optimization. (Chapter 2.1)
      • Boyd-Vandenberghe, Convex Optimization. (Chapter 9.1)
    • Convexity
      • Nocedal-Wright, Numerical Optimization. (Chapter 1)
      • Boyd-Vandenberghe, Convex Optimization. (Chapter 2 and 3)
      • CMU, Convex Optimization (Lecture 2 and 4)
      • Stanford CS 229 (Tutorial)
      • UCSD ECE 273 (Tutorial)
    • Constrained Optimization
      • Nocedal-Wright, Numerical Optimization. (Chapter 12.1)
• Jan 27, 2022. Lecture 5.
  • Topics: Optimization II. (Hand-written Note)
  • Slides from last year. (PDF)
  • Reading:
    • Tutorial on Optimization.
    • Python demo for gradient descent
    • Gradient Descent
      • Boyd-Vandenberghe, Convex Optimization. (Chapter 9.2-9.4)
      • Nocedal-Wright, Numerical Optimization. (Chapter 3.1-3.3)
      • Y. Nesterov, “Introductory lectures on convex optimization”, Chapter 2.
    • Stochastic Gradient Descent
      • CMU 10.725 Lecture
      • Kleinberg et al. (2018) “When Does SGD Escape Local Minima”

Week 4

• Feb 1, 2022. Lecture 6.
  • Topics: Linear Separability. (Hand-written Note)
  • Slides from last year. (PDF)
  • Reading:
    • Python demo for constrained optimization
    • Separating Hyperplane:
      • Duda, Hart and Stork’s Pattern Classification, Chapter 5.1 and 5.2.
      • Princeton ORFE-523, Lecture 5 on Separating hyperplane.
      • Cornell ORIE-6300, Lecture 6 on Separating hyperplane
      • Caltech, Lecture Note
• Feb 3, 2022. Snow Day.

Week 5

• Feb 8, 2022. Lecture 8.
  • Topics: Bayesian Classifier (Hand-written note)
  • Slides from last year. (PDF)
  • Reading:
    • Python demo for Bayesian decision rule
    • High Dimensional Gaussian
      • Bishop, Pattern Recognition and Machine Learning, Chapter 2.3
      • Stanford CS 229 Tutorial on Gaussian
    • Bayesian Decision Rule
      • Bishop, Pattern Recognition and Machine Learning, Chapter 4.1
      • Duda, Hart and Stork’s Pattern Classification, Chapter 2.1, 2.2, 2.6
      • UCSD ECE 271A, Lecture 4 and 5
  • Reading: Same as last lecture.
• Feb 10, 2022. Lecture 9.
  • Topics: Classification Error and ROC curves (Hand-written Note)
  • Slides from last year. (PDF)
  • Reading:
    • Python Demo of Bayesian Decision Rule with Gaussian Model
    • Probability of Error:
      • Duda, Hart and Stork’s Pattern Classification, Chapter 2.7, 3.1.
      • Poor, Intro to Signal Estimation and Detection, Chapter 2.
    • ROC Curve
      • ECE645 Note.

Week 6

• Feb 15, 2022. Lecture 10.
  • Topics: Parameter Estimation I. (Hand-written Note)
  • Slides from last year. (PDF)
  • Reading:
    • Duda, Hart and Stork’s Pattern Classification, Chapter 3.2
    • Iowa State EE 527
    • Purdue ECE 645, Lecture 18-20
    • UCSD ECE 271A, Lecture 6
    • Univ. Orleans
• Feb 17, 2022. Lecture 11.
  • Topics: Parameter Estimation II / Logistic Regression I. (Hand-written Note)
  • Slide from last year.
    • Bayesian Prior (PDF)
    • Logistic Regression. (PDF)
  • Reading:
    • Python Demo for Maximum-a-Posteriori
    • Bayesian Parameter Estimation
      • Duda, Hart and Stork’s Pattern Classification, Chapter 3.3-3.5
      • Bishop, Pattern Recognition and Machine Learning, Chapter 2.4
      • M. Jordan (Berkeley)
      • CMU Note
      • A. Kak (Purdue)
    • Logistic Regression (Machine Learning Perspective)
      • Bishop, Pattern Recognition and Machine Learning, Chapter 4.3
      • Hastie-Tibshirani-Friedman’s Elements of Statistical Learning, Chapter 4.4
      • CMU Lecture
      • Stanford Language Processing
    • Logistic Regression
      • Duke Lecture
      • Princeton Lecture

Week 7

• Feb 22, 2022. Lecture 12.
  • Topics: Logistic Regression II. (Hand-written Note)
  • Slide from last year. PDF
  • Reading:
    • Python Demo for Logistic Regression
    • Logistic Regression-related reading of the previous lecture
• Feb 24, 2022. Lecture 13.
  • Topics: Kernel Trick Hand-written Note
  • Slide from last year. PDF
  • Reading list:
    • Python Demo
    • Kernel Method:
      • Learning from Data (Chapter 3.4)
      • CMU 10701 Lecture 4
      • Berkeley CS 194 Lecture 7
      • Oxford C19 Lecture 3
    • Kernel Regression in Computer Vision:
      • Bilateral Filter
      • Takeda and Milanfar, “Kernel regression for image processing and reconstruction”, IEEE Trans. Image Process. (2007)

Week 8

• Mar 1, 2022. Lecture 14.
  • Topics: Intro to Neural Networks (Hand-written Note)
  • Reading:
    • TensorFlow Playground
    • Python demo for Convolutional Neural Networks
    • Michael Nielsen, Neural Networks and Deep Learning
    • Duda, Hart, Stork, Pattern Classification, Chapter 5
    • Bishop, Pattern Recognition and Machine Learning, Chapter 5
    • Stanford CS 231N
    • CMU Machine Learning Course
    • Cornell CS 5740
• Mar 3, 2022. Lecture 15.
  • Topics: Convolutional Structures and Back Propagation (Hand-written Note)
  • Reading:
    • CMU Deep Learning Course

Week 9

• Mar 8, 2022. Lecture 16.
  • Topics: Convolutional Structures II (Hand-written Note)
  • Reading:
    • Python Demo of CNN on MNIST
    • Checkerboard Effect of Transpose Convolution
• Mar 10, 2022. Lecture 17.
  • Topics: Recurrent Networks and Transformers
  • Reading:
    • CMU Deep Learning Course
    • Stanford CS 231N

Week 10

• Spring Vacation

Week 11

• Mar 22, 2022. Lecture 18.
  • Topics: Attention and Transformers
  • Reading:
    • Python demo of self-attention
    • CMU Deep Learning Course - Attention
    • CMU Deep Learning Course - Transformer
    • Yannic Video - Attention is All You Need
• Mar 24, 2022, Lecture 19
  • Topics: Adversarial Attacks (Hand-written Note)
  • Slide from last year. (PDF)
  • Reading:
    • Min-distance Attacks
    • Max Loss Attacks and Regularization Attacks

Week 12

• Mar 29, 2022. Lecture 20.
  • Topics: Probability Inequality (Hand-written Note)
  • Slide from last year. (PDF)
  • Reading:
    • Python Demo of Convergence
    • Abu-Mustafa, Learning from Data, Chapter 2.
    • Martin Wainwright, High Dimensional Statistics, Cambridge University Press 2019. (Chapter 2)
    • Cornell Note
    • CMU Note
    • Stanford Note
• Mar 31, 2022. Lecture 21.
  • Topics: Is Learning Feasible? (Hand-written Note)
  • Slide from last year. (PDF)
  • Reading:
    • Abu-Mustafa, Learning from Data, Chapter 3.1

Week 13

• Apr 5, 2022. Lecture 22.
  • Topics: Probably-Approximately Correct. (Hand-written Note)
  • Slide from last year. (PDF)
• Apr 7, 2022. Lecture 23.
  • Topics: Generalization Bound (Hand-written Note)
  • Slide from last year. (PDF)

Week 14

• Apr 12, 2022. Lecture 24.
  • Topics: Growth Function and VC Dimension. (Hand-written Note)
  • Growth Function. (PDF)
  • VC Dimension. (PDF)
  • Reading:
    • Yasar Abu-Mostafa, Learning from Data, chapter 2.1
    • Mehrya Mohri, Foundations of Machine Learning, Chapter 3.2
• Apr 14, 2022. Lecture 25. (Hand-written Note)
  • Topics: Sample and Model Complexity

Week 15

• Apr 19, 2022. Lecture 26.
  • Topics: Bias and Variance (Hand-written Note)
  • Slide from last year. (PDF)
  • Reading:
    • Python Demo of Bias and Variance
    • Yasar Abu-Mostafa, Learning from Data, chapter 2.2
    • Chris Bishop, Pattern Recognition and Machine Intelligence, chapter 3.2
    • Duda, Hart, and Stork, Pattern Classfication, chapter 9.3
• Apr 21, 2022. Lecture 27.
  • Topics: Overfitting (Hand-written Note)
  • Slide from last year. (PDF)
  • Reading:
    • Python Demo of Overfitting
    • Yasar Abu-Mostafa, Learning from Data, chapter 2.3, 4.1

Week 16

• Apr 26, 2022. Lecture 28.
  • Topics: Regularization & Validation (Hand-written Note)
  • Slide from last year. (PDF)
  • Reading:
    • Python Demo of Regularization
• Apr 28, 2022. Lecture 29.
  • Topics: Validation & Conclusion (Hand-written Note)
  • Slide from last year. (PDF)
  • Reading:
    • Python Demo of Validation
    • Bill Freeman. How to write a good CVPR submission. (PDF)

---