This course covers practical algorithms and the theory for machine learning from a variety of perspectives. Topics include supervised learning (generative/discriminative learning, parametric/non-parametric learning, deep neural networks, support vector machines), unsupervised learning (clustering, dimensionality reduction, kernel methods) and learning theory (bias/variance tradeoffs, VC theory, large margins). The course will also discuss recent applications of machine learning, such as computer vision, data mining, natural language processing, speech recognition and robotics.

Introduction to Probability and Statistics, Linear Algebra, Algorithms.

1. Supervised Learning

• Linear regression, overfitting, regularization, sparsity

• Bias-variance tradeoff

• Logistic regression

• Naive Bayes

• Decision trees and instance-based learning

• Ensemble methods: boosting and bagging

• Perceptron

• Neural networks and deep learning: DNNs, RNNs, CNNs

• SVM and kernels

• Sample complexity, PAC and VC dimension

2. Unsupervised Learning

• Clustering: k-means, mixture models, expectation maximization, k-medoids, spectral clustering

• Dimensionality reduction: PCA, Kernel PCA, Factor Analysis, ICA, CCA, MDS

• Matrix and tensor factorization

• Topic modeling

• Model selection

Homeworks are due at the beginning of the class on the specified dates. No late homeworks or projects will be accepted.

• Homeworks: 4 HWs (50%)

• Exams: Midterm and Final (50%)

Homework consist of both analytical questions and programming assignments. Programming assignments must be done via Python. Both codes and results of running codes on data must be submitted.

Exams consist of analytical questions from topics covered in the class. Students are allowed to bring a single cheat sheet to the exam.

• [KM] Kevin P. Murphy, Machine Learning: A Probabilistic Perspective. [Required]

• [CB] Christopher Bishop, Pattern recognition and machine learning. [Optional]

• [KF] Daphne Koller and Nir Friedman, Probabilistic Graphical Models. [Optional]

Lecture 1: Introduction to ML, Linear Regression

• Chapter 1 and 2 from KM book.

Lecture 2: Linear Regression, Robust Regression, Overfitting, Regularization

• Chapter 7 from KM book.

Lecture 3: Point Estimation, Maximum Likelihood Estimation, MAP Estimation

• Chapter 3 from KM book.

Lecture 4: Bayesian Learning, Generative Modeling for Classification

• Chapter 3 from KM book.

Lecture 5: Generative Modeling for Classification: Naive Bayes, Gaussian Discriminant Analysis

• Chapters 3 and 4 from KM book.

Lecture 6: Discriminative Modeling for Classification: Logistic Regression, Softmax Regression

• Chapter 8 from KM book.

Lecture 7: Perceptron Algorithm, Functional and Geometric Margins, Support Vector Machines (SVM)

• Chapters 8 and 14 from KM book.

Lecture 8: Support Vector Machines (SVM), Max-Margin Classification, Lagrange Duality, KKT Conditions

• Chapter 14 from KM book.

Lecture 9: Kernels, Kernel SVM, Soft-Margin SVM, SMO Algorithm, Multi-Class SVM

• Chapter 14 from KM book.

Lecture 10: Ensemble Methods, Bagging, Boosting

• Chapter 16 from KM book.

Lecture 11: Neural Networks, Architectures, Activations, Outputs, Forward Propagation

• See piazza for reading materials.

Lecture 12: Feed Forward NNs, Forward and Backward Propagation, Training via Backpropagation Algorithm

• See piazza for reading materials.

Lecture 13: Dimensionality Reduction, PCA, Kernel PCA

• Chapter 12 and 14 from KM book.

Lecture 14: Nonlinear Dimensionality Reduction, Locally Linear Embedding

• See piazza for reading materials.

Lecture 15: Dimensionality Reduction via NNs and Autoencoders, Sparsity, Training Autoencoders

• See piazza for reading materials.

Lecture 16: Convolutional Neural Networks, Architectures, Training, Examples

• See piazza for reading materials.

Lecture 17: Convolutional Neural Networks, Architectures, Training, Examples

• See piazza for reading materials.

Lecture 18: Centroid Clustering via KMeans, Subspace Clustering via KSubspaces

• See piazza for reading materials.

Lecture 19: Similarity Graphs, Graph Laplacian and its Properties, Spectral Clustering

• Chapter 25 from KM book.

Lecture 20: Spectral Clustering, Graph-Cuts

• Chapter 25 and 22 from KM book.

Lecture 21: Latent Variable Models, EM Algorithm, Mixture of Gaussians

• Chapter 11 from KM book.

Probability Review

• Review of Probability and Statistics (by David Blei)
• Another Good Review of Probability Theory (by Arian Maleki)

Linear Algebra Review

• Review of Linear Algebra (by Zico Kolter)
• Another Good Review of Linear Algebra (by Chaouki Abdallah) — Caution: There are some minor typos in the examples.
• Matrix Cookbook (collection of useful operations on matrices, by Petersen and Pedersen)

All students in the course are subject to the Northeastern University's Academic Integrity Policy. Any submitted report/homework/project by a student in this course for academic credit should be the student's own work. Collaborations are only allowed if explicitly permitted. Per CCIS policy, violations of the rules, including cheating, fabrication and plagiarism, will be reported to the Office of Student Conduct and Conflict Resolution (OSCCR). This may result in deferred suspension, suspension, or expulsion from the university.