Machine Learning -- CS 453, Department of Computer Science, PU

CS453 Machine Learning
Spring 2022
Nazar Khan

The ability of biological brains to sense, perceive, analyse and recognise patterns can only be described as stunning. Furthermore, they have the ability to learn from new examples. Mankind's understanding of how biological brains operate exactly is embarrassingly limited. However, there do exist numerous 'practical' techniques that give machines the 'appearance' of being intelligent. This is the domain of statistical pattern recognition and machine learning. Instead of attempting to mimic the complex workings of a biological brain, this course aims at explaining mathematically well-founded techniques for analysing patterns and learning from them.

This course is a mathematically involved introduction into the wonderful world of Machine Learning. It will prepare students for further study/research in the areas of Pattern Recognition, Computer Vision, Data Analysis, Natural Language Processing, Speech Recognition, Machine Translation, Autonomous Driving and other areas attempting to solve Artificial Intelligence (AI) type problems.

CS 453 is an undergraduate course worth 3 credit hours.

Lectures: Tuesday and Thursday, 9:45 a.m. - 11:15 a.m. @ https://meet.google.com/stg-pjvm-vnb
Office Hours: Tuesday, 1:30 p.m. - 2:30 p.m. @ https://meet.google.com/njc-gvuy-wtj

Prerequisites

Python
Basic Calculus (Differentiation, Partial derivatives, Chain rule)
Linear Algebra (Vectors, Matrices, Dot-product, Orthogonality, Eigenvectors)
Basic Probability (Bernoulli, Binomial, Gaussian, Discrete, Continuous)

Books and Other Resources

No single book will be followed as the primary text. Helpful online and offline resources include:

Pattern Recognition and Machine Learning by Christopher Bishop, 2006
Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2017. Available online
Neural Networks and Deep Learning by Michael Nielsen, 2016. Available online
Deep Learning with Python by J. Brownlee
Deep Learning with Python by Francois Chollet
Videos

Administrative

Attendance Sheet (Accessible only through your PUCIT email account)
Grading sheet (Accessible only through your PUCIT email account)

Lectures

#	Date	Topics	Slides	Material	Readings	Assignments
1	March 1	Course Details Introduction to Machine Learning	Introduction to Machine Learning		"Brain: inner workings of the grey matter", Richard Walker "The human brain in numbers: a linearly scaled-up primate brain", Herculano-Houzel. "The Brain Is Both Neurocomputer and Quantum Computer", Stuart R. Hameroff
2	March 3	Curve Fitting Over-fitting vs. Generalization	Curve Fitting
3	March 8	Linear Regression	Linear Regression
4	March 10	Minimization	Linear Regression
5	March 15	Regularized Linear Regression	Linear Regression
	March 17	No class due to sports week
6	March 22	Linear Classification Generalized linear model Orthogonality of w and decision surface	Linear Classification
7	March 24	Linear Discriminant Functions - I Two-class Least Squares Classification -- C(x)=step(w'x)	Linear Classification
	March 28 - April 8	Spring Break
8	April 12	Linear Discriminant Functions -- I Multiclass Least Squares Classification -- C(x)=argmax(W'x) Design and Target Matrices Pseudoinverse	Linear Classification
9	April 14	Linear Discriminant Functions -- I Fisher's Linear Discriminant -- J(w)=w'S_bw / w'S_ww	Fisher's Linear Discriminant		The Matrix Cookbook
10	April 19	Linear Discriminant Functions -- II Perceptron -- y(x)=step(w'φ(x))	Perceptron
11	April 21	Gradient Descent Batch Sequential Stochastic Mini-batch	Gradient Descent
12	April 26	Perceptron Training AND Gate OR Gate Linear Separability XOR Gate	Perceptron Training
13	April 28	Revision
	May2 - May 6	Eid Break
	May 13	Mid-Term Exam
14	May 17	Universal Approximation Theorem
15	May 19	Loss Functions for Machine Learning	Loss Functions
16	May 24	Activation Functions for Machine Learning	Activation Functions
17	May 26	Neural Networks Forward Propagation	Neural Networks
18	May 31	Neural Networks Backward Propagation -- I Chain rule of differentiation	Neural Networks
19	June 2	Neural Networks Backward Propagation -- II Multivariate chain rule of differentiation	Neural Networks
20	June 7	Vanishing Gradients	Vanishing Gradients
21	June 9	Variations of Gradient Descent	Variations of Gradient Descent
22	June 14	Momentum-based Gradient Descent	Momentum-based Gradient Descent
23	June 16	Automatic Differentiation	Automatic Differentiation	Notes