Computer Vision -- CS 565, Department of Computer Science, PU

CS565 Computer Vision
Spring 2021
Nazar Khan

Human beings (and even animals) "look" at the real-world and extract extremely accurate information extremely efficiently. Computers can fail catastrophically at this task! In this course we look into why "Vision" is a difficult problem to solve and we go through successful, mathematically well-founded techniques used to solve the Vision problem.

This course is a useful application of mathematical concepts from Linear Algebra and Calculus. Therefore, the students could do well by brushing up on their Linear Algebra, Calculus and programming skills before taking this class. The techniques learned here can be useful for other areas such as Image Processing, Machine Learning, Artificial Intelligence and Computer Graphics.

CS 565 is a graduate course worth 3 credit hours.

Lectures: Monday and Wednesday, 11:20 a.m. - 12:45 p.m. @ https://meet.google.com/xon-iyqo-zym
Office Hours: Wednesday, 2:00 p.m. - 3:00 p.m. @ https://meet.google.com/njc-gvuy-wtj
Recitations: Friday, 8:30 a.m. - 10:00 a.m @ http://meet.google.com/fqo-spqo-eam
TA: Adeela Islam

Grading:

Assignments	35%
Project	15%
Quizzes	10%
Mid-Term	15%
Final	25%

Prerequisites

Python
Basic Calculus (Differentiation, Partial derivatives, Chain rule)
Linear Algebra (Vectors, Matrices, Dot-product, Orthogonality, Eigenvectors)

Grades

Grading sheet (Accessible only through your PUCIT email account)

Lectures

#	Date	Topics	Slides	Videos	Recitations	Readings	Miscellaneous
1	May 24	Introduction What is Computer Vision? Computer Vision vs. Biological Vision -- The Grand Deception! Applications of Computer Vision Course Details	Introduction to Computer Vision	Video
2	May 31	Background Math Notation Inner Product Euclidean Norm Outer Product Matrix and Vector Calculus Matrices as Linear Operators Eigenvectors Optimisation Constrained Optimisation Taylor Series Approximation Cartesian vs. Image axis	Background Mathematics	Video			Take-home Quiz 1 Due: Friday, June 11
3	June 7	Image Filtering Convolution Properties of convolution Mean Filtering Gaussian Filtering Non-linear filtering cannot be performed via convolution	Image Filtering	Video
4	June 9	Derivative Approximations Gradient magnitude and angle via partial derivatives atan vs. atan2 Derivative approximations via Taylor's formula 1st-derivative using central difference 2nd-derivative using central difference 1st-derivative using forward difference 1st-derivative using backward difference	Derivative Approximations	Video	Friday June 11: Recitation 0 Python Numpy Working with images Video Recitation 0		Take-home Quiz 2 Due: Friday, June 11
5	June 14	Derivative Filtering & Edge Detection Convolution masks for derivative filtering Smoothing before computing derivatives Forward, Backward and Central Difference Derivative of Gaussian Filter Sobel operator Naive Edge Detection The Canny Edge Detector Gradient computation Non-maxima suppression Hysteresis thresholding	Derivative Filtering & Edge Detection	Video
6	June 16	The Structure Tensor Comparing patches -- SSD Quadratic form for SSD Taylor's approximation in SSD Weighted SSD via Gaussian convolution Structure Tensor -- Geometry and Algebra	Structure Tensor	Video	Friday June 18: Recitation 1 OpenCV Image Filtering Edge Detection Video Processing Video Recitation 1		Take-home Quiz 3
7	June 21	Corner Detection Harris Corner Detector Rohr Corner Detector Scale Space and Gaussian Pyramids	Corner Detection	Video
8	June 23	Local Image Descriptors Sum-of-Squared-Differences (SSD) Normalized Cross Correlation (NCC) SIFT Keypoint Detection Descriptor Matching SIFT Descriptors	Local Image Descriptors	Video	Friday June 25: Recitation 2 Contour Detection Corner Detection Histogram Equalization Image Thresholding Video Recitation 2		Take-home Quiz 4
9	June 28	Hough Transform Slope-intercept representation of lines Polar representation of lines Line detection Circle detection Speed-up via gradient information	Hough Transform	Video
10	June 30	Transformations Matrix ≡ Linear Transformation Scaling, Shear, Rotation Translation is not linear in ℝ^2 Homogenous Coordinates make translation linear in ℙ^2 2D Affine Transformation (Scaling, Shear, Rotation, Translation) 6 degrees of freedom 2D Projective Transformation (Homography) 8 degrees of freedom	2D Spatial Transformations	Video	Friday July 2: Recitation 3 SIFT and SURF in OpenCV Hough lines in OpenCV Hough circles in OpenCV Video Recitation 3		Take-home Quiz 5 Assignment 1 Assignment 2 Assigned: Wed. June 30 Due: Thurs. July 8
11	July 5	Estimating and Applying Transformations Recovering best affine transformation from correspondences Affine Image Warping Recovering best projective transformation from correspondences -- Direct Linear Transform (DLT) Projective Image Warping	Estimating Transformations	Video
12	July 7	Part 1: Image Warping Problems with naive warping Inverse transformation Bilinear Interpolation Part 2: Robust Estimation Outliers Least-squares models are sensitive to outliers Robust Estimation via RANSAC Line Fitting Homography Estimation	Part 1: Image Warping Part 2: RANSAC	Part 1 Video Part 2 Video	Friday July 9: Recitation 4 Affine Transformation using opencv Estimation of Affine transformation Projective transformation (Homography) using opencv Applications of homography Object tracking in videos Image stitching Video 1 Video 2 Recitation 4		Take-home Quiz 6
	July 12 - July 30	Break
13	August 2	Optic Flow -- Local I Gray value constancy Linearized Optic Flow Constraint (OFC) Aperture Problem Normal Flow Local Method of Lucas & Kanade	Optic Flow -- Local I	Video
14	August 4	Optic Flow -- Local II Local Method of Lucas & Kanade Flow Classification via Structure Tensor Flow Visualization		Video	Friday August 6: Recitation 5 Optic Flow in OpenCV Motion Estimation Tracking Video Recitation 5		Take-home Quiz 7
	August 12	Mid-Term Exam
15	August 16	Optic Flow -- Global Global Method of Horn & Schunck Data Term Smoothness Term Regularization Parameter Functions versus Functionals Calculus of Variations Euler-Lagrange Equations Fixed-point Iterations	Optic Flow -- Global	Video
	August 18	Muharram Holiday
16	August 23	Camera Geometry Camera Obscura Pinhole Camera Model Camera Matrix Intrinsic Parameters Projection Extrinsic Parameters	Camera Geometry	Video
17	August 25	Camera Anatomy Camera Center Why do parallel lines meet in images? Points at infinity Vanishing points What do the columns of P tell us? What do the rows of P tell us?	Camera Anatomy	Video	Friday August 27: Recitation 6 Camera Calibration in OpenCV Video Recitation 6		Assignment 3 Assigned: Wed. August 25 Due: Wed. September 1 Assignment 4 Assigned: Mon. August 30 Due: Thurs. September 2
18	September 1	Camera Calibration Camera matrix factorization Intrinsic matrix K Extrinsic rotation matrix R Extrinsic translation vector t Calibration via checkerboard images Intrinsic Checkerboard to image plane homographies Estimation of K via Cholsky decomposition Estimation of extrinsic parameters	Camera Calibration	Video
19	September 1	Epipolar Geometry Background Math Cross-product Points and Lines in Homogenous Coordinates Epipoles Epipolar Lines Epipolar Geometry Epipolar Plane Fundamental Matrix Epipolar Constraint Estimation of Fundamental Matrix Enforcement of Rank-2 Constraint	Epipolar Geometry	Video	Friday August 27: Recitation 7 Track and replace object in video Pose estimation and augmented reality Video Recitation 7
	September 4 - September 11	COVID Break
20	September 13	Stereo Reconstruction Projective Ambiguity Stereo Reconstruction via Fundamental Matrix Stereo Correspondence Triangulation	Stereo Reconstruction	Video
21	September 15	Deep Learning -- I Deep Learning is unavoidable Machine Learning Deep Learning The Artificial Neuron Neural Networks Loss Functions	Deep Learning	Video	Friday September 17: Recitation 8 Applications of Deep Learning in Computer Vision Video Recitation 8
22	September 20	Deep Learning -- II Multiclass Classification Activation Functions Regularization		Video
23	September 22	Convolutional Neural Network (CNN) Convolutional Filters Subsampling Fully Connected Layers 1x1 Convolutions Depthwise separable Convolutions Transposed Convolutions Unpooling Fully Convolutional Networks Semantic Segmentation Residual Blocks	Convolutional Neural Network (CNN)	Video	Friday September 24: Recitation 9 Detection, Classification and Segmentation via Mask R-CNN Ground-truth annotations Training on custom data Video Recitation 9
24	September 27	Object Detection, Classification and Segmentation via Mask R-CNN Feature Pyramid Network (FPN) Region Proposal Network (RPN) RoIAlign Classification Bounding Box Regression Instance Segmentation	Object Detection, Classification and Segmentation via Mask R-CNN	Video
25	September 29	Conclusion What was covered? What were the general principles? What mathematical ideas were important? What was not covered?	Conclusion	Video
	October 8	Final Exam