Topics include: convex sets and convex functions convex optimization problems least-square problems optimal control problems Lagrangian duality theory. This course provides a comprehensive coverage of the theoretical foundation and numerical algorithms for convex optimization with engineering applications. The final course project is expected to be on a topic at the intersection of information theory and machine learning. The second half will develop information theoretic bounds on the generalization error in statistical learning. The first half of the course will focus on one-shot approaches in multiuser information theory and discuss some applications to machine learning. This course is designed for students with a background in communication systems and information theory, interested in doing research in machine learning. Top ECE1504H Statistical Learning Exclusions: ECE421H, CSC411H1/CSC2515H, ECE1513H This course deals with fundamental limits on communication, including the following topics: entropy, relative entropy and mutual information: entropy rates for stochastic processes differential entropy data compression the Kraft inequality Shannon-Fano codes Huffman codes arithmetic coding channel capacity discrete channels the random coding bound and its converse the capacity of Gaussian channels the sphere-packing bound coloured Gaussian noise and water-filling rate-distortion theory the rate-distortion function multiuser information theory. Topics include algebraic coding theory: finite fields, linear codes, cyclic codes, BCH codes and decoding, Reed-Solomon codes iterative decoding: codes defined on graphs, the sum-product algorithm, low-density parity-check codes, turbo codes. This course provides an introduction to error control techniques, with emphasis on decoding algorithms. Topics include random vectors, random convergence, random processes, specifying random processes, Poisson and Gaussian processes, stationarity, mean square derivatives and integrals, ergodicity, power spectrum, linear systems with stochastic input, mean square estimation, Markov chains, recurrence, absorption, limiting and steady-state distributions, time reversibility, and balance equations. Introduction to the principles and properties of random processes, with applications to communications, control systems, and computer science.

Note: The course catalogues, the SGS Calendar, and ACORN list all graduate courses associated with ECE – please note that not all courses will be offered every year.

Information entropy