Information theory studies the fundamental limits of the representation and transmission of information. This course provides an introduction to information theory, studying fundamental concepts such as probability, information, and entropy and examining their applications in the areas of data compression, coding, communications, pattern recognition and probabilistic inference.

This is a public repository for an information theory course that has run at the ANU as a 2nd year undergraduate course in second semester since 2009. From 2014 it was double-badged as a Masters level course that contained extra material and assessment.

I developed the material for this course with Edwin Bonilla (2009–2012) and Aditya Menon (2013-). Marcus Hutter contributed guest lectures on Kolmogorov complexity and algorithmic information theory (2009-).

The material in this course is largely based on David MacKay’s excellent book Information Theory, Inference, and Learning Algorithms, which is freely available to download.

As secondary texts, we use Cover & Thomas’s Elements of Information Theory and Chris Bishop’s Pattern Recognition and Machine Learning.

- Lecture 1: Administration & Overview
- Lecture 2: Introduction & Motivation
- Lecture 3: Probability Theory & Bayes Rule
- Lecture 4: Bayesian Inference
- Lecture 5: Probability Distributions
- Lecture 6: Entropy
- Lecture 7: KL Divergence & Mutual Information
- Lecture 8: Fundamental inequalities
- Lecture 9: Probabilistic inequalities & Applications
- Lecture 10: Ensembles, Typicality & AEP
- Lecture 11: Introduction to compression; Uniform coding
- Lecture 12: The Source Coding Theorem
- Lecture 13: Symbol Codes for Lossless Compression
- Lecture 14: Source Coding Theorem for Symbol Codes
- Lecture 15: Interval and Arithmetic Coding
- Lecture 16: Arithmetic Coding (continued)
- Lecture 17: Lempel-Ziv Coding
- Lecture 18: Noisy Channels
- Lecture 19: The Noisy-Channel Coding Theorem
- Lecture 20: The Noisy-Channel Coding Theorem (continued)
- Lecture 21: Computing Channel Capacity; Review
- Lecture 22: Hamming Codes