“An Introduction to Information Theory”, Fazlollah M. Reza, Dover 1991. • “
Mathematical Foundations of Information Theory”, A. Khonchin, Dover 1957.
Important ...
EE515 Introduction to Information Theory - II Winter 2008
University of Washington Dept. of Electrical Engineering
Handout 1: Course Overview Prof: Jeff A. Bilmes
Lecture 1, April 4, 2008
This course will be part two of a thorough introduction to information theory. Topics will include information measures, data compression to the entropy limit (source coding) and various source coding algorithms such as Lempel-Ziv and arithmetic coding, Kolmogerov complexity, some coding theory (including ECC and other modern codes) and a bit on codes on graphs, method of types, differential entropy, maximum entropy, rate-distortion theory, alternating minimization, variational inference, and information geometry in general, including computing the capacity of a channel, network information theory, and more on the gaussian channel. Additional topics will include information theory as it is applicable to pattern recognition, natural language processing, computer science and complexity, biological science, evolution, social science, and last but not least communications. Lecturer: Prof. Jeff Bilmes 418 EE/CS Bldg. Class Reader/TA: To be decided. Course web page: http://ssli.ee.washington.edu/courses/ee515 Where/When: We will meet Wed/Fri from 10:30am-12:20pm in EEB-003. First lecture will be April 4th, 2008. No lecture on April 30th. We will schedule two makup lectures once our schedules and the class has settled. Texts: The main text is “Elements of Information Theory” by Thomas Cover and Joy Thomas, Second Edition. 2006. Other texts we will draw from include “A First Course in Information Theory”, by Raymond W. Yeung, 2002 and also a book by D. MacKey entitled “Information Theory, Inference, and Learning Algorithms”, Cambridge U. Press. We will use this text for some of the coding theory (e.g., Arithmetic coding) we will cover. There will be additional handouts made available on the course web page. Prerequisites: Prerequisites: Basic probability and statistics and EE514 (or an equivalent intro to information theory class), or consent of the instructor. Knowledge of Matlab or some numerical-capable programming language. The course is open to students in all UW departments. Homework: There will be a new problem set assignment every 2 to 3 weeks (about 3-5 problem sets for the quarter). You will have approximately 1 to 1.5 weeks to solve the problem set. Problem sets might also include matlab (or programming) exercises, so you will need to have access to matlab, please let me know if you currently do not. The problem sets that are longer will take longer to do, so please do not wait until the night before they are due to start them. Tests: We will have an in-class midterm and an in-class final. The tests might or might not last the entire 2 hours. Grading: Grades will be based on a combination of the final (33.3%) and midterm (33.3%) exams, and on homework (33.3%). Reserve Books in the engineering/math libraries: • “A First Course in Information Theory”, by Raymond W. Yeung, 2002. Lec 1- Page 1
Lec 1- Page 2
• “Elements of information theory”, 1991, Thomas M. Cover, Joy A. Thomas, Q360.C68 • “Information theory and reliable communication”, 1968, Robert G. Gallager. Q360.G3 • “Information theory and statistics”, 1968, Solomon Kullback, Math QA276.K8 • “Information theory : coding theorems for discrete memoryless systems”, 1981, Imre Csisz´ar and J´anos K¨orner. Q360.C75 Other Texts you might find interesting. • “The Theory of Information and Coding: 2nd Edition” Robert McEliece, Cambridge, April 2002 • “An Introduction to Information Theory: Symbols, Signals, and Noise”, John R. Pierce, Dover 1980 • “Information Theory”, Robert Ash, Dover 1965 • “An Introduction to Information Theory”, Fazlollah M. Reza, Dover 1991 • “Mathematical Foundations of Information Theory”, A. Khonchin, Dover 1957. Important Dates: Midterm: TBD Final: TBD
