Exercise sheet 2

Dr. Torsten Enÿlin Prof. Dr. Erwin Frey Henrik Junklewitz Niels Oppermann

Information Theory and Signal Reconstruction Summer term 2011

Exercise sheet 2 Exercise 1 Derive Bayes' Theorem for probability density functions, P(x|y) =

P(y|x)P(x) , P(y)

(1)

from the corresponding statement for probabilities.

Exercise 2: Coin Toss Consider the following coin toss experiment: • A large number n of coin tosses are performed and the results are stored in a data vector n d(n) = (d1 , . . . , dn ) ∈ {0, 1} , where 0 and 1 represent the possible outcomes head and tail. • Individual tosses are independent from each other. • All tosses are done with the same coin with an unknown bias λ ∈ [0, 1], i.e. P(di |λ) = λdi (1 − λ)1−di .

Assume that a fraction f of the n coin tosses yielded head.

a) Derive the Gaussian approximation of the PDF P(λ|d(n) ) around its maximum. You can use a

saddle point approximation, i.e. identify the maximum and taylor-expand the logarithm around it up to second order.

b) Use this Gaussian approximation to derive an approximation for P(dn+1 |d(n) ). c) Now calculate the exact posterior mean for λ, i.e. P(dn+1 |d

(n)

).

hλiP(λ|d(n) ) , and the exact expression for

Exercise 3 You have locked k persons into a room. Assume that the probability of a person to have his/her birthday is the same for every day of the year. Assume further a constant number of days per year.

a) How high is the probability that the birthday of at least January?

q of these people is on the rst of

b) How high is the probability of at least two persons in the room having their birthday on the same day?

c) For which k is this probability larger than 50%? Exercise 4 You are observing a region of the sky with a photodetector. An average number of photons λ makes it's way from the sky into your detector per time interval.

a) What is the PDF for the photon counts per time interval? b) Calculate the expectation value and the standard deviation for the photon counts per time interval.

c) Use the PDF to calculate the expectation value m denotes the photon counts per time interval.

1/3

D

m! (m−q)!

E (m|λ)

for any q ∈ {0, 1, . . . , m}, where


Exercise 5: Gaussian Distribution


a) Given a one-dimensional Gaussian PDF G(x, σ 2 ) =

x2 1 exp − 2σ 2 (2πσ 2 )1/2

(2)

and an arbitrary number m ∈ R, calculate the following integrals explicitly (do not use Wick's theorem): h1iG(x,σ2 ) (hint: Square the problem and then use polar coordinates.),

2 hxiG(x,σ2 ) , hxiG(x−m,σ2 ) , x G(x−m,σ2 )

b) Given a multi-dimensional Gaussian PDF G(s, D) = p

1 exp − s† D−1 s , 2 |2πD| 1

(3)

calculate the following integrals explicitly (do not use Wick's theorem)" hsi sj sk iG(s,D) ,

hsi sj sk sl iG(s,D)

Exercise 6 You are interested in three numbers, s = (s1 , s2 , s3 ) ∈ R3 . Your measurement device, however, only measures three dierences between the numbers, according to (4) (5) (6)

d1 = s1 − s2 + n1 d2 = s2 − s3 + n2 d3 = s3 − s1 + n3

with some noise vector n ∈ R3 . Assume a Gaussian prior P(s) = G(s, S) for s and a Gaussian PDF for the noise, P(n) = G(n, N ), with Nij = σn2 δij .

a) Assume that the prior is degenerate, i.e. S −1 = 0. Write down the response matrix, try to give the posterior P(s|d), and explain why this is problematic.

b) Now assume that Sij = σs2 δij and σs = σn . Work out the posterior in this case (using a computer

algebra system for the matrix operations is okay; see e.g. http://www.sagemath.org for a free, open-source, python-based one)

2/3


Exercise 7: Functional tting


You have conducted a measurement of a quantity at k positions {xi }i , yielding k data points {(xi , di )}i . Now you want to t some function to these data points. To this end, you write the function as a linear combination of m basis functions {fj (x)}j , i.e. f (x) =

m X

(7)

sj fj (x).

j=1

If, for example, you were to t a second order polynomial, f (x) = s2 x2 + s1 x1 + s0 , you could choose the monomials as basis functions. The tting process now comes down to determining the coecients {sj }j , allowing for some Gaussian and independent measurement error, i.e. di =

m X

(8)

sj fj (xi ) + ni .

j=1

Assume that you don't know anything about the coecients a priori, i.e. S −1 = 0, where Sij = hsi sj iP(s) .

a) Write down the response matrix for this problem. b) For a given set of basis functions, how many data points are at least necessary for the calculation of the posterior mean of the coecients?

c) Now let's make a linear t. Assuming σi2

= n2i P(n) = const, choose two basis functions and

work out the explicit formula for the posterior mean of the two coecients.

Please hand in your answers as far as possible during the lecture on Monday, May 23rd.

3/3