Slides are based on the book: A. Papoulis and S.U. Pillai,. "Probability, random
variables and stochastic processes",. McGraw Hill (4th edition), 2002.
Fakultät Informatik – Institut für Systemarchitektur – Professur Rechnernetze
Probability Theory Waltenegus Dargie Chair for Computer Networks Slides are based on the book book: A. A Papo Papoulis lis and S S.U. U Pillai Pillai, "Probability, random variables and stochastic processes", McGraw Hill (4th edition), 2002.
Definition • Deals with the study of random phenomena, h which hi h under d repeated t d experiments yield different outcomes that h have certain t i underlying d l i patterns tt about b t them. • When an experiment is performed under these conditions, certain elementary events ξi occur in different but completely uncertain ways. • One can assign nonnegative number, Pi(ξi) as the p probability y of the event ξi in various ways.
2
Definition • Laplace’s Classical Definition – The probability of an event A is defined a-priori p without actual experimentation p (provided all the outcomes are equally likely) y) Number of outcomes favorable to A P( A) = Total number of p possible outcomes
• Relative Frequency Definition nA n→∞ n
P ( A) = lim
where nA is the number of occurrences of A and n is the total number of trials.
3
Probability Sets • The totality of all ξi, known a-priori, constitutes a set Ω, the set of all p outcomes experimental Ω = {ξ 1 , ξ 2 , L , ξ k , L } • For F any subsets b t A, A B, B C, C … off Ω Ω, – If ξ ∈ A then ξ ∈ Ω A ∪ B = { ξ | ξ ∈ A or ξ ∈ B} A ∩ B = { ξ | ξ ∈ A andd ξ ∈ B} A = { ξ | ξ ∉ A} 4
Probability Sets A A
A
B
A∩ B
A∪ B A1 A
A2
Aj
A
Ai
B
A∩ B =φ
A
B
An
Ai ∩ A j = φ , and
UA
i
= Ω.
i =1
5
De-Morgan’s Laws
A
B
A∪ B
A
B
A∪ B
A∪ B = A∩ B ;
A
B
A
B
A∩ B
A∩ B = A∪ B 6
Probability Field • A collection of subsets of a nonempty set Ω forms a field F (i)
Ω∈ F
(ii) If A ∈ F , then A ∈ F (iii) If A ∈ F and B ∈ F , then A ∪ B ∈ F.
• Hence Hence, the following sets, sets all, all belong to F (the term event is used only to members b off F): F) F =
{Ω , A , B , A , B , A ∪ B , A ∩ B , A ∪ B , L }
7
Axioms of Probability 1. Probability is a nonnegative number P ( A) ≥ 0
2 The probability of a whole set is 2. unity P ((Ω Ω) = 1
3 The probability of mutually 3. exclusive sets is the union of the probability b bilit off each h sett If A ∩ B = φ , then
P ( A ∪ B ) = P ( A) + P ( B ). 8
Axioms of Probability • From the axioms it follows that: A ∪ A = Ω , P( A ∪ A ) = P ( Ω ) = 1 .
A∩ A∈φ ,
P( A ∪ A) = P ( A) + P( A) = 1 or P( A) = 1 − P ( A).
A ∩ { φ } = { φ }. P ( A ∪ { φ }) = P ( A ) + P (φ ) . A ∪ {φ } = A , P {φ } = 0 . 9
Conditional Dependencies • In N independent trials, suppose NA, NB, NAB denote d t the th number b off times ti events t A, A B and AB occur respectively. According to th ffrequency iinterpretation the t t ti off probability, b bilit for large N NA NB N AB P ( A) ≈ , P( B) ≈ , P ( AB ) ≈ . N N N
10
Conditional Dependencies • Among the NA occurrences of A, only NAB off them th are also l found f d among the th NB occurrences of B. Thus the ratio N AB N AB / N P ( AB ) = = NB NB / N P( B)
• The above expression is a measure of “the the event A given that B has already occurred”.. Conditional dependencies are occurred expressed using Bayes’ Theorem: P ( AB ) P( A | B) = , P( B)
P( B) ≠ 0.
11
Conditional Dependencies • We have P( A | B) =
P ( AB ) ≥ 0 ≥ 0, P(B) > 0
P ((Ω ΩB) P(B) = = 1, P(B) P(B) A ∩ C = 0.
P (Ω | B ) =
• Given
P( A ∪ C | B) =
P (( A ∪ C ) ∩ B ) P ( AB ∪ CB ) = . P(B) P( B)
AB ∩ AC = φ − − > P ( AB ∪ CB ) = P ( AB ) + P ( CB ). P ( AB ) P (CB ) P( A ∪ C | B) = + = P ( A | B ) + P (C | B ) P( B) P( B) 12
Conditional Dependencies • Properties of conditional probabilities 1 1. B ⊂ A, AB = B, P( A | B) =
2 2.
P ( AB ) P ( B ) = =1 P(B) P(B)
A ⊂ B, AB = A, P( A | B) =
P ( AB ) P ( A ) = > P ( A ). P(B) P(B)
13
Conditional Dependencies 3. One can use the conditional probability to express the probability of a complicated event in terms of “simpler” simpler related events. • Let A1 , A2 , L , An are pair wise disjoint sets and their union is Ω. Thus Ai A j = φ , and n
U
Ai = Ω .
i=1
• Thus B = B ( A1 ∪ A2 ∪ L ∪ An ) = BA1 ∪ BA2 ∪ L ∪ BAn . 14
Conditional Probability • Since P(B) =
Ai ∩ A j = φ ⇒ BAi ∩ BA j = φ , n
∑
i =1
P ( BA i ) =
n
∑
i =1
P ( B | A i ) P ( A i ).
• If A and B are independent independent, then: P ( AB ) P ( A)P ( B ) P(A | B) = = = P ( A ). ) P(B) P(B)
• Generally, y, P ( B | Ai ) P ( Ai ) P ( Ai | B ) = = P(B)
P ( B | Ai ) P ( Ai ) n
∑
i =1
,
P ( B | Ai ) P ( Ai ) 15
Independence • Independence: Events A and B are independent if P ( AB ) = P ( A) P ( B ). )
• This sa also so implies p es tthat at tthe e following o o g are also independent: A, B ; A, B ; A, B
16
Independence • The event of zero probability is independent p of every y other event. If P(A) ( )= 0, P ( AB ) ≤ P ( A) = 0 ⇒ P ( AB ) = 0, AB ⊂ A
• Independent events cannot be mutually exclusive, P ( A ) > 0, P ( B ) > 0
P ( AB ) > 0.
• More generally, a family of events {Ai} are said to be independent, if for every finite sub collection Ai , Ai , L , Ai , we have 1
2
⎛ n P ⎜⎜ I Aik ⎝ k =1
⎞ ⎟⎟ = ⎠
n
n
∏ P( A k =1
ik
) ). 17
Independence • Given a union of n independent events: A = A1 ∪ A2 ∪ A3 ∪ L ∪ A n ,
• Then by De-Morgan’s law and using their independence, the following holds A = A1 A 2 L A n
P ( A ) = P ( A1 A 2 L A n ) =
n
n
∏ P ( A ) = ∏ (1 − P ( A )). i
i =1
i =1
i
n
P ( A ) = 1 − P ( A ) = 1 − ∏ (1 − P ( Ai )) , i =1
18
Independence • Example: Three switches connected in parallel operate independently. Each switch remains closed with probability p. (a) Find the probability of receiving an input signal at the output. (b) Find the probability that switch S1 is open given i that th t an input i t signal i l is i received i d att the th output. s1 Input
s2 s3
Output 19
Independence • To answer (a), let Ai = “Switch Si is closed” and P(Ai) = p, for i = 1, 2, 3. Since the switches operate independently, we have P ( Ai A j ) = P ( Ai ) P ( A j ); P ( A1 A2 A3 ) = P ( A1 ) P ( A2 ) P ( A3 ).
• Let R = “input signal is received at the output” Hence, output”. Hence R = A1 ∪ A2 ∪ A3 . P( R) = P( A1 ∪ A2 ∪ A3 ) = 1 − (1 − p)3 = 3 p − 3 p 2 + p 3. 20
Independence • Alternatively P ( R ) = P ( R | A1 ) P ( A1 ) + P ( R | A1 ) P ( A1 ). P ( R | A1 ) = 1 P ( R | A1 ) = P ( A2 ∪ A3 ) = 2 p − p 2
P ( R ) = p + ( 2 p − p 2 )(1 − p ) = 3 p − 3 p 2 + p 3 ,
21
Independence • Note that the events A1, A2, A3 do not form a partition, since they are not mutually exclusive. Obviously any two or all three switches can be closed (or open) simultaneously Therefore, simultaneously. Therefore P ( A1 ) + P ( A2 ) + P ( A3 ) ≠ 1 .
22
Independence • To answer (b), we apply Bayes theorem P ( R | A1 ) P ( A1 ) ( 2 p − p 2 ) (1 − p ) 2 − 2 p + p2 P ( A1 | R ) = = = . 2 3 2 3 P(R) 3p −3p + p 3p −3p + p
• Beca Because se of the s symmetry mmetr of the s switches, itches we also have P ( A1 | R ) = P ( A2 | R ) = P ( A3 | R ).
23
Bernoulli Trial • Knowledge of the success or failure of an E Event t A in i an n independent i d d t trial t i l is i fundamental for many interesting problems bl in i communication i ti and d computer t networks • Often our interest is to find out the probability that Event A occurs exactly k times, k ≤ n, in n trials (Po(ω)), given the probability of A occurring in a single trial is p 24
Bernoulli Trial P0 (ω) = P({ξi1 ,ξi2 ,L,ξik ,L,ξin }) = P({ξi1 })P({ξi2 })LP({ξik })LP({ξin }) = P( A)P( A)LP( A) P( A)P( A)LP( A) = pk qn−k . 144 42444 3 144 42444 3 k
n−k
• However the k occurrences of A can occur in any particular location inside ω. • Let ω 1 , ω 2 , L , ω N represent all such events in which A occurs exactly k times. times Then: " A occurs exactly k times in n trials" = ω1 ∪ ω2 ∪L∪ ωN . ωi are mutually exclusive and equi-probable 25
Bernoulli Trial • Thus: P(" A occurs exactly k times in n trials" ) N
= ∑ P0 (ωi ) = NP0 (ω ) = Np N k q n−k , i =1
• Where: ⎛n⎞ n( n − 1) L ( n − k + 1) n! N= = = ⎜⎜ ⎟⎟ k! ( n − k )! k! ⎝ k ⎠
• Generally: y
Pn (k ) = P(" A occurs exactly k times in n trials" ) ⎛ n ⎞ k n −k = ⎜⎜ ⎟⎟ p q , k = 0,1,2,L, n, ⎝k ⎠
26
Bernoulli Trial • Similarly, we may be interested to find out th probability the b bilit off att least l t k occurrences in i an n trial, in which case: P(X 0 ∪ X1 ∪ L ∪ X n) =
n
∑
k =0
⎛ n ⎞ k n−k P ( X k ) = ∑ ⎜⎜ ⎟⎟ p q . k =0 ⎝ k ⎠ n
• Note that Bernoulli trial consists of repeated independent and identical experiments each of which has only two outcomes (A or its complement), with P ( A) = p, and P ( A) = q. 27
Bernoulli Trial • Often, it is interesting to determine the most likely value of k for a given n and p • The most probable value of k is the number which maximizes (see the figure below) P (k ). n
Pn (k ) n = 12,
p = 1 / 2.
k
28
Bernoulli Trial • To obtain this value, consider the ratio: Pn ( k − 1) n! p k − 1 q n − k + 1 ( n − k )! k ! k q = = . k n−k Pn ( k ) ( n − k + 1)! ( k − 1)! n! p q n − k +1 p
• Thus Pn ( k ) ≥ Pn ( k − 1), if k (1 − p ) ≤ ( n − k + 1) p or k ≤ ( n + 1) p .Thus Pn ( k ) as a function of k increases until k = ( n + 1) p • If k is an integer, or the largest integer kmax is less than ( n + 1) p , Pn(k) represents the most likely number of successes in n trials trials. 29
Bernoulli’s Theorem • Let A denote an event whose probability of occurrence in i a single i l trial t i l is i p. If k denotes the number of occurrences of A in n independent i d d t trials, t i l then th ⎛⎧ k ⎫⎞ pq P ⎜⎜ ⎨ − p > ε ⎬ ⎟⎟ < . 2 nε ⎭⎠ ⎝⎩ n
• The above expression states that the frequency definition of probability of an event kn and its axiomatic definition ( p) can be made compatible to any degree of accuracy. 30
Bernoulli’s Theorem • Proof n −1
n
n n! n! k n −k k n −k = = p q ( ) p q k P k k ∑ ∑ ∑ n ( n − k )! k! k =0 k =1 k =1 ( n − k )! ( k − 1)! n −1 ( n − 1)! n! i +1 n − i −1 =∑ p q = np ∑ p i q n −1−i i = 0 ( n − 1 − i )! i! i = 0 ( n − i − 1)! i! n −1
= np ( p + q ) n −1 = np .
• Proceeding in a similar manner: n
n
n n! n! k n −k k P ( k ) = k p q = p k q n −k ∑ ∑ ∑ n ( n − k )! ( k − 1)! k =0 k =1 k = 2 ( n − k )! ( k − 2 )! 2
n
n! +∑ p k q n − k = n 2 p 2 + npq pq . k =1 ( n − k )! ( k − 1)! 31
Bernoulli’s Theorem • Note that: k − p >ε n n
∑
k =0
is equivalent to ( k − np ) 2 > n 2ε 2 ,
( k − np ) Pn ( k ) > 2
n
∑ ( k − np ) k =0
2
Pn ( k ) =
n
∑
k =0
n 2 ε 2 Pn ( k ) = n 2 ε 2 .
n
∑k k =0
n
2
Pn ( k ) − 2 np ∑ k Pn ( k ) + n 2 p 2 k =0
= n 2 p 2 + npq − 2 np ⋅ np + n 2 p 2 = npq .
32
Bernoulli’s Theorem • The middle expression in the previous slide can be expressed as: n
∑ ( k − np ) k =0
2
Pn ( k ) = ≥
∑ (εk − np )
k − np ≤ n
2
Pn ( k ) +
∑ (εk − np )
k − np > n
2 2 2 ( k − np ) P ( k ) > n ε ∑ n
k − np > n ε
= n 2ε 2 P { k − np > n ε }.
∑
k − np > n ε
2
Pn ( k )
Pn ( k )
• Rearranging and combining terms yields: ⎛⎧ k ⎫ ⎜ − p > ε ⎬ P⎜⎨ ⎭ ⎝⎩ n
⎞ pq ⎟⎟ < . 2 n ε ⎠
33
