(channel coding): Hamming codes. •. : 1. Simon Haykin, Communication Systems
(2001). 2. John Proakis, Communication Systems Engineering (2002). CC Lab ...
3C ddddd
dddd (Digital Communication) Source Coding and Channel Coding
ddd Communication and Coding Laboratory Dept. of Electrical Engineering, National Chung Hsing University
Error Correcting Codes
d d d d : d d (Communication) ddddddddddddd • dddd • ddddddddd • 4Gdddd • dddddd(WiFi)ddddddd(WiMax) • dddddddddd
CC Lab, EE, NCHU
1
Error Correcting Codes
d d d d (Digital Communication) dddddddddddd(ddddddd): • ddddddd • ddd (source coding): Huffman codesd • ddd (channel coding): Hamming codesd • ddddddd: 1. Simon Haykin, Communication Systems (2001) 2. John Proakis, Communication Systems Engineering (2002)
CC Lab, EE, NCHU
2
Error Correcting Codes
PART I: d d d d d d
CC Lab, EE, NCHU
3
Error Correcting Codes
• dddddddddddddddddd • dddddddddddddddd1837dddd (Samuel Morse)ddddddddddddddddddddddddd dddddddd • ddddddddddddddd (Harry Nyquist)ddd (Norbert Weiner)ddd (Stephanie U. Rice)dShannon (Claude E. Shannon)ddddddddddddddddddddddd
CC Lab, EE, NCHU
4
Error Correcting Codes
M-Taiwan
CC Lab, EE, NCHU
5
Error Correcting Codes
Wireless Communication
CC Lab, EE, NCHU
6
Error Correcting Codes
An overview
'
Digital Communication
'
Source coding
ddd &
dddd &
$
Information Theory
%
'?
Cryptography
ddd & CC Lab, EE, NCHU
$ %
? dddd
7
'
-
Channel coding
ddd &
$ %
$ %
Error Correcting Codes
CODING • Coding for efficient communication over bandlimited channels – source coding – data compression (dddd) • Coding for secure communication over wiretap channels – cryptography – cryptology – cryptoanalysis • Coding for reliable communication over noisy channels – channel coding – error correcting code (ddddd) – forward error control code (ddddd) CC Lab, EE, NCHU
8
Error Correcting Codes
dddddddd • ddddddddddddddd(voice)ddd(data)dd d(multimedia)ddddddddddddddddddddd • dddddddddddddddddddddddddd • dddddddddddddddddddddddddddd ddddddddddddddd • dddddddddd(sampling)ddd(quantization)d • dddddddddddddddddddddddddddd ddddddd • dddddd: dddddddddddddd • dddddd: ddddddddddddd CC Lab, EE, NCHU
9
Error Correcting Codes
• dddddddddddddddddddddddddddd dddddddddddddddddddddddddddd dd • dddddddddddddddddddddddddddd dddddddddddddddddddddddddddd ddd • ddddddddddddddd HDTVdddddddddd dddddddddddd
CC Lab, EE, NCHU
10
Error Correcting Codes
CC Lab, EE, NCHU
11
Error Correcting Codes
ddddd
Figure 1: (a) analog signal (b) digital signal after sampling and quantization
CC Lab, EE, NCHU
12
Error Correcting Codes
• r(t) = s(t) ⊗ h(τ ; t) + n(t) (linear time-variant channel dddddd) • r(t) = s(t) ⊗ h(t) + n(t) (linear time-invariant channel ddddddd) • r(t) = s(t) + n(t) (additive noise channel ddddddd) CC Lab, EE, NCHU
13
Error Correcting Codes
CC Lab, EE, NCHU
14
Error Correcting Codes
CC Lab, EE, NCHU
15
Error Correcting Codes
• dddddddddddddddddddddddddddd dddddddd0d1dddd • dddddddddddddddddddddddddddd (noise) dddddd • dddd: dddddddddddddd 1. dddd (source coding)d dddddddddddddd ddddddddddddddd dddddddd (data compression)d 2. dddd (channel coding)d dddddddddddddd dd ddddddddddd ddddddddddddd ddddddddddd (forward error control code)d
CC Lab, EE, NCHU
16
Error Correcting Codes
Example: d d d d
• Discrete-time
– u = (u1 , . . . , uk ) (information vector) – c = u · G = (c1 , . . . , cn ) (codeword)
– r = c + e = (r1 , . . . , rn ) (received vector) – ci ∈ X = {0, 1} (input alphabet)
– ri ∈ Y = {0, 1} or Y = R (output alphabet) • Memoryless
– p(r|c) = p(r1 |c1 ) · · · p(rn |cn ), need p(yj |xi ), 1 ≤ i ≤ q, 1 ≤ j ≤ Q CC Lab, EE, NCHU
17
Error Correcting Codes
18
d d d d d : BSC 1-p
0
0 p
input
output
p
1
1
1-p
Figure 2: Binary Symmetric Channel, dddddd • Channel probability p(1|0) = p(0|1) = p
and
p(0|0) = p(1|1) = 1 − p
• BSC capacity C = 1 − H(p) = 1 + p log2 p + (1 − p) log2 (1 − p)
• Decoding: Hamming distance CC Lab, EE, NCHU
Error Correcting Codes
• Encoding: u ∈ F2k = {0, 1}k → c = u · G ∈ C ⊂ F2n = {0, 1}n • Modulation in transmitter: binary sequence =⇒ complex sequence =⇒ waveform c = (c1 , . . . , cn ) ∈ C
=⇒
x = (x1 , . . . , xn ) ∈ C n
=⇒
s(t) = (s1 (t), . . . , sn (t))
• Channel: r(t) = s(t) + e(t) • Demodulation in receiver: waveform =⇒ complex sequence =⇒ binary sequence r(t) =⇒ r = (r1 , . . . , rn ) ∈ C n =⇒ y = (y1 , . . . , yn ) ∈ F2n • Hard decoding: y ∈ F2n → c ∈ C → u ˆ ∈ F2k
CC Lab, EE, NCHU
19
Error Correcting Codes
20
d d d d d : AWGN c
r =c+e
+ e
Figure 3: Additive White Gaussian Noise Channel, dddddddddd • Channel probability (e ∼ N (0, N0 /2))
2
−(ri −ci ) 1 p(ri |ci ) = √ e N0 πN0
• AWGN capacity
C = W log2 (1 +
Eb C ) N0 W
• Decoding: Euclidean distance CC Lab, EE, NCHU
Error Correcting Codes
• Encoding: u ∈ F2k → c = u · G ∈ C ⊂ F2n • Modulation in transmitter: binary sequence =⇒ complex sequence =⇒ waveform c = (c1 , . . . , cn ) ∈ C
=⇒
x = (x1 , . . . , xn ) ∈ C n
=⇒
s(t) = (s1 (t), . . . , sn (t))
• Channel: r(t) = s(t) + e(t) • Demodulation in receiver: waveform =⇒ complex sequence r(t) =⇒ r = (r1 , . . . , rn ) ∈ C n • Soft decoding: r ∈ C n → c ∈ C → u ˆ ∈ F2k • Iterative decoding: r1 ∈ C n → r2 ∈ C n · · · → rl ∈ C n → u ˆ ∈ F2k CC Lab, EE, NCHU
21
Error Correcting Codes
PART II: d d d d 1. Introduction to source coding 2. Huffman codes
CC Lab, EE, NCHU
22
Error Correcting Codes
• ddddddddddddddddddddddddddd dd dddddddddddddddddddddddddd dddddddddddddddddd(data compression, source coding)d • dddddddddddddddddddddddddd ddDVD(MPEG-4)ddddd dddddddddddddd dddddddddddddd • dddddddddddddddddddddddddddd ddddddddddddddddddddd • dddddddddd(redundency)dddddddddddd ddddddddddddddddddddd
CC Lab, EE, NCHU
23
Error Correcting Codes
24
อૠ(statistical) ᑓڤ
ၦ֏ (quantization) ᑓڤ
ࠢڗ (dictionary) ᑓڤ
ITU-T G.7XX H.26X IS-XXXX ଃᚘᜍ ီಛᚘᜍ (Audio Compression) (Vudio Compression)
GSM-XXX
MPEG-4 x x x x
(Data compression) ᠏ངᒳᒘ (transform coding) ᑓڤ
ࠡה (others) ᑓڤ
ᐙቝᚘᜍ (Image Compression)
JPEG-2000
JPEG ቃྒྷᒳᒘ (prediction coding)ᑓ ڤ
։᙮ᒳᒘ (subband coding)ᑓڤ
Figure 4: ddddddddddddddd CC Lab, EE, NCHU
Error Correcting Codes
• dddddddddddddddddddddddddd 512 · 512 · 3 = 3 · 218 bytes. • ddd56kbpsddddddd 786488 ∗ 8/56K =112ddddd ddddddddddddddd34dddddddd3.2ddd ddddddd
W hy D ata Com pression ?
786488 bytes
23116 bytes,Cr=34.0
CC Lab, EE, NCHU
25
Error Correcting Codes
• ddddddd0d1ddddddddddddddddddd ddddddddddddddd(0d1)dddddddddd dddd(00d01d10d11)d • ddddddddndddddddddddddddd2n dd dddddddddddd(dd) ddddddd dddddd ddd(Fix length coding, FLC)d • Question: dddddddd2n dddddddddddnddd dddddd?
CC Lab, EE, NCHU
26
Error Correcting Codes
27
• Example: dddddddddddddddddd {A, B, A, B, A, C, A, D} • dddddddddddd symbol
dd
code I
code II
A
4/8
00
0
B
2/8
01
10
C
1/8
10
110
D
1/8
11
111
• code I d FLC (Fix length coding), code II d VLC (variable length coding)d
CC Lab, EE, NCHU
Error Correcting Codes
• code I ddsymbolddddddddd 2 · 2/8 + 2 · 4/8 + 2 · 1/8 + 2 · 1/8 = 2bits • code II ddsymbolddddddddd 1 · 4/8 + 2 · 2/8 + 3 · 1/8 + 3 · 1/8 = 1.75bits • dddddddddddddddddddddddddddd ddddddddddd • dddddddddddddddd dddddddddddd ddddddddddddddddddddddddddd ddddddddd(coding redundancy)d
CC Lab, EE, NCHU
28
Error Correcting Codes
• dddddddddddddddddddddddddddd ddddddddd: ddddddbitsdddddddbitsd • code I or code IIddddddddddddddddddddd ddd(instantaneous)d • dddddddddddddddddddddddddddd dddddddddddddddddddddddddddd dd(prefix)d
CC Lab, EE, NCHU
29
Error Correcting Codes
30
S1 0 S2 0 1 S3 0 1
1 S4
Figure 5: code II ddd CC Lab, EE, NCHU
Error Correcting Codes
d d d d d (Huffman coding) • dddddddddddd1952dddddddddddddd ddddJPEGddddd • Huffmanddddd 1. dddddddddddddd 2. d(1)dddddddddddddddddd 3. dd(2)ddddd0d1ddd 4. dddddddddd(1), (2), and (3)ddd 5. ddddddddddd
CC Lab, EE, NCHU
31
Error Correcting Codes
• Example: dddddddddd{A, B, C, D, E}dddddd d{15, 7, 6, 6, 5}d • ddFLCdddd3 · (15 + 7 + 6 + 6 + 5) = 117dbitsd • dVLCddddddddd {A = 0, B = 100, C = 101, D = 110, E = 111} ddd1 · 15 + 3 · (7 + 6 + 6 + 5) = 87dbitsd
CC Lab, EE, NCHU
32
Error Correcting Codes
33
A 0
B
0
1
0
C
1
D 0
E
1
1
Figure 6: ddd CC Lab, EE, NCHU
Error Correcting Codes
PART III: d d d d 1. Introduction: Coding history and coding type 2. Coding method: Hamming codes 3. Coding research: application, practice, theory, and connection
CC Lab, EE, NCHU
34
Error Correcting Codes
Summary: Error Correcting Codes • Origin: 1948 paper, ”A Mathematical Theory of Communication”
It is possible to transmit information over a channel of capacity C with arbitrarily small error at a rate R if and only if R < C.
• Idea: Add redundant bits to information bits Block codes(ddd) vs. Convolutional codes(ddd) • Goal: Trade-off between reducing
R W
Eb (ddd) N0
and Pb (ddd) with the price of
(dddd) and adding encoder/decoder complexity (dd
dddddddd).
CC Lab, EE, NCHU
35
Error Correcting Codes
dddddddd • 1948: Information theory is born, Hamming and Golay codes • 1948-58: {block codes, convolutional codes} ⊂ codes on algebra • 1958-68: BCH, LDPC, concatenated codes, and algebraic decoding • 1968-78: BCJR, Chase and Viterbi decoding • 1978-88: {Goppa code, AG code} ⊂ codes on geometry {TCM and BCM} ⊂ coded modulation
• 1988-98: {Turbo codes, LDPC codes} ⊂ codes on graph {trellis, tailbiting trellis} ⊂ factor graph
• 1998-2008: Multilevel codes, space time codes, and iterative decoding • 2008-: Network coding, Dirty paper codes, and coding application CC Lab, EE, NCHU
36
Error Correcting Codes
CC Lab, EE, NCHU
Mathematics Communication
cryptography
⊂ Informatin Theory ⊂
source coding
channel coding
37
Error Correcting Codes
• Golay codes (1949), Hamming codes (1950), RM codes (1954) • linear block codes (1956), convolutional codes (1955) CC Lab, EE, NCHU
38
Error Correcting Codes
• BCH (1959), RS (1960), LDPC (1963), concatenated codes (1966) • BCH algebraic decoding (1960) and Viterbi decoding (1967) CC Lab, EE, NCHU
39
Error Correcting Codes
• Goppa codes (1970), JPL bound (1977) • Chase (1972), BCJR (1974) decoding, multilevel codes (1977) CC Lab, EE, NCHU
40
Error Correcting Codes
• A-G codes (1982), Tanner Graph (1981), coset codes (1988) • TCM (1982), V.32 (1984), SOVA (1989) CC Lab, EE, NCHU
41
Error Correcting Codes
• Trellis of block codes (1993), Factor Graph (1995) • Turbo codes (1993), iterative decoding (1993) CC Lab, EE, NCHU
42
Error Correcting Codes
43
dddddddd
• ddddddddddddddddddddd c(i) ddddd dddd u(i) ddddddd G(D) = G dddddd c(i) = u(i)G CC Lab, EE, NCHU
(1)
Error Correcting Codes
44
• dddddddddddddddddddddddddddd dddd M ddddd c(i)dddddddd c(i) = u(i)G0 + u(i − 1)G1 + · · · + u(i − M )GM CC Lab, EE, NCHU
(2)
Error Correcting Codes
Decoding
CC Lab, EE, NCHU
45
Error Correcting Codes
dddddddddd dddddddddddd(Codes on algebra)dddd(Codes on graph)ddddd(Codes on geometry)dddd • ddddddddddddd (finite field)dddd (group theory)dddddd • dddddddddddddd(factor graph)dddddd(sum product algorithm)dddddd • dddddddddddddd(algebraic geometry)dddd d(finite geometry) dddddd
CC Lab, EE, NCHU
46
Error Correcting Codes
codes on algebra, d d d
CC Lab, EE, NCHU
47
Error Correcting Codes
CC Lab, EE, NCHU
48
Error Correcting Codes
x4
x6
x2
x7
x5
x3 x1
Figure 7: A [7, 7, 1] code • No constraint on xi and there are 27 vectors.
CC Lab, EE, NCHU
49
Error Correcting Codes
x4
x6
x2
x7
x5
x3 x1
Figure 8: The [7, 6, 2] parity check code • The number of 1 in these 7 positions are even. CC Lab, EE, NCHU
50
Error Correcting Codes
I.e., the code is the set satisfying the following: C = {x ∈ F27 : Hx = 0} x1 x2 x3 h i Hx = 1 1 1 1 1 1 1 x4 x5 x 6 x7
CC Lab, EE, NCHU
51
=0
Error Correcting Codes
x4
x6
x2
x7
x5
x3 x1
Figure 9: A [7, 1, 7] repetition code • The value in these 7 positions are the same. CC Lab, EE, NCHU
52
Error Correcting Codes
I.e., the code is the set satisfying the following: C = {x ∈ F27 : Hx = 0} x1 1 1 0 0 0 0 0 x2 1 0 1 0 0 0 0 x3 1 0 0 1 0 0 0 Hx = x4 = 1 0 0 0 1 0 0 x5 1 0 0 0 0 1 0 x 6 1 0 0 0 0 0 1 x7
CC Lab, EE, NCHU
53
0 0 0 0 0 0
Error Correcting Codes
x4
x6
x2
x7
x5
x3 x1
Figure 10: A [7, 4, 3] Hamming code • The number of 1 in each circle is even. CC Lab, EE, NCHU
54
Error Correcting Codes
I.e., the code is the set satisfying the following: C = {x ∈ F27 : Hx = 0} x1 x2 0 0 0 1 1 1 1 x3 0 Hx = 0 1 1 0 0 1 1 x4 = 0 1 0 1 0 1 0 1 x5 0 x 6 x7
CC Lab, EE, NCHU
55
Error Correcting Codes
56
Codes on graph, d d d Example: g(x1 , x2 , x3 , x4 ) = fA (x1 )fB (x1 , x2 , x3 )fC (x3 , x4 )fD (x3 )
1
2
3
4
fA
fB
fC
fD
4 1
g 2
3
S = {x1 , x2 , x3 , x4 } and Q = {fA , fB , fC , fD }
CC Lab, EE, NCHU
Error Correcting Codes
• A factor graph of a linear code C[n, k] with parity check matrix H = hh1 , h2 , . . . , hn−k i • [(x1 , x2 , . . . , x6 ) ∈ C] (a complicated function)
= [x1 ⊕ x3 ⊕ x4 = 0] · [x1 ⊕ x2 ⊕ x5 = 0] · [x2 ⊕ x3 ⊕ x6 = 0]
CC Lab, EE, NCHU
57
Error Correcting Codes
• A factor graph of a trellis of a linear code C[n, k] T (C) = [(s0 , c1 , s1 ) ∈ T1 ][(s1 , c2 , s2 ) ∈ T2 ] · · · [(sn−1 , cn−1 , sn ) ∈ Tn ]
CC Lab, EE, NCHU
58
Error Correcting Codes
• A factor graph of the turbo code and its decoding
CC Lab, EE, NCHU
59
Error Correcting Codes
Coding Research
CC Lab, EE, NCHU
60
Error Correcting Codes
dd 1. ddddddddddddddddddddd 2. BSCdddddddd 3. AWGNdddddddd 4. ddHuffmandddddd 5. [7, 4, 3] Hammingdddddddd 6. ddddddddddd 7. ddddddddddddd
CC Lab, EE, NCHU
61