3C (Digital Communication) Source Coding and Channel Coding

6 downloads 153 Views 2MB Size Report
(channel coding): Hamming codes. •. : 1. Simon Haykin, Communication Systems (2001). 2. John Proakis, Communication Systems Engineering (2002). CC Lab ...
3C ddddd

dddd (Digital Communication) Source Coding and Channel Coding

ddd Communication and Coding Laboratory Dept. of Electrical Engineering, National Chung Hsing University

Error Correcting Codes

d d d d : d d (Communication) ddddddddddddd • dddd • ddddddddd • 4Gdddd • dddddd(WiFi)ddddddd(WiMax) • dddddddddd

CC Lab, EE, NCHU

1

Error Correcting Codes

d d d d (Digital Communication) dddddddddddd(ddddddd): • ddddddd • ddd (source coding): Huffman codesd • ddd (channel coding): Hamming codesd • ddddddd: 1. Simon Haykin, Communication Systems (2001) 2. John Proakis, Communication Systems Engineering (2002)

CC Lab, EE, NCHU

2

Error Correcting Codes

PART I: d d d d d d

CC Lab, EE, NCHU

3

Error Correcting Codes

• dddddddddddddddddd • dddddddddddddddd1837dddd (Samuel Morse)ddddddddddddddddddddddddd dddddddd • ddddddddddddddd (Harry Nyquist)ddd (Norbert Weiner)ddd (Stephanie U. Rice)dShannon (Claude E. Shannon)ddddddddddddddddddddddd

CC Lab, EE, NCHU

4

Error Correcting Codes

M-Taiwan

CC Lab, EE, NCHU

5

Error Correcting Codes

Wireless Communication

CC Lab, EE, NCHU

6

Error Correcting Codes

An overview

'

Digital Communication

'

Source coding

ddd &

dddd &

$ 

Information Theory

%

'?

Cryptography

ddd & CC Lab, EE, NCHU

$ %

? dddd

7

'

-

Channel coding

ddd &

$ %

$ %

Error Correcting Codes

CODING • Coding for efficient communication over bandlimited channels – source coding – data compression (dddd) • Coding for secure communication over wiretap channels – cryptography – cryptology – cryptoanalysis • Coding for reliable communication over noisy channels – channel coding – error correcting code (ddddd) – forward error control code (ddddd) CC Lab, EE, NCHU

8

Error Correcting Codes

dddddddd • ddddddddddddddd(voice)ddd(data)dd d(multimedia)ddddddddddddddddddddd • dddddddddddddddddddddddddd • dddddddddddddddddddddddddddd ddddddddddddddd • dddddddddd(sampling)ddd(quantization)d • dddddddddddddddddddddddddddd ddddddd • dddddd: dddddddddddddd • dddddd: ddddddddddddd CC Lab, EE, NCHU

9

Error Correcting Codes

• dddddddddddddddddddddddddddd dddddddddddddddddddddddddddd dd • dddddddddddddddddddddddddddd dddddddddddddddddddddddddddd ddd • ddddddddddddddd HDTVdddddddddd dddddddddddd

CC Lab, EE, NCHU

10

Error Correcting Codes

CC Lab, EE, NCHU

11

Error Correcting Codes

ddddd

Figure 1: (a) analog signal (b) digital signal after sampling and quantization

CC Lab, EE, NCHU

12

Error Correcting Codes

• r(t) = s(t) ⊗ h(τ ; t) + n(t) (linear time-variant channel dddddd) • r(t) = s(t) ⊗ h(t) + n(t) (linear time-invariant channel ddddddd) • r(t) = s(t) + n(t) (additive noise channel ddddddd) CC Lab, EE, NCHU

13

Error Correcting Codes

CC Lab, EE, NCHU

14

Error Correcting Codes

CC Lab, EE, NCHU

15

Error Correcting Codes

• dddddddddddddddddddddddddddd dddddddd0d1dddd • dddddddddddddddddddddddddddd (noise) dddddd • dddd: dddddddddddddd 1. dddd (source coding)d dddddddddddddd ddddddddddddddd dddddddd (data compression)d 2. dddd (channel coding)d dddddddddddddd dd ddddddddddd ddddddddddddd ddddddddddd (forward error control code)d

CC Lab, EE, NCHU

16

Error Correcting Codes

Example: d d d d

• Discrete-time

– u = (u1 , . . . , uk ) (information vector) – c = u · G = (c1 , . . . , cn ) (codeword)

– r = c + e = (r1 , . . . , rn ) (received vector) – ci ∈ X = {0, 1} (input alphabet)

– ri ∈ Y = {0, 1} or Y = R (output alphabet) • Memoryless

– p(r|c) = p(r1 |c1 ) · · · p(rn |cn ), need p(yj |xi ), 1 ≤ i ≤ q, 1 ≤ j ≤ Q CC Lab, EE, NCHU

17

Error Correcting Codes

18

d d d d d : BSC 1-p

0

0 p

input

output

p

1

1

1-p

Figure 2: Binary Symmetric Channel, dddddd • Channel probability p(1|0) = p(0|1) = p

and

p(0|0) = p(1|1) = 1 − p

• BSC capacity C = 1 − H(p) = 1 + p log2 p + (1 − p) log2 (1 − p)

• Decoding: Hamming distance CC Lab, EE, NCHU

Error Correcting Codes

• Encoding: u ∈ F2k = {0, 1}k → c = u · G ∈ C ⊂ F2n = {0, 1}n • Modulation in transmitter: binary sequence =⇒ complex sequence =⇒ waveform c = (c1 , . . . , cn ) ∈ C

=⇒

x = (x1 , . . . , xn ) ∈ C n

=⇒

s(t) = (s1 (t), . . . , sn (t))

• Channel: r(t) = s(t) + e(t) • Demodulation in receiver: waveform =⇒ complex sequence =⇒ binary sequence r(t) =⇒ r = (r1 , . . . , rn ) ∈ C n =⇒ y = (y1 , . . . , yn ) ∈ F2n • Hard decoding: y ∈ F2n → c ∈ C → u ˆ ∈ F2k

CC Lab, EE, NCHU

19

Error Correcting Codes

20

d d d d d : AWGN c

r =c+e

+ e

Figure 3: Additive White Gaussian Noise Channel, dddddddddd • Channel probability (e ∼ N (0, N0 /2))

2

−(ri −ci ) 1 p(ri |ci ) = √ e N0 πN0

• AWGN capacity

C = W log2 (1 +

Eb C ) N0 W

• Decoding: Euclidean distance CC Lab, EE, NCHU

Error Correcting Codes

• Encoding: u ∈ F2k → c = u · G ∈ C ⊂ F2n • Modulation in transmitter: binary sequence =⇒ complex sequence =⇒ waveform c = (c1 , . . . , cn ) ∈ C

=⇒

x = (x1 , . . . , xn ) ∈ C n

=⇒

s(t) = (s1 (t), . . . , sn (t))

• Channel: r(t) = s(t) + e(t) • Demodulation in receiver: waveform =⇒ complex sequence r(t) =⇒ r = (r1 , . . . , rn ) ∈ C n • Soft decoding: r ∈ C n → c ∈ C → u ˆ ∈ F2k • Iterative decoding: r1 ∈ C n → r2 ∈ C n · · · → rl ∈ C n → u ˆ ∈ F2k CC Lab, EE, NCHU

21

Error Correcting Codes

PART II: d d d d 1. Introduction to source coding 2. Huffman codes

CC Lab, EE, NCHU

22

Error Correcting Codes

• ddddddddddddddddddddddddddd dd dddddddddddddddddddddddddd dddddddddddddddddd(data compression, source coding)d • dddddddddddddddddddddddddd ddDVD(MPEG-4)ddddd dddddddddddddd dddddddddddddd • dddddddddddddddddddddddddddd ddddddddddddddddddddd • dddddddddd(redundency)dddddddddddd ddddddddddddddddddddd

CC Lab, EE, NCHU

23

Error Correcting Codes

24

อૠ(statistical) ᑓ‫ڤ‬

ၦ֏ (quantization) ᑓ‫ڤ‬

‫ࠢڗ‬ (dictionary) ᑓ‫ڤ‬

ITU-T G.7XX H.26X IS-XXXX ፿ଃᚘᜍ ီಛᚘᜍ (Audio Compression) (Vudio Compression)

GSM-XXX

MPEG-4 x x x x

(Data compression) ᠏ངᒳᒘ (transform coding) ᑓ‫ڤ‬

ࠡ‫ה‬ (others) ᑓ‫ڤ‬

ᐙቝᚘᜍ (Image Compression)

JPEG-2000

JPEG ቃྒྷᒳᒘ (prediction coding)ᑓ ‫ڤ‬

։᙮ᒳᒘ (subband coding)ᑓ‫ڤ‬

Figure 4: ddddddddddddddd CC Lab, EE, NCHU

Error Correcting Codes

• dddddddddddddddddddddddddd 512 · 512 · 3 = 3 · 218 bytes. • ddd56kbpsddddddd 786488 ∗ 8/56K =112ddddd ddddddddddddddd34dddddddd3.2ddd ddddddd

W hy D ata Com pression ?

786488 bytes

23116 bytes,Cr=34.0

CC Lab, EE, NCHU

25

Error Correcting Codes

• ddddddd0d1ddddddddddddddddddd ddddddddddddddd(0d1)dddddddddd dddd(00d01d10d11)d • ddddddddndddddddddddddddd2n dd dddddddddddd(dd) ddddddd dddddd ddd(Fix length coding, FLC)d • Question: dddddddd2n dddddddddddnddd dddddd?

CC Lab, EE, NCHU

26

Error Correcting Codes

27

• Example: dddddddddddddddddd {A, B, A, B, A, C, A, D} • dddddddddddd symbol

dd

code I

code II

A

4/8

00

0

B

2/8

01

10

C

1/8

10

110

D

1/8

11

111

• code I d FLC (Fix length coding), code II d VLC (variable length coding)d

CC Lab, EE, NCHU

Error Correcting Codes

• code I ddsymbolddddddddd 2 · 2/8 + 2 · 4/8 + 2 · 1/8 + 2 · 1/8 = 2bits • code II ddsymbolddddddddd 1 · 4/8 + 2 · 2/8 + 3 · 1/8 + 3 · 1/8 = 1.75bits • dddddddddddddddddddddddddddd ddddddddddd • dddddddddddddddd dddddddddddd ddddddddddddddddddddddddddd ddddddddd(coding redundancy)d

CC Lab, EE, NCHU

28

Error Correcting Codes

• dddddddddddddddddddddddddddd ddddddddd: ddddddbitsdddddddbitsd • code I or code IIddddddddddddddddddddd ddd(instantaneous)d • dddddddddddddddddddddddddddd dddddddddddddddddddddddddddd dd(prefix)d

CC Lab, EE, NCHU

29

Error Correcting Codes

30

S1 0 S2 0 1 S3 0 1

1 S4

Figure 5: code II ddd CC Lab, EE, NCHU

Error Correcting Codes

d d d d d (Huffman coding) • dddddddddddd1952dddddddddddddd ddddJPEGddddd • Huffmanddddd 1. dddddddddddddd 2. d(1)dddddddddddddddddd 3. dd(2)ddddd0d1ddd 4. dddddddddd(1), (2), and (3)ddd 5. ddddddddddd

CC Lab, EE, NCHU

31

Error Correcting Codes

• Example: dddddddddd{A, B, C, D, E}dddddd d{15, 7, 6, 6, 5}d • ddFLCdddd3 · (15 + 7 + 6 + 6 + 5) = 117dbitsd • dVLCddddddddd {A = 0, B = 100, C = 101, D = 110, E = 111} ddd1 · 15 + 3 · (7 + 6 + 6 + 5) = 87dbitsd

CC Lab, EE, NCHU

32

Error Correcting Codes

33

A 0

B

0

1

0

C

1

D 0

E

1

1

Figure 6: ddd CC Lab, EE, NCHU

Error Correcting Codes

PART III: d d d d 1. Introduction: Coding history and coding type 2. Coding method: Hamming codes 3. Coding research: application, practice, theory, and connection

CC Lab, EE, NCHU

34

Error Correcting Codes

Summary: Error Correcting Codes • Origin: 1948 paper, ”A Mathematical Theory of Communication”

It is possible to transmit information over a channel of capacity C with arbitrarily small error at a rate R if and only if R < C.

• Idea: Add redundant bits to information bits Block codes(ddd) vs. Convolutional codes(ddd) • Goal: Trade-off between reducing

R W

Eb (ddd) N0

and Pb (ddd) with the price of

(dddd) and adding encoder/decoder complexity (dd

dddddddd).

CC Lab, EE, NCHU

35

Error Correcting Codes

dddddddd • 1948: Information theory is born, Hamming and Golay codes • 1948-58: {block codes, convolutional codes} ⊂ codes on algebra • 1958-68: BCH, LDPC, concatenated codes, and algebraic decoding • 1968-78: BCJR, Chase and Viterbi decoding • 1978-88: {Goppa code, AG code} ⊂ codes on geometry {TCM and BCM} ⊂ coded modulation

• 1988-98: {Turbo codes, LDPC codes} ⊂ codes on graph {trellis, tailbiting trellis} ⊂ factor graph

• 1998-2008: Multilevel codes, space time codes, and iterative decoding • 2008-: Network coding, Dirty paper codes, and coding application CC Lab, EE, NCHU

36

Error Correcting Codes

CC Lab, EE, NCHU

Mathematics Communication

 

cryptography

⊂ Informatin Theory ⊂



source coding

  

  

channel coding

37

Error Correcting Codes

• Golay codes (1949), Hamming codes (1950), RM codes (1954) • linear block codes (1956), convolutional codes (1955) CC Lab, EE, NCHU

38

Error Correcting Codes

• BCH (1959), RS (1960), LDPC (1963), concatenated codes (1966) • BCH algebraic decoding (1960) and Viterbi decoding (1967) CC Lab, EE, NCHU

39

Error Correcting Codes

• Goppa codes (1970), JPL bound (1977) • Chase (1972), BCJR (1974) decoding, multilevel codes (1977) CC Lab, EE, NCHU

40

Error Correcting Codes

• A-G codes (1982), Tanner Graph (1981), coset codes (1988) • TCM (1982), V.32 (1984), SOVA (1989) CC Lab, EE, NCHU

41

Error Correcting Codes

• Trellis of block codes (1993), Factor Graph (1995) • Turbo codes (1993), iterative decoding (1993) CC Lab, EE, NCHU

42

Error Correcting Codes

43

dddddddd

• ddddddddddddddddddddd c(i) ddddd dddd u(i) ddddddd G(D) = G dddddd c(i) = u(i)G CC Lab, EE, NCHU

(1)

Error Correcting Codes

44

• dddddddddddddddddddddddddddd dddd M ddddd c(i)dddddddd c(i) = u(i)G0 + u(i − 1)G1 + · · · + u(i − M )GM CC Lab, EE, NCHU

(2)

Error Correcting Codes

Decoding

CC Lab, EE, NCHU

45

Error Correcting Codes

dddddddddd dddddddddddd(Codes on algebra)dddd(Codes on graph)ddddd(Codes on geometry)dddd • ddddddddddddd (finite field)dddd (group theory)dddddd • dddddddddddddd(factor graph)dddddd(sum product algorithm)dddddd • dddddddddddddd(algebraic geometry)dddd d(finite geometry) dddddd

CC Lab, EE, NCHU

46

Error Correcting Codes

codes on algebra, d d d

CC Lab, EE, NCHU

47

Error Correcting Codes

CC Lab, EE, NCHU

48

Error Correcting Codes

x4

x6

x2

x7

x5

x3 x1

Figure 7: A [7, 7, 1] code • No constraint on xi and there are 27 vectors.

CC Lab, EE, NCHU

49

Error Correcting Codes

x4

x6

x2

x7

x5

x3 x1

Figure 8: The [7, 6, 2] parity check code • The number of 1 in these 7 positions are even. CC Lab, EE, NCHU

50

Error Correcting Codes

I.e., the code is the set satisfying the following: C = {x ∈ F27 : Hx = 0}  x1   x2    x3 h i  Hx = 1 1 1 1 1 1 1  x4    x5   x  6 x7

CC Lab, EE, NCHU

51



       =0      

Error Correcting Codes

x4

x6

x2

x7

x5

x3 x1

Figure 9: A [7, 1, 7] repetition code • The value in these 7 positions are the same. CC Lab, EE, NCHU

52

Error Correcting Codes

I.e., the code is the set satisfying the following: C = {x ∈ F27 : Hx = 0}     x1  1 1 0 0 0 0 0      x2    1 0 1 0 0 0 0           x3    1 0 0 1 0 0 0        Hx =    x4  =   1 0 0 0 1 0 0           x5    1 0 0 0 0 1 0      x    6  1 0 0 0 0 0 1 x7

CC Lab, EE, NCHU

53

0 0 0 0 0 0

            

Error Correcting Codes

x4

x6

x2

x7

x5

x3 x1

Figure 10: A [7, 4, 3] Hamming code • The number of 1 in each circle is even. CC Lab, EE, NCHU

54

Error Correcting Codes

I.e., the code is the set satisfying the following: C = {x ∈ F27 : Hx = 0}   x1    x2          0 0 0 1 1 1 1  x3  0          Hx =  0 1 1 0 0 1 1   x4  =  0       1 0 1 0 1 0 1  x5  0    x   6  x7

CC Lab, EE, NCHU

55

Error Correcting Codes

56

Codes on graph, d d d Example: g(x1 , x2 , x3 , x4 ) = fA (x1 )fB (x1 , x2 , x3 )fC (x3 , x4 )fD (x3 )

1

2

3

4

fA

fB

fC

fD

4 1

g 2

3

S = {x1 , x2 , x3 , x4 } and Q = {fA , fB , fC , fD }

CC Lab, EE, NCHU

Error Correcting Codes

• A factor graph of a linear code C[n, k] with parity check matrix H = hh1 , h2 , . . . , hn−k i • [(x1 , x2 , . . . , x6 ) ∈ C] (a complicated function)

= [x1 ⊕ x3 ⊕ x4 = 0] · [x1 ⊕ x2 ⊕ x5 = 0] · [x2 ⊕ x3 ⊕ x6 = 0]

CC Lab, EE, NCHU

57

Error Correcting Codes

• A factor graph of a trellis of a linear code C[n, k] T (C) = [(s0 , c1 , s1 ) ∈ T1 ][(s1 , c2 , s2 ) ∈ T2 ] · · · [(sn−1 , cn−1 , sn ) ∈ Tn ]

CC Lab, EE, NCHU

58

Error Correcting Codes

• A factor graph of the turbo code and its decoding

CC Lab, EE, NCHU

59

Error Correcting Codes

Coding Research

CC Lab, EE, NCHU

60

Error Correcting Codes

dd 1. ddddddddddddddddddddd 2. BSCdddddddd 3. AWGNdddddddd 4. ddHuffmandddddd 5. [7, 4, 3] Hammingdddddddd 6. ddddddddddd 7. ddddddddddddd

CC Lab, EE, NCHU

61