EUROPEAN COOPERATION IN THE FIELD OF SCIENTIFIC AND TECHNICAL RESEARCH
COST 259 TD(99) . Thessaloniki, January 1999
EURO-COST
Source:
U.K University of York
Comparison of iterative decoder performance with union bounds for short frame turbo-codes A. G. Burr and G. P. White Dept. of Electronics, University of York, York YO10 5DD, U.K. Email:
[email protected]
Abstract We consider short frame turbo-codes, suitable for delay-sensitive services such as conversational speech, or for encoding single ATM cells. We compare the uniform interleaver bound of Benedetto and Montorsi, and a union bound obtained by estimating the weight distribution due to the actual pseudo-random interleaver, with the simulated BER using an iterative decoder. We show that the uniform interleaver bound is significantly pessimistic at high SNR, where the random interleaver union bound is quite close. However, at lower SNR the BER exceeds the union bound, showing that it does not achieve ML decoding. We also consider half rate turbo-codes formed by puncturing various proportions of the data and parity sequences, and adapt the uniform interleaver bounds for these codes. We show that while the union bounds predict better performance for datapunctured codes, in practice they perform better only at high SNR and for limited proportions of data puncturing.
Acknowledgements This work has been performed with the support of the U.K. Rngineering and Physical Sciences Research Council It has also been submitted to Les Annales des Telecommunications for possible publication
Comparison of iterative decoder performance with union bounds for short frame turbo-codes A. G. Burr and G. P. White, Dept. of Electronics, University of York, U.K.
1. Introduction The discovery of Turbo-codes by Berrou, Glavieux and Thitijashima [1] in 1993 demonstrated at last a means of closely approaching the Shannon bounds on the capacity of a communication channel. However, to achieve this performance requires very large interleavers, corresponding to large latency and delay. For delay-sensitive services such as conversational speech, latency is typically limited to a few hundred bits. Further, the application of turbo-codes has also been proposed in systems using ATM, in which the frame (or cell) length is normally only 424 bits (including headers). Hence there has been considerable recent interest [2, 3] in the performance of turbo-codes with comparatively short interleavers, for short frame or low delay applications. This is also the main interest in the work described in this paper. There are two aspects to Turbo-codes that result in their unique combination of powerful error-control properties and computationally feasible decoding. The first is the combination of two (or more) very simple component codes, linked by a large pseudo-random interleaver, which results in a very complex composite code with particularly useful distance properties. The second is the iterative decoding algorithm which enables simple decoders for the component codes to decode the full composite code, and whose structure leads to the name "Turbo-codes". One of the questions this paper examines is the extent to which the iterative decoder can provide true maximum-likelihood decoding of the whole code. At the same time it casts some light on the distance properties of the composite code. The distance properties of Turbo-codes, as described by the weight spectrum, are largely determined by the exact interleaver chosen. For long frame codes it is computationally prohibitive to determine the precise weight spectrum, or weight distribution, due to a given pseudo-random interleaver, although weight spectra have been examined for some other interleavers [4-6]. For this reason, and especially to examine the influence of interleaver size on Turbo-code performance, Benedetto and Montorsi [7] introduced the concept of the uniform interleaver, which could be used to determine expected weight distributions for a given interleaver length. We examine this concept in more detail in the next section. However, this work was partly motivated by the observation that the performance of some practical short frame Turbo-codes as obtained by simulation and as reported in the literature significantly exceeds that suggested by the uniform interleaver bounds. Hence in the central part of this paper we give results for the actual weight distribution given by a particular pseudo-random interleaver, and compare it with that obtained from the uniform interleaver. The weight distribution of a code can be used to obtain a union upper bound on the bit error ratio (BER) of a code, assuming ideal maximum likelihood (ML) decoding. Union bounds are typically quite tight at high signal to noise ratio (SNR), and are certainly asymptotically tight. Thus if the actual BER of a decoder exceeds the union bound, it shows that ML decoding is not achieved. Puncturing of the component codes is commonly used to provide codes of rate greater than one third, which is the "natural" rate of a Turbo-code. For example, the code of [1] was half rate, created by puncturing every other bit of each of the two component code streams. Conventionally, only the code bits have been punctured in this way, not the data. By a slight adaptation of Benedetto and Montorsi's technique, we derive uniform interleaver bounds for codes in which the data as well as the code bits have been punctured. Comparison with simulation results casts further light on the performance of the iterative decoding algorithm.
2. Uniform interleaver bounds We here adapt the derivation of the uniform interleaver bounds given by Benedetto and Montorsi slightly in order to allow for the possibility of data-punctured codes. Our terminology is also slightly different. C Let A ( w,d ) denote the conditional weight distribution of the code C, i.e. the number of code sequences of code weight d and data weight w. (This differs from the otherwise correspondent IRWEF of Benedetto and
Montorsi in that the code weight here is the whole weight of the code sequence, not just the redundancy weight). The length of C is the interleaver length, N. The uniform interleaver is a probabalistic device which maps a data sequence of weight w to any other sequence of the same weight with equal probability. If a randomly-chosen data word D of weight w is presented to the encoder C, then the probability that the codeword C will have weight d: Pr{wt (C) = d wt( D) = w} = C1
C 1 (2N)
π C2
..(1)
C 2 (N)
Multiplex
D (N)
A C( w,d ) N Cw
Cp (3N)
Fig. 1. Structure of parallel-concatenated (turbo) encoder where NCw denotes the binomial coefficient. Suppose a rate one third turbo-code Cp is formed by the parallel concatenation of codes C1 and C2, as shown in Fig. 1. The composite codeword Cp (of length 3N) is made up of a multiplex of the length 2N systematic codeword C1 and the length N parity-only sequence C2. (Note that this differs from the usual model of the turbo-encoder in that the data sequence is treated as part of the component codeword C1). Then the expected number of sequences of total weight d and data weight w in Cp is given by:
{
}
AC P ( w,d )= NCw Pr (wt (C1 ) + wt (C2 ) = d)wt ( D) = w d
∑ Pr{(wt(C1 ) = d1 & wt (C2 ) = d − d1)wt (D) = w}
N
= Cw
d1 =0
AC1 (w,d1 )AC2 (w,d − d1)
d
=
∑
..(2)
N
d1 =0
Cw
which is of course the expected conditional weight distribution of Cp. As we shall see, this approach allows us to determine uniform interleaver bounds for turbo-codes in which the data sequence is punctured, as well as the component code sequences. The expected conditional weight distribution of Cp can then be used to obtain a union bound on the BER of the turbo-code. This, of course, implicitly assumes ML decoding. The error-event probability for an error sequence with data weight w and code weight d is upper-bounded by: Cp
Pev ( w,d ) ≤ A
(w,d )Q 2dR
Hence the BER is bounded by:
Eb N0
..(3)
3N w Cp E E A (w,d )Q 2dR b = ∑ ed ( d )Q 2dR b N N0 d=d N0 w,d min
Peb ≤ ∑
..(4)
where ed(d), the data-weighted distance profile is given by: ed (d ) =
N
w Cp
∑NA
(w,d )
..(5)
w=1
The bound given by (4) is an upper bound on the BER expected over an ensemble of random interleavers. By a random coding argument like Shannon's we can see that there must be a least one interleaver among the ensemble which performs at least as well as the average. Hence it is an upper bound on the BER of the "best" turbo-code, which uses the best interleaver, and hence may be described as the uniform interleaver bound. It is not clear, however, how tight this bound may be, particularly since the iterative decoder is not provably ML. Some light is cast on this question below, however.
Data-weighted codeword multiplicity
C Rather than using the transfer function approach, we have determined A ( w,d ) for the two component encoders by means of a trellis-based algorithm implemented directly in C, which enumerates all paths of given data weight w and code weight d through the complete length N trellis. In this way we avoid the need Benedetto and Montorsi encounter to extrapolate the weight distribution to higher values of d. This is then used to Cp calculate A (w, d) and henceed(d), using (2) and (5). The resulting data-weighted distance profile is shown in Fig. 2, plotted on a logarithmic scale. We observe the exponential increase of the profile with distance described in [7], but note that it in fact flattens somewhat at higher distance with a finite interleaver length. 1.
× 10 6
10000 100 1 0.01 0.0001 1.
× 10 - 6
20
40
60
Codeword distance
80
100
Fig. 2. Data-weighted distance profile for rate one third length 256 code, obtained using uniform interleaver.
0.1
BER 0.01 0.001 0.0001 0.00001 1. × 10
-6
1. × 10
-7
ub
sim
0
1
2
3
4
Bit energy to noise density ratio, dB
5
Fig. 3. Comparison of simulated BER (sim) with uniform interleaver bound (ub) for rate one third length 256 code. Fig. 3 compares the simulated performance of a rate one third length 256 turbo-code using a pseudo-random interleaver with the uniform interleaver bound. Eight iterations of the decoder have been used. We note that the simulated performance significantly outperforms the bound at high SNR. We cannot tell, however, to what extent the divergence is due to the tightness of the union bound (which is known to diverge significantly at lower SNR) and to what extent it is because the actual interleaver is better than the ensemble average. In the next section, therefore, we find the actual weight distribution for the pseudo-random interleaver (which is feasible for a short frame turbo-code), which then gives a union bound for this specific code.
Weight spectrum for pseudo-random interleaver The conditional weight distribution of the code has been determined by a combination of Monte Carlo methods and a directed search for the lowest weight code sequences. The code in question is based on that proposed by Berrou et al [8], using four state recursive component encoders, feedback polynomial 1 + x + x2, feedforward polynomial 1 + x2. For each data weight up to w = 7 a selection of data sequences was generated and encoded, and the code weight determined. For data weights w up to 3 all NCw possible sequences were used; for higher weights a randomlychosen sample was generated, and an estimate of the true conditional weight distribution obtained from the (C p sampled distribution A (w, d) as: C Aˆ p (w, d) =
NC ( w ACp ( w,d )
Ns
..(6)
where Ns is the size of the sample. Where not all sequences were tested (i.e. Ns < NCw ), a directed search was undertaken to ensure that the lowest weight sequences had been found. All ~N shifts of the data sequences known to give lowest weight in encoder C1 were generated, interleaved and fed to encoder C2, and any low combined weights were noted. Cp Fig. 4 shows the estimated conditional weight distribution Aˆ (w, d) for values of w up to 7. Note that because of the termination of the encoder at the end of the input block there are a small number of sequences of information weight 1, despite the fact that both encoders require at least two data bits to return them to the zero state. In order to show the whole range of values this graph has been plotted on a logarithmic scale: this has suppressed the random variations for larger values of multiplicity. Note also the parabolic trend of all the
curves: this shows that on a linear plot they would fit a Gaussian "bell" shape, at least up to weights of about 300. w=7 × 10
Codeword multiplicity
1.
10
1.
× 10 8
1.
× 10 6
w=6 w=5 w=4 w=3
10000
w=2 100
w=1 50
100
150
200
250
300
350
Total codeword weight Fig. 4. Conditional weight distribution of rate one third length 256 turbo-code using pseudo-random interleaver Fig. 5 shows the resulting data-weighted distance profile of the code, compared with that derived from the Cp unform interleaver (in grey). This has been computed from the seven elements of Aˆ (w, d) shown in Fig.4, and thus may be incomplete. However, it is evident from Fig. 5 that this will have an effect only for larger values of d, greater than 50, since the evident trends show that any contribution below this will be negligible. Cp We note in fact in Fig. 5 that the curve begins to flatten above d = 50; with more terms of Aˆ (w, d) included it might continue more linearly. This would affect the union bound computed from it. Note that the two curves follow the same general trend, but there is much more random variability in the pseudo-random interleaver profile. This is particularly significant at very low values of d: we note especially that many of the multiplicities below d = 20 are in fact zero, and the minimum distance is 14, while the uniform interleaver profile suggests a minimum distance of 8. This is likely to be the reason for the deviation of the simulated BER from the uniform interleaver bound at high SNR. Data-weighted multiplicity
1.
× 10 6
10000 100 1 0.01 0.0001
1.
× 10 - 6
10
20
30
40
50
60
70
80
Codeword distance, d Fig. 5. Data-weighted distance profile of turbo-code with pseudo-random interleaver compared to uniform interleaver
3. Comparison with iterative decoder performance The estimated data-weighted distance profile for the pseudo-random interleaver can be used to calculate a union bound on the performance of this turbo-code with ML decoding, using equation (4). The result is shown in Fig. 6 compared with the simulated BER and the uniform interleaver bound shown in Fig. 3. It will be observed that the two bounds differ markedly at high SNR, and that the pseudo-random interleaver bound is much closer to the simulated result. At lower SNR the two bounds converge, and indeed if more terms Cp of Aˆ (w, d) were included, as mentioned above, the two might cross at low SNR. 0.1
BER 0.01
x x sim
0.001
x
0.0001
x 0.00001
x
1. × 10 −6
ub x x
1
2
3
rb 4
Bit energy to noise density ratio
5
Fig. 6. Simulated BER (sim) of rate one third length 256 turbo-code compared with uniform interleaver bound (ub) and union bound obtained from pseudo-random interleaver (rb) However, at lower SNR the simulated result diverges upward from the union bound. Since the union bound is an upper bound on BER for ML decoding, this implies that in this range (below a bit energy to noise desnsity ratio a little over 2 dB for this code) the iterative decoder no longer provides maximum likelihood decoding. It is evident however from the results in Berrou et al [8] that for much longer interleavers the deviation from ML decoding is less.
4. Performance of punctured codes Puncturing is required in most practical turbo-coded systems to give an overall code rate of at least one half. (Equivalently we may use higher rate recursive-systematic convolutional codes, as suggested by Benedetto and Montorsi recently). In this section we determine uniform interleaver bounds for punctured codes, and again compare them with union bounds derived from the estimated weight spectrum of the actual pseudo-random interleaver, and with the simulation results. Conventionally in turbo-codes only the two code streams have been punctured. Here we also consider patterns in which the data stream is punctured. Our adaptation of Benedetto and Montorsi's technique allows uniform interleaver bounds to be obtained here also, again casting light on the difference between the performance of the iterative decoder and ML decoder performance. Since we describe the conditional weight distributions of the component codes in terms of the total code weight, rather than taking data and parity separately, we may simply use the conditional weight distributions of the punctured component codes, even if the data is punctured. This approach has been used to determine uniform interleaver bounds for three puncture patterns, each giving an overall rate half code. The patterns are listed in Table 1, in which "d" denotes the data stream, "c1" the parity stream from code 1 and "c2" the parity tream from code 2. Pattern A is the conventional pattern used by Berrou et al [1, 8] in which the data is not punctured; in B one in four of the data bits is punctured, and correspondingly fewer of the code bits; while in C all the data bits are punctured and none of the parity bits.
d
A c1
c2
1 1
1 0
0 1
d
B c1
c2
1 1 1 0
1 0 1 1
1 1 0 0
d
C c1
c2
0
1
1
Table 1. Puncture patterns Fig. 7 shows the resulting uniform interleaver bounds. This shows that puncture patterns in which the data is punctured might be expected to perform better than the conventional patterns in which the parity only is punctured, if decoding were maximum likelihood. (We assume that the weight distributions using the pseudorandom interleaver will bear the same relations to one another as with the uniform interleaver). The reason for this can be illustrated by considering some of the lowest weight sequences generated by each component encoder. For example, the data sequences 1010.., 100010.. etc generate parity sequences 1110.., 110110.. which tend to have larger weight than the corresponding data sequences. Hence puncturing the parity sequences is likely to reduce the weight more than puncturing the data sequences. 1.
BER
0.1
A
0.01
B
0.001
C
0.0001 0.00001 1. ¥ 10
-6
0
1
2
3
4
5
6
Bit energy to noise density ratio, dB Fig. 7. Uniform interleaver bounds for half rate punctured turbo-codes using puncture patterns of Table 1. However, Fig. 8 shows the simulated BER for these codes. Here puncture pattern C performs extremely poorly, while pattern B improves on A only at high SNR. This suggests, once again, that the iterative decoder is not performing ML decoding under some conditions, and especially at low SNR. Consider now the code which is presented to each of the component decoders. For puncture pattern A this is a rate 2/3 code with a minimum distance of 3. For puncture pattern B, code C1 is again rate 2/3, with the same minimum distance (although a different puncture pattern), but C2 is rate 4/5, with minimum distance 2. For puncture pattern C, both component codes are unit rate, and thus have unity minimum distance, which explains the extremely poor performance of this code. This supports the conventional wisdom stated by Robertson [9], among others, that the codes must be systematic, although it is possible under some conditions that partially data punctured codes may perform better than equivalent rate non-data punctured codes.
+
+
x 0.1 o
o x
BER
+
C
+
+
+
+
o x
0.01
o x
0.001
x o
0.0001
A x o
B
0.00001 1. ¥ 10
x o
-6
0.5
1
1.5
2
2.5
3
Bit energy to noise density ratio, dB Fig. 8. Simulated BER for half rate punctured turbo-codes using puncture patterns of Table 1. 0.1 x
BER
x x
0.01
x
0.001
x
0.0001
x
0.00001
1. ¥ 10
sim ub x
rb
-6
1
2
3
4
5
Bit energy to noise density ratio, dB Fig. 9. Simulated BER (sim) of half rate length 256 turbo-code using puncture pattern A, compared with uniform interleaver bound (ub) and union bound obtained from pseudo-random interleaver (rb) Fig. 9 compares the simulated BER of the half rate code with puncture pattern A with the uniform interleaver bound and the pseudo-random interleaver union bound. It again shows that the union bound is much closer that the uniform interleaver bound at high SNR, and again the simulated result deviates from ML performance at low SNR.
5. Conclusions For short frame turbo-codes, suitable for limited latency services such as speech, we compare union bounds on the BER with the simulated BER using an iterative (turbo) decoder. We consider the uniform interleaver bounds of Benedetto and Montorsi and a union bound based on the actual weight spectrum of the turbo-code estimated using a combination of exhaustive search and a Monte Carlo technique. We show that at high SNR the latter is much closer to the simulated result than the uniform interleaver bound, which is rather pessimistic, because in fact the minimum distance of the code using the pseudo-random interleaver of Berrou and Glavieux is significantly greater than the uniform interleaver approach suggests. At lower SNR, however, the simulated BER deviates above the union bound, indicating that the iterative decoder no longer provides an ML solution.
We consider also codes in which the basic rate one third turbo code is punctured to give rate half. By a slight adaptation of Benedetto and Montorsi's technique, we are able to compute uniform interleaver bounds for codes in which the data sequence is punctured, as well as the two parity sequences. We compare uniform interleaver bounds for puncture patterns giving half rate codes in which varying proportions of the data sequence are punctured, and find that these bounds suggest that if ML decoding were achieved it would be preferable to puncture data than parity sequences. However, comparison with simulation shows that this is true only up to a point. A partially data punctured code performs better at high SNR than pure parity puncturing, but worse at lower SNR, and, as might be expected, a fully data punctured code performs very poorly indeed. This again shows that under some conditions the iterative decoder does not provide ML decoding.
References 1.
Berrou, C. Glavieux, A and Thitimajshima, P: "Near Shannon limit error-correcting coding and decoding: Turbo-codes" Proc. ICC'93, Geneva, May 1993, pp 1064-1070
2.
Jung, P. "Comparison of Turbo-Code Decoders Applied to Short Frame Transmission Systems" IEEE Journal on Selected Areas in Communications, vol. 14, no. 3, April 1996
3.
Valenti, M. C. and Woerner, B. D. "Variable latency turbo codes for wireless multimedia applications" Proc. Int. Symp. on Turbo Codes and Related Topics, Brest, France pp 216-219, Sept. 1997
4.
Barbelescu, A. and Pietrobon, S. "On the interleaver design for turbo-codes" Electronics Letters, Dec 1994, vol. 30, no. 25, pp 2107-2108
5.
Barbelescu, A. and Pietrobon, "On terminating the trellis of turbo-codes in the same state", Electronics Letters, Jan 1995, vol. 31, no. 1, pp 22-23
6.
Perez, L. C, Segners, J, and Costello, D. J. "A distance spectrum interpretation of turbo-codes" IEEE Trans. Information Theory, vol 42, no. 6, pp 1698-1709, Nov. 1996
7.
Benedetto, S and Montorsi, G: "Unveiling Turbo-codes: Some Results on Parallel Concatenated Coding Schemes", IEEE Trans. Information Theory, vol. 42 no. 6 pp 409-428, March 1996.
8.
Berrou, C. and Glavieux, A "Near optimum error correcting coding and decoding: Turbo-codes" IEEE Trans. Communications, vol 44, no. 10, October 1996, pp 1261-1270
9.
Robertson, P. Hoeher, P. and Villebrun, E. "Optimal and sub-optimal Maximum A Posteriori algorithms suitable for turbo-decoding" European Transactions on Telecommunications, vol. 18, no. 2, pp 119-25, March-April 1997
