Bootstrap Techniques for Signal Processing Abdelhak ...

Bootstrap Techniques for Signal Processing

Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

1

Abdelhak M. Zoubir ¨ Darmstadt Technische Universitat Fachgebiet Signalverarbeitung Institut fur ¨ Nachrichtentechnik Merckstr. 25, 64283 Darmstadt E-Mail: [email protected] URL: www.spg.tu-darmstadt.de

c A.M. Zoubir

ATiSSP Sommersemester 2007

1


Signal Processing Group

Outline Signal Processing Group

2

➠ Introduction Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

➠ Principle of the Bootstrap ➠ The Non-Parametric Bootstrap ➠ Examples ➠ Bias Estimation ➠ Variance Estimation ➠ Confidence Interval Estimation for the Mean ➠ The Parametric Bootstrap ➠ An Example: Confidence Interval Estimation for the Mean ➠ An Example of Bootstrap Failure c A.M. Zoubir


2


Outline (Cont’d) Signal Processing Group

3


➠ Principle of the Bootstrap ➠ The Dependent Data Bootstrap ➠ An Example: Variance Estimation for AR Parameter Estimates ➠ The Principle of Pivoting ➠ An Example: Confidence Interval for the Mean ➠ Variance Stabilisation ➠ Examples ➠ Correlation Coefficient c A.M. Zoubir


3


Outline (Cont’d) Signal Processing Group

4

➠ Hypothesis Testing with the Bootstrap


➠ An Example: Testing the Frequency Response for Zero ➠ Signal Detection ➠ Bootstrap Matched-Filter ➠ Tolerance Interval Bootstrap Matched-Filter ➠ Applications in Signal processing ➠ Noise Floor Estimation in OTH-Radar ➠ Confidence Intervals for Flight Parameters in Passive Sonar ➠ Detection of Buried Landmines Using GPR ➠ Summary ➠ Acknowledgements c A.M. Zoubir


4


Introduction Signal Processing Group

5

➠ Most techniques for computing variances of parameter estimators or for setInstitut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

ting confidence intervals assume a large sample size [Bhattacharya & Rao (1976)].

➠ In many signal processing problems large sample methods are inapplicable [Fisher & Hall (1991)].

➠ The bootstrap was introduced [Efron (1979)] to calculate confidence intervals for parameters when few data are available.

➠ The bootstrap has been shown to solve many other problems which would be too complicated for traditional statistical analysis [Hall (1992), Efron & Tibshirani (1993), Shao & Tu (1995)]. c A.M. Zoubir


5


Introduction (Cont’d) Signal Processing Group

6


➠ The bootstrap does with a computer what the experimenter would do in practice, if it were possible.

➠ The observations are randomly re-assigned, and the estimates re-computed. ➠ These assignments and re-computations are done many times and treated as repeated experiments.

➠ In an era of exponentially increasing computational power, computer-intensive methods such as the bootstrap are affordable.

c A.M. Zoubir


6



7


Applications of the bootstrap to real-life problems have been reported in (see also special session at ICASSP-94 [64])

➠ radar signal processing [Nagaoka & Amai (1990,1991)], ➠ sonar signal processing [Krolik et al. (1991), Reid et al. (1996)], ➠ geophysics [Fisher & Hall (1989,1990,1991), Tauxe et al. (1991)], ➠ biomedical engineering [Haynor & Woods (1989), Banga & Ghorbel (1993)], ➠ control [Lai & Chen (1995)],

c A.M. Zoubir


7



8


➠ atmospheric environmental research [Hanna (1989)], ¨ ➠ vibration analysis [Zoubir & Bohme (1991,1995)].

➠ power systems [Herman (1996)], ➠ computer vision [Lange et al. (1998), Kanatani & Ohta (1998)], ➠ image analysis [Archer & Chan (1996)], ➠ nuclear technology [Yacout et al. (1996)], ➠ metrology [Ciarlini (1997), Cox et al. (1997)], ➠ financial engineering [Bhar & Chiarella (1996), Ankenbrand & Tomassini (1996)]. c A.M. Zoubir


8




9

➠ Bootstrap methods are potentially superior to large sample techniques for small sample sizes [Hall (1992)].

➠ A danger exists when applying bootstrap techniques in some circumstances where standard approaches are judged inappropriate and in such circumstances the bootstrap may also fail [Freedman (1981)].

➠ Special care is therefore required when applying the bootstrap in real-life situations [Young (1994)].

c A.M. Zoubir


9


Problem Statement Signal Processing Group

10

X = {X1 , X2 , . . . , Xn } be a random sample from a completely unspecified distribution F . Let θ denote an unknown characteristic of F , such as its


Let

mean or variance.

Find the distribution of θˆ, an estimator of sample X . Possible solution:

θ, derived from the

repeat the experiment a sufficient number of times and

approximate the distribution of θˆ by the so obtained empirical distribution. Problem: may be inapplicable for cost reasons or because the experimental conditions are not reproducible.

c A.M. Zoubir


10


Principle of the Bootstrap Signal Processing Group

11

BOOTSTRAP WORLD


REAL WORLD Unknown Probability Model

F

Estimated Probability Model

Observed Data

^

X=(x1 , x2 , ..., xn )

F

θ=s(x ) Statistic of interest ^

Bootstrap Sample

x*=(x*1 , x*2 , ..., x*n )

θ*=s(x *) Bootstrap replication ^

Basic idea: simulate the probability mechanism of the real world by substituting the unknowns with estimates derived from the data. c A.M. Zoubir


11



12




12


The Non-Parametric Bootstrap Signal Processing Group


13

1. Conduct the experiment to obtain the random sample

{X1 , X2 , . . . , Xn } and calculate the estimate θˆ from X .

X

=

ˆ , which puts equal mass 1/n at each 2. Construct the empirical distribution F observation X1

= x1 , X2 = x2 , . . . , Xn = xn .

3. From the selected

Fˆ , draw a sample X ∗ = {X1∗ , X2∗ , . . . , Xn∗ }, called

the bootstrap (re)sample.

4. Approximate the distribution of θˆ by the distribution of θˆ∗ derived from

c A.M. Zoubir


13


X ∗.

The Bootstrap Procedure Signal Processing Group

14 Measured Data

-


Conduct Experiment

-

Compute Statistic

x

- θ(x) ˆ

? Resample

Bootstrap Data

-

x∗ 1 x∗ 2

-

c A.M. Zoubir


θ1 ˆ∗

Statistic

-

Compute

ˆ ˆ ˆ (θ) F Θ θ2 ˆ∗

Statistic

. . .

-

Compute

ˆ ˆ ˆ (θ) F Θ

. . .

x∗ N

14

-

Compute

Generate EDF,

-

6 - θˆ

-

∗ θˆN

Statistic



15




15


Example: Bias Estimation Signal Processing Group

16

Consider the problem of estimating the variance of an unknown distribution Fµ,σ , Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

based on the random sample

ˆb2 . exist, σ û2 and σ

X = {X1 , . . . , Xn }. Two different estimators

It can be easily shown that

Eσ ˆ u2

=σ

2

and

Eσ ˆ b2

= (1 −

1 2 )σ n

With the bootstrap we estimate the bias b(ˆ σ2)

= Eσ ˆ 2 −σ 2 by E∗ σ ˆ ∗2 −ˆ σ 2 , where σ ˆ 2 is the maximum likelihood estimate of σ 2 , i.e., σ ˆb2 and E∗ is expectation w.r.t. bootstrap sampling. c A.M. Zoubir


16


Example (Cont’d) Signal Processing Group

17

Step 0. Experiment. Collect the data into X Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

ˆb2 . estimates σ û2 and σ

= {X1 , . . . , Xn }. Compute the

Step 1. Resampling. Draw a sample of size n, with replacement, from X . Step 2. Calculation of the bootstrap estimate. Calculate the bootstrap esti-

ˆb2 were computed û2 and σ ˆb∗2 from X ∗ in the same way σ σ û∗2 and σ but with the resample X ∗ .

mates

Step 3. Repetition. Repeat Steps 1 and 2 to obtain a total of

N bootstrap

∗2 ∗2 ∗2 ∗2 ,...,σ û,N . estimates σ û,1 ,...,σ ˆb,N and σ ˆb,1

Step 4. and c A.M. Zoubir

σu∗2 ) Bias Estimation. Estimate b(ˆ σu2 ) by b∗ (ˆ PN ∗2 ∗2 2 ˆb2 . ˆb,i − σ σb ) = 1/N i=1 σ b(ˆ σb ) by b∗ (ˆ


17

= 1/N

PN


2 ∗2 − σ ˆ σ ˆ b i=1 u,i


18

We considered a sample of size

n = 5 from the standard normal distribution.

N = 999 and 1000 Monte Carlo simulations, we obtained the following σb∗2 ). histograms for b∗ (ˆ σu∗2 ) and b∗ (ˆ 450

200

400

180

160

350

140 300

Frequency of occurence

Frequency of occurence


With

250

200

120

100

80

150 60 100

40

50

0 −0.15

20

−0.1

−0.05 0 Bootstrap bias estimate

0.05

0.1

0 −0.9

−0.8

−0.7

−0.6

−0.5 −0.4 −0.3 Bootstrap bias estimate

−0.2

−0.1

0

Bootstrap estimation of b(ˆ σu2 ),

Bootstrap estimation of b(ˆ σb2 ),

sample mean: -4.9400e-04 (=0 !).

sample mean: -0.1573 (=-0.2 !).

c A.M. Zoubir


18



19




19


Example: Variance Estimation Signal Processing Group

20

Consider the problem of finding the variance σ 2ˆ of an estimator θˆ of θ , based on Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

θ

the random sample X

= {X1 , . . . , Xn } from the unknown distribution Fθ .

➠ If tractable, one may derive an analytic expression for σθ2ˆ. ➠ Alternatively, one may use asymptotic arguments to compute an estimate σ ˆθ2ˆ for σ 2ˆ. θ Problem: In many situations the conditions for the above are not fulfilled. Solution: The bootstrap provides a simple and accurate alternative to approx-

ˆ ∗2 imate σ 2ˆ by σ ˆ . θ

c A.M. Zoubir

θ


20



21


Step 0. Experiment. Conduct the experiment and collect the random data into the sample X

= {X1 , . . . , Xn }.

Step 1. Resampling. Draw a random sample of size n, with replacement, from

X. Step 2. Calculation of the bootstrap estimate. Evaluate the bootstrap estimate

θˆ∗ from X ∗ calculated in the same manner as θˆ but with the resample X ∗ replacing X . Step 3. Repetition. Repeat Steps 1 and 2 many times to obtain a total of ∗ bootstrap estimates θˆ1∗ , . . . , θˆB . Typical values for 200 [Efron & Tibshirani (1993)]. c A.M. Zoubir


21

B

B are between 25 and




22

Step 4. Estimation of the variance of θˆ. Estimate the variance σ 2ˆ of θˆ by θ

σˆ

2

BOOT

2 P P B B ˆ∗ 1 1 ∗ ˆ = B−1 b=1 θb − B b=1 θb .

➠ Suppose Fµ,σ is N (10, 25) and we wish to estimate σµˆ based on a random sample X of size n = 50. ➠ Following the above procedure with B = 25, a bootstrap estimate of the 2 = 0.49 as compared to the true σµ2ˆ = 0.5. variance of µ ˆ is found to be σ ˆBOOT c A.M. Zoubir


22



replacements 23

X = {X1 , . . . , Xn }


Observations

Resamples

X1∗

X2∗

XB∗

Bootstrap statistics

θˆ1∗

θˆ2∗

∗ θˆB

σ ˆθ2∗ ˆ∗

Variance estimation

c A.M. Zoubir


23

P B 1 ˆ∗ − θ = B−1 b b=1

1 B

PB ˆ∗ 2 b=1 θb



24 140

100 Frequency of occurence


120

80

60

40

20

0 0.44

∗2(1)

Histogram of σ ˆµˆ and B

c A.M. Zoubir

0.46

∗2(2)

,σ ˆµˆ

0.48

0.5

0.52 0.54 0.56 Bootstrap variance estimate

∗2(1000)

,...,σ ˆµˆ

0.58

0.6

0.62

, based on a random sample of size n

= 25. ATiSSP Sommersemester 2007

24


= 50


25


➠ Principle of the Bootstrap ➠ The Non-Parametric Bootstrap ➠ Examples ➠ Bias Estimation ➠ Variance Estimation ➠ Confidence Interval Estimation for the Mean ➠ The Parametric Bootstrap ➠ Confidence Interval Estimation for the Mean ➠ An Example of Bootstrap Failure c A.M. Zoubir


25


Example: Confidence Interval for the Mean


26

X = {X1 , . . . , Xn } be from some unknown distribution Fµ,σ . We wish to find an estimator and a 100(1 − α)% interval for µ. Let


Let

µˆ = A confidence interval for

X1 +...+Xn . n

µ is found by determining the distribution of µ ˆ, and

finding µ ˆL , µ Û such that

Pr(ˆ µL

≤ µ ≤ µˆ U ) = 1 − α.

The distribution of µ ˆ depends on the distribution of the Xi ’s, which is unknown. If

n is large, the distribution of µ ˆ could be approximated by the normal distribution as per the central limit theorem. What if n is small? c A.M. Zoubir


26



27


Step 0. Experiment.

Conduct

the

experiment

and

collect

X1 , . . . , Xn into X . Suppose Fµ,σ is N (10, 25) and X = {−2.41, 4.86, 6.06, 9.11, 10.20, 12.81, 13.17, 14.10, 15.77, 15.79} is of size n = 10. The mean of all values in X is µ ˆ = 9.95. Step 1. Resampling.

Draw a sample of 10 values,

with replace-

X. One might obtain the bootstrap resample X ∗ = {9.11, 9.11, 6.06, 13.17, 10.20, −2.41, 4.86, 12.81, −2.41, 4.86}.

ment, from

Note that some values from the original sample appear more than once while others do not appear at all. Step 2. Calculation of the bootstrap estimate. Calculate the mean of X ∗ . The mean of all 10 values in X ∗ is µ ˆ∗1

c A.M. Zoubir


27

= 6.54.



28

Step 3. Repetition. Repeat Steps 1 and 2 to obtain

N bootstrap estimates


ˆ∗N . Let N = 1000. µ ˆ∗1 , . . . , µ µ ˆ. Sort the bootstrap estiˆ∗(N ) . We might get 3.48, ˆ∗(2) ≤ . . . ≤ µ mates to obtain µ ˆ∗(1) ≤ µ 3.39, 4.46, . . . , 8.86, 8.88, 8.89, . . . , 10.07, 10.08, . . . , 14.46, 14.53, 14.66.

Step 4. Approximation of the Distribution of

− α)% bootstrap confidence interˆ∗(q2 ) ), where q1 = ⌊N α/2⌋ and q2 = N − q1 + 1. For val is (ˆ µ∗(q1 ) , µ α = 0.05 and N = 1000, q1 = 25 and q2 = 976, and the 95% confidence interval is found to be (6.27, 13.19) as compared to the theoretical (6.85, 13.05).

Step 5. Confidence Interval. The 100(1

c A.M. Zoubir


28



29 0.3

0.2 Density function


0.25

0.15

0.1

0.05

0 4

Histogram of

6

8

10 Bootstrap estimates

12

14

16

ˆ∗1000 , based on the random sample X = {−2.41, 4.86, ˆ∗2 , . . . , µ µ ˆ∗1 , µ

6.06, 9.11, 10.20, 12.81, 13.17, 14.10, 15.77, 15.79}, together with the density function of a Gaussian variable with mean 10 and variance 2.5. c A.M. Zoubir


29



30 1 0.9 0.8

Density function


0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 −2

−1.5

−1

−0.5 0 0.5 Boostrap estimates

1

1.5

2

Histogram of 1000 bootstrap estimates of the mean of the t4 -distribution and the kernel probability density function obtained from 1000 Monte Carlo simulations. The 95 % confidence interval is (-0.896,0.902) and (-0.886,0.887) based on the bootstrap and Monte Carlo, respectively. c A.M. Zoubir


30



31


➠ The procedure described above can be substantially improved because the interval calculated is, in fact, an interval with coverage less than the nominal value [Hall (1988)].

➠ Later, we shall discuss another way that will lead to a more accurate confidence interval for the mean.

➠ The computational expense to calculate the confidence interval for µ is approximately N times greater than the one needed to compute µ ˆ. ➠ This is acceptable given the ever-increasing capabilities of today’s computers.

c A.M. Zoubir


31



32




32


The Parametric Bootstrap Signal Processing Group

33


➠ Bootstrap sampling can be carried out parametrically. ➠ If one has partial knowledge of F , one may use Fˆθˆ instead of Fˆ . ➠ Draw N samples of size n from the parametric estimate of F

Fˆθˆ −→ (x∗1 , x∗2 , . . . , x∗n) and proceed as before.

➠ When used in a parametric way, the bootstrap provides more accurate answers, provided the model is correct.

c A.M. Zoubir


33



34




34


Example: Confidence Interval for the Mean


35


Step 0. Experiment.

Conduct

the

experiment

and

collect

X1 , . . . , Xn into X . Suppose Fµ,σ is N (10, 25) and X = {−2.41, 4.86, 6.06, 9.11, 10.20, 12.81, 13.17, 14.10, 15.77, 15.79} is of size n = 10. The mean of all values in X is µ ˆ = 9.95 and the sample variance is σ ˆ 2 = 33.15. Step 1. Resampling.

Draw

a

sample

of

10

values,

with

Fµˆ,ˆσ . We might obtain X ∗ = {7.45, 0.36, 10.67, 11.60, 3.34, 16.80, 16.79, 9.73, 11.83, 10.95}.

replacement,

from

Step 2. Calculation of the bootstrap estimate. Calculate the mean of X ∗ . The mean of all 10 values in X ∗ is µ ˆ∗1

c A.M. Zoubir


35

= 9.95.



36


Step 3. Repetition. Repeat Steps 1 and 2 to obtain

N bootstrap estimates

ˆ∗N . Let N = 1000. µ ˆ∗1 , . . . , µ Step 4. Approximation of the Distribution of µ ˆ. Sort the bootstrap estimates to obtain µ ˆ∗(1)

ˆ∗(N ) . ≤µ ˆ∗(2) ≤ . . . ≤ µ

− α)% bootstrap confidence interˆ∗(q2 ) ), where q1 = ⌊N α/2⌋ and q2 = N − q1 + 1. For val is (ˆ µ∗(q1 ) , µ α = 0.05 and N = 1000, q1 = 25 and q2 = 976, and the 95% confidence interval is found to be (6.01, 13.87) as compared to the theoretical (6.85, 13.05).

Step 5. Confidence Interval. The 100(1

c A.M. Zoubir


36



37 0.3

0.2 Density function


0.25

0.15

0.1

0.05

0

Histogram of

4

6

8

10 Bootstrap estimates

12

14

16

ˆ∗1000 , based on the random sample X = {−2.41, 4.86, ˆ∗2 , . . . , µ µ ˆ∗1 , µ

6.06, 9.11, 10.20, 12.81, 13.17, 14.10, 15.77, 15.79}, together with the density function of a Gaussian variable with mean 10 and variance 2.5. c A.M. Zoubir


37



38




38


An Example of Bootstrap Failure Signal Processing Group

39

X ∼ U(0, θ) and X = {X1 , X2 , . . . , Xn }. We wish to estimate θ by θˆ ˆ . The Maximum Likelihood (ML) estimator of θ is given ˆˆ (θ) and its distribution F Θ by θˆ = X(n) .


Let

➠ To obtain an estimate of the density function of θˆ we sample with replacement from the data and each time estimate θˆ∗ from X ∗ . ˆ and estimate θˆ∗ from X ∗ (para➠ Alternatively, we could sample from U(0, θ) metric bootstrap).

➠ We ran an example with θ = 1, n = 50 and N = 1000. The ML estimate of θ was found to be θˆ = 0.9843. c A.M. Zoubir


39


An Example of Bootstrap Failure (Cont’d)


40

The non-parametric bootstrap shows that approximately 62% of the values of θˆ∗

n −→ ∞.

ˆ = 1 − (1 − 1/n)n −→ 1 − e−1 ≈ 0.632 as = θ)

700

120

600

100

Frequency of occurrence

500 Frequency of occurrence


equal θˆ. In fact, Pr(θˆ∗

400

300

80

60

40 200

20

100

0 0.65

0.7

0.75

0.8 0.85 theta hat star

0.9

0.95

Non-parametric Bootstrap c A.M. Zoubir


40

1

0 0.65

0.7

0.75

0.8 0.85 theta hat star

0.9

Parametric Bootstrap


0.95

1


41




41


The Dependent Data Bootstrap Signal Processing Group

42


➠ The assumption of i.i.d. data can break down in practice either because the data is not independent or because it is not identically distributed, or both.

➠ We can still invoke the bootstrap principle if we knew the model that generated the data [Efron & Tibshirani (1993), Bose (1988), Kreiss & Franke (1992), Paparoditis (1996), Zoubir (1993)].

➠ For example, a way to relax the i.i.d. assumption is to assume that the data is identically distributed but not independent such as in autoregressive (AR) models.

c A.M. Zoubir


42


The Dependent Data Bootstrap (Cont’d)


43

➠ If no plausible model such as AR is available for the probability mechanism Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

generating stationary observations, we could make the assumption of weak dependence.

➠ Strong mixing processesa , for example, satisfy the weak dependence condition.

➠ The moving blocks bootstrap [Kunsch (1989), Liu & Singh (1992), Politis & ¨ Romano (1992,1994)] has been proposed for bootstrapping weakly dependent data. a

Loosely speaking a process is strong mixing if observations far apart (in time) are almost

independent [Rosenblatt (1985)].

c A.M. Zoubir


43


The Moving Blocks Bootstrap Signal Processing Group

44 Observed Data

Amplitude

5

5

−5

4

1

3

0

10

20

30

40 50 60 Resampled Blocks

70

80

90

100

Amplitude

5

1

2

3

4

5

0

−5

0

10

20

30

40

50 60 Bootstrap Data

70

80

90

100

0

10

20

30

40

50 60 Sample number

70

80

90

100

5 Amplitude


2

0

0

−5

The moving block bootstrap randomly selects blocks of the original data (top) and concatenate them together (centre) to form a resample (bottom). In this example, the block size is l c A.M. Zoubir

= 20 and n = 100. ATiSSP Sommersemester 2007

44


Other Block Bootstrap Methods Signal Processing Group


45

➠ The circular blocks bootstrap [Politis & Romano (1992), Shao & Yu (1993)] allows blocks which start at the end of the data and wrap around to the start.

➠ The blocks of blocks bootstrap [Politis et al. (1992)] uses two levels of blocking to estimate confidence bands for spectra and cross-spectra.

➠ The stationary bootstrap [Politis & Romano (1994)] allows blocks to be of random lengths instead of a fixed length.

c A.M. Zoubir


45



46


➠ Principle of the Bootstrap ➠ The Dependent Data Bootstrap ➠ An Example: Variance Estimation for AR Parameter Estimators ➠ The Principle of Pivoting ➠ An Example: Confidence Interval for the Mean ➠ Variance Stabilisation ➠ Examples ➠ Correlation Coefficient c A.M. Zoubir


46


An Example: Variance Estimation in AR Models 47

We generate n observations xt , t


= 0, . . . , n − 1, from


Xt + a · Xt−1 = Zt , Zt is white Gaussian noise with EZt = 0, cZZ (u) = σZ2 δ(u), and a such that |a| < 1. where

After de-trending the data, we fit the AR(1) model to the observation

xt . With

Pn−|u|−1

xt xt+|u| for 0 ≤ |u| ≤ n − 1, we calculate the ˆ = −ˆ cxx (1)/ˆ cxx (0), which has Maximum Likelihood Estimate (MLE) of a, a approximate variance‡ σ âˆ2 = (1 − a2 )/n. cˆxx (u) = 1/n

t=0

σ âˆ2 can be found in the nonGaussian case and is a function of a and the variance and kurtosis of Zt [Porat & Friedlander ‡

under some regularity conditions an asymptotic formula for

(1989)]. c A.M. Zoubir


47



48

n observations xt , t = 0, . . . , n − 1, from an auto-regressive process of order one, Xt .


Step 0. Experiment.

Conduct the experiment and collect

Step 1. Calculation of the residuals. With the Maximum Likelihood Estimate a ˆ of a, define the residuals zˆt Step 2. Resampling.

= xt + a ˆ · xt−1 for t = 1, 2, . . . , n − 1.

Create a bootstrap sample

x∗0 , x∗1 , . . . , x∗n−1 by

∗ , with replacement, from the residuals zˆ1∗ , zˆ2∗ , . . . , zˆn−1 ax∗t−1 + zˆt∗ , t = zˆ1 , zˆ2 , . . . , zˆn−1 , then letting x∗0 = x0 , and x∗t = −ˆ 1, 2, . . . , n − 1.

sampling

c A.M. Zoubir


48



49

Step 3. Calculation of the bootstrap estimate.

After centring the time se-

ˆ∗ , using the above formulae but based on x∗0 , x∗1 , . . . , x∗n−1 , obtain a x∗0 , x∗1 , . . . , x∗n−1 .


ries

Step 4. Repetition. Repeat steps 2–3 a large number of times,

N = 1000,

ˆ∗N . ˆ∗2 , . . . , a say, to obtain a ˆ∗1 , a ˆ∗N , approximate the variance ˆ∗2 , . . . , a Step 5. Variance estimation. From a ˆ∗1 , a of a ˆ by

σˆ aˆ∗2 c A.M. Zoubir

=


1 N −1

49

PN

∗ (ˆ a i=1 i

−

1 N

PN

∗ 2 a ˆ i=1 i ) .



50 6

4 Density function


5

3

2

1

0 −0.8

−0.7

ˆ∗1000 for ˆ∗2 , . . . , a Histogram of a ˆ∗1 , a for a was a ˆ

−0.5 −0.4 Bootstrap estimates

−0.3

−0.2

−0.1

a = −0.6, n = 128 and Zt Gaussian. The MLE

= −0.6351 and σ âˆ = 0.0707. The bootstrap estimate was σ âˆ∗ = 0.0712

as compared to σ âˆ c A.M. Zoubir

−0.6

= 0.0694 based on 1000 Monte Carlo simulations.


50



51




51


The Principle of Pivoting Signal Processing Group


52

➠ A statistic T (X, θ) is called pivotal if it possesses a fixed probability distri´ (1967), Lehmann (1986)]. bution independent of θ [Cramer ➠ Bootstrap confidence intervals or tests have excellent properties even for relatively low fixed resample number [Hall (1992)].

➠ For example, one can show that the coverage error in confidence interval estimation with the bootstrap is Op (n−1 ) as compared to Op (n−1/2 ) when using the normal approximation.

➠ The accuracy claimed holds whenever the statistic is asymptotically pivotal [Hall & Titterington (1989), Hall (1992)].

c A.M. Zoubir


52



53




53


An Example: Confidence Interval Estimation


54


We consider the construction of a confidence interval for the mean. Let

X =

{X1 , . . . , Xn } be a random sample from some unknown FµX ,σX . We wish to find an estimator of µX with a 100(1 − α)% confidence interval. Let

2 µ ˆX and σ ˆX be the sample mean and the sample variance of X , respec-

tively. Alternatively to the previous example, we will base our method for finding a confidence interval for µX on the statistic

µˆ Y =

µ ˆX −µX σ ˆ

,

σ ˆ is the standard deviation of µ ˆX . The statistic has asymptotically for large n a distribution free of unknown parameters. where

c A.M. Zoubir


54



55

Step 0. Experiment. Conduct the experiment and collect the random data into


the sample X

= {X1 , X2 , . . . , Xn }.

Step 1. Parameter estimation. Based on deviation σ ˆ , using a nested bootstrap.

X , calculate µ ˆX and its standard

Step 2. Resampling. Draw a random sample, ment, from X .

X ∗ of n values, with replace-

Step 3. Calculation of the pivotal statistic. Calculate the mean of all values in

X ∗ and using a nested bootstrap, calculate σ ˆ ∗ . Then, form

µˆ ∗Y = c A.M. Zoubir


55

µX µ ˆ∗X −ˆ σ ˆ∗



56

Step 4. Repetition. Repeat Steps 2-3 many times to obtain a total of

N boot-


strap estimates µ ˆ∗Y,1 , . . . , µ ˆ∗Y,N . Step 5. Ranking. Sort the bootstrap estimates to obtain

... ≤ µ ˆ∗Y,(1000) .

ˆ∗Y,(2) ≤ µ ˆ∗Y,(1) ≤ µ

ˆ∗Y,(q2 ) ) is an interval containing (1 − Step 6. Confidence Interval. If (ˆ µ∗Y,(q1 ) , µ α)N of the means µ ˆ∗Y , where q1 = ⌊N α/2⌋ and q2 = N − q1 + 1, then

(ˆ µX −

∗ σˆ µˆ Y,(q2), µˆ X

−

∗ σˆ µˆ Y,(q1))

is a 100(1 − α)% confidence interval for µX . The interval is called percentile-t confidence interval [Efron (1987), Hall (1988)]. c A.M. Zoubir


56



57

X as before, we obtained the confidence interval (3.54, 13.94) as compared to (6.01, 13.87). Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

For the same random sample

➠ This interval is larger than the one obtained earlier and enforces the statement that the interval obtained there has coverage less than the nominal 95%.

➠ It also yields better results than an interval derived using the assumption that µ ˆY is N (0, 1)- or the (better) approximation that µ ˆY is tn−1 -distributed. ➠ The interval obtained here accounts for skewness in the underlying population or other errors [Hall (1988,1992), Efron & Tibshirani (1993), Zoubir (1993)]. c A.M. Zoubir


57



58




58


Variance Stabilisation Signal Processing Group

59

To ensure pivoting, usually the statistic is “studentised”, i.e., we form


Tˆ =

ˆ θ−θ σ ˆθˆ

The percentile-t method is particularly applicable to location statistics, such as the sample mean, sample median, etc. [Efron & Tibshirani (1993)]. However, for more general statistics, it may not be accurate. Problem:

Studentising results in confidence intervals with erratically varying

lengths and end points. Solution: Pivoting often does not hold unless an appropriate variance stabilising transformation is applied first. How do we “automatically” get a variance stabilising transformation? c A.M. Zoubir


59


Variance Stabilisation (Cont’d) Signal Processing Group

60


Step 1. Estimation of the variance stabilising transformation. (a) Generate B1 bootstrap samples Xi∗ from X and for each calculate the value of the statistic θî∗ , i

= 1, . . . , B1 . For example B1 = 100.

B2 bootstrap samples from Xi∗ , i = 1, . . . , B1 , and calculate σ î∗2 , a bootstrap estimate for the variance of θî∗ , i = 1, . . . , B1 . For example B2 = 25.

(b) Generate

(c) Estimate the variance function against

ζ(θ) by smoothing the values of σ î∗2

θî∗ , using, for example, a fixed-span 50% “running lines”

smoother [Hastie & Tibshirani (1990)].

c A.M. Zoubir


60


Variance Stabilisation (Cont’d) Signal Processing Group


61

Step 1. Estimation of the variance stabilising transformation.

ˆ from (d) Estimate the variance stabilising transformation h(θ)

h(θ) =

Rθ

{ζ(s)}−1/2ds.

Step 2. Bootstrap quantile estimation. Generate

B3 bootstrap samples and

h(θî∗ ) for each sample i. Approximate the distribution of ˆ − h(θ) by that of h(θˆ∗ ) − h(θ) ˆ. h(θ)

compute θî∗ and

c A.M. Zoubir


61



62


➠ Principle of the Bootstrap ➠ The Dependent Data Bootstrap ➠ An Example: Variance Estimation for AR Parameter Estimates ➠ The Principle of Pivoting ➠ An Example: Confidence Interval for the Mean ➠ Variance Stabilisation ➠ Examples ➠ Correlation Coefficient

c A.M. Zoubir


62


Example: Correlation coefficient Signal Processing Group


63

θ = ̺ be the correlation coefficient of two unknown populations, and let ̺ˆ and σ ˆ 2 be estimates of ̺ and the variance of ̺ˆ, respectively, based on X = {X1 , . . . , Xn } and Y = {Y1 , . . . , Yn }.

Let

X ∗ and Y ∗ be resamples, drawn with replacement from X and Y , respectively, and let ̺ ˆ∗ and σ ˆ ∗2 be bootstrap versions of ̺ˆ and σ ˆ2.

Let

By repeated resampling from

00

- Tˆ (X)

- Tˆ ≷ λ

K H

Receiver Structure c A.M. Zoubir


91


Receiver

Signal Detection with the Matched Filter


92

-


x

- ×

Ps

6

-

1 √ σ s′ s

- ˆ T

s

Tˆ =

c A.M. Zoubir

s√′ x σ s′ s

=


s′√ Ps x σ s′ s

92

∼N

θ

√

s′ s ′ ′ , 1 , P = ss /s s . s σ


Performance of the Matched Filter Signal Processing Group

93

−5

=1 0

=1 0 FA

P

=1 P

FA

−4

0

−3 FA

P

FA

=1 0

−2

P

FA

P

FA

P 0.99

Detection probability


=1 0

=1 0

−1

0.999

−6

0.9999

0.95 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.05

0.01

0

1

2

3

4

5

6

7

8

9

Voltage signal−to−noise ratio

c A.M. Zoubir


93


Signal Detection with the CFAR Matched Filter


94

s

? -×


-P s

x

′

√

s′ s ·? ·

- I −P

s

-k · k2

-×

-

6

6

√

1/(n − 1)

T = c A.M. Zoubir

√ ′ P x σ 2 s′ s s √ ′ s x (I−Ps )x/(n−1)σ 2


94

∼ tn−1

√

-≷ λ

θ s′ s σ


Performance of the CFAR Matched Filter 95

0.9999

N=∞

N=21 N=11

0.99

Detection probability


0.999

N=6

N=3

0.95 0.9 0.8

N=2

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.05

0.01

0

1

2

3

4

5

6

7

8

9

Voltage signal−to−noise ratio

c A.M. Zoubir


95



Limitations of the Matched Filter Signal Processing Group

96

➠ The matched filter and the CFAR matched filter are designed (and optimal) Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

for Gaussian interference.

➠ Although they show high probability of detection in the non-Gaussian case, they are unable to maintain the preset level of significance for small sample sizes.

➠ The matched filter fails in the case where the interference/noise is non-Gaussian and the data size is small.

➠ The goal is to develop techniques which require little in the way of modelling and assumptions.

c A.M. Zoubir


96



97





97


The Bootstrap in Signal Detection: Motivation


98

➠ The Bootstrap can provide solutions for detecting signals in non-Gaussian Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

interference, e.g., high-resolution radar and radar at low grazing angles.

➠ The bootstrap is particularly attractive when only a limited number of samples is available for signal detection.

➠ The bootstrap is also attractive when detection is associated with estimation, e.g., estimation of the number of signals.

➠ The bootstrap has been successfully applied to signal detection in areas such as wireless communications, landmine detection, radar, sonar and biomedical engineering.

c A.M. Zoubir


98


The Bootstrap in Signal Detection: Principle 99

Measured Data

-


Conduct Experiment

-

Compute

ˆ

T -

-

Statistic

x

? Resample

Bootstrap Data

-

x∗ 1 x∗ 2

-

c A.M. Zoubir


Tˆ1∗

-

Statistic

-

Compute

Tˆ2∗

-

Statistic

. . .

-

Compute

x∗ B

99

-

ˆ ∗ , . . . , Tˆ ∗ Sort T 1 B to get

∗ ∗ Tˆ(1) ≤ . . . ≤ Tˆ(B) Set λ

. . . Compute


∗ , = Tˆ(q)

q = ⌈(1 − α)(B + 1)⌉ ∗ TˆB

-

Statistic


Decision

The Bootstrap Matched Filter Signal Processing Group

100

Step 0. Experiment.

Run the experiment and collect the data

xt , t =


0, . . . , n − 1.

ˆθˆ, and Step 1. Estimation. Compute the LSE θˆ of θ , σ

ˆ θ−θ ˆ T = σˆ ˆ θ

θ=0

.

Step 2. Resampling. Compute the residuals

ˆ t, w ˆt = xt − θs

t = 0, . . . , n − 1,

and after centering, resample the residuals, assumed to be i.i.d., to obtain

w ˆt∗ , t = 0, . . . , n − 1. c A.M. Zoubir


100


The Bootstrap Matched Filter (Cont’d)


101

Step 3. Bootstrap Test Statistic. Compute new measurements


ˆ t + wˆ ∗ , x∗t = θs t

t = 0, . . . , n − 1 ,

the LSE θˆ∗ based on the resamples x∗t , t

Tˆ ∗ =

= 0, . . . , n − 1, and

θˆ∗ −θˆ . ∗ σ ˆ ˆ∗ θ

Step 4. Repetition. Repeat Steps 2 and 3 a large number of times to obtain

Tˆ1∗ , . . . , TˆN∗ . ∗ ∗ Step 5. Bootstrap Test. Sort Tˆ1∗ , . . . , TˆN to obtain Tˆ(1)

H if Tˆ c A.M. Zoubir

∗ , where q = ⌊(1 − α)(N + 1)⌋. ≥ Tˆ(q)


101

∗ ≤ · · · ≤ Tˆ(N ) . Reject


The Bootstrap Matched Filter (Cont’d)


102 Measured Data Estimate


Conduct Experiment

θ

x

ˆ θ

Compute Statistic

ˆ T

Decision

ˆ ˆ = x − θs w

Resample & Reconstruct Bootstrap Data

x∗ 1

Compute Statistic

ˆ∗ T 1

Compute Statistic

ˆ∗ T 2

ˆ∗ , . . . , Sort T 1

ˆ∗ T N

to get

x∗ 2

ˆ∗ ≤ . . . ≤ T ˆ∗ T (1) (N ) ˆ∗ set T

x∗ B

c A.M. Zoubir


102

Compute Statistic

ˆ∗ T B

, (q) q = ⌊(1 − α)(N + 1)⌋


Performance of the Bootstrap Matched Filter


103


➠ The bootstrap matched filter is consistent, ie, Pd → 1 as n → ∞. ➠ One can show [Ong & Zoubir (2000)] that −1

Pf = α + O p n −1/2 α+O n for the matched filter for |st | < ∞, t ∈ Z.

➠ If st = cos(ωt + φ), then

Pf = α + O p n

−3/2

(α + O (n−1 ) for the matched filter) c A.M. Zoubir


103


Simulation Results (Gaussian Case)


104


Let st

= sin(2πt/6), n = 10, N = 999 (25 for variance estimation), α = 5%,

and the number of independent runs be 5,000.

N (0, 1), SNR=7 dB Pˆf [%]

Pˆd [%]

Matched Filter (MF)

5

99

CFAR MF

5

98

Boot. (σ known)

4

99

Boot. (σ unknown)

5

98

Detector

c A.M. Zoubir


104


Simulation Results (Non-Gaussian Case)


105

Let st

= sin(2πt/6), n = 10, N = 999 (25 for variance estimation), α = 5%,


and the number of independent runs be 5,000.

P4

Wt ∼ i=1 ai N (µi , σi2 ) with a = (0.5, 0.1, 0.2, 0.2), µ = (−0.1, 0.2, 0.5, 1), σ = (0.25, 0.4, 0.9, 1.6), and SNR = -6 dB Pˆf [%]

Pˆd [%]

Matched Filter (MF)

8

83

CFAR MF

7

77

Boot. (σ known)

5

77

Boot. (σ unknown)

7

79

Detector

c A.M. Zoubir


105


Simulation Results (Cont’d) Signal Processing Group

106 (a)

2

fa

P /α

3

0 10

20

30

40

50

60

70

80

90

100

60

70

80

90

100

50 60 Sample size

70

80

90

100

(b)

2

fa

P /α

3

1 0 10

20

30

40

50 (c)

3 2

fa

P /α


1

1 0 10

20

30

40

Ratio of actual vs. nominal false alarm rate for the detection of a DC in Gaussian (top), skewed (middle) and heavy tailed (bottom) noise. c A.M. Zoubir


106



107 (a)

2

fa

P /α

3

0 10

20

30

40

50

60

70

80

90

100

60

70

80

90

100

50 60 Sample size

70

80

90

100

(b)

2

fa

P /α

3

1 0 10

20

30

40

50 (c)

3 2

fa

P /α


1

1 0 10

20

30

40

Ratio of actual vs. nominal false alarm rate for the detection of a sinusoid in Gaussian (top), skewed (middle) and heavy tailed (bottom) noise. c A.M. Zoubir


107



108





108


Tolerance Interval Bootstrap Matched Filter 109


➠ The thresholds calculated in the bootstrap detectors are random variables


conditional on the data.

➠ We would like to set β to specify our confidence that the set level is maintained. ∗ must fulfill ➠ The required threshold T(r) ∗ |H) ≤ α) ≥ β. Pr(Pr(Tˆ > T(r)

➠ Distribution free tolerance interval theory gives r as the smallest positive integer which satisfies the relation

FB (α; B − r + 1, r) ≥ β

where FB (x; λ1 , λ2 ) is the beta cdf with parameters λ1 , c A.M. Zoubir


109

λ2 .


Aside - Tolerance Interval Theory Signal Processing Group


110

➠ Problem : Given an iid sample X1 , . . . , Xn , find two order statistics, X(r) and X(n−s+1) such that [X(r) , X(n−s+1) ] will encompass at least 100(1 − α)% of the underlying distribution, F (x), with probability at least β , R X(n−s+1) dF (x) ≤ 1 ≥ β. Pr 1 − α ≤ X(r) ➠ From the theory of distribution free tolerance intervals we must find r and s such that

1 − FB (1 − α; n − r − s + 1, r + s) ≥ β, where FB (1 − α; λ1 , λ2 ) is the beta cdf with parameters λ1 and λ2 [Wilks

1941].

c A.M. Zoubir


110


Tolerance Interval Bootstrap Matched Filter



111

➠ Proceed through steps 0 to 4 of previous bootstrap matched filter procedure. ➠ Step 5 is modified for tolerance intervals : Step 5. Bootstrap Test. Let the α be the level of significance, β , be the probability that this level is kept and B be the number of bootstrap resamples. ∗ ∗ T1∗ , . . . , TB∗ into increasing order to obtain T(1) . Re≤ · · · ≤ T(B) ject H if Tˆ ≥ Tˆ ∗ , where r is the smallest positive integer which satisfies

Sort

(r)

FB (α; B − r + 1, r) ≥ β .

c A.M. Zoubir


111


Simulations Signal Processing Group

112


➠ A signal, s = sin(2πt/9), t = 0, . . . , 9, i.e., n = 10, in Gaussian and Laplace noise. B = 1000, SNR = 5dB, α = 0.05 and β = 0.95. ➠ Estimated β using bootstrap detectors with and without tolerance intervals. β (nominal 0.95)

c A.M. Zoubir


Gaussian

Laplace

BMF

0.5146

0.4003

BCMF

0.4684

0.5549

TBMF

0.9643

0.9398

TBCMF

0.9584

0.9734

112


Example - Real Data Signal Processing Group


113

➠ Detection of sinusoids in non-stationary, non-Gaussian noise, α = 0.05, β = 0.95, window length of 21 samples. BCMF PFA = 0.0562, TBCMF PFA = 0.0463. 5

1.5

x 10

1

0.5

0

−0.5 data, α=0.05 β=0.95 target present BMF detection TBMF detection

−1

−1.5 1000

c A.M. Zoubir


1200

113

1400

1600

1800

2000 sample

2200

2400

2600

2800

3000


Summary Signal Processing Group

114

➠ Bootstrap detectors, on average, keep the false alarm rate close to the set Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

level when :

➠ The underlying noise distribution is unknown and probably non-Gaussian. ➠ The noise exhibits some non-stationarity. ➠ Using distribution free tolerance intervals we can specify our confidence that the false alarm rate will not exceed the set level.

➠ Simulations verify the accuracy of tolerance interval bootstrap detection. ➠ With real non-Gaussian non-stationary data the tolerance interval based bootstrap detector maintains the set level. c A.M. Zoubir


114



115





115


Applications in Signal Processing Signal Processing Group

116

The relatively simple examples showed that the bootstrap is powerful. This sugInstitut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

gests its use in real-life applications and more complex problems. We show now how the bootstrap can be applied for:

➠ Noise Floor Estimation in Radar [Zoubir & Boashash (1996)] ➠ Confidence Intervals for Flight Parameters in Passive Sonar [Reid et al. (1996), Zoubir & Boashash (1998)].

➠ Detection of Buried Landmines Using GPR [Zoubir et al. (1999,2000,2001,2002)]. ➠ Image Processing [Zoubir & Iskander (2007)]. ➠ Micro-Doppler Radar [Zoubir & Iskander (2007)]. c A.M. Zoubir


116



117

➠ Signal Detection ➠ Bootstrap Matched-Filter


➠ Tolerance Interval Bootstrap Matched-Filter ➠ Applications in Signal processing ➠ Noise Floor Estimation in OTH-Radar ➠ Confidence Intervals for Flight Parameters in Passive Sonar ➠ Detection of Buried Landmines Using GPR ➠ Image Processing ➠ Micro-Doppler Radar ➠ Summary ➠ Acknowledgements c A.M. Zoubir


117


Noise Floor Estimation in OTH-Radar


118


IONOSPHERE

F LAYER

E LAYER

TRANSMITTER BEAM

BEAM TO RECEIVER FROM TARGET

350 km 150 km

TRANSMITTER EARTH’S SURFACE

800-3000 km RECEIVER

c A.M. Zoubir


118


Noise Floor Estimation in OTH-Radar (Cont’d)


119

➠ The noise floor is one of the main radar performance statistics. Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

➠ It is computed by taking a trimmed mean of power estimates in the Doppler spectrum, after excluding the low Doppler power associated with any stationary targets or ground clutter.

➠ The radar operator requires an optimal value for the trim and also supplementary information describing the accuracy of the computed noise floor estimate.

➠ The limited number of samples available and the non-Gaussian nature of the noise field make it particularly difficult to estimate confidence measures reliably. This suggests the use of the bootstrap. c A.M. Zoubir


119


Optimal Selection of the Trim Signal Processing Group


120

➠ The goal is to use the bootstrap to optimally select the amount of trimming, γ , say. ➠ The problem can be stated as one of estimation of θ from a class of estimates ˆ {θ(γ) : γ ∈ A} indexed by a parameter γ , which is selected according to some optimality criterion.

➠ It is reasonable to use the estimate that leads to a minimum variance. Beˆ may depend on unknown parameters, the optimal cause the variance of θ(γ) parameter γo is unknown.

c A.M. Zoubir


120


Optimal Selection of the Trim (cont’d)



121

ˆ ➠ The objective is to find γ by estimating the unknown variance of θ(γ) and select γ which minimises the variance estimate. ➠ The problem of trimming has been solved theoretically [Jaeckel (1971)] for the mean of symmetric distributions. The trim is, however, restricted to the interval [0, 25]%.

➠ In addition to the fact that for a small sample size asymptotic results are invalid, explicit expressions for the asymptotic variance may not be available outside the interval [0, 25]%.

c A.M. Zoubir


121


An Example: Selection of the Trim for the Mean


122


➠ Let X = {X1 , . . . , Xn } be a random sample drawn from a distribution function F . ➠ Let X(1) , . . . , X(n) denote the order statistics. ➠ For an integer γ less then n/2, the γ -trimmed mean based on the sample X is given by

µ ˆ(γ) =

1 n−2γ

Pn−γ

i=γ+1 X(i)

➠ For an asymmetric distribution we use the γκ-trimmed mean:

µ ˆ(γ, κ) =

c A.M. Zoubir


122

1 n−γ−κ

Pn−κ

i=γ+1 X(i) ,

γ, κ < n/2 .


Example Signal Processing Group

123


Step 1. Initial conditions. Select the initial trim γ

= 0 and κ = 0.

Step 2. Resampling. Using a pseudo-random number generator, draw a large number, say N , of independent samples

∗ ∗ }, . . . , XN∗ = {XN∗ 1 , . . . , XN∗ n } , . . . , X1n X1∗ = {X11 of size n from X . Each sample is taken with replacement. Step 3. Calculation of the bootstrap statistic. For each bootstrap sample

j = 1, . . . , N , calculate the trimmed mean

µ ˆ∗j (γ, κ) c A.M. Zoubir

=


1 n−γ−κ

123

Pn−κ

∗ X i=γ+1 j(i) ,

Xj∗ ,

j = 1, . . . , N.


Example (cont’d) Signal Processing Group

124


Step 4. Variance estimation. Using bootstrap based trimmed mean values, calculate the estimate of the variance

σ ˆ ∗2 =

1 N −1

PN j=1

µ ˆ∗j (γ, κ) −

1 N

2 ∗ . µ ˆ j=1 j (γ, κ)

PN

Step 5. Repetition. Repeat steps 2–4 using different combinations of the trim

γ and κ. Step 6. Optimal trim. Choose the setting of the trim that results in a minimal variance estimate.

c A.M. Zoubir


124


Noise Floor Estimation in OTH-Radar(Cont’d) 125


➠ In radar, we estimate residuals for the Doppler spectrum by taking the ratio of the periodogram and a spectral estimate of the raw data. The residuals are


used for resampling to generate bootstrap spectral estimates as in a previous example.

➠ We proceed as in the example for the trimmed mean, replacing Xi by CˆXX (ωi ), where ωi = 2πi/n, i = 1, . . . , n, are discrete frequencies and n is the number of observations.

➠ The procedure makes the assumption that the spectral estimates are i.i.d. for distinct discrete frequencies.

➠ The optimal noise floor is found by minimising the bootstrap variance estimate of the trimmed spectrum w.r.t. (γ, κ). c A.M. Zoubir


125





126

Step 1. Initial conditions. Select the initial trim γ

= κ = 0.

B , of indepenˆ ∗(1) (ω1 ), . . . , Cˆ ∗(1) (ωM )}, . . . , X ∗ = dent samples X1∗ = {C B XX XX ∗(B) ∗(B) {CˆXX (ω1 ), . . . , CˆXX (ωM )}

Step 2. Resampling.

Draw a large number,

say

Step 3. Calculation of the trimmed mean. For each bootstrap sample Xj∗ , j

1, . . . , B , calculate the trimmed mean µ ˆ∗j (γ, κ)

c A.M. Zoubir

=


1 M −γ−κ

126

∗(j) ˆ i=γ+1 CXX (ωi ),

Pn−κ

j = 1, . . . , B.


=

Optimal Noise Floor in OTHR (cont’d)



127

Step 4. Variance estimation. Using bootstrap based trimmed mean values, calculate the estimate of the variance

σ ˆ ∗2 =

1 B−1

PB j=1

µ ˆ∗j (γ, κ) −

1 B

2 ∗ µ ˆ . j=1 j (γ, κ)

PB

Step 5. Repetition. Repeat steps 2–4 using different combinations of the trim

γ and κ. Step 6. Optimal trim. Choose the setting of the trim that results in a minimal variance estimate.

c A.M. Zoubir


127


Optimal Noise Floor in OTHR Signal Processing Group

128 X0 , . . . , Xn−1


Measurements

Variance estimation

∗2 σ ˆµ ˆ ∗ (γ,κ) =

1 = M − Mγ − Mκ

MX −M κ

1

N X

N − 1 j=1

0

128

N 1 X

N j=1

∗

12

µ ˆ j (γ, κ)A

ˆ ∗(j) (ω(i) ) C XX

i=M γ+1

γ,κ


µ ˆ∗ B (γ, κ)

∗ @µ ˆ j (γ, κ) −

Find arg min

c A.M. Zoubir

∗ XB

ˆ∗ µ ˆ∗ 2 (γ, κ) 1 (γ, κ) µ

Trimmed mean of resamples for (γ, κ)

µ ˆ∗ b (γ, κ)

X2∗

X1∗

Spectral resamples

∗2 σ ˆµ ˆ ∗ (γ,κ)


How to Bootstrap Power Spectra? Signal Processing Group

129

We will consider a kernel estimate of the spectrum CXX (ω) of a stationary signal


Xt , t = 0, . . . , n − 1, CˆXX (ω; h) =

1 nh

PM

k=−M

➠ K(·) is symmetric and nonnegative,

K

ω−ωk h

IXX (ωk ) ,

➠ IXX (ωk ) is the periodogram at ωk ➠ ωk = 2πk/n, −M ≤ k ≤ M . ➠ h is the bandwidth of the kernel, ➠ M denotes the largest integer ≤ n/2 c A.M. Zoubir


129


How to Bootstrap Power Spectra? (Cont’d) 130


➠ Asymptotically for large n and with m = ⌊(h n − 1)/2⌋,


CˆXX (ω1 ), . . . , CˆXX (ωM ) are independently ∼ CXX (ωk ) χ24 m+2 /(4 m + 2), k

= 1, . . . , M .

¨ ➠ Alternatively, we use (see [Franke & Hardle (1992)]) the approximate regression

IXX (ωk ) , k = 1, . . . , M , εk = CXX (ωk ) ˆXX (ω) or of functions thereof. to find the distribution of C c A.M. Zoubir


130


How to Bootstrap Power Spectra? (Cont’d)



131

Step 1. Compute Residuals. Choose an hi

> 0 which does not depend on ω

and compute

εˆk =

IXX (ωk ) ˆXX (ωk ;hi ) , C

k = 1, . . . , M.

Step 2. Rescaling. Rescale the empirical residuals to

ε˜k =

εˆk , εˆ

k = 1, . . . , M, εˆ =

1 M

PM

ˆj j=1 ε

.

Step 3. Resampling. Draw independent bootstrap residuals ε˜∗1 , . . . , ε˜∗M from the empirical distribution of ε˜1 , . . . , ε˜M .

c A.M. Zoubir


131





132

Step 4. Bootstrap estimates. With a bandwidth g , find ∗ IXX (ωk ) = CˆXX (ωk ; g) ε˜∗k , k = 1, . . . , M, ∗ P M ω−ω 1 ∗ k CˆXX (ω; h) = n h k=−M K IXX (ωk ). h

For confidence bands estimation, we would repeat Steps 3 − 4 and find c∗U (and

proceed similarly for c∗L ) such that Pr∗

c A.M. Zoubir

√


ˆXX (ω;g) ˆ ∗ (ω;h)−C C XX nh ˆXX (ω;g) C

132

≤ c∗U = γ .




133 Measured Data Conduct Experiment

x

` ´ IXX ωk

Find Periodogram

Smooth Periodogram

` ´ ˆ C XX ωk


k = 1, . . . , M `

ǫ ˆ ωk

´

´ ` IXX ωk ´ ` = ˆ ωk C XX

Resample Bootstrap Data

` ´ ǫ ˆ∗ 1 ωk ´ ` ǫ ˆ∗ 2 ωk

´ ` ǫ ˆ∗ B ωk

c A.M. Zoubir


133

Reconstruct & Estimate



´ ` ˆ ∗(1) ω C k XX ` ´ ˆ ∗(2) ω C k XX

Confidence Sort

` ´ ˆ ∗(1) ω C ,... k XX ` ´ ˆ ∗(B) ω C k XX

´ ` ˆ ∗(N ) ω ...,C k XX


Interval



134 70

C XX C

NN

TM

50 Power Spectrum [dB]


60

40

30

20

10 −0.5

−0.4

−0.3

−0.2

−0.1 0 0.1 Normalized Doppler Frequency

0.2

0.3

0.4

0.5

Doppler spectrum and estimated noise floor for a real over-the-horizon radar return (no target present). With B c A.M. Zoubir


ˆ γ, κ = 99, θ(ˆ ˆ ) = 2.0738 × 103 . 134


Noise Floor Estimation in OTH-Radar (Cont’d)


135 70

C XX C

65

NN

TM

Target

55 Power Spectrum [dB]


60

50

45

40

35

30

25

20 −0.5

−0.4

−0.3

−0.2

−0.1 0 0.1 Normalized Doppler Frequency

0.2

0.3

0.4

0.5

Doppler spectrum and estimated noise floor for a real over-the-horizon radar return (target present). With B c A.M. Zoubir

ˆ γ, κ = 99, θ(ˆ ˆ ) = 8.9513 × 103 .


135



136





136


Confidence Intervals for an Aircraft’s Flight Parameters 137


OVERFLYING AIRCRAFT

ACOUSTIC EMISSION

MICROPHONE

Schematic of the passive acoustic scenario.

c A.M. Zoubir


137



The Passive Acoustic Approach Signal Processing Group

138

➠ Information about the flight parameters is contained in the acoustic signal as Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

heard by the stationary observer.

➠ The frequency of the observed acoustic signal s(t) from the over-flying aircraft undergoes a time varying Doppler shift.

➠ Thus, the phase of s(t) undergoes a time varying rate of change. A simple model for the aircraft acoustic signal, as heard by a stationary observer, is given by j2π

X(t) = z(t) + U (t) = Aejφ(t) + U (t) = Ae where z(t) c A.M. Zoubir

Rt

−∞

f (τ )dτ

= s(t) + j H[s(t)] and H[·] is the Hilbert transform.


138


+ U (t) ,

Problem Description (Cont’d) Signal Processing Group


139

v

θ(t)

R(t)

h

r(t)

Schematic of the geometric arrangement of the aircraft and observer in terms of the physical parameters of the model. c A.M. Zoubir


139


Problem Description (Cont’d) Signal Processing Group

140


Context: Estimation of an aircraft’s flight parameters (height, speed, range and acoustic frequency) from the acoustic signal as heard by a stationary observer.

Problem:

To estimate an aircraft’s flight parameters using passive acoustic

techniques and to determine confidence bounds given only a single acoustic realisation.

Solution: Estimate the aircraft flight parameters from the time varying phase or frequency of the observed acoustic signal. Use bootstrap techniques to estimate the confidence bounds. c A.M. Zoubir


140


The Passive Acoustic Model Signal Processing Group

141


An observer phase model

φ(t) describes the phase of the observed acoustic

signal in terms of the flight parameters and is given by

φ(t) =

f a c2 2π c2 −v2

t−

q

h2 c+v 2 t2 c+2v 2 th c3

+ φ0 , −∞ < t < ∞ ,

where fa is the source acoustic frequency, c is the speed of sound in the medium,

v is the constant velocity of the aircraft, h is the constant altitude of the aircraft and φ0 is an initial phase constant.

c A.M. Zoubir


141


The Passive Acoustic Model (Cont’d)


142 16000 14000

phase (radians)


12000 10000 8000 6000 4000 2000 0

−15

−10

−5

0

5

10

15

time (s)

Typical Time varying acoustic phase profile of an over-flying aircraft as described by the observer phase model. For this example,

v = 75 m/s, h = 300 m, fa = 85 Hz and

t0 = 0 s. The cartwheel symbol marks the time at which the aircraft is directly overhead the observer. c A.M. Zoubir


142




143


From the phase model, the instantaneous frequency (IF), relative to the stationary observer, can be expressed as

f (t) =

1 dφ(t) 2π dt

fa c2 = c2 −v2 1 − √

v 2 (t+h/c) h2 (c2 −v 2 )+v 2 c2 (t+h/c)2

.

f (t), or φ(t), −∞ < t < ∞ and c, the aircraft parameters collected in the vector θ = (fa , h, v, t0 )′ can be uniquely determined from the For a given

phase or observer IF model.

c A.M. Zoubir


143




144 115 110 105

frequency (Hz)


100 95 90 85 80 75 70 65

−15

−10

−5

0

5

10

15

time (s)

Typical Time varying acoustic frequency profile of an over-flying aircraft as described by the observer frequency model. For this example, v and t0

= 75 m/s, h = 300 m, fa = 85 Hz

= 0 s. The cartwheel symbol indicates the time at which the aircraft is directly

overhead the observer. c A.M. Zoubir


144


A Bootstrap Approach Signal Processing Group

145

Step 0. Conduct Experiment. Collect and sample the data to obtain

Xt , t =


−n/2, . . . , n/2 − 1. Xt to provide a ˆt which approximates the true phase φt . non-decreasing function φ

Step 1. Phase Unwrapping. Unwrap the phase of the signal

ˆ , an initial estimate of the aircraft parameθ ˆt in a leastters by fitting the non-linear observer phase model φt;θ to φ

Step 2. Initial Estimation. Obtain

squares sense. Step 3. Compute residuals. Compute the residuals

εˆt = φ

ˆ

t;θ

c A.M. Zoubir


145

− φˆt ,

t = −n/2, · · · , n/2 − 1 .


A Bootstrap Approach (Cont’d) Signal Processing Group

146

Step 4. Variance estimation. Compute σ î , a bootstrap estimate of the standard Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

deviation of θî , i

= 1, . . . , 4.

X ∗ = {ˆ ε∗−n/2 , · · · , εˆ∗n/2−1 }, with replacement, from X = {ˆ ε−n/2 , · · · , εˆn/2−1 } and construct

Step 5. Resampling. Draw a random sample

φˆ∗t = φ

ˆ

t;θ

+ εˆ∗t .

Step 6. Bootstrap estimates. Obtain and record the bootstrap estimates of the ∗ ˆ ˆ∗ in a leastaircraft parameters θ by fitting the observer phase model to φ t

squares sense.

c A.M. Zoubir


146


A Bootstrap Approach (Cont’d) Signal Processing Group

147


Step 7. Bootstrap variance estimate. Estimate the standard deviation of θî∗ using a nested bootstrap step and compute and record

Tî∗ =

θî∗ −θî σ î∗

,

i = 1, . . . , 4 .

Step 8. Repetition. Repeat Steps 5 through 7 a large number of times N . Step 9. Sorting. Order the bootstrap estimates as

∗ ∗ ≤ ··· ≤ ≤ Tî,(2) Tî,(1)

∗ Tî,(N ) and compute the (1 − α)100% confidence interval ∗ î − Tˆ∗ σ (θî − Tî,(q σ ˆ , θ i i,(q2 ) î ) , 1)

where q1 c A.M. Zoubir

= N − ⌊N α/2⌋ + 1 and q2 = ⌊N α/2⌋.


147


Simulation Results Signal Processing Group

148


➠ An n = 320 point passive acoustic signal zt is generated at three levels of SNR (15 dB, 20 dB and 30 dB).

➠ We set h = 304.8 m, v = 102.89 m/s, fa = 1 Hz, t0 = 0 s, sampling frequency fs = 8 Hz and bootstrap variables B1 = 25, B2 = 100 and B3 = 1000. ➠ The bootstrap confidence bounds are computed and compared with those obtained by Monte Carlo simulation where the aircraft parameters are calculated for 1000 independent realisations of zt .

c A.M. Zoubir


148



149

SNR

Actual


dB

30

20

15

h [m]

v [m/s]

t0 [s]

fa [Hz]

304.8

102.89

0.000

1.000

BS

MC

BS

MC

BS

MC

BS

MC

Upper Bound

308.40

308.0

102.94

103.04

0.004

0.002

1.000

1.000

Lower Bound

301.4

301.9

102.67

102.74

-0.005

-0.006

1.000

1.000

Interval length

6.6

6.1

0.27

0.30

0.009

0.008

0.000

0.000

Upper Bound

309.5

314.5

103.17

103.34

0.011

0.016

1.000

1.000

Lower Bound

290.8

295.2

102.26

102.44

-0.020

-0.016

0.999

0.999

Interval length

18.7

19.3

0.91

0.90

0.031

0.032

0.001

0.001

Upper Bound

315.7

320.7

103.54

103.65

0.040

0.027

1.001

1.001

Lower Bound

286.0

288.6

102.14

102.12

-0.008

-0.027

0.999

0.999

Interval length

29.7

32.1

1.40

1.53

0.048

0.054

0.002

0.002

The bootstrap derived 95% confidence bounds for each of the parameters are compared with the 95% confidence bounds determined by Monte Carlo simulation.

c A.M. Zoubir


149



300

300

200

200

count

count

150 100 0 0.9992

300

100

102.5

103

(b) speed (m/s) (bootstrap)

300

count 290

295

300

305

310

315

(c) height (m) (bootstrap) count

100 0 −0.03

−0.02

−0.01

0

0.01

0.02

(d) time reference (s) (bootstrap)

0.03

1.0008

102.5

103

103.5

(f) speed (m/s) (Monte Carlo)

200 100

300

200

1.0004

100

0 285

320

1

200

300

100

0.9996

(e) source frequency (Hz) (Monte Carlo)

0 102

103.5

200

0 285

0 0.9992

1.0008

count

count

1.0004

200

300

count

1

(a) source frequency (Hz) (Bootstrap)

0 102

count


300

0.9996

100

290

295

300

305

310

315

320

(g) height (m) (Monte Carlo)

200 100 0 −0.03

−0.02

−0.01

0

0.01

0.02

0.03

(h) time reference (s) (Monte Carlo)

The histograms of the bootstrap distribution (left column) are compared with the histograms obtained by Monte Carlo simulation (right column) for each of the parameters at 20 dB SNR. The true values are indicated by the dashed line. c A.M. Zoubir


150


Real Experiment Signal Processing Group

151

seconds of latitude (relative to the ground location)


−100

−50

ground location

0

50

100

150

−150

−100

−50

0

50

100

seconds of longitude (relative to the ground location)

The complete flight path of an aircraft acoustic data experiment conducted at Bribie Island showing five fly-overs. The ground location is indicated by the cartwheel symbol. c A.M. Zoubir


151


The Instantaneous Frequency Signal Processing Group

152

90

frequency (Hz)


85

80

75

70

65

60

55 0

5

10

15

20

25

time (s)

Doppler signature of an acoustic signal.

c A.M. Zoubir


152


Real Experiment (Cont’d) Signal Processing Group

153


Run 1.

Run 2.

Run 3.

Run 6.

h [m]

v [m/s]

t0 [s]

fa [Hz]

Nominal Value

149.05

36.61

0.00

76.91

Upper Bound

142.03

36.50

0.02

76.99

Lower Bound

138.16

36.07

-0.03

76.92

Interval length

3.87

0.43

0.05

0.07

Nominal Value

152.31

52.94

0.00

77.90

Upper Bound

161.22

53.27

-0.04

78.18

Lower Bound

156.45

52.64

-0.06

78.13

Interval length

4.77

0.62

0.02

0.05

Nominal Value

166.52

47.75

0.00

75.94

Upper Bound

232.30

57.29

0.10

76.48

Lower Bound

193.18

53.47

-0.14

75.87

Interval length

39.13

3.83

0.24

0.60

Nominal Value

233.01

56.56

0.00

77.65

Upper Bound

243.02

57.15

0.74

77.96

Lower Bound

209.72

53.69

0.68

77.80

Interval length

33.30

3.46

0.06

0.16

The bootstrap derived 95% confidence bounds for four real test signals using the unwrapped phase based parameter estimator. c A.M. Zoubir


153


Interpretation of the Results Signal Processing Group

154


➠ The proposed bootstrap and unwrapped phase based estimation techniques can be used to provide a practical flight parameter estimation scheme.

➠ The techniques do not assume any statistical distribution for the parameter estimates and can be applied in the absence of multiple acoustic realisations.

➠ The obtained confidence bounds are in close agreement with those obtained from Monte Carlo simulations.

➠ Similar experiments with Central Finite Difference (CFD) estimates were performed. The confidence bounds presented here are much tighter than those of the CFD based estimates.

c A.M. Zoubir


154



155





155


Detection of Buried Landmines Using GPR


156

➠ In many post-war zones, landmines of little or no metal content have been Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

found. Metal detectors are ineffective in such regions.

➠ Ground Penetrating Radar (GPR) can detect discontinuities in the electric properties of the soil, and therefore has the potential to detect buried plastic landmines.

➠ GPR is sensitive to changes in the soil conditions, e.g. moisture content and soil type, and signal processing may be used to improve detection of targets.

➠ Robust detection requires adaptive techniques to allow for changes in the soil whilst still detecting targets.

c A.M. Zoubir


156


Application to Real Data Signal Processing Group

157


➠ An FR-127-MSCB Impulse GPR (ImGPR) system developed by the CSIRO has been used.

➠ 127 soundings per second, each composed of 512 samples with 12 bit accuracy.

➠ Bistatic bow-tie antennae transmitting wide-band, ultra-short duration pulses. Center frequency of 1.4GHz and 80% bandwidth.

➠ Surrogate targets for PMN, PMN-2 and M14 AP mines with non-metallic casings were used.

c A.M. Zoubir


157


Impulse GPR System Signal Processing Group


158

The ImGPR runnig over a sand box.

c A.M. Zoubir


158


and Surrogate Landmines Signal Processing Group


159

Surrogate Landmines.

c A.M. Zoubir


159


Minelike Surrogate Targets Signal Processing Group


160

Target

Surrogate

Dimensions

Orientation

diam×hght ST-AP(1)

M14

52 × 42 mm

not critical

ST-AP(2)

PMN

118 × 50mm

sensitive

ST-AP(3)

PMN2

115 × 53mm

sensitive

Minelike surrogate targets used.

c A.M. Zoubir


160


Signal Model and Preliminary Investigations


161


➠ It is difficult to define a complete, physically-motivated signal model of the GPR backscatter waveform.

➠ Extensive data analysis has shown that a polynomial amplitude - polynomial phase model is suitable,

gt =

hP

Pa n=0

an tn

i

h P i b exp j Pm=0 bm tm + zt

➠ zt is stationary interference and the amplitude and frequency modulation are described by polynomials of order Pa and Pb respectively.

c A.M. Zoubir


161


Signal Model Investigations: An Example


162


➠ The signal model has been applied to GPR returns from clay soil, containing a small plastic target, denoted ST-AP(1) – a surrogate for the M14 antipersonnel (AP) mine.

➠ ST-AP(1) has a PVC casing and is filled with paraffin wax. ➠ A solid stainless steel cylinder 5 cm in diameter and length is also present, denoted by SS05x05.

➠ Phase and amplitude parameters are estimated using the DPT and the method of least-squares respectively.

c A.M. Zoubir


162


Results Signal Processing Group

163 4

2

x 10

0

−1

−2

0

2

4

6 Time (ns)

8

10

12

0

2

4

6 Time (ns)

8

10

12

6000 4000 Amplitude


Amplitude

1

2000 0 −2000 −4000

Top: Fit of the polynomial amplitude - polynomial phase model (bold line) to data (fine line) from non-homogeneous soil. Bottom: Residual of the fit to data.

c A.M. Zoubir


163


Test on the Parameters Signal Processing Group

164

It is suggested that the mean of this distribution changes in the presence of a target. b0

−0.5

−1

0

100

200

300

400

500

600

700

800

900

1000

0 100 −3 x 10

200

300

400

500

600

700

800

900

1000

0

200

300

400

500 600 Distance (steps)

700

800

900

1000

0.3

b1

0.28 0.26 0.24 0

b2


0

−0.5

−1

100

Phase coefficients from an order 2 model. Vertical dotted lines indicate the approximate position of ST-AP(1) (left) and SS05x05 (right) targets. c A.M. Zoubir


164


Test on the Parameters (Cont’d) Signal Processing Group

165

➠ For the data set shown b2 is seen to decrease in the presence of a target, however, in some cases, in particular when the soil type is loam, b2 increases


with the presence of a target. We suggest

H : b2 = b2,0 K : b2 6= b2,0 ➠ The obvious test statistic is T =

ˆb2 −b2,0 σ ˆˆb 2

where σ ˆˆb2 is an estimate of the standard deviation of ˆb2 .

➠ Since the exact distribution of ˆb2 is unknown and has unknown variance, we use the bootstrap to determine thresholds. c A.M. Zoubir


165



166

From a background region of 100 samples, estimates of b2 are seen in a Q − Q plot.

Normal Probability Plot

0.99 0.98 0.95 0.90

0.75 Probability


0.997

0.50

0.25

0.10 0.05 0.02 0.01 0.003 −3.8

−3.6

−3.4

−3.2 Data

−3

−2.8

−2.6 −4

x 10

Normal probability plot of third phase coefficient estimate, ˆ b2 , from a background region. c A.M. Zoubir


166


Bootstrap-Based Detection Signal Processing Group

167

Step 1. If gt for t

= 0, ..., n − 1 is a sampled GPR return signal, fit a polynomial

amplitude - polynomial phase model to the data. From the model, an estimate Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

of the second order phase parameter, ˆ b2 , is made. Step 2. Form the residual signal rt

= gt − gˆt , where gˆt is the polynomial

amplitude-polynomial phase model corresponding to estimated parameters.

Step 3. Whiten rt by removing an AR model of suitable order to obtain the innovations zt . Step 4. Re-sample from zt

n times using the Block Bootstrap [Politis (1998)] to

obtain zt∗ . Step 5. Repeat Step 4. Step 6. Generate c A.M. Zoubir

N times to obtain zt∗1 , ..., zt∗N .

N bootstrap residual signals rt∗i , for i = 1, ..., N by filtering


167


Bootstrap-Based Detection Signal Processing Group

168


zt∗i with the AR process obtained in Step 3. Step 7. Generate N bootstrap signals gt∗i

= gˆt + rt∗i , for i = 1, ..., N .

Step 8. Estimate the third phase coefficient from gt∗i to obtain ˆ b∗i 2 for i Step 9. Calculate the bootstrap statistics T

∗i

=

ˆb∗i −ˆb2 2 σ ˆˆ∗

for i

= 1, ..., N .

= 1, ..., N .

b2

Step 10. Compare the test statistic

T =

ˆb2 −b2,0 σ ˆˆb

to the empirical distribution of

2

T ∗ . Reject H if T is in the tail of the distribution, otherwise retain H .

c A.M. Zoubir


168


Discussion Signal Processing Group

169

➠ Shown in the following figures are sample results for three surrogate mines Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

as well as various other “targets”. The location of the first target in each scenario is around the 300th trace, while the second target is around the 690’th trace.

➠ Overall, the results are very encouraging. It can be said that the targets have been correctly detected, while false alarms appear to be concentrated in areas near the targets – rather than true false alarms that occur far from the targets in background only areas.

➠ These near-target false alarms may be triggered by disturbances to the soil structure caused by the burying of the target. c A.M. Zoubir


169



170

200 300 400 500 100

200

300

400

500

100

200

300

400 500 Sample

600

700

800

900

−3

x 10 2 Test statistic


Depth samples

100

1.5 1 0.5 0 0

Top:

600

700

800

900

B -scan with ST-AP(1) and SS05x05 targets buried at 5 cm in clay.

Bottom: Corresponding test statistics (thin line) and detection decision (thick line). c A.M. Zoubir


170



171 Depth samples

100 200 300

500 100

200

300

400

500

600

700

800

100

200

300

400 500 Sample

600

700

800

900

−3



400

1.5 1 0.5 0 0

Top:

900

B -scan with ST-AP(2) target (parallel orientation) and an aluminium soft-

drink can buried at 5 cm in clay. Bottom: Corresponding test statistics (thin line) and detection decision (thick line). c A.M. Zoubir


171



172 Depth samples

100 200 300

500 100

200

300

400

500

600

700

800

100

200

300

400 500 Sample

600

700

800

900

−3



400

1.5 1 0.5 0 0

Top:

900

B -scan with ST-AP(3) target (perpendicular orientation) and shrapnel buried

at 5 cm in clay. Bottom: Corresponding test statistics (thin line) and detection decision (thick line). c A.M. Zoubir


172



173





173


An Image Processing Example Signal Processing Group


174

➠ We consider an example of fitting functions such as circles or ellipses to the outlines of the pupil and limbus (iris) in eye images [Iskander et al. (2004)].

➠ Performance for extracting eye features can be evaluated with synthetic images.

➠ For real images and particularly for the limbus [Morelande et al. (2002)] these procedures need to be benchmarked against manual operators.

c A.M. Zoubir


174


An Image Processing Example (Cont’d)



175

Typical example of a slit lamp image of an eye with manually selected points (yellow crosses) around the limbus (i.e., iris outline) area.

c A.M. Zoubir


175


An Image Processing Example (Cont’d) 176


➠ The number of points (xi , yi ), i = 1, 2, . . . , n, where xi and yi denote the


horizontal and vertical point position is small.

➠ A linear least-squares procedure can be applied:        2 2 x 1 + y1 2x1 2y1 1 ε1 p1    .     . . . .   =  .. .. .. ..  ·  p  +  ..      2   p3 x2n + yn2 2xn 2yn 1 εn where p1

  

= x0 , p2 = y0 , p3 = x20 + y02 − R2 and εi , i = 1, 2, . . . , n, Y =X ·P +E

➠ A solution is given by ˆ = XT X P c A.M. Zoubir




176

−1

XT Y .


An Image Processing Example (Cont’d)


177


➠ The question is how well a particular operator can fit a circle to the limbus. ➠ One way is to select 8 data point pairs (considered in our example) 1000 times, say.

➠ Although feasible, this task is laborious and the results would most definitely be affected by the subsequently decreasing commitment of the operator.

➠ The alternative is the bootstrap. ➠ The selected data points (xi , yi ), i = 1, 2, . . . , n, form a particular pattern and are not i.i.d, unlike the modelling errors εi , i = 1, . . . , n, collected in the random sample ε = {ε1 , ε2 , . . . , εn }. c A.M. Zoubir


177


The Bootstrap Procedure Signal Processing Group

178

ˆ and construct an estimate of the Step 1. Estimate x0 , y0 , and R, collected in P


ˆ circle using Y

ˆ. =X ·P

ˆ Step 2. Calculate the residuals E

= Y − Yˆ .

∗ ˆ Step 3. Create a set of bootstrap residuals E by resampling with replacement ˆ. from E ∗ ∗ ˆ ˆ ˆ Step 4. Create a new estimate of the circle by Y = X · P + E .

Step 5. Estimate a new set of parameters from the newly created bootstrap sample

∗ ˆ Pb

T

= X X

−1

∗ ˆ X Y . T

∗ ∗ ∗ ˆ ˆ ˆ Step 6. Repeat Steps 3-5 B times to obtain a set of P 1 , P 2 , . . . , P B , from

which empirical distributions can be obtained. c A.M. Zoubir


178


The Bootstrap Procedure-Results Signal Processing Group

179 120

Frequency of occurrence


100

80

60

40

20

0 5.92

5.94

5.96

5.98

6

6.02

6.04

Bootstrap estimates of limbus radius

6.06

An example of the distribution (histogram) of the limbus radius obtained with the bootstrap method is shown

c A.M. Zoubir


179



180





180


Micro-Doppler Analysis Signal Processing Group

181

➠ Doppler often arises in engineering applications where radar, ladar, sonar


and ultra-sound measurements are made.

➠ Due to the relative motion of an object with respect to the measurement system.

➠ If the motion is harmonic, for example due to vibration or rotation, the resulting signal can be well modeled by an FM process [Huang et al. 1990].

➠ Estimation of the FM parameters may allow one to determine physical properties such as the angular velocity and displacement of the vibrational/rotational motion which can in turn be used for classification.

➠ Objective: estimate the FM parameters along with a measure of accuracy, such as confidence intervals. c A.M. Zoubir


181


Micro-Doppler Analysis (Cont’d) Signal Processing Group

182

➠ Assume the following AM-FM signal model:


s(t) = a(t) exp{jϕ(t)} where a(t; α)

=

Pq

k α t and α = (α0 , . . . , αq ) . k=0 k

➠ The phase modulation for a micro-Doppler signal is described by a sinusoidal function: ϕ(t) = −D/ωm cos(ωm t + φ). ➠ The instantaneous frequency (IF) of the signal is given by: dϕ(t) ωi (t; β) , = D sin(ωm t + φ), dt where β = (D, ωm , φ) are termed the FM or micro-Doppler parameters. ➠ Given {x(k)}nk=1 of X(t) = s(t) + V (t), the goal is to estimate the microDoppler parameters as well as their confidence intervals.

c A.M. Zoubir


182



183


➠ The estimation of the phase parameters is performed using a time-frequency Hough transform (TFHT) [Barbarossa and Lemoine1996,Cirillo, Zoubir & Amin 2006].

➠ The TFHT is based on the pseudo Wigner-Ville (PWVD) distribution: (L−1)/2

Pxx [k, ω) =

X

l=−(L−1)/2

h[k]x[k + l]x∗ [k − l]e−j2ω ,

k = −(L − 1)/2, . . . , n − (L − 1)/2, where h[k] is a windowing function of duration L samples.

for

c A.M. Zoubir


183



184

➠ The TFHT based on the PWVD is given by: n−(L−1)/2+1


H(β) =

X

Pxx [k, ωi (n; β)),

k=−(L−1)/2

where ωi (t) is described above and β

= (D, ωm , φ).

➠ An estimate of β is obtained from the location of the largest peak of H(β). ➠ Once the phase parameters have been estimated, the phase term is demodulated and the amplitude parameters are estimated via least-squares.

➠ Given estimates for α and β , the residuals are obtained. ➠ We whiten the residuals using a fitted AR model. The innovations are resampled, filtered and added to the estimated signal term. c A.M. Zoubir


184




185

➠ The results shown here are based on an experimental radar system, operating at carrier frequency fc = 919.82 MHz. ➠ After demodulation, the in-phase and quadrature baseband channels are sampled at fs = 1 kHz. ➠ The radar system is directed toward a spherical object, swinging with a pendulum motion, which results in a typical micro-Doppler signature.

➠ The PWVD of the observations is computed and shown below:

c A.M. Zoubir


185



186 PWVD of the radar data

200

200

150

150

100

100 Frequency (Hz)

Frequency (Hz)


PWVD of the radar data

50 0 −50

50 0 −50

−100

−100

−150

−150

−200

−200

−250

0.05

0.1

0.15

0.2

0.25 0.3 Time (sec)

0.35

0.4

0.45

−250

(a)

0.05

0.1

0.15

0.2

0.25 0.3 Time (sec)

0.35

0.4

0.45

(b)

The PWVD of the radar data (a). The PWVD of the radar data and the microDoppler signature estimated using the TFHT (b).

c A.M. Zoubir


186



187 Real (in−phase) component

Time−domain waveform of residuals 0.01 Re{ r(n) } Im{ r(n) }

Data AM−FM Fit

0.005

Amplitude (V)

0.02 0 −0.02 −0.04

0

−0.005

−0.01

−0.015

0

0.05

0.1

0.15

0.2

0.25 0.3 0.35 Time (sec) Imaginary (quadrature) component

0.4

0.45

0.5

0

0.05

0.1

0.15

0.2

0.25 Time (sec)

0.3

0.35

0.4

0.45

0.5

Estimated spectrum of residuals

0.04

−20 Periodogram AR Fit

Data AM−FM Fit

0.02

−30

−40 Power (dBW)

Amplitude (V)


Amplitude (V)

0.04

0

−50

−60

−70

−0.02

−80

−0.04

0

0.05

0.1

0.15

0.2

0.25 0.3 Time (sec)

0.35

0.4

(c)

0.45

0.5

−90 −500

−400

−300

−200

−100

0 f (Hz)

100

200

300

400

(d)

The real and imaginary components of the radar signal are with their estimated counterparts (c). The real and imaginary parts of the residuals and their spectral estimates (d). c A.M. Zoubir


187


500


188 Boostrap confidence interval for ω

Boostrap confidence interval for D

m

5

450 Bootstrap dist. Initial estimate 95% CI

4.5


4

Bootstrap dist. Initial estimate 95% CI

400 350

3.5

300

3 250 2.5 200 2 150

1.5

100

1

50

0.5 0 71.7

71.8

71.9

72 D/(2π) (Hz)

72.1

72.2

72.3

0 3.037

3.038

3.039

3.04

3.041 3.042 ω /(2π) (Hz)

3.043

3.044

3.045

3.046

m

(a)

(b)

The bootstrap distributions and 95% confidence intervals for the FM parameters

D (a) and ωm (b) using N = 500. c A.M. Zoubir


188



189





189


Summary Signal Processing Group

190


➠ It is often necessary to find the sampling distributions of parameter estimators, so that the respective means and variances can be calculated, and more generally, confidence intervals for the true parameters can be set.

➠ Most techniques for computing variances or confidence intervals assume that the size of the available set of sample values is sufficiently large, so that “asymptotic” results can be applied.

➠ In many signal processing problems this assumption cannot be made because, for example, the process is non-stationary and only small portions of stationary data are considered.

c A.M. Zoubir


190


Summary (Cont’d) Signal Processing Group

191


➠ Bootstrap techniques are an alternative to asymptotic methods. ➠ The bootstrap does with a computer what the experimenter would do in practice, if it were possible: they would repeat the experiment.

➠ With the bootstrap, the observations are randomly re-assigned, and the estimates re-computed. This process is done thousands of times to simulate repeated experiments.

➠ In an era of exponentially increasing computational power, such computerintensive methods are becoming increasingly attractive.

c A.M. Zoubir


191



192

➠ The tutorial provides the fundamental concepts and methods needed by the Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

signal processing practitioner to decide when and how to apply the bootstrap successfully.

➠ The tutorial focuses on the independent data bootstrap. The assumption of i.i.d. data can break down in practice either because the data is not independent or because it is not identically distributed, or both.

➠ The bootstrap can still be invoked if we knew the model that generated the data. In other cases we can make the reasonable assumption that the data is identically distributed but not independent such as in autoregressive processes. c A.M. Zoubir


192



193


➠ For confidence interval estimation or hypothesis testing, it is essential that the statistic used is asymptotically pivotal.

➠ It has been shown that working on a variance stable scale is better than studentising.

➠ When a variance stabilising transformation is not known, bootstrap can be used to estimate it.

➠ Many applications have been presented to demonstrate the power of the bootstrap. Special care is however required when applying the bootstrap in real-life situations.

c A.M. Zoubir


193


Acknowledgements Signal Processing Group

194

➠ Many of the examples and simulations were implemented by my former and Institut fur ¨ Nachrichtentechnik-Fachgebiet Signalverarbeitung

current PhD students. I wish to thank them for their support.

➠ The passive acoustic problem was the subject of Dr. David Reid’s PhD. He collected the data and implemented the algorithms.

➠ The data from the JINDALEE over-the-horizon radar system and the GPR was provided by the Defence Science & Technology Organisation in Salisbury, South Australia.

➠ The Australian Research Council is acknowledged for providing partial financial support for some of the projects.

c A.M. Zoubir


194


Bibliography Signal Processing Group


195

This presentation is based on

➠ A. M. Zoubir and D. R. Iskander. Bootstrap Techniques for Signal Processing. Cambridge University Press, 2004.

➠ A. M. Zoubir and B. Boashash. The Bootstrap and Its Application in Signal Processing. IEEE Signal Processing Magazine, 15(1):56–76, 1998.

c A.M. Zoubir


195


Cited References Signal Processing Group


196

[1] H. Akaike. Comments on “On Model Structure Testing in System Identification”. International J. Control, 27:323–324, 1978. [2] T. W. Anderson. An Introduction to Multivariate Statistical Analysis. John Wiley and Sons, 1984. [3] T. Ankenbrand and M. Tomassini. Predicting Multivariate Financial Time Series Using Neural Networks: the Swiss Bond Case. In Proceedings of the IEEE/IAFE 1996 Conference on Computational Intelligence for Financial Engineering (CIFEr), pages 27–33, New York, USA, 1996. IEEE. [4] G. Archer and K. Chan. Bootstrapping Uncertainty in Image Analysis. In Proceedings of the 12th Symposium in Computational Statistics, pages 193–198, Heidelberg, Germany, 1996. Physica-Verlag. c A.M. Zoubir


196




197

[5] C. Banga and F. Ghorbel. Optimal Bootstrap Sampling for Fast Image Segmentation: Application to Retina Image. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-93), Minnesota, 1993. [6] R. Bhar and C. Chiarella. Bootstrap Algorithms and Applications. In Proceedings of the IEEE/IAFE 1996 Conference on Computational Intelligence for Financial Engineering (CIFEr), pages 168–182, New York, USA, 1996. IEEE. [7] R. N. Bhattacharya and R. R. Rao. Normal Approximation and Asymptotic Expansions. Wiley, New York, 1976. ¨ [8] J. F. Bohme and D. Maiwald. Multiple Wideband Signal Detection and Tracking from ¨ ¨ editors, Proceedings of the 10th Towed Array Data. In M. Blanke and T. Soderstr om, IFAC Symposium on System Identification (SYSID 94), volume 1, pages 107–112, Copenhagen, July 1994. Danish Automation Soc. [9] A. Bose. Edgeworth Correction by Bootstrap in Autoregressions. The Ann. of Statist., 16:1709–1722, 1988. [10] D. R. Brillinger. Time Series: Data Analysis and Theory. Holden-Day, 1981. [11] P. Ciarlini. Bootstrap Algorithms and Applications. In Advanced Mathematical Tools c A.M. Zoubir


197



198

in Metrology III, pages 12–23, Singapore, 1997. World Scientific.


[12] M. G. Cox, P. M. Harris, M. J. T. Milton, and P. T. Woods. A method for evaluating trends in ozone-concentration data and its application to data from the UK Rural Ozone Monitoring network. In Advanced Mathematical Tools in Metrology III, pages 171–177, Singapore, 1997. World Scientific. ´ Mathematical Methods of Statistics. Princeton University Press, 1967. [13] H. Cramer. [14] P. M. Djurić and S. M. Kay. Parameter Estimation of Chirp Signals. IEEE Transactions on Acoustics, Speech and Signal Processing, 38(12):2118–2126, December 1990. [15] B. Efron. Bootstrap Methods. Another Look at the Jackknife. The Ann. of Statist., 7:1–26, 1979. [16] B. Efron. Computers and the Theory of Statistics: Thinking the Unthinkable. SIAM Review, 4:460–480, 1979. [17] B. Efron. Better Bootstrap Confidence Interval. J. Amer. Statist. Assoc., 82:171– 185, 1987. [18] B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall, c A.M. Zoubir


198



199

1993.


[19] B. G. Ferguson. A Ground Based Narrow-Band Passive Acoustic Technique for Estimating the Altitude and Speed of a Propeller Driven Aircraft. J. Acoust. Soc. Am., 92(3):1403–1407, 1992. [20] B. G. Ferguson and B. G. Quinn. Application of the Short-Time Fourier Transform and the Wigner-Ville Distribution to the Acoustic Localization of Aircraft. J. Acoust. Soc. Am., 96(2):821–827, 1994. [21] N. I. Fisher and P. Hall. Bootstrap Confidence Regions for Directional Data. J. Amer. Statist. Assoc., 84:996–1002, 1989. [22] N. I. Fisher and P. Hall. New Statistical Methods for Directional Data – I. Bootstrap Comparison of Mean Directions and the Fold Test in Palaeomagnet. Geophys. J. International, 101:305–313, 1990. [23] N. I. Fisher and P. Hall. Bootstrap Algorithms for Small Samples. J. Statist. Plan. Infer., 27:157–169, 1991. [24] N. I. Fisher and P. Hall. General Statistical Test for the Effect of Folding. Geophys. J. International, 105:419–427, 1991. c A.M. Zoubir


199



200

[25] R. A. Fisher. On the “Probable Error” of a Coefficient of Correlation Deduced from a Small Sample. Metron, 1:1–32, 1921.


¨ [26] J. Franke and W. Hardle. On Bootstrapping Kernel Estimates. The Ann. of Statist., 20:121–145, 1992. [27] D. A. Freedman. Bootstrapping Regression Models. The Ann. of Statist., 9:1218– 1228, 1981. [28] P. Hall. Theoretical Comparison of Bootstrap Confidence Intervals (with discussion). The Ann. of Statist., 16:927–985, 1988. [29] P. Hall. The Bootstrap and Edgeworth Expansion. Springer-Verlag New York, Inc., 1992. [30] P. Hall and D. M. Titterington. The Effect of Simulation Order on Level Accuracy and Power of Monte Carlo Tests. J. R. Statist. Soc. B, 51:459–467, 1989. [31] S. R. Hanna. Confidence Limits for Air Quality Model Evaluations, as Estimated by Bootstrap and Jackknife Resampling Methods. Atmospheric Environment, 23:1385– 1398, 1989. [32] T. Hastie and R. Tibshirani. Generalized Additive Models. Chapman & Hall, 1990. c A.M. Zoubir


200




201

[33] D. R. Haynor and S. D. Woods. Resampling Estimates of Precision in Emission Tomography. IEEE Transactions on Medical Imaging, 8:337–343, 1989. [34] R. Herman. Using the Bootstrap Method for Evaluating Variations Due to Sampling in Voltage Regulation Calculations. In Proceedings of the Sixth Southern African Universities Power Engineering Conference (SAUPEC-96), pages 235–238, Witwatersrand, South Africa, 1996. Univ. Witwatersrand. [35] A. O. Hero III, J. A. Fessler, and M. Usman. Exploring Estimator Bias-Variance Tradeoffs Using the Uniform CR Bound. IEEE Transactions on Signal Processing, 44(8):2026–2041, 1996. [36] M. J. Hinich. Testing for Gaussianity and Linearity of a Stationary Time Series. J. Time Series Anal., 3:169–176, 1982. [37] L. B. Jaeckel. Some Flexible Estimates of Location. Ann. Math. Statist., 42:1540– 1552, 1971. [38] K. Kanatani and N. Ohta. Optimal Robot Self-Localization and Reliability Evaluation. In Proceedings of the 5th European Conference on Computer Vision (ECCV’98), volume 2, pages 796–808, Berlin, Germany, 1998. Springer-Verlag. c A.M. Zoubir


201



202

[39] S. Kay. Modern Spectral Estimation. Theory and Application. Prentice Hall, 1988.


[40] J. P. Kreiss and J. Franke. Bootstrapping Stationary Autoregressive Moving Average Models. J. Time Ser. Anal., 13:297–319, 1992. [41] J.-P. Kreiss and G. Lien. Bootstrapping autoregressions with infinite order. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-94), volume 6, pages 97–100, Adelaide, South Australia, April 1994. [42] E. Kreyszig. Introductory Functional Analysis with Applications. J. Wiley & Sons, 1989. [43] J. Krolik, G. Niezgoda, and D. Swingler. A bootstrap approach for evaluating source localization performance on real sensor array data. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP91), volume 2, pages 1281–1284, Toronto, Canada, May 1991. [44] H. R. Kunsch. The Jackknife and the Bootstrap for General Stationary Observations. ¨ The Ann. of Statist., 17:1217–1241, 1989. [45] D. Lai and G. Chen. Computing the Distribution of the Lyapunov Exponent from c A.M. Zoubir


202




203

Time Series: The One-dimensional Case Study. International Journal of Bifurcation and Chaos in Applied Sciences and Engineering, 5:1721–1726, 1995. [46] J. M. Lange, H. M. Voigt, S. Burkhardt, R. Gobel, S. Burkhardt, and R. Gobel. The Intelligent Inspection Engine—a Real-Time Real-World Visual Classifier System. In IEEE International Joint Conference on Neural Networks Proceedings, pages 1810– 1815, New York, USA, 1998. IEEE. [47] E. L. Lehmann. Testing Statistical Hypotheses. Wadsworth and Brooks, 1991. [48] R. Y. Liu and K. Singh. Moving Blocks Jackknife and Bootstrap Capture Weak Dependence. In LePage and Billard, editors, Exploring the limits of bootstrap, New York, 1992. John Wiley. [49] S. L. Marple Jr. Digital Spectral Analysis with Applications. Prentice-Hall, 1987. [50] R. G. Miller. The Jackknife - A Review. Biometrika, 61:1–15, 1974. [51] S Nagaoka and O. Amai. A Method for Establishing a Separation in Air Traffic Control Using a Radar Estimation Accuracy of Close Approach Probability. J. Japan Ins. Navigation, 82:53–60, 1990. [52] S Nagaoka and O. Amai. Estimation Accuracy of Close Approach Probability for c A.M. Zoubir


203



204

Establishing a Radar Separation Minimum. J. Navigation, 44:110–121, 1991.


[53] H. Ong, D. R. Iskander, and A. M. Zoubir. Detection of a Common Non-Gaussian Signal in Two Sensors Using The Bootstrap. In Proceedings of the IEEE Signal Processing Workshop on Higher-Order Statistics, pages 463–467, Banff, Alberta, Canada, 1997. [54] H. Ong and A. M. Zoubir. Non-Gaussian Signal Detection from Multiple Sensors using the Bootstrap. In Proceedings of the International Conference on Information, Communications & Signal Processing (ICICS’97), volume 1, pages 340–344, Singapore, September 1997. [55] E. Paparoditis. Bootstrapping Autoregressive and Moving Average Parameter Estimates of Infinite Order Vector Autoregressive Processes. J. Multivar. Anal., 57:277– 296, 1996. [56] D. N. Politis and J. P. Romano. A General Resampling Scheme for Triangular Arrays of α-Mixing Random Variables with Application to the Problem of Spectral Density Estimation. The Ann. of Statist., 20:1985–2007, 1992. [57] D. N. Politis and J. P. Romano. Bootstrap Confidence Bands for Spectra and CrossSpectra. IEEE Transactions on Signal Processing, 40:1206–1215, 1992. c A.M. Zoubir


204



205

[58] D. N. Politis and J. P. Romano. The Stationary Bootstrap. Journal Am. Stat. Assoc., 89:1303–1313, 1994.


[59] B. Porat and B. Friedlander. Performance Analysis of Parameter Estimation Algorithms Based on High-Order Moments. International J. Adaptive Control and Signal Processing, 3:191–229, 1989. [60] M. B. Priestley. Spectral Analysis and Time Series. Academic Press, 1981. [61] D. C. Reid, A. M. Zoubir, and B. Boashash. The Bootstrap Applied to Passive Acoustic Aircraft Parameter Estimation. In Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP-96), volume VI, pages 3153– 3156, Atlanta, May 1996. [62] D. C. Reid, A. M. Zoubir, and B. Boashash. Aircraft Parameter Estimation based on Passive Acoustic Techniques Using the Polynomial Wigner-Ville Distribution. J. Acoust. Soc. Am., 102(1):207–223, 1997. [63] M. Rosenblatt. Stationary Sequences and Random Fields. Birkhauser, Boston, 1985. [64] Special Session. The Bootstrap and Its Applications. In Proceedings of the IEEE c A.M. Zoubir


205



206

International Conference on Acoustics, Speech and Signal Processing (ICASSP94), volume VI, pages 65–79, Adelaide, Australia, 1994. [65] J. Shao. Bootstrap Model Selection. J. Am. Statist. Assoc., 91:655–665, 1996.


[66] J. Shao and D. Tu. The Jackknife and Bootstrap. Springer, 1995. [67] Q. Shao and H. Yu. Bootstrapping the Sample Means for Stationary Mixing Sequences. Stochastic Processes and Their Appl., 48:175–190, 1993. [68] R. H. Shumway. Replicated Time-Series Regression: An Approach to Signal Estimation and Detection. In D. R. Brillinger and P. R. Krishnaiah, editors, Handbook of Statistics, volume 3, pages 383–408. North-Holland, 1983. [69] T. Subba Rao and M. M. Gabr. Testing for Linearity of a Stationary Time Series. J. Time Series Anal., 1:145–158, 1980. [70] A. Swami, J. M. Mendel, and C. L. Nikias. Higher-Order Spectral Analysis Toolbox for use with MATLAB. The MathWorks, Inc., 1995. [71] L. Tauxe, N. Kylstra, and C. Constable. Bootstrap Statistics for Paleomagnetic Data. J. Geophysical Research, 96:11723–11740, 1991. [72] S. Tretter. Estimating the Frequency of a Noisy Sinusoid. IEEE Transactions on ATiSSP

c A.M. Zoubir

Sommersemester 2007

206



207

Information Theory, 31:832–835, 1985.


[73] J. K. Tugnait. Two-Channel Test for Common Non-Gaussian Signal Detection. IEE Proceedings-F, 140:343–349, 1993. [74] A. M. Yacout, S. Salvatores, and Y. G. Orechwa. Degradation Analysis Estimates of the Time-to-Failure Distribution of Irradiated Fuel Elements. Nuclear-Technology, 113(2):177–189, 1996. [75] G. A. Young. Bootstrap: More Than a Stab in the Dark? Statistical Science, 9:382– 415, 1994. [76] Y. Zhang, D. Hatzinakos, and A. N. Venetsanopoulos. Bootstrapping Techniques in the Estimation of Higher-Order Cumulants from Short Data Records . In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-93), volume IV, pages 200–203, Minnesota, 1993. [77] A. M. Zoubir. Bootstrap: Theory and Applications. In F. T. Luk, editor, Advanced Signal Processing Algorithms, Architectures and Implementations, volume 2027, pages 216–235, San Diego, July 1993. Proceedings of SPIE. [78] A. M. Zoubir. Multiple Bootstrap Tests and Their Application. In Proceedings of c A.M. Zoubir


207




208

the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-94), volume VI, pages 69–72, Adelaide, Australia, 1994. [79] A. M. Zoubir and B. Boashash. Advanced Signal Processing for Over-The-Horizon Radar: Application of Bootstrap Methods. Technical report, Signal Processing Research Centre, Queensland University of Technology, March 1996. DSTO Report ref. 96/2–Part I-IV. [80] A. M. Zoubir and B. Boashash. The Bootstrap and Its Application in Signal Processing. IEEE Signal Processing Magazine, 15(1):56–76, 1998. ¨ [81] A. M. Zoubir and J. F. Bohme. Bootstrap Multiple Hypotheses Tests for Optimizing Sensor Positions in Knock Detection. In Proceedings of 13th World Congress on Computation and Applied Mathematics, IMACS’91, pages 955–958, Dublin, 1991. ¨ [82] A. M. Zoubir and J. F. Bohme. Application of Higher Order Spectra to the Analysis and Detection of Knock in Combustion Engines. In B. Boashash, E. J. Powers, and A. M. Zoubir, editors, Higher-Order Statistical Signal Processing, pages 269–290. John Wiley, 1995. ¨ [83] A. M. Zoubir and J. F. Bohme. Multiple Bootstrap Tests: An Application to Sensor Location. IEEE Transactions on Signal Processing, 43:1386–1396, 1995. c A.M. Zoubir


208




209

[84] A. M. Zoubir and D. R. Iskander. A Bispectrum Based Gaussianity Test Using The Bootstrap. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-96), volume V, pages 3029–3032, Atlanta, 1996. [85] A. M. Zoubir and D. R. Iskander. On Bootstrapping the Confidence Bands for Power Spectra. Technical report, Signal Processing Research Centre, Queensland University of Technology, January 1996. [86] A. M. Zoubir and D. R. Iskander. Bootstrap Model Selection for Polynomial Phase Signals. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP-98), pages 2229–2232, Seattle, USA, May 1998. [87] A. M. Zoubir and D. R. Iskander. Bootstrapping Bispectra: An Application to Testing for Departure from Gaussianity of Stationary Signals. IEEE Transactions on Signal Processing, March 1999. [88] A. M. Zoubir and D. R. Iskander. Bootstrap Based Modelling of a Class of NonStationary Signals. IEEE Transaction on Signal Processing, (under review). [89] I. Koutrouvelis. Regression-type estimation of the parameters of stable laws. Journal of the American Statistical Association, 75(372), 1980. c A.M. Zoubir


209




210

[90] D. Middleton. Statistical-physical models of electromagnetic interference. IEEE Transactions on Electromagnetic Compatibility, EMC-19(3):106–27, August 1977. [91] J. Shao and D. Tu. The Jackknife and Bootstrap. Springer-Verlag, 1996. [92] S.S. Wilks. Determination of sample sizes for setting tolerance limits. Annals of Mathematical Statistics, 12:91–96, 1941.

c A.M. Zoubir


210
