Two-dimensional Empirical Mode Decomposition for

0 downloads 0 Views 6MB Size Report
Step 4. Subtract the h(i,j)(t) by the mean envelop to obtain the sifting result h(i,j+1)(t), ... end, we can decompose the signal X(t) into several IMFs and a residual rn(t). ... of DφL along eight sampling directions while the negative openness ψL.
Two-dimensional Empirical Mode Decomposition for the fusion of multispectral and panchromatic images Tee-Ann Teo*a, Chi-Chung Laub, Liang-Chien Chenc Department of Civil Engineering, National Chiao Tung University, No. 1001, University Road, Hsinchu 30010, Taiwan b Energy and Environment Research Laboratories, Industrial Technology Research Institute, No. 195, Section 4, Chung Hsing Road, Chutung, Hsinchu 31040, Taiwan. c Center for Space and Remote Sensing Research, National Central University, No. 300, Jhungda Road, Jhungli City, Taoyuan 32001, Taiwan a

ABSTRACT Image fusion is an important technique to integrate high resolution panchromatic image and low resolution multispectral image. The fused image enhances the capability for image interpretation. EMD (Empirical Mode Decomposition) is an effective method to decompose the linear or nonlinear signals into a set of IMFs (Intrinsic Mode Functions). The characteristics of EMD may apply to image fusion technique. The objective of this investigation is to establish a novel image fusion method using a 2-D EMD. The idea of image fusion based on EMD is to decompose the panchromatic and multispectral images into their IMFs. Then, we replace the high frequency IMF of multispectral image by high frequency IMF of panchromatic image. Finally, the image fusion is performed by reconstructing the mixed IMFs. The experimental results indicate that the proposed method may produce a fused image that preserves spatial and spectral information. Keywords: image fusion, image processing, multi-resolution techniques, Empirical Mode Decomposition

1. INTRODUCTION Nowadays, most of the optical sensors are capable of acquiring high spatial resolution panchromatic and low spatial resolution multispectral bands simultaneously, e.g., QuickBird, IKONOS, SPOT Series, etc. Due to the physical constraint of sensors, the resolution of panchromatic band is better than multispectral ones in an optical sensor. In order to overcome this problem, image fusion techniques (also known as color fusion, pan sharpening, or resolution merging) is widely used to obtain an image with both high spatial and high spectral information. The approaches of image fusion may be categorized into three types, namely, Color transformation approach, Statistical approach, and Multi-scale approach. Intensity-Hue-Saturation (IHS) Transform [1] is one of the traditional fusion algorithms using color transformation. This method converts the multispectral image into intensity, hue and saturation bands. Then, the intensity of multispectral image is replaced by high spatial panchromatic image. However, this method is limited to the number of bands. The statistical approach includes Principle Component Analysis (PCA) [2], Independent Component Analysis (ICA) [3], and etc. The PCA is a statistical approach to convert a multispectral image into several components based on Eigen vectors and values. Then, the first component with large variance is replaced by high spatial panchromatic image and performs the inverse PCA. The process is similar to IHS Transform; and, it is not constrained by the number of bands. However, this method may cause large color distortion. The multi-scale fusion approach is widely used in image fusion as the advantage of multi-scale analysis may improve the fusion results. The multi-scale approach includes Wavelet Transform [4], Empirical Mode Decomposition (EMD) [5], and etc. The Wavelet approach transforms the original images into several high and low frequency layers. Then, it replaces the high frequency of multispectral by panchromatic ones. Finally, an inverse Wavelet transform is selected to construct the fused image. *[email protected]; phone 886 3 5712121 ext. 54957; fax 886 3 5716257.

MIPPR 2009: Multispectral Image Acquisition and Processing, edited by Jayaram K. Udupa, Nong Sang, Laszlo G. Nyul, Hengqing Tong, Proc. of SPIE Vol. 7494 74940T · © 2009 SPIE · CCC code: 0277-786X/09/$18 · doi: 10.1117/12.832541 Proc. of SPIE Vol. 7494 74940T-1

EMD (Empirical Mode Decomposition) is an effective method to decompose the linear or nonlinear signals into a set of IMFs (Intrinsic Mode Functions). The EMD method is widely used in the one-dimensional signal processing as well as two-dimensional image processing. Due to its merits in treating non-stationary signals, EMD has been applied in many image processing applications such as image fusion, noise reduction, texture analysis, image compression, image zooming, feature extraction and others. The objective of this investigation is to establish a novel image fusion procedure using a two-dimensional EMD. The proposed scheme comprises three major steps: (1) the decomposition of panchromatic and multispectral images using image EMD, (2) image fusion using mixed IMFs of panchromatic and multispectral images, and, (3) quality assessment of the fused image. The test data include QuickBird images in a sub-urban area. The quality assessment is discussed from two different aspects, i.e., visual and quantity aspects. Meanwhile, the fusion results from modified IHS, PCA and Wavelet methods are also provided for comparison.

2. METHODOLOGY In the beginning, we introduce the basic ideas and procedures of 1-D EMD. Then, the 1-D EMD is extended to a 2-D EMD. Finally, the description of EMD-based image fusion is shown in the third part. 2.1

One-Dimensional EMD

The objective of EMD is to decompose signals into limited IMFs. An IMF is defined as a function in which the number of extreme points and the number of zero crossings are the same or differ by one [6]. The IMFs are obtained by an iterative process called Sifting Process. The brief description of Sifting Process is shown as below. Step 1. Find the local maxima and minima from current input signal h(i,j)(t), where i is the number of IMF and j is the number of iteration. In the first iteration, h(1,1)(t) is the original time series signal X(t). Step 2. Compute the upper and lower envelopes u(i,j)(t) and l(i,j)(t) by interpolating the local minima and maxima using the cubic splines interpolation. Step 3. Compute the mean envelop m(i,j)(t) from the upper and lower envelopes, as shown as (1). m ( i , j ) (t ) = [ u ( i , j ) (t ) + l ( i , j ) (t )] / 2 .

(1)

Step 4. Subtract the h(i,j)(t) by the mean envelop to obtain the sifting result h(i,j+1)(t), as shown as (2). If h(i,j+1)(t) satisfies the requirement of IMF, then h(i,j+1)(t) is IMFi(t). Subtract the original by this IMFi(t) to obtain residual ri(t). ri(t) is treated as the input data and then Step 1 is repeated. If h(i,j+1)(t) does not satisfy the requirement of IMF, h(i,j+1)(t) is treated as the input data and then Step 1 is repeated. h ( i , j +1) (t ) = h ( i , j ) (t ) − m ( i , j ) ( t ).

(2)

The stopping criterion of generating an IMF is the numbers of zero-crossing and extreme are the same during the iteration. The procedure is repeated to obtain all the IMFs until the residual r(t) is smaller than a predefined value. In the end, we can decompose the signal X(t) into several IMFs and a residual rn(t). The decomposition of a signal X(t) can be written as Equation (3). Equation (3) shows that X(t) can be reconstructed from the IMFs and residual without losing the information. More details of EMD’s basic theory are discussed and can be found in [6]. n

X (t ) = ∑ IMFi (t ) + rn (t ).

(3)

i =1

2.2

Two-Dimensional EMD

The 2-D EMD determines and interpolates the extreme points in the 2-D space rather than the 1-D space. The objective of EMD is to decompose signals into limited IMFs. An IMF is defined as a function in which the number of extreme points and the number of zero crossings are the same or differ by one. The IMFs are obtained by an iterative process called Sifting Process. The brief description of 2-D EMD is summarized as below.

Proc. of SPIE Vol. 7494 74940T-2

Step 1. The generation of extreme points In the generation of extreme points, we need to locate the local maxima and minima of image h(i,j)(p,q), where i is the number of IMF and j is the number of the iteration. In the first iteration, h(1,1)(p,q) is the original image X(p,q). The morphological filter is often used to locate the local maxima and minima. However, it is not easy to extract the extreme points in the low frequency iteration. In order to overcome this problem, we propose a surface operator called “openness” [7]. The openness is defined to measure the surface reliefs in zenith and nadir angles as shown in Fig. 1a. The positive openness φL is defined as the average of DφL along eight sampling directions while the negative openness ψL is the corresponding average of DψL along them. Figures 1b and 1c show the positive and negative openness in the scale L. In the high frequency layer, L should be smaller to extract the local extreme points. On the contrary, L should be larger in the low frequency iteration. The openness is more reasonable to locate the local extreme in different scales. Moreover, the extreme point selection is related to the surrounding points in different scales rather than neighboring points.

(b)

(c)

(a)

Fig. 1. Illustration of surface openness. (a) elevation profile, (b) positive openness, (c) negative openness.

Step 2. The interpolation for mean envelope Once we found the extreme points, we perform the spline interpolation for upper envelopes u(i,j)(p,q) and lower envelopes l(i,j)(p,q) using local maxima and minima points. Then, we compute the mean envelope m(i,j)(p,q)from the upper and lower envelopes. (4)

m ( i , j ) ( p , q ) = [u ( i , j ) ( p , q ) + l ( i , j ) ( p , q )] / 2 .

Step 3. The stopping criterion We subtract the input image h(i,j)(p,q) by the mean envelope to obtain the sifting result h(i,j+1)(p,q). If sifting result satisfies the requirement of IMF, it is selected as an IMF. We then subtract the original signal by the sifting result to obtain residuals. The residuals are treated as the next input data and Step 1 is repeated. If sifting result does not satisfy the requirement of IMF, sifting result is treated as the input data and Step 1 is repeated.

(a)

(b)

(c)

(d)

Fig. 2. An example of two-dimensional EMD, (a) original image, (b) IMF1, (c) IMF2, (d) Residual.

Proc. of SPIE Vol. 7494 74940T-3

The stopping criterion of generating an IMF is when the envelope mean signal is close to zero. The sifting procedure is repeated to obtain all the IMFs until the residual is smaller than a predefined value. In the end, we can decompose the image into several high to low frequency IMFs and a residual. Fig. 2 is an example of 2-D EMD. The original image is decomposed into two IMFs and a residual. 2.3

EMD-Based Image Fusion

The idea of EMD-based image fusion is similar to the traditional approach. It uses the high frequency panchromatic IMF to replace the high frequency IMF of multispectral image. Then, we construct the mixed IMFs to obtain the fused image. The data preprocessing is to co-register the panchromatic and multispectral image into the same system; and, to resample the multispectral image to the same size of panchromatic image. The overall procedure of EMD-based image fusion is described as follows. First, we decompose these two images into several IMFs and a residual using EMD. Then, we replace the first IMF of multispectral image with the first IMF of panchromatic image. Finally, the fused image is obtained by reconstructing the mixed IMFs of multispectral image. A schematic representation of the proposed method is shown in Fig. 3. High Frequency Pan IMF Panchromatic Image

EMD

Low Frequency Pan IMFs, Residual High Frequency Ms IMF

Multispectral Image

EMD

Reconstruction of IMFs

Fused Image

Low Frequency Ms IMFs, Residuals

Fig. 3. Workflow of EMD-based image fusion.

2.4

Quality Evaluation of the Fused Image

The quality assessment is discussed from two different aspects, i.e., visual and quantity aspects. The quantity assessment is referred to both spatial and spectral quality. In other words, the fusion method should improve the spatial resolution while preserve the spectral contents. A number of indices are selected in the quantity evaluation of fused image. We compare the fused image and original multispectral image to ensure the spectral fidelity. The three major spectral indices are bias of mean, bias of standard deviation and correlation coefficient. The spatial indices are the entropy of an image. (1) Bias of Mean This index regards to the mean value of the difference between fused and multispectral images. The ideal value is zero. (2) Bias of Standard Deviation This index regards to the standard deviation of the difference between fused and multispectral images. This index is used to evaluate the distribution of bias. The ideal value is zero. (3) Correlation Coefficient This index measures the correlation between fused image and original multispectral image. The higher the correlation between the fused and original image, the better estimation of the spectral values is obtained. The idea value is one.

Proc. of SPIE Vol. 7494 74940T-4

(4) Entropy Entropy represents the information in an image. This index shows the overall detail information in the image. The greater the entropy of a fused image, the more information is included in the image.

3. TEST DATA In order to evaluate the performance and efficiency of the proposed method, the experiments are carried out on QuickBird satellite images. The land cover of QuickBird satellite image is in a sub-urban area. The nominal spatial resolution of QuickBird panchromatic and multispectral images are 0.7m and 2.8m, respectively. These two images were taken on the same path. The quality assessment includes visual and quantity aspects. In the visual aspect, we compare the fused image and the original multispectral image visually. Four major indices are selected in the quantity, i.e., bias of mean, bias of standard deviation, correlation coefficient, and entropy of image. We also use the commercial software ERDAS Image 9.2 to fuse the images with different methods including modified IHS, PCA and Wavelet. These images were used to compare with the image fused by EMD method. There are some parameters that have to be setup in the experiment. As the purpose of EMD is image fusion, we only decompose the image into two components, i.e. high frequency and reminder layer. The stopping criterion is 99% of mean envelope when it is smaller than 2 pixels. The window of openness is 5 to 15 pixels in different iteration. Both threshold of minimum points for positive openness and threshold of maximum points for negative openness are smaller than 75 degrees.

4. RESULTS Fig. 4 provides the results of different fusion methods. Visual inspection provides a comprehensive comparison between fused images. The PCA method has the largest color distortion. All these methods may improve the spatial and spectral resolution of the images. The main difference between these methods is shown as the zoomed in images in Fig. 4. The result of 2-D EMD is shown as Fig 5a. Fig 5b shows the fused image with edge effect using Wavelet approach. Among these multi-scale fusion approaches, the 2-D EMD provides promising results.

(a)

(b)

(c)

(d) (e) (f) Fig. 4. Comparison of different fusion methods using QuickBird images, (a) panchromatic image, (b) multispectral image, (c) 2-D EMD, (d) modified IHS, (e) PCA, (f) Wavelet.

Proc. of SPIE Vol. 7494 74940T-5

(a) (b) Fig. 5. Details of the three fusion methods, (a) 2-D EMD, (b) Wavelet.

In the quantitative evaluation, the indices as stated before are selected to evaluate the fusion performance. The quantitative indices value is calculated and given in Table 1. The modified IHS has the smaller bias of mean, but the correlation is relatively low when comparing to other methods. The PCA method has the largest color distortion. This statistical assessment results is the same as the visual inspection. The color distortion of modified IHS and Wavelet is better than EMD method. It is caused by the replacement of IMFs in different ranges. The correlation of EMD method is slightly better than the other ones. Table 1. Statistical information of QuickBird image.

Item Original MS 2-D EMD Modified IHS PCA

Wavelet

Band 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

Bias Mean 0.00 0.00 0.00 -10.06 -10.02 -10.05 2.25 2.47 -5.99 -178.47 -159.94 -81.75 -2.21 -1.32 -3.01

Bias Std. 0.00 0.00 0.00 70.44 70.71 70.46 61.41 45.46 68.65 70.17 63.01 35.38 47.56 44.56 46.17

Correlation 1.00 1.00 1.00 0.94 0.93 0.93 0.91 0.94 0.87 0.90 0.90 0.97 0.94 0.94 0.95

Entropy 6.33 6.54 7.07 6.32 6.20 6.66 6.58 6.68 6.89 5.57 5.70 7.11 6.26 6.41 7.07

5. CONCLUSIONS In this investigation, an EMD-based image fusion method is proposed. The experimental results are summarized as follows: (1) The proposed method may produce a fused image with smaller distortion. (2) A 2-D EMD is proposed to decompose the image into several IMFs. The proposed decomposition method is subject to investigation for future applications.

ACKNOWLEDGEMENT This investigation was partially supported by the Industrial Technology Research Institute of Taiwan.

Proc. of SPIE Vol. 7494 74940T-6

REFERENCES 1. W. J. Carper, T. M. Lillesand, and R. W. Kiefer, “The use of Intensity-Hue-Saturation transformations for merging SPOT panchromatic and multispectral image data,” Photogrammetry Engineering and Remote Sensing 56(4), 459467 (1990). 2. J. P. S. Chavez, S. C. Sides, and J. A. Anderson, “Comparison of three difference methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic,” Photogrammetric Engineering and Remote Sensing 57(3), 295-303 (1991). 3. G. Zhang, L. Wang, and H. Zhang, “A fusion algorithm of high spatial and spectral resolution images based on ICA,” International Archives of Photogrammetry Remote Sensing and Spatial Information Sciences 37(B7), 1295-1300 (2008). 4. N. Jorge, O. Xavier, F. Octavi et al., “Multiresolution-based imaged fusion with additive wavelet decomposition,” IEEE Transactions on Geoscience and Remote sensing 37(3), 1204-1211 (1999). 5. J. Wang, J. Zhang, and Z. Liu, “EMD based multi-scale model for high resolution image fusion,” Geo-spatial Information Science 11(1), 31-37 (2008). 6. N. E. Huang, Z. Shen, S. R. Long et al., “The empirical mode decomposition and the Hilbert spectrum for nonlinear and nonstationary time series analysis,” Proceedings of the Royal Society of London A454, 903-995 (1998). 7. R. Yokoyama, M. Shirasawa, and R. Pike, “Visualizing topograhpy by openness: a new application of image processing to Digital Elevation Methods,” Photogrammetry Engineering and Remote Sensing 68(3), 257-265(2002).

Proc. of SPIE Vol. 7494 74940T-7