Enhancement of Spectral Resolution for Remotely Sensed ...

Thisarticlehasbeenacceptedforinclusioninafutureissueofthisjournal.Contentisfinalaspresented,withtheexceptionofpagination. IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING

1

Enhancement of Spectral Resolution for Remotely Sensed Multispectral Image Xuejian Sun, Lifu Zhang, Senior Member, IEEE, Hang Yang, Taixia Wu, Yi Cen, and Yi Guo

Abstract—Hyperspectral (HS) remote sensing has an important role in a wide variety of fields. However, its rapid progress has been constrained due to the narrow swath of HS images. This paper proposes a spectral resolution enhancement method (SREM) for remotely sensed multispectral (MS) image, to generate wide swath HS images using auxiliary multi/hyper-spectral data. Firstly, a set number of spectra of different materials are extracted from both the MS and HS data. Secondly, the approach makes use of the linear relationships between multi and hyper-spectra of specific materials to generate a set of transformation matrices. Then, a spectral angle weighted minimum distance (SAWMD) matching method is used to select a suitable matrix to create HS vectors from the original MS image, pixel by pixel. The final result image data has the same spectral resolution as the original HS data that used and the spatial resolution and swath were also the same as for the original MS data. The derived transformation matrices can also be used to generate multitemporal HS data from MS data for different periods. The approach was tested with three image datasets, and the spectra-enhanced and real HS data were compared by visual interpretation, statistical analysis, and classification to evaluate the performance. The experimental results demonstrated that SREM produces good image data, which will not only greatly improve the range of applications for HS data but also encourage more utilization of MS data. Index Terms—Date fusion, hyperspectral (HS) image, multispectral (MS) image, multitemporal, spectral resolution enhancement.

I. I NTRODUCTION

A

LONG WITH the development of imaging spectroscopy, hyperspectral (HS) remote sensing is playing an increasingly important role in a wide variety of fields, especially for mineral exploration [1], precision agriculture [2], military target identification [3], forest research [4], earth science research [5], and environmental monitoring [6], [7]. Compared to conventional multispectral (MS) optical sensors, HS sensors can Manuscript received April 25, 2014; revised July 27, 2014; accepted September 01, 2014. This work was supported in part by the National Natural Science Foundation of China (Project 41371362), in part by the Special research funding for public benefit industries from National Ministry of Environmental Protection (Project 2011467071), and in part by China Geological Survey (Project 1212011120222). (Corresponding author: Hang Yang.) X. Sun, L. Zhang, H. Yang, T. Wu, and Y. Cen are with the State Key Laboratory of Remote Sensing Science, Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing 100101, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]). Y. Guo is with the Digital Productivity and Services Flagship, Commonwealth Science and Industry Organization (CSIRO), North Ryde, NSW 2113, Australia (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/JSTARS.2014.2356512

obtain images with hundreds of spectral bands and provide near-laboratory-quality reflectance spectra. The data produced by HS sensors contain much more spectral information than MS data do, greatly extending the range of remote sensing applications. However, because of difficulties with instrument design, there is a tradeoff between the spectral and spatial performance of optical sensors in terms of the signal-to-noise ratio (SNR) and data acquisition rate [8]. As a consequence of this, HS sensors have more bands but low spatial resolution and narrow swath. Therefore with a higher spectral resolution, it becomes difficult to obtain high spatial resolution and wide data coverage. In recent decades, a wide range of satellite data, including MS data and HS data, have been used in military, civil, and commercial studies. Different spatial, spectral, and timeresolved data are required for different types of studies [9]. However, owing to the restrictions on spatial resolution and swath, HS data captured from satellites on-orbit are not suitable for studies in many fields. Take an example of data from Hyperion carried on the Earth-Observing 1 (EO-1) satellite. The spatial resolution of the data is 30 m with a swath of only 7.5 km [10]. Applications such as mineral exploration, land cover mapping, crop disease detection, and vegetation research usually need HS data with higher spatial resolution and wider coverage [11]–[14]. The scope of research in these areas is usually limited to a relatively small range, and better results can often be obtained with higher spatial resolution. However, a massive amount of accumulated MS data has been collected around the world, and these MS data usually have higher spatial and time resolution and a wider swath than HS data. For example, MS images captured by the Advanced Land Imager (ALI) (also carried on EO-1) have the same spatial resolution as Hyperion products, but have a swath of 37 km, and the images captured by the Landsat Thematic Mapper 5/Enhanced Thematic Mapper (TM/ETM+) have a swath of 185 km [15], [16]. As the Landsat satellites and EO-1 satellite operate on the same track, the HS or MS sensors onboard can obtain data from the same location with a very small time difference. The Système Pour l’Observation de la Terre (Spot-5) MS sensor can obtain image data with a 60-km swath and a spatial resolution as fine as 10 m [17]. Therefore, it would be very useful to develop methods that can get effective HS data through enhancing the spectral resolution of MS data, especially in situations where the former are necessary but hard to acquire. Data spectral resolution enhancement using different datasets is a form of data fusion. To obtain HS data with high spatial resolution, many studies used methods combining spectral information contained in HS images with spatial information

1939-1404 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Thisarticlehasbeenacceptedforinclusioninafutureissueofthisjournal.Contentisfinalaspresented,withtheexceptionofpagination. 2

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING

contained in MS/panchromatic images, where the fused image has the spectral characteristics of the HS image and the spatial characteristics of the MS/panchromatic image. For example, Eismann et al. [18], [19] proposed a maximum a posteriori/stochastic mixing model (MAP/SMM) to construct subpixel information to enhance the spatial resolution of an HS image by using a higher resolution coincident panchromatic image. A technique presented by Zhang et al. [20] works in the wavelet domain and is based on a Bayesian estimation for the fusion of MS and HS images. Liu et al. [21] simulated EO-1 Hyperion data from ALI MS data based on the universal pattern decomposition method (UPDM). An image sharpening method called Color Resolution Improvement Software Package (CRISP) proposed by Winter et al. [22] is used to enhance the spatial resolution of an HS image by combining information contained in a registered MS image. A coupled nonnegative matrix factorization (CNMF) unmixing algorithm proposed by Yokoya et al. [23] also shows good performance in spatial resolution enhancement of HS data. However, these fused or reconstructed HS image products are usually limited to the same range as the original HS images. Few attempts have been made to estimate wide swath HS data. In this paper, we propose a method to generate HS data by enhancing the spectral resolution of MS data when the swath of the former is consistent with the latter. We have termed this the spectral resolution enhancement method (SREM). Three datasets obtained by the Airborne Visible Infrared Imaging Spectrometer (AVIRIS) and the EO-1 satellite were used to evaluate the performance of the method and confirm its ability to generate multitemporal images. This paper is organized as follows. Section II describes the theory of SREM. Section III presents the datasets and the treatment processes used in the experiment to evaluate the performance of the method. Experimental results and discussion are presented in Section IV. We conclude the paper in Section V. II. S PECTRAL R ESOLUTION E NHANCEMENT M ETHOD The basic principle of SREM is based on the information fusion of a low-spatial-resolution HS image and a high-spatialresolution MS image of the same scene. The relationships between these two images are dependent on the correlation of the content, rather than a priori physical knowledge of the scene. Once the images are registered, the low-spatialresolution HS image can be resampled to the same size as the MS image using the nearest neighbor method simultaneously. Each HS or MS pixel spectrum occupies a column in the corresponding matrix. The HS image is described as (1) H = hTK(1) , hTK(2) , . . . , hTK(R) where hT K(i) is the column vector of each pixel spectrum (with K bands) at a spatial location designated by index i (R is the total number of pixels). In a similar manner, the MS image (with L bands) is denoted as (2) M = mTL(1) , mTL(2) , . . . , mTL(R) .

In order to establish the relationship between the HS image and the MS image, according to the method used in [22], a transformation matrix is applied GM = H + r

(3)

where G is a K × L matrix which acts to approximately reconstruct the HS image. r is assumed to be the Gaussian random error. The transformation matrix can be obtained by least squares estimation as follows: −1 . G = HM T M M T

(4)

To have enough information to calculate G in (4), the number of pixels must be no less than that of the bands in the MS image, which is easy to satisfy. An estimated HS spectrum is then obtained by ⎡ ⎢ ⎢ ⎢ ⎣

h1(i) h2(i) .. . hK(i)

⎤

⎡

g(1,1) g(1,2) ⎥ ⎢ g(2,1) g(2,2) ⎥ ⎢ ⎥ = ⎢ .. .. ⎦ ⎣ . . g(K,1) g(K,2)

⎤ ⎡ m1(i) · · · g(1,L) ⎢ m2(i) · · · g(2,L) ⎥ ⎥ ⎢ .. ⎥ · ⎢ .. .. . . ⎦ ⎣ . · · · g(K,L) mL(i)

T hT K(i) = GmL(i)

⎤ ⎥ ⎥ ⎥ (5) ⎦ (6)

T where hT K(i) is the estimated HS vector and mL(i) is the given MS vector. A new HS image is generated using (6) pixel by pixel over the entire MS image. This procedure can be described as

H = GM

(7)

where H is the estimated HS image. An estimated HS image is created from an MS image using a simple linear transformation. However, the generated HS image is limited in the scope of the MS image and the original HS image registration area because the transformation matrix G is a global average optimization for the overlapped region. Therefore, G is not performing well for the range outside the registration area. More importantly, the spectral information contained in the generated HS data is also not sufficient for analysis. There are two main reasons for poor quality of spectral information in generated image (H ). One is that there are not enough bands in MS image and the band positions are not well distributed [22]. The second and more important reason relates to the complexity of the feature types in the scene. Less diverse the features are, better the method performs. Consider a situation where there is only one material in the scene. The spectra used in (4) are very similar if we neglect the background. An extreme case is that all spectra are identical. Then the generated image will be identical to the original HS image. This implies that, the simpler the material constituent, the more stable the transformation matrix G is, which results in a better H . On the basis of the above, we can extract some of the spectra of different endmembers, which include all the materials in the overlapped region, from both registered HS and MS images. The number of the spectra of each endmember must be equal to or greater than the band number of the MS image.

Thisarticlehasbeenacceptedforinclusioninafutureissueofthisjournal.Contentisfinalaspresented,withtheexceptionofpagination. SUN et al.: ENHANCEMENT OF SPECTRAL RESOLUTION FOR REMOTELY SENSED MS IMAGE

3

These spectra will be divided into N groups representing N different endmembers. In a similar way to (3), the relationship between the HS set and the MS set of every material can be denoted as (8) G (q)M (q) = H(q) + r(q) T T is an where M (q) = m(q)T L(1) , m(q)L(2) , . . . , m(q)L(W ) L × W matrix representing the set of spectral column vectors T T T is a m(q)T L(j) ; H(q) = h(q)K(1) , h(q)K(2) , . . . , h(q)K(W ) K × W matrix representing the set of spectral column vectors h(q)T K(j) (j is in the range from 1 to W ); W is the number of pixel spectra collected from the material of class q (q is in the range from 1 to N ). A specific transformation matrix G (q) can then be calculated according to (4) when W is equal to or greater than L. N types of materials extracted from the images will generate N transformation matrices. A spectrum h (q)T K(i) with K bands can then be obtained from the matrix G (q) by multiplying mT L(i) with L bands from an MS image pixel, in accordance with (6). However, there are N available transformation matrices, and the same number of h (q)T K(i) will be produced. Therefore, the correct transformation matrix must be selected. To solve this problem, a spectral matching method is used to match the generated spectrum h (q)T K(i) with the average of original HS set H(q) which is calculated by ⎤ ⎡ W H(q) = ⎣ h(q)TK(j) ⎦ W .

Fig. 1. Flowchart of SREM.

(9)

j=1

The best matching of h (q)T K(i) will indicate the cor rect G (q) to use as a suitable transformation matrix. Then, we developed a spectral angle weighted minimum distance (SAWMD) matching method to choose the transformation matrix in SREM. SAWMD is defined as follows: n SAW M D(q)(i) = EM D(q)(i) · 1 − cos SAM (q)(i) (10) where

EM D(q)(i) = h (q)TK(i) − H(q) (11) 2 ⎤ ⎡ h (q)TK(i) , H(q) −1 ⎣ ⎦. (12) SAM (q)(i) = cos T h H(q) (q) · K(i) 2

2

EM D(q)(i) is the Euclidean distance of h (q)T K(i) and H(q); SAM (q)(i) is the spectral angle of the two spectra. The more similar the spectra, the smaller the spectral angle and SAWMD. Parameter n is used to adjust the degree of influence of spectral angle metric to SAW M D(q)(i) . In our experiment, we set it to 1 which produced satisfactory results. The SAWMD method gives full consideration to the role of the spectral shape and distance in spectral matching, and it satisfies the requirements of the SREM process.

After calculating the SAWMD of every spectrum generated by different transformation matrices, the correct HS spectrum can be obtained as follows: T hT K(i) = G (c) mL(i)

(13)

where c is given by SAWMD(c)(i) = min SAWMD(1)(i) , SAWMD(2)(i) , . . . , SAWMD(N )(i) . (14) This process can be performed over the whole MS image pixel by pixel, even beyond the scope of the original HS image. The final spectra-enhanced product H , with the same spatial resolution and swath as the MS image and the same spectral resolution as the original HS image, will then be obtained. Consider the case that material features of the same region for different years over the same period are nearly consistent; therefore, the derived G (q) can be used to enhance the spectral resolution of MS data of other time range, without additional auxiliary HS data. Its feasibility was tested by the experiment on Dataset III, and the application conditions are discussed in Section IV-D. Fig. 1shows the flowchart of the SREM process. III. DATA AND E XPERIMENT A. Test Data We used three sets of remote sensing data to test SREM. Dataset I consists of an AVIRIS HS image as well as its



Fig. 2. RGB composites of the Indian Pines (channels 43, 21, 11 for RGB) and the ground-truth reference map.

Fig. 4. RGB composites of the generated images in Dataset I: (a) and (b) are the cropped HS and spectrally resampled MS images, respectively. TABLE I AVIRIS 155-BAND S UBSET U SED IN T HIS S TUDY

Fig. 3. RGB composites of the Hyperion and ALI images: (a) and (b), (c) and (d) are Hyperion and ALI images in Dataset II and III, respectively.

subset, and a simulated TM MS image generated by spectrally resampling the original AVIRIS data. Dataset II includes real Hyperion and ALI images, which were collected simultaneously over some overlapped regions. Dataset III also consisted of Hyperion and ALI images covering the same region, but obtained at a different time. The original HS image in Dataset I was the famous 92AV3C source of data corresponding to a spectral image (145×145 pixels, 220 nonzero bands) acquired by the AVIRIS sensor in 1992 over the Indian Pine Test site in Northwestern Indiana, with a spatial resolution of 20 m. The image showed an agricultural site with several kinds of crops, which was freely available courtesy of Purdue University includes the ground truth reference data needed for assessment (see Fig. 2). We used the original HS data to generate a narrow swath HS image and a spectrally resampled MS image as the source images. The original HS data were also served as the ground truth to evaluate the simulated result. The study area of the next two datasets was located in Suihua in the south-central region of Heilongjiang Province in northeastern China and was part of the Hulan River basin of the Songnen plain (see Fig. 3). Dataset II covering the study area was collected on June 11, 2011. The centers of the Hyperion and ALI images were located at 46.7536◦ N, 127.0320935◦ E, 46.7151◦ N, and 126.7952265◦ E, respectively. Dataset III was collected on June 22, 2010, covering almost exactly the same area. Each Hyperion scene has a width of 7.5 km and each ALI scene has a width of 37 km. ALI data have nine MS bands, with the spatial resolution of 30 m, which is the same as Hyperion data. It covers a wavelength range from the visible to shortwave infrared and provides resolutions similar to those obtained by Landsat. The Hyperion sensor monitors has 242 bands, and its L1R product downloaded from the United States Geological

Survey (USGS) provides 196 effective 10-nm-wide bands from 400 to 2500 nm. We used the Hyperion and ALI data in Dataset II to generate transformation matrix from overlapped regions. The Hyperion data were also used to evaluate the spectral quality of the generated image. In Dataset III, the ALI image served as historical data from which we attempted to generate a multitemporal wide swath HS image using the same transformation matrix generated from the second dataset, whereas the Hyperion image served as real data to test the result. B. Data Preparation Preprocessing of the remote sensing data in the three datasets is necessary. First, an atmospheric correction was performed on the EO-1 data and the AVIRIS data using Fast Line-ofsight Atmospheric Analysis of Spectral Hypercubes (FLAASH: available in ENVI 4.8 package) to obtain surface reflectance data. Then, the Hyperion images were finely registered to the time corresponding to the ALI images using a first-order polynomial interpolation and nearest-neighbor resampling in the overlapped regions. For the AVIRIS image, an image of size 145×40 was cropped from the left-top corner of the original AVIRIS data as the narrow swath HS data, and a 6-band image was obtained by spectral resampling according to the spectral response curve of the TM data as the wide swath MS data (see Fig. 4). Because of the lack of calibration, low SNR, strong water vapor absorption, and serious vertical stripes, subsets of 155 AVIRIS bands and 133 Hyperion bands were selected for our study (Tables I and II). C. Data Processing In SREM, the same number of pixels of the endmembers must be extracted from both the registered HS and MS images. For Dataset I, 12 endmembers were extracted according to the


TABLE II H YPERION 133-BAND S UBSET U SED IN T HIS S TUDY

ground-truth image of the overlapped region (including three unlabeled classes, see Fig. 5). Fifteen pixels of every endmember were selected to extract spectra that were stored in a two-dimensional matrix. For Dataset II, with carefully examining terrain types in the registration area of the Hyperion and ALI images and by the auxiliary of the method of vertex component analysis (VCA) [24], we collected the spectra of eight endmembers (see Fig. 6). Twenty-five pixels of every endmember were selected. However, the numbers of extracted spectra of each endmember do not need to be the same. Transformation matrices were then, respectively, derived from the spectral matrix of the different materials using (4) and (8). The average HS spectra were calculated (see Fig. 7) as a standard spectral vector H(q) that was used to select the correct transformation matrix for each MS pixel. Wide swath HS images can be generated pixel by pixel by using the MS images of the first two datasets and the method described by (8) and (10). This process was also performed on Dataset III using the same transformation matrices obtained from Dataset II. IV. R ESULTS AND D ISCUSSION We used six bands of TM data and nine bands of ALI data to generate 155 bands of AVIRIS data and 133 bands of Hyperion data. We evaluated the results by comparisons based on visual interpretation, statistical characteristics, and classification effects. Note that the number of spectra of each material used in the matrix derivation in Dataset II was only 25, although there were millions of pixels available. In the overlapped area, the quality of the spectra-enhanced image was consistent with the original HS image for other areas. Therefore, for Datasets II and III, we tested the spectra-enhanced image in the overlapped area to evaluate the general results of the whole dataset. In order to highlight the advantages of the proposed method and its difference from the previous work, a comparison between SREM and the method of CRISP in [22] was carried out. Note that in CRISP, for the purpose of improving the spectral quality of the simulated result, a filtering method such as wavelet filter banks or butterworth filters is used to merge the simulated HS image with the original HS image. However, there were no original HS data in the nonoverlapped region for CRISP to complete its whole process. Therefore, in the comparison, the filtering process in CRISP was taken out. A transformation matrix derived from the overlapped image was used to enhance the spectral resolution of the whole MS image, as in (3) and (4).

5

A. Comparison Between Spectra-Enhanced and Real Data by Visual Interpretation To consider the general appearance of the spectra-enhanced image and the real images by a visual interpretation, we selected the specific bands as red, green, and blue (RGB) values to show the false-color composite image (see Figs. 8–10). The spectra of typical objects extracted from the same pixel location of the real HS data and spectra-enhanced data by SREM and CRISP were displayed to show their differences. From the figures, we can see that on one hand, the false color composites of the spectraenhanced images (no matter estimated by SREM or CRISP) and real HS images of Datasets I and II were very consistent, both generally and in detail. On the other hand, however, there were obvious differences between the images of Dataset III, especially for the manmade objects in the middle of the wide swath images generated by CRISP. The spectral curves of the three typical objects from the spectra-enhanced images by SREM and CRISP also showed significant differences. The simulation approach of CRISP is a global optimization method. Therefore, the simulated spectrums lost a lot of low-frequency information and many spectral features of the objects were changed. Especially for the objects (such as the river in Dataset II) that occupy a small number of pixels of the whole overlapped area, obvious spectral distortion will appear in the results [as shown in Fig. 9(d)]. Different from the result of CRISP, the spectra obtained by SREM retained most of the detailed features and was very close to the real one, which is very important for application. To compare the details of the regions of Datasets II and III, we selected a small area enclosed by rectangles in Figs. 9 and 10 to show the differences. This area covers the major material types in the whole images and adequately indicates the quality of the results. Then, we selected band 3 (average central wavelength 468.73 nm, correlation coefficient 0.9782), band 29 (725.47 nm, 0.9591), band 78 (1253.83 nm, 0.9422), and band 124 (2122.78 nm, 0.9971) of the spectra-enhanced and real AVIRIS images of Dataset I, and band 2 (498.04 nm, 0.9681), band 80 (1568.22 nm, 0.9888), band 103 (2052.45 nm, 0.9696), and band 131 (2335.01 nm, 0.9805) of the spectraenhanced and real Hyperion images of Dataset II, and band 2 (498.04 nm), band 84 (1608.61 nm), band 103 (2052.45 nm), and band 131 (2335.01 nm) of the spectra-enhanced and real Hyperion images in Dataset III, on the basis of correlation coefficients between each pair of spectra-enhanced and real bands (see Figs. 11–13) to show the differences in the gray scale image (see Figs. 14–16). Interpretation of each group of images in Figs. 14 and 15 clearly showed that the tone, texture, and border of objects had no obvious differences between the spectra-enhanced and real data. Particularly for the images of band 124 in Fig. 14(d) and (h), band 80 in Fig. 15(b) and (f), and band 84 in Fig. 16(b) and (f), where the spectra-enhanced and real HS images had the highest correlation coefficient, both images appeared to be similar. Almost the same result was obtained from a comparison of the images from band 103 in Fig. 15(c) and (g) and Fig. 16(c) and (g), except for a little deviation which was caused by the influence of stripe noises in the original image. There were a small amount of noises in



Fig. 5. Reflectance spectra of different endmembers extracted from overlapped regions of the cropped AVIRIS and simulated TM images in Dataset I: (a1) to (l1) and (a2) to (l2) show the HS and MS spectra of corn-notill, corn-min, corn, grass/pasture, grass/trees, soybean-notill, soybean-min, soybean-clean, wheat, class-unlabeled1, class-unlabeled2, and class-unlabeled3, orderly.

Fig. 6. Reflectance spectra of different endmembers extracted from overlapped regions of Hyperion and ALI images in Dataset II: (a1) to (h1) and (a2) to (h2) show the HS and MS spectra of vegetation, bare soil, river, pond, cropland, human-made feature, cloud, and the shadow of cloud, orderly.


7

Fig. 7. (a) and (b) show the average HS curves of different materials plotted in Figs. 5 and 6, respectively.

Fig. 10. Visual comparison of the result of Dataset III: (a) to (c) and (d) display the false color composite and typical object spectra of the real Hyperion data, spectra-enhanced data by SREM and CRISP, respectively.

B. Comparing Spectra-Enhanced and Real HS Data by Statistical Analysis Fig. 8. Visual comparison of the result of Dataset I: (a) to (c) and (d) display the false color composite and typical object spectra of the real AVIRIS data, spectra-enhanced data by SREM and CRISP, respectively.

Fig. 9. Visual comparison of the result of Dataset II: (a) to (c) and (d) display the false color composite and typical object spectra of the real Hyperion data, spectra-enhanced data by SREM and CRISP, respectively.

the images estimated by SREM in Fig. 16, because the spectra of these pixels in the original data were abnormal from others. The images estimated by CRISP in Fig. 16 displayed a distinctive difference in luminance, which explained the color distortion and spectral distortion in Fig. 10. It should be noted that the images of band 131 in Figs. 15(h) and 16(h) were also very well spectra-enhanced although their centre wavelength (2335.01 nm) was outside the wavelength coverage of the ALI bands (from 443 nm to 2215 nm). This indicated that the SREM could accurately simulate not only HS bands covered by the MS band, but also those that were not covered.

Statistical analysis is a very important method to evaluate the spectra-enhancement results. First, the correlation coefficients [25] between each band of the spectra-enhanced and corresponding real images in the whole region for Dataset I and the overlapped region for Datasets II and III were calculated (see Figs. 11–13). The higher the correlation coefficients were, better the method performed. As shown in Fig. 11, 153 of the 155 bands generated by SREM had correlation coefficients >0.95, indicating that the value of each spectra-enhanced band was very similar to the real one. In Datasets II and III (Figs. 12 and 13), though the average correlation coefficients (0.9819 and 0.9748) of SREM were a little lower than that in Dataset I (0.9859), nearly all of the bands had correlation coefficients >0.95 and most bands had a higher value than that of CRISP. The correlation coefficients of a small number of bands in Datasets II and III were relatively low, which is likely caused by low reflectivity or stripe noise [see Figs. 15(c) and 16(c)]. To make a detail comparison of the spectra-enhanced images generated by SREM and real images, we selected several bands with different correlation coefficients to perform a linear regression with a fixed slope of 1. In Dataset I, these bands were band 3 (0.9782), band 18 (0.9954), band 29 (0.9591), band 51 (0.9798), band 78 (the lowest correlation coefficient 0.9422), and band 124 (the highest correlation coefficient 0.9971). In Dataset II, they were band 2 (the lowest correlation coefficient 0.9681), band 29 (0.9804), band 75 (0.9886), band 80 (the highest correlation coefficient 0.9888), band 103 (0.9696), and band 131 (0.9805). In Dataset III, they were band 2 (the lowest



Fig. 11. Correlation coefficients between the spectra-enhanced and real HS data of 155 bands of Dataset I.

Fig. 14. Visual comparison of the result of Dataset I in the selected bands: (a) to (d) show the real gray images of band 3, band 29, band 78, and band 124, respectively; (e) to (h) and (i) to (l) show the spectra-enhanced gray images of the corresponding bands estimated by SREM and CRISP.

Fig. 12. Correlation coefficients between the spectra-enhanced and real HS data of 133 bands in the overlapped regions of Dataset II.

Fig. 15. Visual comparison of the result of Dataset II in the selected bands: (a) to (d) show the real gray images of band 2, band 80, band 103, and band 131, respectively; (e) to (h) and (i) to (l) show the spectra-enhanced gray images of the corresponding bands estimated by SREM and CRISP.

Fig. 13. Correlation coefficients between the spectra-enhanced and real HS data of 133 bands in the overlapped regions of Dataset III.

correlation coefficient = 0.9481), band 21 (correlation coefficient = 0.9538), band 57 (correlation coefficient = 0.9845), band 84 (the highest correlation coefficient = 0.9850), band 103 (correlation coefficient = 0.9575), and band 123 (correlation coefficient = 0.9742). We sampled 10 000 pixels via the generate random sample module of ENVI 4.8 from both the

spectra-enhanced and real images in the overlapped regions. The regression analysis scatter diagrams with regression equations, R square values, and root-mean-square (RMS) error are shown in Figs. 17–19. The data points for all the bands in Dataset I and II were clustered around the fitted line with a very small intercept (not higher than 0.0049). They all had very high R2 values and small RMS, and their fitted lines were very close to the line 1:1, indicating that these bands were well spectraenhanced and highly similar to the real ones. The data points


9

Fig. 18. Linear regression analysis with a fixed slope of 1 between the selected bands of spectra-enhanced images by SREM and real images in Dataset II: (a) to (f) are scatter diagrams with the regression equation, R2 value, and RMS for bands 2, 29, 75, 80, 103, and 131, respectively. Fig. 16. Visual comparison of the result of Dataset III in the selected bands: (a) to (d) show the real gray images of band 2, band 84, band 103, and band 131, respectively; (e) to (h) and (i) to (l) show the spectra-enhanced gray images of the corresponding bands estimated by SREM and CRISP.

Fig. 19. Linear regression analysis with a fixed slope of 1 between the selected bands of spectra-enhanced images by SREM and real images in Dataset III: (a) to (f) are scatter diagrams with the regression equation, R2 value, and RMS for bands 2, 21, 57, 84, 103, and 123, respectively. Fig. 17. Linear regression analysis with a fixed slope of 1 between the selected bands of spectra-enhanced images by SREM and real images in Dataset I: (a) to (f) are scatter diagrams with the regression equation, R2 value, and RMS for bands 3, 18, 29, 51, 78, and 124, respectively.

for bands in Dataset I (see Fig. 17) clustered around the fitted line more tightly than those in Dataset II (Fig. 18) and the data points for bands in Dataset III (see Fig. 19), and particularly for band 2 and 21, were loosely scattered and did not cluster around the fitted line as tightly as those in Datasets I and II. This was primarily due to the complex nature of the spectra of high reflectivity objects (such as cloud) in Datasets II and III and also because that there were no registration error between the spectra-enhanced and original images in Dataset I. The R2 value of band 103 in Datasets II and III was relatively lower than the others because of the stripes that exist in this band of the original Hyperion image. However, the R2 values of band 103 were still high enough to suggest good linearity.

To give a comprehensive evaluation by objective and quantitative analysis, the spectra-enhanced images were compared to the original images in the overlapped regions using three indices: spectral angle mapper (SAM), erreur relative globale adimensionnelle de synthèse (ERGAS) [26], and universal image quality index (UIQI) [27]. Table III showed the average cumulative value of the three indices. SAM is related to spectral distortion, while ERGAS is mainly related to radiometric distortion. They should take values as low as possible (ideally 0). UIQI models any distortion as a combination of three different factors: loss of correlation, luminance distortion, and contrast distortion. The dynamic range of UIQI is [-1,1] (ideally 1). As shown in Table III, SREM performs much better than CRISP when considering no matter SAM, ERGAS, or UIQI. The values of SAM and UIQI got from SREM reached a satisfactory level. The values of ERGAS were relatively high, especially for Datasets II and III. ERGAS usually should be



TABLE III AVERAGE C UMULATIVE Q UALITY I NDICES B ETWEEN THE O RIGINAL AND S PECTRA -E NHANCED I MAGES

Fig. 21. Classification results for Dataset II in the selected region: (a) to (e) display the selected ROI and the classification images of real Hyperion data, spectra-enhanced data by SREM and CRISP, and the ALI data, respectively.

Fig. 20. Classification results for Dataset I in the selected region: (a) to (d) displayed the classification images of the real AVIRIS data, spectra-enhanced data by SREM and CRISP, and the spectrally resampled TM data, respectively.

smaller than 3 in MS image fusion evaluation which indicates very good performance. But we find that it is hard to achieve this level for the HS evaluation with hundreds of bands, especially for that the tested image were simulated from an MS image using a small amount of spectral samples. As the spectral intensity of the same object in the MS image is often different, the result obtained by SREM showed relative poor performance in the average RSME, leading to a bigger value of ERGAS. However, this index has still demonstrated the advantage of SREM beyond CRISP. Overall, Table III showed good quality of our results. C. Comparing Classification

Spectra-Enhanced

and

Real

Data

Fig. 22. Classification results for Dataset III in the selected region: (a) to (e) display the selected ROI and the classification images of real Hyperion data, spectra-enhanced data by SREM and CRISP, and the ALI data, respectively.

by

To further evaluate the performance of the method, classification using the support vector machine (SVM) classifier [28], [29] was performed on Datasets I, II, and III (see Figs. 20–22). In Dataset I, 10% labeled samples of per class in the ground reference were randomly chosen for training and the rest for validation, leaving us approximately 1141 training pixels. Fig. 20 showed the classification maps for the real AVIRIS, spectra-enhanced, and simulated TM data. Table IV presented the overall accuracy (OA), kappa coefficient (κ), as well as the producer accuracy (PA), and the user accuracy (UA) of the individual classes and the average PA and UA of all the classes. In Dataset II, by photo-interpretation using high-resolution color image, the region of interest (ROI) was delineated with seven land cover classes. One thousand three hundred and

sixty-eight ground reference pixels were collected and subdivided into: cropland-1, woods-1, cropland-2, woods-2, bare soil, town, and trees. Also 10% of the ground reference pixels were randomly chosen for training and the rest for validation. Fig. 21 showed the ROI and the classification maps for the real Hyperion, spectra-enhanced, and ALI data. Table V presented the OA, κ, PA, and UA of the individual classes, and the average PA and UA of all the classes. In Dataset III, 1276 ground reference pixels were collected and subdivided into: cropland-1, woods-1, cropland-2, woods-2, bare soil, town, and trees. In a similar manner, the results were shown in Fig. 22 and Table VI. From Fig. 20 and Table IV, the results demonstrated that, compared to the classification map of simulated TM data, the other two maps of the real AVIRIS data and spectra-enhanced


11

TABLE IV C LASSIFICATION ACCURACY ON DATASET I

TABLE V C LASSIFICATION ACCURACY ON DATASET II

data by SREM led to better performance. Especially for the classes of “Grass/Pasture” and “Wheat,” many more misclassifications were made in the classification result of simulated TM data. One can observe obviously improvement in OA for the real AVIRIS and spectra-enhanced data by SREM, with respect to the simulated TM data. They both showed

an improvement of the order of 6%. Conversely, the result of the spectra-enhance data by CRISP did not show noticeable improvement than the MS data. The classification map of the spectra-enhanced AVIRIS data by SREM was very close to the one of the real data, except a lower PA and UA for “Bldg-Grass-Tree-Drives” only. It was worth noting



TABLE VI C LASSIFICATION ACCURACY ON DATASET III

that the classes of “Alfalfa,” “Grass/pasture-mowed,” “Haywindrowed,” “Oats,” “Woods,” “Bldg-Grass-Tree-Drives,” and “Stone-steel towers” were not contained in the overlapped region for spectra extraction. However, most of them showed very good performance in the classification results got from SREM. This indicated that the SREM was an effective method to obtain good quality data for HS remote sensing applications. From Fig. 21 and Table V, it was clear that the classification results for the four kinds of data were very similar to each other. Because the classes contained in the scene were quite distinguishable. However, the OA of the spectra-enhanced data by SREM was 97.40%, even a little higher than that of the real data, indicating that our method could generate HS data from the real MS data successfully when the quality of spectral information was very close to that of the real HS data. The results calculated from Dataset III also displayed very high accuracy, which showed very good application potential of the spectral-enhanced multitemporal image obtained by SREM. D. Discussion The feature spectra used in the SREM were collected discrepantly by different endmember extraction method. The performance of the approach was therefore partly influenced by how the chosen spectra are representing different materials and there might be spectra that were not included to obtain the best result. However, the spectra extraction process is not necessary for every group of images because a transformation matrix library for specific sensors can be established and derived from large amounts of historical data with respect to various places and times. We could use the matrix stored in the library to obtain a spectra-enhanced image from multitemporal MS data without the auxiliary of the real HS data, and the feasibility of this idea will be verified in our future work.

The results of image simulation by CRISP mainly depend on the only transformation matrix derived from a part of the images in the overlapped area. The difference of feature component in the scene could have a significant impact on the results, especially for the materials occupying a small portion. Unlike CRISP, SREM is more stable and better deals with small sample. It is worth mentioning that ALI is not the only source for real MS images; any MS images with spatial resolution higher than (or equal to) the original HS image (such as from TM or Spot-5) would be suitable. In Dataset II of this study, we chose Hyperion and ALI data mainly to evaluate the method because both types of data were captured on the same satellite platform with almost no time difference. By using these data, we could obtain registered images with high accuracy and better evaluate the method. However, the image registration process in this paper served mainly to easy spectra extraction and result evaluation. High precision of image registration is not necessary if the HS and MS spectra of different objects are available. These spectra might be extracted from images or obtained directly from ground spectral equipment. Airborne HS data and satellite high-resolution MS data could also be used to simulate large data coverage and meet the needs of airborne data applications. V. C ONCLUSION An SREM for remotely sensed MS image has been presented, with the aim of generating wide swath HS data and improving MS data utilization. By using SREM, a spectralenhanced MS image with spectral resolution the same as that of the original HS data and swath the same as that of the original MS data can be generated. These transformation matrices were also used to simulate HS data from MS data for different years over the same period. We compared the spectra-enhanced and real data in the overlapped region by visual interpretation,


statistical analysis, and classification application. The results showed that images generated by the SREM are very similar to real data, demonstrating the excellent performance of our method. SREM is relatively straightforward and very easy to implement thanks to its simplicity. This is valuable when the HS MS image pair have millions of pixels. R EFERENCES [1] T. Tukiainen and B. Thomassen, “Application of airborne hyperspectral data to mineral exploration in North-East Greenland,” Geol. Surv. Den. Greenl. Bull., vol. 20, pp. 71–74, 2010. [2] A. Baiano, C. Terracone, G. Peri, and R. Romaniello, “Application of hyperspectral imaging for prediction of physico-chemical and sensory characteristics of table grapes,” Comput. Electron. Agric., vol. 87, pp. 142–151, Sep. 2012. [3] P. W. T. Yuen and M. Richardson, “An introduction to hyperspectral imaging and its application for security, surveillance and target acquisition,” Imag. Sci. J., vol. 58, no. 5, pp. 241–253, Oct. 2010. [4] H. Latifi, F. Fassnacht, and B. Koch, “Forest structure modeling with combined airborne hyperspectral and LiDAR data,” Remote Sens. Environ., vol. 121, pp. 10–25, Jun. 2012. [5] D. P. Shrestha, D. E. Margate, F. van der Meer, and H. V. Anh, “Analysis and classification of hyperspectral data for mapping land degradation: An application in southern Spain,” Int. J. Appl. Earth Observ. Geoinf., vol. 7, no. 2, pp. 85–96, Aug. 2005. [6] M. Govender, K. Chetty, and H. Bulcock, “A review of hyperspectral remote sensing and its application in vegetation and water resource studies,” Water SA, vol. 33, no. 2, pp. 145–151, Apr. 2007. [7] B. Zhang, D. Wu, L. Zhang, Q. Jiao, and Q. Li, “Application of hyperspectral remote sensing for environment monitoring in mining areas,” Environ. Earth Sci., vol. 65, no. 3, pp. 649–658, Feb. 2012. [8] N. Mayumi and A. Iwasaki, “Image sharpening using hyperspectral and multispectral data,” in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), 2011, pp. 519–522. [9] C. E. Woodcock and A. H. Strahler, “The factor of scale in remote sensing,” Remote Sens. Environ., vol. 21, no. 3, pp. 311–332, Apr. 1987. [10] M. Folkman, J. Pearlman, L. Liao, and P. Jarecke, “EO-1/Hyperion hyperspectral imager design, development, characterization, and calibration,” in Proc. SPIE Hyperspectral Remote Sens. Land Atmos., 2001, vol. 4151, pp. 40–51. [11] S. Bhattacharya, T. J. Majumdar, A. S. Rajawat, M. K. Panigrahi, and P. R. Das, “Utilization of Hyperion data over Dongargarh, India, for mapping altered/weathered and clay minerals along with field spectral measurements,” Int. J. Remote Sens., vol. 33, no. 17, pp. 5438–5450, 2012. [12] S. Pignatti et al., “Evaluating Hyperion capability for land cover mapping in a fragmented ecosystem: Pollino National Park, Italy,” Remote Sens. Environ., vol. 113, no. 3, pp. 622–634, Mar. 2009. [13] T. Al-Moustafa, R. P. Armitage, and F. M. Danson, “Mapping fuel moisture content in upland vegetation using airborne hyperspectral imagery,” Remote Sens. Environ., vol. 127, pp. 74–83, Dec. 2012. [14] G. H. Mitri and I. Z. Gitas, “Mapping post-fire forest regeneration and vegetation recovery using a combination of very high spatial resolution and hyperspectral satellite imagery,” Int. J. Appl. Earth Observ. Geoinf., vol. 20, pp. 60–66, Feb. 2013. [15] D. R. Hearn et al., “EO-1 advanced land imager overview and spatial performance,” in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), 2001, pp. 897–900. [16] P. S. Barry et al., “EO-1 hyperion hyperspectral aggregation and comparison with EO-1 advanced land imager and Landsat 7 ETM+,” in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), 2002, pp. 1648–1651. [17] J. P. Gleyzes et al., “SPOT5: System overview and image ground segment,” in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), 2003, pp. 300–302. [18] M. T. Eismann and R. C. Hardie, “Application of the stochastic mixing model to hyperspectral resolution, enhancement,” IEEE Trans. Geosci. Remote Sens., vol. 42, no. 9, pp. 1924–1933, Sep. 2004. [19] R. C. Hardie, M. T. Eismann, and G. L. Wilson, “MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor,” IEEE Trans. Image Process., vol. 13, no. 9, pp. 1174–1184, Sep. 2004. [20] Y. F. Zhang, S. De Backer, and P. Scheunders, “Noise-resistant waveletbased Bayesian fusion of multispectral and hyperspectral images,” IEEE Trans. Geosci. Remote Sens., vol. 47, no. 11, pp. 3834–3843, Nov. 2009.

13

[21] B. Liu, L. Zhang, X. Zhang, B. Zhang, and Q. Tong, “Simulation of EO-1 Hyperion data from ALI multispectral data based on the spectral reconstruction approach,” Sensors, vol. 9, no. 4, pp. 3090–3108, Apr. 2009. [22] M. E. Winter, E. M. Winter, S. G. Beaven, and A. J. Ratkowski, “Hyperspectral image sharpening using multispectral data,” in Proc. IEEE Aerosp. Conf., 2007, pp. 1–9. [23] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 2, pp. 528–537, Feb. 2012. [24] J. M. P. Nascimento and J. M. B. Dias, “Vertex component analysis: A fast algorithm to unmix hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 4, pp. 898–910, Apr. 2005. [25] Z. Shi, Z. An, and Z. Jiang, “Hyperspectral image fusion by the similarity measure-based variational method,” Opt. Eng., vol. 50, no. 7, p. 077006, 2011. [26] T. Ranchin and L. Wald, “Fusion of high spatial and spectral resolution images: The ARSIS concept and its implementation,” Photogramm. Eng. Remote Sens., vol. 66, no. 1, pp. 49–61, 2000. [27] Z. Wang and A. C. Bovik, “A universal image quality index,” IEEE Signal Process. Lett., vol. 9, no. 3, pp. 81–84, Mar. 2002. [28] V. N. Vapnik, “An overview of statistical learning theory,” IEEE Trans. Neural Netw., vol. 10, no. 5, pp. 988–999, Sep. 1999. [29] F. Melgani and L. Bruzzone, “Classification of hyperspectral remote sensing images with support vector machines,” IEEE Trans. Geosci. Remote Sens., vol. 42, no. 8, pp. 1778–1790, Aug. 2004.

Xuejian Sun received the B.E. degree in surveying and mapping engineering from Tongji University, Shanghai, China, in 2010, and currently he is pursuing the Ph.D. degree in cartography and geographical information systems at the Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing, China. His research interests include remote sensing data processing, hyperspectral remote sensing feature extraction, hyperspectral data fusion, and superresolution reconstruction.

Lifu Zhang (S’04–M’05–SM’14) received the B.E. degree in photogrammetry and remote sensing from the Department of Airborne Photogrammetry and Remote Sensing, Wuhan Technical University of Surveying and Mapping (WTUSM), Wuhan, China, in 1992, the M.E. degree in photogrammetry and remote sensing from the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, WTUSM, in 2000, and the Ph.D. degree in photogrammetry and remote sensing from the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China, in 2005. From 2003 to 2004, he was a Visiting Researcher with the Department of Information and Computer Sciences, Nara Women’s University, Nara, Japan. From 2005 to 2007, he was a Postdoctoral Researcher with the Institute of Remote Sensing and Geographic Information System, School of Earth and Space Sciences, Peking University, Beijing, China. In 2011, he was an Advanced Visiting Researcher with the Earth Science and Resource Engineering, Commonwealth Scientific and Industrial Research Organization (CSIRO), Sydney, Australia. In 2007, he joined the former Institute of Remote Sensing Applications (IRSA), Chinese Academy of Sciences (CAS), Beijing, China, where he was a Research Professor and a Scientist Team Leader. From November 2012, IRSA was merged and given a new name of Institute of Remote Sensing and Digital Earth (RADI), CAS, where he is a Professor and the Head of the Hyperspectral Remote Sensing Laboratory, RADI, CAS. His research interests include hyperspectral remote sensing, imaging spectrometer system development, and its applications. Dr. Zhang is a Member of SPIE, a Senior Member of the Academy of Space Science of China, and is also a Committeeman of Chinese National Committee of the International Society for Digital Earth (CNISDE), a Vice Chairman of Hyperspectral Earth Observation Committee (HEOC), CNISDE, and a Standing Committeeman of the Expert Committee of China Association of Remote Sensing Applications.



Hang Yang received the B.Ag. degree in agricultural education from the Department of Agricultural, Laiyang Agricultural College, Qingdao Agricultural University, Qingdao, China, in 2001, the M.S. degree in ecology from China Agricultural University (CAU), Beijing, China, in 2004, and the Ph.D. degree in cartography and geographical information systems with the Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing, China, in 2011. His research interests include hyperspectral thermal infrared remote sensing, atmospheric correction, and information extraction using hyperspectral remote sensing.

Taixia Wu received the B.Ag. degree in forestry remote sensing from Nanjing Forestry University, Nanjing, China, in 1999, the M.S. degree in cartography and GIS from the College of Urban and Environmental Sciences, Northeast Normal University, Changchun, China, in 2006, and the Ph.D. degree in remote sensing from Peking University, Beijing, China, in 2010. Currently, he is an Associate Research Fellow with the Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing, China, since 2010. His research interests include hyperspectral remote sensing.

Yi Cen received the Ph.D. degree in photogrammetry and remote sensing from the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China, in 2008. Currently, she is a Postdoctoral Researcher in Remote Sensing with the Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing, China. Her research interests include hyperspectral remote sensing application, emphasis on environmental monitoring, carbon circle, and greenhouse gases.

Yi Guo received the B.E. degree in electrical engineering from North China University of Technology, Beijing, China, in 1998, and the Ph.D. degreefrom the University of New England, Armidale, Australia, in 2008. Since 2008, he has been a Research Scientist with Commonwealth Science and Industry Organization (CSIRO), Sydney, Australia. His research interests include machine learning and computational statistics for big data.