Multimodal Medical Volumetric Data Fusion Using 3-D ... - IEEE Xplore

8 downloads 431 Views 1MB Size Report
Dec 16, 2013 - Abstract—Traditional two-dimensional (2-D) fusion framework usually suffers from the loss of the between-slice information of the third ...
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 1, JANUARY 2014

197

Multimodal Medical Volumetric Data Fusion Using 3-D Discrete Shearlet Transform and Global-to-Local Rule Lei Wang, Bin Li∗ , and Lianfang Tian

Abstract—Traditional two-dimensional (2-D) fusion framework usually suffers from the loss of the between-slice information of the third dimension. For example, the fusion of three-dimensional (3-D) MRI slices must account for the information not only within the given slice but also the adjacent slices. In this paper, a fusion method is developed in 3-D shearlet space to overcome the drawback. On the other hand, the popularly used average–maximum fusion rule can capture only the local information but not any of the global information for it is implemented in a local window region. Thus, a global-to-local fusion rule is proposed. We firstly show the 3-D shearlet coefficients of the high-pass subbands are highly non-Gaussian. Then, we show this heavy-tailed phenomenon can be modeled by the generalized Gaussian density (GGD) and the global information between two subbands can be described by the Kullback–Leibler distance (KLD) of two GGDs. The finally fused global information can be selected according to the asymmetry of the KLD. Experiments on synthetic data and real data demonstrate that better fusion results can be obtained by the proposed method. Index Terms—Three-dimensional (3-D) medical image fusion, 3-D Shearlet transform, generalized Gaussian density (GGD), Kullback–Leibler distance.

I. INTRODUCTION ULTIMODAL medical image fusion technologies facilitate better applications of medical imaging for they provide an easy access for doctors to recognize the lesion structures and functional change by studying the data of anatomical and functional modalities. For example, the combination of the positron emission tomography (PET) and computed tomogra-

M

Manuscript received February 11, 2013; revised May 18, 2013, July 13, 2013, and August 15, 2013; accepted August 15, 2013. Date of publication August 21, 2013; date of current version December 16, 2013. This work is supported by National Natural Science Foundation of China under Grant 61305038, Grant 61273249, and Grant 61105062, the Natural Science Foundation of Guangdong Province, China under Grant S2012010009886 and Grant S2011010005811, the Fundamental Research Funds for the Central Universities (SCUT) under Grant 2013ZZ045, the Key Laboratory of Autonomous Systems and Network Control of Ministry of Education, the National Engineering Research Center for Tissue Restoration and Reconstruction, and the Guangdong Key Laboratory for Biomedical Engineering (SCUT of China). Asterisk indicates corresponding author. L. Wang and L. Tian are with the School of Automation Science and Engineering, South China University of Technology, Guangzhou 510640, China (e-mail: [email protected]; [email protected]). ∗ B. Li is with the School of Automation Science and Engineering, South China University of Technology, Guangzhou 510640, China (e-mail: binlee@ scut.edu.cn). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TBME.2013.2279301

phy (CT) imaging can be used to concurrently view the tumor activity by visualizing the anatomical and physiological characteristics in oncology [1]. The fusion of CT and magnetic resonance imaging (MRI) is helpful for the neuronavigation in skull base tumor surgery [2] and the combination of the PET and MRI is useful for the diagnosis of the hepatic metastasis [3]. Due to the great need in practical applications, different fusion technologies have been developed in recent years, all of which can be generally classified into three levels [4]: pixel level, feature level, and decision level. Medical image fusion usually employs the techniques at the pixel level. According to whether multiscale decomposition (MSD) is applied, the pixellevel fusion methods can be roughly classified into MSD-based or non-MSD-based methods. Compared to the latter, the former performs better for salient image features can be captured in different scales, which are more suitable to the mechanism of the human vision [5], [6]. Though quite good results have been reported by these methods, there is still much room to improve the fusion performance for their following limitations: 1) Most of these methods are only implemented in twodimensional (2-D) space. The results are not of the same quality as those of the three-dimensional (3-D) methods due to the loss of between-slice information. For example, the fusion of the 3-D MRI and PET slices must account for the information content not only within the given slice but also the cross and adjacent slices. The 2-D fusion framework, however, fails to do this. 2) The traditional 3-D image fusion methods usually suffer from bad image representations. For example, the edges and the contours in the images cannot be well represented by the well-known 3-D wavelet transform. This is because the source images can be decomposed into only three highpass subbands in each level by the wavelet transform, losing the directional sensitivity. 3) The popularly used average–maximum formed rules are implemented in a local region of the current subband. Thus, the MSD coefficients only know the local relationship in a small region but not any of the global relationship between the two corresponding high-pass subbands. To deal with the earlier limitations, this paper presents a novel 3-D medical image fusion method. The special characteristics of this paper are: 1) Compared with the existing shearlet-based image fusion methods, our method is applied in the 3-D shearlet transform space. Besides, we also show the validity of the 2-D version of the proposed method.

0018-9294 © 2013 IEEE

198

Fig. 1.

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 1, JANUARY 2014

Common framework of the MSD-based fusion methods.

2) Compared with the other well-known MSD tools, such as the wavelet transform, the shearlet transform provides better image representations since the source images can be decomposed into more than three high-pass subbands in each level. Therefore, more features information and directional sensitivity in different subbands can be captured. 3) A global-to-local fusion rule is proposed to combine the 3-D shearlet coefficients. We show the heavy-tailed phenomenon commonly exists in the shearlet coefficients subbands and the global relationship between two high-pass subbands can be described by the Kullback–Leibler distance (KLD). After calculating the global relationship, an average scheme is implemented between the global relationship and the local relationship to produce the final fusion results. The remainder of this paper is organized as follows: Several previous methods are reviewed in Section II. The details of the proposed method are presented in Section III. Experimental results and discussions are shown in Section IV. Finally, the whole paper is concluded in Section V. II. RELATED WORK A. Background Nowadays, it has been widely reported that the MSD-based medical image methods outperform the other methods [7]–[9], for the source images can be decomposed into the low-pass subbands and the high-pass subbands in different levels to be more suitable to the mechanism of the human vision. The common framework of such methods is shown in Fig. 1: In this framework, the source images are firstly decomposed into different levels and different directions in each level. Then, the low-pass subbands and high-pass subbands are combined under the fusion rules. Finally, the fused results are obtained by the inversion of the corresponding MSD tool. Thus, the fusion performance is highly determined by the MSD tools and the fusion rules. The Laplacian pyramid transform and the wavelet transform are two of the most popular MSD tools in image fusion [7], [10], [11]. These methods, however, often produce undesirable side effects in the final fusion results, such as the block artifacts and the reduced contrast, which may result in the wrong diagnosis [7]. The reason is that wavelet-like tools decompose the source images into only three high-pass subbands, and the limited high-pass subbands result in that wavelets cannot well represent the sharp image features [7], [11]. As one

of the state-of-the-art MSD tools, the shearlet transform [12] has been reported to be the better MSD tool than the discrete wavelet transform in image fusion [13]–[15] for it decomposes the source images into more than the wavelet-like vertical, horizontal and diagonal high-pass subbands. Therefore, more directional information can be captured. In addition, compared with the curvelet transform, contourlet transform, which have been successfully introduced into medical image fusion in [16], [17], the shearlet transform has better mathematical properties. For example, different from the contourlet transform, the number of directions for shearing the images is not restricted. Furthermore, compared with the inversion of the contourlet transform, the implementation of the inversion of the shearlet transform is more efficient computationally, more details can be found in [12], [18]. The recently reported methods on the applications of the shearlet transform in image fusion domain, however, are only implemented in 2-D space. Just as we know, the performance of the 3-D shearlet transform in medical fusion has never been reported yet. As for the fusion rules, the average–maximum formed fusion schemes have been popularly employed [4], [7], [9], [11], [13], [15], [16]. These schemes, however, suffer from the loss of the global relationship since they are only implemented in a local window region. Though several alterative rules have been proposed, such as the pulse-coupled neural network [19] and the self-generating neural network [20], they still suffer from the same problem because the inputting of the neural network is also calculated in a local region. Substantially speaking, the fused coefficients are only determined according to the relationship of its neighborhood but not any of the global relationship. Since the information of the fused subbands is completely from the two subbands of the source images, the fused subbands should contain not only the local information but also the global information between them. Therefore, the combination of the local information and the global information will improve the fusion performance. According to the experiments, we find there is obvious heavy-tailed phenomenon in each high-pass subband. The heavy-tailed phenomenon can be well modeled by the generalized Gaussian density (GGD). Furthermore, the relationship of two GGDs can be measured by the KLD. The asymmetry of the KLD therefore inspires an efficient scheme to describe the global relationship between two high-pass subbands during the fusion procedure. B. 3-D Shearlet Transform An efficient MSD tool is one of the foundations for the MSDbased multimodal medical image fusion. Fig. 2 shows an intuitive example of applying the shearlet transform and the wavelet transform on a circle. It is found that the circle can be decomposed into more high-pass subbands in each level than the only vertical, horizontal and diagonal subband of the wavelet transform. Therefore, more features information and directional sensitivity in different levels can be captured by the shearlet transform. The 3-D shearlet transform mainly consists of two steps: 3-D Laplacian pyramid filter for the multiscale partition and

WANG et al.: MULTIMODAL MEDICAL VOLUMETRIC DATA FUSION

199

Fig. 2. Example of the shearlet transform and the wavelet transform on a circle. It shows only three directions can be obtained in each level by the wavelet transform (vertical, horizontal, and diagonal direction) but more than three directions can be obtained by the shearlet transform (four directions in the first level and six directions in the second level).

Fig. 4. Marginal statistics of every two 3-D high-pass subbands for the MRI data (T1, T2, and Pd) in Fig. 9. The kurtosis of each distribution shows the 3-D shearlet coefficients in each high-pass subband are highly non-Gaussian.

Fig. 3. Shapes of the wavelet and the shearlet in 3-D space. (a) Four 3-D wavelets; (b) one 3-D shearlet.

pseudo-spherical Fourier transform for the directional localization. This paper does not focus on the introduction of the 3-D shearlet transform. The readers can find the details in [21]. In Fig. 3, the shapes of the 3-D wavelet and the 3-D shearlet are shown. Mapping into the 2-D plane, every four vertices of one wavelet form a square and it is an approximate trapezium for one shearlet. It is the different shapes that result in the shearlets are able to provide better image representations than the wavelets for the edges of the circle. III. METHOD A. Marginal Statistics of the 3-D Shearlet Coefficients Fig. 4 plots the histograms of two high-pass subbands of the finest level, respectively for the MRI data of three modalities: T1, T2, and Pd that are shown in Fig. 9. It shows that all the distributions are characterized by a very sharp peak at the zero amplitude and the extended tails in both sides of the peak (this is the so-called heavy-tailed phenomenon). The kurtosis of each distribution is respectively measured as 20.91, 22.93, 79.71, 40.01, 50.36, and 129.81, which is significantly larger than the value 3 of the Gaussian distribution. By testing 448 high-pass subbands from all the data used in Section IV, it is found that the histograms of the high-pass subbands in different levels all yield similar distributions. Thus, the marginal distributions of the 3-D high-pass subbands coefficients are highly non-Gaussian. B. GGD for the 3-D Shearlet Coefficients In the earlier section, we have shown the universal existence of the heavy-tailed phenomenon. Therefore, how to model it is the key problem since it is obviously not shown as the typ-

Fig. 5. GGD for the 3-D shearlet high-pass subbands that are shown in Fig. 4. It shows that the heavy-tailed phenomenon can be well described by the GGD model.

ical Gaussian distribution. We here propose to use the GGD to describe the heavy-tailed phenomenon. The GGD is defined as [22], [23] p(x; α, β) =

β exp(−(|x|/α)β ) 2αΓ(1/β)

(1)

where Γ(·) is the Gamma function. In this definition, the parameter α determines the width of the probability density function (PDF) peak and the parameter β is related to the decreasing rate of the peak. Particularly, the Gaussian and the Laplacian PDF are the special cases when β = 2 and β = 1, respectively. How to estimate α and β can be found in [24]. A short description on the estimation procedure is detailed in the Appendix. Fig. 5 shows the heavy-tailed phenomenon in each high-pass subband can be well approximated by the GGD.

200

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 1, JANUARY 2014

pass subbands are only the approximation of the source images. For convenience, we here use the typical averaging method to produce the fused low-pass subbands. The high-pass subbands usually contain the important features information, such as the edges and the corners, in different directions. Without loss of generality, let A, B denote the source images (or volumes) to be fused, respectively, and let F denote the fused results. Let Cλl,k (i, j) denote the high-pass coefficient located at (i, j) in the lth subband at the kth decomposition level λ = A, B. The procedure of calculating the fused coefficient CFl,k (i, j) by the maximum scheme is described as  l,k l,k l,k CA (i, j), RA (i, j) ≥ RB (i, j) l,k CF (i, j) = (4) l,k l,k l,k CB (i, j), RA (i, j) < RB (i, j)

Fig. 6. KLD between two GGDs. (a) An example of the KLD with the parameter β; (b) an example of the KLD with the parameter α; (c) an example on the asymmetry of the KLD between two high-pass subbands.

C. KLD Between Two GGDs By the earlier section, the PDF of the 3-D shearlet coefficients in each high-pass subband can be completely defined via the GGD. Therefore, the global relationship of two subbands can be described by the relationship of two GGDs. According to information theory [23], two GGDs can be measured by the KLD, which is defined as [24], [25]  p(x; θq ) dx. (2) KLD(p(x; θq )||p(x; θi )) = p(x; θq ) log p(x; θi ) Substituting (1) into (2) and after some manipulations, the closed form of the KLD between two PDFs is [24] KLD(p(.; α1 , β1 )||p(.; α2 , β2 ))    β 2 β1 α2 Γ(1/β2 ) Γ(β2 + 1/β1 ) α1 1 − . (3) = log + β2 α1 Γ(1/β1 ) α2 Γ(1/β1 ) β1 An example of the KLD between two GGDs is shown in Fig. 6. According to the information theory [23], the KLD measures the difference between two probability distributions (supposing they are P and Q). Specifically, the KLD of Q from P is a measurement of the information lost when Q is used to approximate P. One of the mathematical properties for KLD is its asymmetry, i.e., it is nonsymmetric. The KLD of Q from P equals to the KLD of P from Q if and only if P = Q. An example on the asymmetry of the KLD between two corresponding high-pass subbands is shown in Fig. 6(c). We here point out that, this asymmetry inspires the proposed global fusion rule in the following section. D. Fusion Rule The fusion rule determines how to transfer the features information of the two subbands into the fused subbands. The low-

where RA and RB are the local features that are computed in a window region centered by (i, j), such as the local energy. Although such methods have been proved to be effective, they suffer from the loss of the global relationship between CAl,k and CBl,k because RA and RB are only locally calculated. To deal with this drawback, we propose a novel fusion rule, named global-to-local fusion rule. In this rule, the fused coefficient subband CFl,k contains two parts: the global part CGl,k and the local part CLl,k . The details of the proposed fusion rule are described as follows. 1) Compute the KLD between CAl,k and CBl,k KLDA B = KLD(CAl,k , CBl,k ) KLDB A = KLD(CBl,k , CAl,k ).

(5)

Because the KLD is nonsymmetric and CAl,k , CBl,k comes from different images, thus KLDA B = KLDB A . 2) Do the global fusion to compute CGl,k  l,k CA , KLDA B < KLDB A l,k CG = (6) CBl,k , KLDA B > KLDB A . The reason being that the KLD measures the global difference between CAl,k and CBl,k . KLDA B < KLDB A means CAl,k contains more information of CBl,k than the information that CBl,k contains CAl,k . And vice versa since KLDA B = KLDB A . 3) Do the local fusion to compute CLl,k  l,k l,k l,k CA (i, j), RA (i, j) ≥ RB (i, j) l,k CL (i, j) = (7) l,k l,k l,k CB (i, j), RA (i, j) < RB (i, j) where Rl,k (i, j) represents the absolute value operation in our experiments. 4) The fused subband CFl,k is calculated by CFl,k =

CGl,k + CLl,k . 2

(8)

5) The fused results can be obtained by applying the inversion of the shearlet transform on CFl,k .

WANG et al.: MULTIMODAL MEDICAL VOLUMETRIC DATA FUSION

201

IV. EXPERIMENTAL RESULTS AND DISCUSSION In this section, the validity of the proposed method is firstly shown on four pairs of 2-D images. Then, the performance is evaluated on normal MRI of three modalities (T1, T2, and Pd) and five groups of MRI with noise. Finally, the performance is evaluated on five groups of real volumetric data. Four methods, i.e., the Laplacian pyramid transform based method [26], discrete wavelet transform based method (Haar basis is used) [11], the shearlet transform based method [13], and the proposed method (LP, wavelet, shearlet, and proposed for short, respectively), are quantitatively evaluated. To give the fair comparison, the source images are all decomposed into the same levels (four levels for 2-D and three levels for 3-D) by the LP, Wavelet and Shearlet, respectively. The average–maximum rule is used in the former three methods. Mutual information (MI) [27], Entropy [27], peak-signal-to-noise ratio (PSNR) [13], structural similarity index metric (SSIM) [28], and QA B /F [29] are selected as the quantitative metrics. In image fusion domain, MI measures the similarity of the image intensity distribution between the corresponding image pairs. We here take the MI as an example to show how to calculate these measurements. More details on the other measurements can be found in the corresponding reference. Let M IA ,F denote the mutual information between the source images A, B, and the fused images F . Let M IB ,F denote the mutual information between the source images B and the fused images F . Without loss of generality, in RGB space, the mutual information can be calculated by MI =

R ,G ,B N 1  1  M IikA ,F + M IikB ,F N i=1 3 2

(9)

k

where N is the number of the slices in the data set. A. Validity of the Proposed Method on 2-D Data In this section, we show the effectiveness of the proposed method on four pairs of 2-D data. The four pairs are: pair 1 [see Fig. 7(a) and (e)], pair 2 [see Fig. 7(b) and (f)], pair 3 [see Fig. 7(c) and (g)], and pair 4 [see Fig. 7(d) and (h)]. In each column of the third row to the sixth row, the fusion results of the four methods (from the top to the bottom: the LP, wavelet, 2-D shearlet, and the proposed method) are shown. Compared with the results of the other three methods, it was found in the fused images that the structural features (in the MRI) and the color information [in the single-photon emission computed tomography (SPECT) and PET] were all well preserved by the proposed method. In Fig. 8, some zoom regions were provided to intuitively show the differences. The waveletbased method produced obvious block artifacts at the edges [see the regions labeled by the arrows in Fig. 8(a), (e), and (g)]. This was because the wavelets could not well represent the features of the images. In Fig. 8(c) and (d), the different results demonstrated that the proposed global-to-local rule outperformed the local fusion rule in capturing the features information. In addition, by the five quantitative metrics in Table I, at least four of five of the metrics could get the best value for each pair by

Fig. 7. Fusion results of the 2-D real brain images. From the top to the bottom are the four pairs: (a) and (e), (b) and (f), (c) and (g), and (d) and (h). In each column of row 3 to row 6, the fusion results of the LP method, wavelet method, the 2-D shearlet method, and the proposed method are shown, respectively.

the proposed method. Therefore, on the whole, the proposed method produced the better fusion results. B. Evaluation Using 3-D Synthetic Data In this section, we present the evaluation of the proposed method on three sets of simulated 3-D MRI brain images (T1, T2, and Pd) from the BrainWeb [30]. The scans in each set were spatially registered for the simulation. Each scan has 181 × 217 × 181 voxels with 12-bit precision, and the size of each voxel is 1 mm3 . The method was implemented in MATLAB and the experiments were performed on HP Workstation Z800 with 8 G RAM. In order to avoid possible out of memory, we selected 32 slices in this experiment. We here show the fusion of the slice no. 28 in Fig. 9. The objective evaluation

202

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 1, JANUARY 2014

Fig. 8. Some zoom regions from Fig. 7 to show the fusion differences: (a) the zoom region of Fig. 7(m); (b) the zoom region of Fig. 7(u); (c) the zoom region of Fig. 7(s); (d) the zoom region of Fig. 7(w); (e) the zoom region of Fig. 7(p); (f) the zoom region of Fig. 7(x); (g) the zoom region of Fig. 7(n); (h) the zoom region of Fig. 7(v). Specially, the MR angiogram reveals a diffuse stenosis in the proximal left middle cerebral artery [labeled by the yellow arrows in Fig. 8(g) and (h)]. The Perfusion SPECT shows these areas have increased uptake of hyperperfusion [labeled by the blue arrows in Fig. 8(g) and (h)]. Therefore, the diagnosis of tumor was considered. Obvious block artifacts are produced in Fig. 8(g) but they are different in Fig. 8(h). The difference in Fig. 8(g) and (h) demonstrates a doctor/radiologist is able to “see” more “useful” information by the proposed method in comparison to the wavelet-based methods.

TABLE I OBJECTIVE EVALUATION RESULTS OF THE REAL 2-D IMAGES

Fig. 9. Fusion results of the synthetic MRI brain images. (a) MRI T1 image; (b) MRI T2 image; (c) MRI Pd image. From the left to the right: (d) to (g) are the T1 and T2 fusion results of the LP, wavelet, 3-D shearlet, and the proposed method, respectively. (h) to (k) are the T1 and Pd fusion results of the LP, wavelet, 3-D shearlet, and the proposed method, respectively. (l) to (o) are the T2 and Pd fusion results of the LP, wavelet, 3-D shearlet, and the proposed method, respectively. (p) to (r) are the zoon regions of (d), (e), and (g), respectively. The arrows in (p), (q), and (r) demonstrate the wavelet method produces heavy artifacts at the edges but the result of proposed method is smooth enough.

results are summarized in Table II. In addition, the proposed method was also evaluated on five sets of 3-D MRI brain images with the noise of 1%, 3%, 5%, 7%, and 9%, respectively. These images were also downloaded from the BrainWeb [30]. We selected 64 slices in this experiment. The average value of the objective metrics is summarized in Table III. We show the fusion of the slice no. 60 in Fig. 10. As shown in Fig. 9, compared with the fusion results of the LP method and the Wavelet method, the results of the shearlet method were much clearer. The edges of the results obtained by the LP method and the wavelet method produced obvious block artifacts but the edges obtained by the proposed method were smooth enough [see Fig. 9(p)–(r)]. Furthermore, most of the five quantitative metrics in Tables II and III could get the best value by the propose method.

WANG et al.: MULTIMODAL MEDICAL VOLUMETRIC DATA FUSION

203

TABLE II OBJECTIVE EVALUATIONS OF THE SYNTHETIC MRI IMAGES

TABLE III AVERAGE VALUE OF THE EVALUATION RESULTS FOR THE MRI IMAGES WITH NOISE

Fig. 11. Fusion results of the MRI and SPECT for a normal man. (a) Five slices of MRI. (b) Five slices of SPECT. (c) to (f) are the fusion results of the LP, wavelet, 3-D shearlet, and the proposed method, respectively.

In addition, compared with the results of the shearlet method using the average–maximum rule, it demonstrated the validities of the proposed fusion rule. This could be interpreted as: the shearlet transform could capture more features information than that of the LP and the wavelet transform. Furthermore, the proposed fusion rule could transfer not only the information in a local window region but also the global information between two corresponding high-pass subbands into the fused images. C. Evaluation on 3-D Real Data

Fig. 10. A fusion example for the MRI data with 9% noise. (a) MRI T1 with 9% noise; (b) MRI T2 with 9% noise; (c) The fusion result of the proposed method

In this section, five groups of real data were used to evaluate the performance. They are: group 1: MRI and SPECT from a normal man; group 2: MRI and SPECT from a man with cavernous hemangioma; group 3: MRI and SPECT from a woman with anaplastic astrocytoma; group 4: MRI and PET from a man with mild Alzheimer’s disease; and group 5: MRI and SPECT from a man with AIDS dementia. All the data can be downloaded from the Whole Brain Atlas [31]. To save space, in this paper, only five continuous slices from group 1, group 3, and group 4 are shown in Figs. 11–13, respectively. The objective evaluations are summarized in Table IV. At craniotomy in Fig. 12, left parietal anaplastic astrocytoma is

204

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 1, JANUARY 2014

Fig. 12. Fusion results of the MRI and SPECT for a man with anaplastic astrocytoma. The regions labeled by arrows show the proposed method can capture more color information and better details at the edges. (a) Five slices of MRI. (b) Five slices of SPECT. (c) The fusion results of the LP method. (d) The fusion results of the Wavelet method. (e) The fusion results of the 3D Shearlet method. (f) The fusion results of the Proposed method.

found. In Fig. 13, globally widened hemispheric sulci are found and they are more prominent in parietal lobes. The regional cerebral metabolism is shown to be markedly abnormal in the PET. Fig. 14 shows an easy example of the fusion for a man with acquired immunodeficiency syndrome (AIDS) dementia. By comparing the results in Fig. 11–13, it was found that the brain structure information of the MRI were all well preserved together with the color information of the SPECT and the PET by the proposed method. The LP method only preserved the morphological structure of brain well but the color information lost. Though the wavelet method preserved the color information well, it resulted in the low contrast at the edges. Therefore, the fusion results of the proposed method provided better visual sensing. Besides, by the five quantitative metrics in Table IV, most of the metrics could get the best value in each group by the proposed method. Therefore, on the whole, the proposed method provided better fusion results. Though good performance has been shown by the proposed method, there is still much work to do. At present, our method

Fig. 13. Fusion results of the MRI and PET for a man with the Mild Alzheimer’s disease. (a) Five slices of MRI. (b) Five slices of PET. (c) to (f) are the fusion results of the LP, wavelet, 3-D shearlet, and the proposed method, respectively.

Fig. 14. Easy fusion example in group 5: the fusion of MRI and SPECT for a man with AIDS dementia. (a) A slice of MRI; (b) a slice of SPECT; (c) the fusion result of the proposed method. It shows both of the information in (a) and (b) can be well combined together by the proposed method.

is not fast enough for clinical applications. This is because the source code was written using MATLAB language. In MATLAB, operations of a large number of iterations are very slow. For clinical applications, we will design a pure C++ platform in the future. In the new platform, the speed will be highly improved with the favor of the parallel computing technology and more CPUs. The readers should specially note that our method may not perform very well when the source images are very similar to each other. This is because the global information

WANG et al.: MULTIMODAL MEDICAL VOLUMETRIC DATA FUSION

205

TABLE IV OBJECTIVE EVALUATIONS OF FIVE GROUPS OF REAL DATA

APPENDIX ESTIMATION OF THE GDD PARAMETERS FOR THE 3-D SHEARLET SUBBANDS In each 3-D shearlet high-pass subband, similar to the procedure in reference [24], we first define the likelihood function of the sample x = (x1 , . . . , xn ), which has the following independent components: L (x; α, β) = log

n 

p (xi ; α, β).

(10)

i=1

It has been shown in [23] that in this case the following likelihood equations have a unique root in probability ∂L(x; α, β) L  β|xi |β α−β =− + =0 ∂α α i=1 α N

(11)

 β  N  |xi | ∂L(x; α, β) L LΨ(1/β)  |xi | = + − log =0 ∂β β β2 α α i=1 (12) where Ψ(·) is the digamma function. When β is fixed and β > 0, the equation has the unique, real, and positive solution 1/β  N β  β |xi | . (13) α ˆ= N i=1

is measured by the asymmetry of the KLD. When the source images are similar to each other, though the asymmetry still exists, it is very weak. Therefore, the global information cannot be efficiently captured in this case, resulting in bad fusion results. According to our experience, the greater the difference between the source images, the better the fusion results will be. If this condition is satisfied, our method can also be applied in other domains, such as the fusion of the remote sensing images, the fusion of the biological images, etc.

V. CONCLUSION AND FUTURE WORK In this paper, we have developed a novel medical image fusion method in the 3-D shearlet transform space. The 3-D shearlet transform provided better image representations than the popularly used 3-D LP transform and the 3-D wavelet transform. In addition, the global-to-local fusion rule was proposed to overcome the limitations of the traditional average–maximum fusion rule. The proposed fusion rule was inspired from the asymmetry of the KLD, which could guarantee the validity of the proposed method in the information theory. Experiments on synthetic data and real data demonstrated the effectiveness of the proposed method by the fusion results of higher quality. In the future, we plan to design a pure C++ platform to reduce the time cost and extend our method for 4D medical image fusion.

Substituting (13) into (12), β is the solution of the following transcendental equation:

N ˆ ) N |xi |βˆ log (β/N βˆ ˆ i=1 |x | Ψ(1/β) i − i=1 = 0. + 1+ N βˆ βˆ βˆ |x i| i=1 (14) The parameter βˆ can be effectively solved by the NewtonRaphson iterative procedure. More details can be found in [24] and [25]. ACKNOWLEDGMENT The authors would like to specially thank Professor D. Labate in the Department of Mathematics, University of Houston, for his kind help on the 3-D shearlet transform. The authors would also like to thank the anonymous reviewers for their helpful comments and kind suggestions. REFERENCES [1] A. L. Grosu, W. A. Weber, and M. Franz, “Reirradiation of recurrent highgrade gliomas using amino acid PET (SPECT)/CT/MRI image fusion to determine gross tumor volume for stereotactic fractionated radiotherapy,” Int. J. Rad. Oncol. Biol. Phys., vol. 73, pp. 511–519, 2005. [2] S. F. Nemec, M. A. Donat, S. Mehrain, K. Friedrich, C. Krestan, C. Matula, H. Imhof, and C. Czerny, “CT-MR image data fusion for computer assisted navigated neurosurgery of temporal bone tumors,” Eur. J. Radiol., vol. 62, pp. 192–198, 2007. [3] O. F. Donati, T. F. Hany, and C. S. Reiner, “Value of retrospective fusion of PET and MR images in detection of hepatic metastases: Comparison with18F-FDG PET/CT an Gd-EOB-DTPA-enhanced MRI,” J. Nucl. Med., vol. 51, no. 5, pp. 692–699, 2010.

206

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 1, JANUARY 2014

[4] G. Piella, “A general framework for multi-resolution image fusion: From pixels to regions,” Inf. Fusion, vol. 4, no. 4, pp. 259–280, 2003. [5] J. Goutsias and H. J. A. M. Heijmans, “Nonlinear multi-resolution signal decomposition schemes. Part I: Morphological pyramids,” IEEE Trans. Image Process., vol. 9, no. 11, pp. 1862–1876, Nov. 2000. [6] H. J. A. M. Heijmans and J. Goutsias, “Nonlinear multi-resolution signal decomposition schemes. Part II: Morphological wavelets,” IEEE Trans. Image Process., vol. 9, no. 11, pp. 1897–1913, Nov. 2000. [7] S. Li, B. Yang, and J. Hu, “Performance comparison of different multiresolution transforms for image fusion,” Inf. Fusion, vol. 12, no. 2, pp. 74– 84, 2011. [8] A. A. Goshtasby and S. Nikolov, “Image fusion: Advances in the state of the art,” Inf. Fusion, vol. 8, no. 2, pp. 114–118, 2007. [9] R. Shen, I. Cheng, and A. Basu, “Cross-scale coefficient selection for volumetric medical image fusion,” IEEE Trans. Biomed. Eng., vol. 60, no. 4, pp. 1069–1079, Apr. 2013. [10] V. S. Petrovic and C. S. Xydeas, “Gradient-based multi-resolution image fusion,” IEEE Trans. Image Process., vol. 13, no. 2, pp. 228–237, Feb. 2004. [11] G. Pajares and J. M. de la Cruz, “A wavelet-based image fusion tutorial,” Pattern Recognit., vol. 37, no. 9, pp. 1855–1872, 2004. [12] G. Easley, D. Labate, and Wang-Q Lim, “Sparse directional image representations using the discrete shearlet transform,” Appl. Comput. Harmon. Anal., vol. 25, pp. 25–46, 2008. [13] Q. Miao, Ch. Shi, P. Xu, M. Yang, and Y. Shi, “A novel algorithm of image fusion using shearlets,” Opt. Commun., vol. 284, pp. 1540–1547, 2011. [14] L. Wang, B. Li, L. F. Tian, Multi-modal medical image fusion using the inter-scale and intra-scale dependencies between image shift-invariant shearlet coefficients. Inf. Fusion (2012). [Online]. Available: http://dx.doi. org/10.1016/j.inffus.2012.03.002 [15] Y. Cao, Sh. Li, and J. Hu, “Multi-focus image fusion by nonsubsampled shearlet transform,” in Proc. 6th Int. Conf. Image Graphics, IEEE, Hefei, China, 2011, pp. 17–21. [16] F. E. Ali, I. M. El-Dokany, A. A. Saad, and F. E. Abd El-Samie, “A curvelet transform approach for the fusion of MR and CT images,” J. Mod. Opt., vol. 57, pp. 273–286, 2010. [17] L. Yang, B. Guo, and W. Ni, “Multimodality medical image fusion based on multi-scale geometric analysis of contourlet transform,” Neurocomputing, vol. 72, pp. 203–211, 2008. [18] K. Guo, D. Labate, W.-Q. Lim, G. Weiss, and E. Wilson, “Wavelets with composite dilations and their MRA properties,” Appl. Comput. Harmon. Anal., vol. 20, pp. 202–236, 2006. [19] Z. Wang and Y. Ma, “Medical image fusion using m-PCNN,” Inf. Fusion, vol. 9, no. 2, pp. 176–185, 2008. [20] H. Jiang and Y. Tian, “Fuzzy image fusion based on modified selfgenerating neural network,” Expert Syst. Appl., vol. 38, pp. 8515–8523, 2011. [21] P. Negi and D. Labate, “3D discrete shearlet transform and video processing,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 2944–2954, 2012. [22] P. Moulin and J. Liu, “Analysis of multi-resolution image denoising schemes using generalized Gaussian and complexity priors,” IEEE Trans. Inf. Theory, vol. 45, pp. 909–919, 1999. [23] T. M. Cover and J. A. Thomas, Elements of Information Theory. New York, NY, USA: Wiley, 1991. [24] M. N. Do and M. Vetterli, “Wavelet-based texture retrieval using generalized Gaussian density and Kullback–Leibler distance,” IEEE Trans. Image Process., vol. 11, no. 2, pp. 146–158, Feb. 2002. [25] K. Sharifi and A. Leon-Garcia, “Estimation of shape parameter for generalized Gaussian distributions in subband decompositions of video,” IEEE Trans. Circuits Syst. Video Technol., vol. 5, no. 1, pp. 52–56, Feb. 1995. [26] M. N. Do and M. Vetterli, “Framing pyramids,” IEEE Trans. Signal Process., vol. 51, no. 9, pp. 2329–2342, Sep. 2003.

[27] S. Daneshvar and H. Ghassemian, “MRI and PET image fusion by combining IHS and retina-inspired models,” Inf. Fusion, vol. 11, pp. 114–123, 2011. [28] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004. [29] C. Xydeas and V. Petrovic, “Objective image fusion performance measure,” Electron. Lett., vol. 36, no. 4, pp. 308–309, 2000. [30] R. K.-S. Kwan, A. C. Evans, and G. B. Pike, “MRI simulation based evaluation of image processing and classification methods,” IEEE Trans. Med. Imag., vol. 18, no. 11, pp. 1085–1097, Nov. 1999. [31] K. A. Johnson and J. A. Becker. The whole brain altas [Online]. Available: http://www.med.harvard.edu/aanlib/

Lei Wang received the B.S. degree in applied mathematics from Ludong University (LDU), Yantai, China, in 2008. He is currently working toward the Ph.D. degree in the School of Automation Science and Engineering, South China University of Technology (SCUT), Guangzhou, China. His current research interests include medical image registration, multimodal medical image fusion, and statistics models in medical image processing and analysis.

Bin Li received the B.S. and Ph.D. degrees from the School of Automation Science and Engineering, South China University of Technology (SCUT), Guangzhou, China, in 2002 and 2007, respectively. He is currently an Associate Professor of Automation Science and Engineering, SCUT. His current research interests include information visualization, medical image processing, and pattern recognition.

Lianfang Tian received the B.S. and M.S. degrees in mechanical engineering from Shandong University of Technology, Jinan, China, and the Ph.D. degree in mechanical and electrical engineering from Harbin Institute of Technology (HIT), Harbin, China, in 1991, 1994, and 1997, respectively. He is currently a Professor of Automation Science and Engineering College, South China University of Technology (SCUT), Guangzhou, China. His current research areas include biomedical image processing, biomedical devices design, robotics, and pattern recognition. Dr. Tian serves as the reviewer for several national and international journals.