A Variational Approach for Combined Segmentation

A Variational Approach for Combined Segmentation and Estimation of Respiratory Motion in Temporal Image Sequences Jan Ehrhardt

Alexander Schmidt–Richberg

Department of Medical Informatics University Medical Center Hamburg-Eppendorf 20246 Hamburg, Germany

Faculty of Computer Science Dresden University of Technology 01062 Dresden, Germany

Heinz Handels Department of Medical Informatics University Medical Center Hamburg-Eppendorf 20246 Hamburg, Germany

Abstract In this paper a variational approach for the combined segmentation and registration of temporal image sequences is presented. The purpose of the proposed method is to estimate respiratory–induced organ motion in temporal CT image sequences and to segment a structure of interest simultaneously. In this model the segmentation of all images in the sequences is obtained by finding a non–linear registration to an initial segmentation in a reference image. A dense non-linear displacement field is estimated using image intensities and segmentation information in the images. Both problems (registration and segmentation) are formulated in a joint variational approach and solved simultaneously. A validation of the combined registration and segmentation approach is presented and demonstrates that the simultaneous solution of both problems improves the segmentation performance over a sequential application of the registration and segmentation steps.

1. Introduction Spatiotemporal image data sets, like 4D CT or dynamic MRI, open up the possibility to model and analyze the dynamic behavior of inner organs. In various medical applications, e.g cardiology or radiotherapy, it is important to generate four-dimensional models of structures of interest that describe the temporal change in organ position and shape.

978-1-4244-1631-8/07/$25.00 ©2007 IEEE

For example, in radiotherapy of the thorax and upper abdomen such models can be used to predict respiratory motion and deformation of organs during treatment in order to enable respiratory gating or robotic radiotherapy [24, 23]. Furthermore, they can be used to optimize radiotherapy plans in order to be less sensitive to breathing-induced organ motion [23]. In our project, 4D data sets are used to analyze the influence of the breathing motion on the dose distribution under a conventional static irradiation. For this an irradiation of the clinical target volume is simulated and 4D CT data acquired during free breathing are used to sum up the applied dose per voxel. Two main problems arise: the structure of interest (here: liver) has to be segmented in the 4D data set and the motion of each voxel has to be estimated in the temporal image sequence. From earlier work it is known that respiratory motion causes complex deformations of inner organs [22], therefore a non-linear registration approach is required. Furthermore the segmentation of the 4D CT data is a difficult problem since these data show a poor soft tissue contrast. We address this problem by using shape prior information for segmentation. Different approaches exist for modeling organ motion by solving the segmentation and registration problem independently [26], by using the segmentation results to guide the registration process [27, 23], or by using the registration results to transfer an initial segmentation to all time frames [22]. However, solutions of both problems depend on each other. In this paper, we present a method for simultaneous segmentation and non–linear registration using a variational

approach. This method is used to determine structures of interest and to estimate respiratory organ motion in 4D CT image sequences. There have already been some attempts in the literature for simultaneous segmentation and registration. Active contour approaches [6, 29, 18, 30], markov random fields [10, 28] and a Bayesian framework based on a maximum aposteriori probability (MAP) estimation [21] were proposed to formulate a joint segmentation and registration for different applications. However, all these approaches are restricted to lower dimensional parametric transformations. Respiratory motion causes complex deformations of inner organs and therefore it might be important to model organ motion as non–linear deformation [11, 22, 3]. In [5] the authors extend the work of [28] to include a B–spline based non–rigid transformation. In [2, 8] the joint segmentation/registration of atlas and study images is formulated by a mixture of Gaussians model to describe image intensities and by a non–linear registration of the study image with tissue probability maps provided by the atlas. Variational approaches based on level set segmentation and non–linear registration where proposed in [7, 1, 13, 25, 12]. However, in these approaches the deformation field is defined only near object surfaces [1, 25] or no shape–prior information is incorporated in the model [25, 13, 12]. In [7] image sequences are segmented into regions of homogeneous motion. However, the assumption of piecewise homogeneous motion does not hold for soft tissue deformations induced by heart beat or respiration. In this paper, a framework for level set based segmentation and non–linear registration using shape prior information is presented. The purpose of the presented method is to estimate respiration induced organ motion in spatiotemporal CT image sequences and to segment a structure of interest simultaneously. Therefore, a dense non-linear displacement field is estimated using image intensities and segmentation information. A segmentation is obtained by a non-linear registration with a prior shape and by using a level set formulation related to the Mumford–Shah functional [17]. Both problems are formulated in a joint variational approach and solved simultaneously. A validation of the combined registration and segmentation approach is presented and demonstrate that the simultaneous solution of both problems within a single mathematical framework improves the segmentation performance over a sequential application of the registration and segmentation steps. The organization in this paper is as follows. First, we recall shortly variational approaches for registration and for level set based segmentation. We then present our way to combine both approaches using shape prior information and outline the numerical method to minimize the resulting energy functional. Experimental results of the model which were applied to temporal CT image sequences of the liver

are shown in section 3.

2. Method In our application, the goal is to segment a structure of interest, e.g. lung, liver, heart, tumor, and to trace the organ motion in a sequence of spatiotemporal CT images I0 (x), I1 (x, ), . . ., In−1 (x, ) acquired throughout the respiratory cycle [14, 15, 9]. Organ motion is commonly described by a displacement vector field which links the location of points in the image frames. Here, we are not interested in the deformation of organ surfaces only, but in a dense deformation field defined throughout the whole image. Furthermore, the outline of anatomical structures is needed for radiotherapy treatment planning and has to be defined in the whole image sequence. As expected, parts of the boundaries are not well defined for most of the objects of interest or they are connected with neighboring organs (see fig. 1 bottom left). Therefore, it seems to be impossible to separate them without prior knowledge. A reference shape for the segmentation is provided for one of the images, let’s say I0 , based on a previous semi–automatic segmentation step. Afterwards, the object of interest is segmented in the whole image sequence and the motion field is estimated by combined segmentation/registration between the reference image I0 and the target images Ij , j = 1, . . . , n − 1.

2.1. Variational non–linear image registration. Due to the wide range of applications a variety of different registration techniques has been developed in the last two decades (see [31] for an survey). Here, we will focus on non–linear intensity–driven approaches. Given two images, a reference R and a target T , the aim of image registration is to find a transformation ϕ(x) = x − u(x) that matches R ◦ ϕ onto T . The general registration problem may be phrased as [16]: J Reg [u] = D[R, T ; u] + αS[u] → min,

(1)

where D models the distance measure, e.g. sum of squared differences (SSD) or mutual information, and S is a regularizer to constrain the calculated transformation to physically meaningful movements1 . The parameter α is used to balance the smootheness of the displacement versus the similarity of the images. Examples for similarity measures D, regularizers S and numerical methods for minimizing eq. (1) are shown in [16]. In our application we choose the SSD measure for the intensity–based registration component and the diffusive 1 In contrast to the notation in other publications, e.g. in [16], in our formulation the reference image R is transformed. Therefore, the image T is called target instead of template.

regularization to ensure a smooth deformation field: 1 D[R, T ; u] = (T (x) − R(x − u(x)))2 dx 2 Ω d 1 ∇ul 2 dx . S[u] = 2 Ω l=1

2.2. Variational level–set segmentation. Level set methods have been utilized for image segmentation in many applications. An important class of these methods derives its evaluation equation from a variational formulation, where an energy functional is minimized. Because strong edge information is not always present along the entire object boundary, purely edge–based models are often inadequate for medical image segmentation [4, 19]. Here, for level set based segmentation a region–based energy term is defined as: J Seg [Φ] = E[T ; Φ] + γI[Φ], where E is the external energy depending on the image function T and I is designed to keep the evolving curve smooth. In our application, these energy terms are defined as I[Φ] = δ (Φ(x)) |∇Φ(x)| dx and Ω H (Φ(x))Fout (x) dx E[T ; Φ] = Ω (1 − H (Φ(x)))Fin (x) dx, + Ω

where Φ : Ω → IR (Ω ⊂ IRd ) is the desired level set function, Fin and Fout are region descriptors for inside and outside the contour, and δ and H are continuous approximations of Dirac and Heaviside distributions (see [18] for a detailed description). Chan and Vese [4] propose the following particular choices for Fin and Fout , which are related to the Mumford–Shah functional: 2

2

Fin (x) = (T (x) − cin ) and Fout (x) = (T (x) − cout ) , where cin and cout denote the mean gray values inside and outside the contour defined by Φ. Other region descriptors are suggested by [18].

shape knowledge is incorporated in the model by a coupling term as suggested in [20]: ˆ (Φ(x) − Φ(x ˆ − u(x)))2 dx . ˆ Φ, u] = 1 N (Φ, Φ) P[Φ; 2 Ω (2) N is a binary function, which restrict the optimization to all pixels within a range of distance from the actual shape: 0 , min(|x|, |y|) > . N (x, y) = 1 , min(|x|, |y|) ≤ Eq. (2) couples the segmentation and registration process by forcing the actual level set segmentation Φ to be close to ˆ ◦ ϕ. Integration of intensity– the deformed shape prior Φ based non–linear registration, level set segmentation and prior shape knowledge can now be done in the following optimization criterion: J [u, Φ]

:= λ1 D[R, T ; u] + λ2 S[u] + λ3 E[T ; Φ] ˆ u, Φ]. + λ4 I[Φ] + λ5 P[Φ; (3)

The aim is to find simultaneously a segmentation and a transformation by minimizing the energy functional J with respect to u and Φ. λi , (i = 1, . . . , 5) are parameters balancing the influence of the five terms in the model. In contrast to other approaches for joint segmentation and non-linear registration [1, 25] the displacement field u is determined for the whole image domain and not only between object surfaces. In regions distant from the contour the displacements are influenced mainly by image intensities. Near the object surface the influence of the coupling term grows. However, due to the smoothing criterion S the segmentation information may affect the transformation in a large image region. Furthermore, we penalize distances between Φ and the ˆ instead to demand Φ = Φ ˆ ◦ ϕ. deformed reference shape Φ ˆ ◦ ϕ are possible dependThus, deviations between Φ and Φ ing on the weighting parameter λ5 . Current non–linear registration methods presume smooth continuous deformation fields, but complex deformations of inner organs are sometimes non–continuous (e.g. motion of the lung along the pleura). Therefore, a correct registration is not possible in every case. By choosing a variable weighting parameter λ5 (x) in eq. (3) prior knowledge about problematic regions can be incorporated in the model.

2.4. Numerical Methods. 2.3. Variational framework for joint segmentation and registration. In our approach we determine the level set segmentation Φ of the target image T and the displacement field u(x) between R and T simultaneously. R and T are two frames of one temporal image sequence and thus a rigid alignment of R and T can be assumed. Furthermore, we assume a ˆ of the reference image R. The prior prior segmentation Φ

For the minimization of eq. (3) a time marching approach was used by finding a steady state solution of the evolution equations. The evolution equations are derived from the Euler-Lagrange equations of eq. (3): ∂u(x,t) = λ1 (R(x ∂t +λ2 ∆u(x)

− u(x)) − T (x))∇R(x − u(x))

ˆ − u(x)) − Φ(x) ∇Φ(x ˆ − u(x)) ˆ Φ(x +λ5 N (Φ, Φ)

Figure 1. Example for temporal CT image slices. Top left: reference image with reference segmentation (peak inhale), top right: target image (peak exhale), bottom left: target image with result of level set segmentation without shape prior (λ1 = λ2 = λ5 = 0), bottom right: target image with result of combined segmentation and registration (white) and contour of manual reference segmentation (black).

∂Φ(x,t) = λ3 δ (Φ) (cout ∂t ∇Φ − λ4 δ (Φ)∇ · |∇Φ|

ˆ + λ5 N (Φ, Φ)

− T (x))2 − (cin − T (x))2

ˆ − u(x)) Φ(x) − Φ(x

A finite difference scheme is applied to discretize the equations.

3. Results The behavior of the algorithm was investigated by segmenting the liver in 2D temporal CT image sequences of four patients acquired during free breathing [14, 9] (fig. 1). The images were acquired with a low radiation dose and show a poor contrast in soft tissues. Therefore, a level set segmentation without shape prior knowledge fails (fig. 1 bottom left). For each patient 3 transversal image slices were selected for three different breathing phases: peak exhale, mid inhale and peak inhale. The liver was segmented by an expert in the resulting 36 image slices. To asses the result of the combined registration and segmentation the peak inhale image is treated as reference and used to segment

the liver contour in the mid inhale and peak exhale images. Three different experiments were performed : in a first experiment, we set λ3 = λ4 = λ5 = 0 to register reference and target image by a diffusive registration approach. The segmentation result is a deformed reference segmentation (Algorithm I). In a second experiment, the result of the registration step is improved by a succeeding segmentation using shape prior information with λ1 = λ2 = 0, (Algorithm II). In the last experiment, we perform the combined segmentation and registration by minimizing eq. (3) (Algorithm III). Each algorithm was run on 24 different image pairs (6 per patient). We determined the necessary parameters , λ1 , . . . , λ5 experimentally for one image pair and reused this set of parameters for all 24 tests (except λi = 0 as described above). The quality of the methods were compared by computing surface distances and overlap coefficients between the ground truth (manual expert segmentation) and the segmentation results of algorithm I, II and III. Here, the dice coefficient |A∩B| |A∪B| is used. Table 1 shows mean (over 6 images)

Patient 01 Patient 02 Patient 03 Patient 04

Alg. I 94.81% 91,74% 95,95% 91,89%

Alg. II 94.98% 92,72% 97,28% 93,30%

Alg. III 95.11% 93,28% 97,50% 93,55%

Table 1. Overlap coefficients (mean values) (in %) between the ground truth and the segmentation maps obtained using diffusive registration (Alg. I), registration and subsequent segmentation (Alg. II) and combined registration and segmentation (Alg. III).

overlap coefficients for each patient and for each of the three algorithms. In table 2 the mean surface distances are shown. It can be observed that combined registration leads to better segmentation results. In fact, algorithm III achieved the best segmentation result for each of the 24 test image pairs.

We run another experiment to asses the registration reproducibility of our method. In the first step, the transformation ϕ was calculated by algorithm I and III. Then, the roles of R and T are interchanged and the inverse ¯ was estimated and the mean distance transformation ϕ ¯ x − ϕ(ϕ(x)) of all pixels was determined. Both methods show a high registration reproducibility and only small differences exist between the methods. The mean distance (in pixel) of all 24 test images was 0.07 (max: 3.05) for algorithm I and 0.08 (max: 3.11) for algorithm III.

In a last experiment we examined the influence of the SSD term D[R, T ; u] on the segmentation and registration results. Therefore we run the above experiments without this additional term (λ1 = 0) to asses how important this formulation is. Compared to algorithm III, for three patient data sets the segmentation result worsened (Dice coefficient: patient 1: 93.7%, patient 2: 92.7%, patient 3: 96.1%), for one patient data set the segmentation result was improved (Dice coefficient patient 4: 95.6%). The displacement field u is only dependent on the regularization term and differences between the current segmentation Φ and the ˆ Therefore the displacement field reference segmentation Φ. is estimated only nearby the object surface. We assessed the registration reproducibility of this method by interchanging the roles of R and T and calculating the mean distance be¯ tween x − ϕ(ϕ(x)). As expected, only a poor registration reproducibility could be discovered (mean distance: 1.8, max 13.2) and this method is not suitable for estimating organ deformations.

4. Discussion and Conclusion In this paper we present a framework for combined intensity–based non–linear registration and image segmentation that incorporates prior shape knowledge. The segmentation is obtained by finding a non–linear registration to a reference shape and a global, dense displacement field is calculated using intensity and segmentation information. Both problems (non–linear registration and segmentation) are formulated in a joint variational approach and solved simultaneously. Experimental segmentation results using temporal CT image sequences showed that the proposed approach performed better than two other segmentation/registration schemes. The registration reproducibility of the combined segmentation/registration was slightly worse compared to stand–alone registration. However, the difference between the two methods is very small and a possible explanation is that new information was introduced for ¯ in the combined calculating the inverse transformation ϕ segmentation/registration process. For the calculation of ϕ the reference segmentation of R is used and for the calcula¯ a reference segmentation of T is required. Furthertion of ϕ more, we showed that the introduction of image intensities in the energy functional has essential impact to the registration accuracy. A drawback of our method is that 6 parameters have to be defined: , λ1 , . . . , λ5 . However, compared to registration and succeeding segmentation (alg. II) only one parameter more is needed for combined segmentation and registration. In future work we will investigate the automatic determination of the parameters. A further straight-forward extension is a 3D implementation of our approach. Furthermore, a more in depth validation for the presented method is required but our studies have led to very promising experimental results. Acknowledgements. We would like to thank Jan Modersitzki and Stefan Heldmann, University of Lübeck, for

Patient 01 Patient 02 Patient 03 Patient 04

Alg. I 1.40 1.79 1.21 2.0

Alg. II 1.35 1.52 1.03 1.67

Alg. III 1.32 1.41 0.95 1.61

Table 2. Surface distances (mean values) in mm between the ground truth and the segmentation result obtained using diffusive registration (Alg. I), registration and subsequent segmentation (Alg. II) and combined registration and segmentation (Alg. III).

helpful discussions and suggestions in the beginning of this project.

References [1] J.-h. An, Y. Chen, F. Huang, et al. A variational PDE based level set method for a simultaneous segmentation and nonrigid registration. In J. S. Duncan and G. Gerig, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2005, volume 3749 of Lecture Notes in Computer Science, pages 286–293. Springer, 2005. [2] J. Ashburner and K. Friston. Unified segmentation. NeuroImage, 26:839–851, 2005. [3] J. M. Blackall, S. Ahmad, M. E. Miquel, J. R. McClelland, D. B. Landau, and D. J. Hawkes. MRI-based measurements of respiratory motion variability and assessment of imaging strategies for radiotherapy planning. Phys Med Biol, 51(17):4147–4169, Sep 2006. [4] T. Chan and L. Vese. Active contours without edges. IEEE Trans. Imag. Proc., 10(2):266–277, Feb. 2001. [5] X. Chen, M. Brady, and D. Rueckert. Simultaneous segmentation and registration for medical images. In C. Barillot, D. R. Haynor, and P. Hellier, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2004, volume 3216 of Lecture Notes in Computer Science. Springer, 2004. [6] Y. Chen, H. Tagare, S. Thiruvenkadam, et al. Using prior shapes in geometric active contours in a variational framework. Int. J. Comp. Vis., 50(3):315–328, 2002. [7] D. Cremers and S. Soatto. Variational space-time motion segmentation. In 9th IEEE International Conference on Computer Vision (ICCV 2003), pages 886–893vol.2. IEEE Computer Society, 13-16 Oct. 2003. [8] E. D’Agostino, F. Maes, D. Vandermeulen, and P. Suetens. A unified framework for atlas based brain image segmentation and registration. In J. P. W. Pluim, B. Likar, and F. A. Gerritsen, editors, Third Int. Workshop on Biomedical Image Registration – WBIR 2006, volume 4057 of Lecture Notes in Computer Science, pages 136–143. Springer, 2006. [9] J. Ehrhardt, R. Werner, D. Säring, T. Frenzel, W. Lu, D. Low, and H. Handels. An optical flow based method for im-

proved reconstruction of 4D CT data sets acquired during free breathing. Medical Physics, 34(2):711–721, 2007. [10] B. Flach, E. Kask, D. Schlesinger, and A. Skulish. Unifying registration and segmentation for multi-sensor images. In L. van Gool, editor, Pattern Recognition, volume 2449 of LNCS, pages 190–197. Springer, 2002. [11] T. Guerrero, G. Zhang, T.-C. Huang, and K.-P. Lin. Intrathoracic tumour motion estimation from CT imaging using the 3D optical flow method. Phys Med Biol, 49(17):4147–4161, Sep 2004. [12] J. Han, B. Berkels, M. Rumpf, J. Hornegger, M. Droske, M. Fried, J. Scorzin, and C. Schaller. A variational framework for joint image registration, denoising and edge detection. In H. Handels, J. Ehrhardt, A. Horsch, H. Meinzer, and T. Tolxdorff, editors, Bildverarbeitung für die Medizin, pages 246–250, Hamburg, 2006. Springer. [13] K. Karantzalos and N. Paragios. Implicit free-formdeformations for multi-frame segmentation and tracking. In N. Paragios, O. D. Faugeras, T. Chan, and C. Schnörr, editors, Variational, Geometric, and Level Set Methods in Computer Vision - VLSM 2005, volume 3752 of Lecture Notes in Computer Science, pages 271–282. Springer, 2005. [14] D.-A. Low, M. Nystrom, E. Kalinin, P. Parikh, J.-F. Dempsey, J.-D. Bradley, S. Mutic, S.-H. Wahab, T. Islam, G. Christensen, D.-G. Politte, and B.-R. Whiting. A method for the reconstruction of four-dimensional synchronized CT scans acquired during free breathing. Med. Phys., 30(6):1254–1263, 2003. [15] W. Lu, P. J. Parikh, I. M. El Naqa, M. M. Nystrom, J. P. Hubenschmidt, S. H. Wahab, S. Mutic, A. K. Singh, G. E. Christensen, J. D. Bradley, and D. A. Low. Quantitation of the reconstruction quality of a four-dimensional computed tomography process for lung cancer patients. Med. Phys., 32:890–901, 2005. [16] J. Modersitzki. Numerical Methods for Image Registration. Oxford University Press, 2003. [17] D. Mumford and J. Shah. Optimal approximation by piecewise smooth functions and associated variational problems. Comm. Pure Applied Math, 42:577–685, 1989.

[18] N. Paragios. A level set approach for shape-driven segmentation and tracking of the left ventricle. IEEE Trans. Med. Imag., 22(6):773–776, June 2003. [19] N. Paragios and R. Deriche. Geodesic active contours and level sets for the detection and tracking of moving objects. IEEE Trans PAMI, 22(3):266–280, March 2000. [20] N. Paragios, M. Rousson, and V. Ramesh. Matching distance functions: A shape-to-area variational approach for globalto-local registration. In A. Heyden, G. Sparr, M. Nielsen, and P. Johansen, editors, 7th European Conference on Computer Vision - ECCV 2002, volume 2350 of Lecture Notes in Computer Science, pages 775–790. Springer, 2002. [21] K. Pohl, J. Fisher, J. Levitt, et al. A unifying approach to registration, segmentation, and intensity correction. In J. S. Duncan and G. Gerig, editors, Medical Image Computing and Computer-Assisted Intervention - MICCAI 2005, volume 3749 of Lecture Notes in Computer Science, pages 310– 318. Springer, 2005. [22] T. Rohlfing, C. R. Maurer, W. G. O’Dell, and J. Zhong. Modeling liver motion and deformation during the respiratory cycle using intensity-based nonrigid registration of gated MR images. Med Phys, 31(3):427–432, Mar 2004. [23] D. Sarrut, V. Boldea, M. Ayadi, et al. Nonrigid registration method to assess reproducibility of breath-holding with ABC in lung cancer. Int J Radiat Oncol Biol Phys, 61(2):594–607, Feb 2005. [24] A. Schweikard, G. Glosser, M. Bodduluri, et al. Robotic motion compensation for respiratory movement during radiosurgery. Comp Aided Surg, 5:263–77, 2000. [25] G. Unal and G. Slabaugh. Coupled PDEs for non-rigid registration and segmentation. In Computer Vision and Pattern Recognition - CVPR 2005, volume 1, pages 168–175, June 2005. [26] R. Werner, J. Ehrhardt, T. Frenzel, D. Säring, W. Lu, D. Low, and H. Handels. Motion artifact reducing reconstruction of 4D CT image data for the analysis of respiratory dynamics. Methods of Information in Medicin, 49:254–260, 2007. [27] L. Weruaga, J. Morales, L. Nez, and R. Verd. Estimating volumetric motion in human thorax with parametric matching constraints. IEEE Trans Med Imaging, 22(6):766–772, Jun 2003. [28] P. Wyatt and J. A. Noble. Map mrf joint segmentation and registration of medical images. Med Image Anal, 7(4):539– 552, 2003. [29] A. Yezzi, L. Zollei, and T. Kapur. A variational framework for integrating segmentation and registration through active contours. Med Imag Anal, 7(2):171–185, 2003. [30] Y. Young and D. Levy. Registration-base morphing of active contours for segmentation of ct scans. Math. Biosci. Eng., 2:79–96, 2005. [31] B. Zitova and J. Flusser. Image registration methods: A survey. Image and Vision Computing, 21(11):977–1000, 2003.