University of South Florida, Department of Computer Science & Engineering, ... A&M University, Department of Electrical & Computer Engineering, College ...
CONTOUR-BASED HIDDEN MARKOV MODEL TO SEGMENT 2D ULTRASOUND IMAGES Xiaoning Qian1 and Byung-Jun Yoon2 1
2
University of South Florida, Department of Computer Science & Engineering, Tampa, FL 33620 Texas A&M University, Department of Electrical & Computer Engineering, College Station, TX 77843
ABSTRACT The segmentation of ultrasound images is challenging due to the difficulty of appropriate modeling of their appearance variations including speckle as well as signal dropout. We propose a novel automatic segmentation method for 2D cardiac ultrasound images based on hidden Markov models (HMMs). By directly exploiting the local image characteristics around contour points in images and integrating them into contour-based HMMs, we solve the segmentation problem by graph matching using an efficient dynamic programming algorithm. Due to the direct integration of local properties in our HMMs, our segmentation method automatically deals with inhomogeneity but avoids the complexities of explicit appearance modeling in classical Maximum A Posteriori (MAP) approaches. The optimization for contour extraction is straightforward and guarantees the global optimal results. We implemented our method to segment the endocardium in short-axis cardiac ultrasound images successfully. The method can also be used for other image modalities with the presence of image inhomogeneity. Index Terms— Image segmentation, hidden Markov model (HMM), contour finding, ultrasound images. 1. INTRODUCTION Endocardium segmentation provides important information for the analysis of cardiac function from medical images [8]. It is a challenging task with cardiac ultrasound images as the appearances of these images are very complex with two major sources of image inhomogeneity: (1) Ultrasound images are formed by recording backscatter from a propagating acoustic wave. Due to scattering from wavelength-scale inhomogeneities in myocardial fibers, ultrasound images are “grainy” and contain speckles as appearance inhomogeneity in the brightness and contrast; (2) Ultrasound images often contain dropout regions with weak signals due to the loss of energy in the acoustic wave, the angle of the wave and backscatter, and shadows from anatomical structures [5, 11]. Different techniques have been proposed to segment cardiac ultrasound images [2, 8]. Many of them are based on the classical Maximum A Posteriori (MAP) estimation approach with deformable models, where an evolving contour, typically represented by a zero level set [9], partitions images into two regions assuming an appearance model for each region. Shape priors have been added to deal with inhomo-
978-1-4577-0539-7/11/$26.00 ©2011 IEEE
705
geneity [3, 9, 10, 16, 19]. These methods perform well in relatively homogeneous images. When images become complex with signal dropout, it is difficult to get accurate and reliable appearance model for these region-based methods. Although further attempts have been made to explicitly model image inhomogeneity in ultrasound images for better segmentation [11, 15, 17], the MAP formulation requires to estimate many additional auxiliary parameters together with the segmentation, which makes the objective function in deformable models highly non-convex and the segmentation can be easily trapped at local minima away from the true boundary. Machine-learning for segmentation has attracted increasing attention in computer vision, and some researchers have proposed its implementation for ultrasound image segmentation to avoid the complexities of explicit appearance modeling [2, 4, 18]. Using large training sets covering appearance variations, these algorithms are capable of dealing with inhomogeneity and achieve good segmentation. However, it is not clear how these learning-based algorithms perform on images with different appearance variations. We focus on segmenting the endocardium from 2D cardiac ultrasound images based on a novel approach. Instead of explicitly modeling the appearance variations with significant inhomogeneity as in existing algorithms, we directly exploit the local image characteristics around the endocardium. By representing both the image to segment and a priori known endocardium contour model as two graphs, we formulate the image segmentation problem as a graph matching problem. We respectively model the two graphs as two hidden Markov models (HMMs) and integrate local image features into their parameterization. Based on this framework, the task of segmentation is transformed into that of finding the most similar pair of state paths in two HMMs. Due to the integration of local properties in our HMMs, our method automatically deals with inhomogeneity but avoids complex modeling. This new formulation allows us to use an efficient dynamic programming algorithm to find the global optimal solution. The complexity of the algorithms increases only linearly with the length of the contour as well as the image size. 2. METHODS To extract the endocardium in the image by graph matching, we first construct a “contour-based HMM” based on a priori known endocardium contour model. We parameterize
ICASSP 2011
$
S
'HOHWLRQ
S X
X
X
%
'HOHWLRQ
X X S
X
X
S
X S
S X
'HOHWLRQ X
X
G&
Y
Y
Y
Y
Y
Y
Y
Y T
Y T Y
Y
Y T
$
Y
X
Y T Y Y T Y
Y
,QVHUWLRQ
S
Y Y
Y T Y Y
Y T Y
Y T Y
T
Y
Y
X X
V/
X
Y
X X V
X
,QVHUWLRQ
S
Y
Y
Y T Y
Y
%
X
Y
X
G,
G&
Fig. 1. Example of a contour model graph GC (A) and an image graph GI (B): each vertex corresponds to either a contour point or a pixel; each edge is defined by the neighborhood system in both graphs; and dashed lines link points with similar local image characteristics. The best matching pair of paths p (in GC ) and q (in GI ) are shown in the figure with bold edges. The points with similar local features are black. Both p and q can skip over points in two graphs respectively, introducing “insertions” or “deletions”.
this “contour-based HMM” based on desired local image features for endocardium, which can either be learned from the training set or be given based on available rough knowledge about the endocardium in images. The image to be segmented is modeled by an “image HMM”, whose parameterization is also based on local image features. We look for the most similar state path in the image HMM for the contour-based HMM to segment the endocardium (Fig. 1). We start with the contour model: U = {u1 , u2 , . . . , uN1 } with N1 points, which are in fact the sample points along the contour. Based on the prior information about these points, we can define the edges between neighboring contour points and represent them as a directed graph GC (Fig. 1A). For our 2D image segmentation, the graph has a linear structure. We focus on closed contour segmentation and there is one edge from uN1 to u1 as shown in the figure. For a given image, we represent it also by a graph GI : V = {v1 , v2 , . . . , vN2 } with N2 = n × m points, which are the pixels in the image of size n × m. The edges in the graph is defined based on the neighborhood system that we adopt. In this paper, we implement 8-neighborhood systems as shown in Fig. 1B. Note that this image graph is undirected. Discrete graph representations for both the contour model and the image are natural and have been widely used [10, 17]. To segment the contour of interest, we formulate a graph matching problem to find the best matching contour q from the image, which connected by highlighted edges, for the given contour model. Figure 1 shows that our task for segmentation transforms to find the optimal pair of paths (p, q) from GC and GI , where the constituent points have the best matching correspondences. Black vertices represent the corresponding contour points with similar local image features. As the contours may have different sizes and can be sampled in different ways, the best matching paths may have flexible number of insertions and deletions as shown in the figure, which provides an automatic mechanism to accommodate both image and geometric variations among different images.
706
X X X X
X
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
V G,
Y
V/
Fig. 2. A. Contour-based HMM: The dots next to the hidden states represent all possible symbols corresponding to virtual nodes in s. For simplicity, the detailed illustration for the probabilistic structure of the hidden states in the HMMs are shown only for the vertices u1 , u6 , and u8 in GC . B. Image HMM: Each vertex corresponds to a hidden state and has an accompanying hidden state. The detailed HMM structure is shown for v29 in GI .
2.1. Hidden Markov models To find the optimal pair of paths for segmentation, we adopt the hidden Markov model (HMM) formalism [12, 13, 14]. We first construct the contour-based HMM for GC . Each vertex ui ∈ U (a contour point) in GC corresponds to a hidden state in the HMM. We represent this hidden state with the same notation ui . The transition between states ui from uj in the HMM is determined by the edges given in GC . At this stage, the resulting HMM has an identical structure as the graph GC . To accommodate insertions and deletions, we add an accompanying state u ˜i for every state ui with an incoming edge from the original state ui and outgoing edges to all the neighboring states of ui . We further allow consecutive insertions or deletions by enabling self-transitions at u ˜i . The HMM for GI can be constructed in a similar way. Figure 2 illustrates the HMMs that correspond to the graphs shown in Fig. 1. In order to find the best matching pair of paths, we introduce a “virtual” path s = s1 s2 . . . sL that has L nodes. A node st in s can be viewed as a symbol emitted jointly by a pair of hidden states (pt , qt ) = (uj , v ) in the respective HMMs. Therefore, the virtual path can naturally couple a path p in GC with another path q in GI , providing a convenient framework for identifying the correspondence between points in the contour model and pixels in the image. Note that, the matching paths p and q contain L states each, but they may contain one or more accompanying states which represent gaps, i.e., insertions or deletions. The proposed framework does not impose any restriction on the number of gaps and their locations in the matching paths. Based on the described HMM structure, we further define a matching score S(p, q) for a pair of matched paths, which should sensibly integrate the similarity h(pt , qt ) between matched states (corresponding to contour points and image pixels respectively) pt = uj and qt = v (1 ≤ t ≤ L) based on local similarity, the connectivity w1 (pt−1 , pt ) between pt−1 = ui and pt = uj , the connectivity w2 (qt−1 , qt ) between qt−1 = vk and qt = v (2 ≤ t ≤ L), and the penalty for possible gaps. We first define two mappings f1 : w1 (ui , uj ) → t1 (uj |ui ) and f2 : w2 (vk , v ) → t2 (v |vk ),
&
%
$
'
Fig. 3. Local feature responses: A. An example cardiac ultrasound image; B. Zoomed region with endocardium; C. Maximum responses from the detector [7]; D. Maximizing orientations with colormap representing 16 orientations marked in the figure.
which convert w1 (ui , uj ) and w2 (vk , v ) to the following transition probabilities between hidden states P (pt = uj |pt−1 = ui )
=
t1 (uj |ui ) = f1 (w1 (ui , uj ));
P (qt = v |qt−1 = vk )
=
t2 (v |vk ) = f2 (w2 (vk , v )).
We define another mapping g : h(uj , v ) → e(uj , v ) that converts h(uj , v ) to the following “pairing” probability P (pt = uj , qt = v ) = e(uj , v ) = g(h(uj , v )),
which is the emission probability of st at a pair of hidden states (pt , qt ) = (uj , v ). The mappings f1 , f2 , and g should be defined such that the resulting transition and emission probabilities are legitimate probability functions [12, 13, 14]. By defining S(p, q) = log P (p, q), we solve image segmentation by finding the optimal pair of state sequences in two HMMs that jointly maximize S(p, q). We devise a dynamic programming algorithm to find the optimal solution in polynomial time. We define γ(t, j, ) as the log-probability of the most probable pair of state sequences of length t(≤ L), where the two underlying states are pt = uj and qt = v . We have the following recursive equation: h i
γ(t, j, ) = max γ(t−1, i, k) + log(t1 (uj |ui ) · t2 (v |vk ) · e(uj , v )) . i,k
We repeat the above iterations until t = L. The maximum score is given by: h i S(p∗ , q∗ ) = max log P (p, q) = max γ(L, j, ), p,q
j,
(1)
where {p∗ , q∗ } = arg maxp,q S(p, q) is the optimal pair of state paths with maximum similarity score in the given HMMs. The computational complexity of the above optimization algorithm is O(LM1 M2 ), where L is the length of the aligned paths that we want to find, M1 is the number of edges in GC , and M2 is the number of edges in GI . 2.2. Parameterization of HMMs The quality of the point matching results obtained from our HMM-based algorithm is determined by the transition and pairing probabilities of the HMMs. We now present an effective parameterization scheme based on local image characteristics around the contour of interest. To capture the local image properties around the contour, we resort to a boundary detector [7], which predicts the posterior probability of a boundary at each pixel by measuring the difference of brightness and texture gradients on the two halves of a local disc at different directions. We apply the detector at three different scales with disk diameters 5, 11, and 21 at 16 uniformly sampled orientations within [0, 2π). Figure 3 shows an example with the maximum responses and the maximizing orientations for one cardiac ultrasound image. The figure shows appearance inhomogeneity with signal
707
dropout near both 2 and 10 o’clock regions. We parametrize our HMMs based on the local features with both the maximum responses and maximizing orientations. Note that it is also straightforward to incorporate other local image features including SURF (Speeded Up Robust Features) [1] and SIFT (Scale Invariant Feature Transform) features [6]. For the image HMM, each hidden state corresponds to a pixel in the original image and hence, all the hidden states have associated local features by running the detector. For our contourbased HMM, we can either learn the desired local features for each state, corresponding to a contour point, from a training set or by manually assigning corresponding features based on available rough knowledge about the endocardium in images. In this paper, we use a manually segmented contour from one training image as our contour model, with which the corresponding local image features are associated. With these local features available for both contour-based and image HMMs, we now define our transition and pairing scores (logarithm of probabilities). First, for the transition scores, we take a simple scheme based on the presence of edges between the corresponding states in the contour-based HMM and image HMM respectively. If there exists an edge, we set the score to be 0; Otherwise, it is −∞ so that the result will not be illegitimate according to the adopted neighborhood system. We define the pairing score between two hidden states that correspond respectively to a contour point and a pixel from the image based on their local features:
8 < log [λk Gw (θi , φk )], log λk , log e(uj , v ) = : −Δ,
if θi > 0 and φk > 0; if θi = 0; if φk = 0,
(2)
where Gw (·, ·) is a similarity function whose value is determined by the difference of the direction between θi and φk , the maximizing orientations at ui and vk . As the backbone of the contour-based HMM is a loop, the scoring function is in fact invariant to rotation. With dropout at ui in the contour model with θi = 0, we take log λk , which is the maximum response of the detector in the range of [0, 1] at vk . As the brightness and contrast changes around endocardium across different images, we do not integrate maximum responses into our contour-based HMM. With φk = 0, we take −Δ to avoid leaking out by penalizing the addition of these points. If either ui or vk is an accompanying hidden state in the HMMs, we set a constant −Δem as the gap penalty. With this parameterization scheme, our HMM-based segmentation automatically takes care of geometric variations of endocardium as well as spatial variations of brightness and contrast around endocardium, including signal dropout. 3. EXPERIMENTAL RESULTS AND DISCUSSIONS We tested our HMM-based segmentation algorithm on 3 sequences of 2D clinical short-axis echocardiac images with satisfactory results. These sequences were acquired from different subjects with an Acuson Sequoia imaging system. We select one image from one of the sequences to construct our contour-based HMM. To ensure that we obtain closed contours, we initialize the segmentation by selecting a starting
A-1
A-2
B-1
A-3
B-2
Fig. 4. Segmentation results of the proposed HMM-based algorithm (Green: manual segmentation; Red: HMM-based segmentation): A. Examples within the same sequence of the image for the contour model construction; B. Examples from different sequences.
Table 1. Average MD and SHD and standard deviations Metric Average STD
Within sequence MD SHD 1.16 3.25 0.16 0.29
4. REFERENCES
Across sequences MD SHD 1.38 6.76 0.15 2.26
[1] Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: SURF: Speeded up robust features. Comput Vis Imag Und, 110(3), 346–359 (2008)
point on the contour of interest. Examples of the segmentation for these sequences are shown for visual evaluation in Fig. 4 with manual segmentations, which were obtained blind to our segmentation results. We also compared our results with manual P segmentation using Ptwo metrics: mean distance mina∈A a−b a∈A minb∈B a−b (MD= + b∈B 2|B| ), measur2|A| ing the global disagreement; and symmetric Hausdorff disb∈B mina∈A a−b ), tance (SHD= maxa∈A minb∈B a−b+max 2 comparing the local differences. A and B are manual and automatic segmentations respectively; |A| and |B| gives the number of points in them. We compute the average values and standard deviations for the segmentations of the images in the same sequence as the image for the contour model construction as well as for the results of the images in different sequences. They are given in Table 1. As the size of speckle in these ultrasound images is about 7 pixels, our results are reasonably close to manual segmentations. We presented a novel HMM-based segmentation algorithm and reported preliminary results for endocardium segmentation in ultrasound cardiac images. By appropriate parameterization, the new algorithm deals with image inhomogeneity but avoids complex explicit modeling of appearance variations as well as geometric variations, which are tackled by gaps allowed at arbitrary locations. With an efficient dynamic programming algorithm, we can extract the best contour that is mathematically optimal. This is a significant merit considering that many available segmentation algorithms, including the MAP approaches, can be easily trapped at local optima. HMM representations of both contours and images are conceptually natural and mathematically flexible to incorporate desired features, and it is straightforward to extend them for multiscale implementation to speed up the algorithm further. We can also combine HMM-based segmentation with classical MAP segmentation by constraining the MAP segmentation around the optimal state path. Finally, we note that our segmentation algorithm is sensitive to the gap penalty, especially when the severity of dropout varies greatly across images. Future research will be focused on further experimental validation of the algorithm and its improvement, in terms of segmentation accuracy and efficiency.
708
[2] Carneiro, G., Georgescu, B., Good, S. Comaniciu, D.: Detection and measurement of fetal anatomies from ultrasound images using a constrained probabilistic boosting tree. IEEE Trans. Med. Imaging, 27, 1342– 1355 (2008) [3] Chen, Y., Huang, F., Tagare, H.D., Rao. M.: A coupled minimization problem for medical image segmentation with priors. Int. J. Comp. Vis., 71, 259–272 (2007) [4] Georgescu, B., Zhou, X., Comaniciu, D., Gupta, A.: Databaseguided segmentation of anatomical structures with complex appearance. CVPR’05, 2, 429–436 (2005) [5] Goodman, J.W.: Laser Speckle and Related Phenomenon. in J.C. Dainty (ed.), Springer-Verlag (1975) [6] Lowe, D.: Distinctive image features from scale-invariant keypoints. Int J Comput Vis, 60(4), 91–110 (2004) [7] Martin, R.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. PAMI, 26(5), 530–549 (2004) [8] Noble, J., Boukerroui, D.: Ultrasound image segmentation: a survey. IEEE Trans. Med. Imaging, 25, 987–1010 (2006) [9] Paragios, N.: A level set approach for shape-driven segmentation and tracking of left ventricle. IEEE Trans. Med. Imaging, 22, 773–776 (2003) [10] Parker, A. D., Hill, A., Taylor, C. J., Cootes, T. F., Jin, X. Y., Gibson, D. G.: Application of point distribution models to the automated analysis of echocardiograms. Computers in Cardiology, 25–28 (1994) [11] Qian, X., Tagare, H. D., Tao, Z.: Segmentation of rat cardiac ultrasound images with large dropout regions. IEEE MMBIA (2006) [12] Qian, X., Sze, S.H., Yoon, B.J.: Querying pathways in protein interaction networks based on hidden Markov models. J Comput Biol, 16(2):145–157 (2009) [13] Qian, X. and Yoon, B.J.: Effective identification of conserved pathways in biological networks using hidden Markov models. PLoS One, 4(12) (2009) [14] Qian, X. and Yoon, B.J.: Shape matching based on graph alignment using hidden Markov models. ICASSP’09, 934–937 (2009) [15] Rousson, M., Cremers, D.: Efficient kernel density estimation of shape and intensity priors for level set segmentation. MICCAI’05, 757–764 (2005) [16] Shen, D., Zhan, Y., Davatzikos, C.: Segmentation of prostate boundaries from ultrasound images using statistical shape model. IEEE Trans. Med. Imaging, 22(4), 539–551 (2003) [17] Xiao, G., Brady, M., Noble, J., Zhang, Y.: Segmentation of Ultrasound B-mode Images with Intensity Inhomogeneity Correction. IEEE Trans. Med. Imaging., 21, 48–57 (2002) [18] Yue, Y., Tagare, H. D.: Learning to Segment Using Machine-Learned Penalized Logisitic Models. IEEE MMBIA (2009) [19] Zhu, Y., Papademetris, X., Sinusas, A.J., Duncan, J.S.: A dynamical shape prior for LV segmentation from RT3D echocardiography. MICCAI (2009)
