Int J CARS (2012) 7:371–387 DOI 10.1007/s11548-011-0645-6
ORIGINAL ARTICLE
Robust bronchoscope motion tracking using sequential Monte Carlo methods in navigated bronchoscopy: dynamic phantom and patient validation Xióngbi¯ao Luó · Marco Feuerstein · Takayuki Kitasaka · Kensaku Mori
Received: 17 March 2011 / Accepted: 6 July 2011 / Published online: 23 July 2011 © CARS 2011
Abstract Purpose Accurate and robust estimates of camera position and orientation in a bronchoscope are required for navigation. Fusion of pre-interventional information (e.g., CT, MRI, or US) and intra-interventional information (e.g., bronchoscopic video) were incorporated into a navigation system to provide physicians with an augmented reality environment for bronchoscopic interventions. Methods Two approaches were used to predict bronchoscope movements by incorporating sequential Monte Carlo (SMC) simulation including (1) image-based tracking techniques and (2) electromagnetic tracking (EMT) methods. SMC simulation was introduced to model ambiguities or uncertainties that occurred in image- and EMT-based bronchoscope tracking. Scale invariant feature transform (SIFT) features were employed to overcome the limitations of image-based motion tracking methods. Validation was performed on five phantom and ten human case datasets acquired in the supine position.
X. Luó (B) · K. Mori Graduate School of Information Science, Nagoya University, Nagoya, Japan e-mail:
[email protected] M. Feuerstein Computer Aided Medical Procedures (CAMP), Technische Universität München, Munich, Germany T. Kitasaka Faculty of Information Science, Aichi Institute of Technology, Toyota, Japan K. Mori Information and Communications Headquarters, Nagoya University, Nagoya, Japan e-mail:
[email protected]
Results For dynamic phantom validation, the EMT–SMC simulation method improved the tracking performance of the successfully registered bronchoscopic video frames by 12.7% compared with a hybrid-based method. In comparisons between tracking results and ground truth, the accuracy of the EMT–SMC simulation method was 1.51 mm (positional error) and 5.44◦ (orientation error). During patient assessment, the SIFT–SMC simulation scheme was more stable or robust than a previous image-based approach for bronchoscope motion estimation, showing 23.6% improvement of successfully tracked frames. Comparing the estimates of our method to ground truth, the position and orientation errors are 3.72 mm and 10.2◦ , while those of our previous image-based method were at least 7.77 mm and 19.3◦ . The computational times of our EMT– and SIFT–SMC simulation methods were 0.9 and 1.2 s per frame, respectively. Conclusion The SMC simulation method was developed to model ambiguities that occur in bronchoscope tracking. This method more stably and accurately predicts the bronchoscope camera position and orientation parameters, reducing uncertainties due to problematic bronchoscopic video frames and airway deformation during intra-bronchoscopy navigation. Keywords Sequential Monte Carlo methods · Bronchoscope motion tracking · Virtual bronchoscopy · Navigated bronchoscopy Introduction With an estimated 222,520 new cases and 157,300 deaths in 2010, respectively, accounting for about 15% of all cancer diagnoses and about 28% of all cancer deaths, lung and bronchus cancer are the leading cause of cancer deaths in the United States [1]. To reduce the incidence rates or the
123
372
mortality of lung and bronchus cancer patients, early diagnosis is greatly needed to allow for more effective treatments in the preliminary stages of cancer. For early detection and treatment of such cancers, physicians usually acquire such pre-interventional information of the patient chest as a noninvasive three-dimensional (3D) computed tomography (CT) scan and perform such invasive bronchoscopic procedures as the transbronchial lung biopsy (TBLB). Such conventional bronchoscopy only provides two-dimensional (2D) information (bronchoscopic video images) and only displays the visible bronchial wall surfaces through the natural orifices of patients and miss anatomical structures under the bronchial surfaces. Because a TBLB procedure must be performed inside the very complex bronchial tree structure, it remains challenging to accurately localize the biopsy needle in the region of interest (ROI) to sample tissue inside the airway tree. To address such disadvantages, current popular processes first use pre-interventional information (e.g., 3D CT datasets) to build virtual bronchoscopic images [2–4] and combine intra-interventional information (live bronchoscopic videos) to construct bronchoscopy navigation systems. They usually provide various guidance information, e.g., the positions of abnormalities, a TBLB-guidance map of the suspicious regions, automatical displays of the anatomical names of the currently observed bronchial branches, and particularly visualizing the anatomical organs beyond the bronchial walls (see Fig. 1). To develop such a navigation system, the most challenging task is to explore a method that precisely and powerfully tracks the bronchoscope camera motion inside the complex airway tree structure. Various techniques have been proposed. Image-based tracking methods usually estimate the bronchoscope camera motion parameters on the basis of image registration techniques that usually compute the similarities between live bronchoscopic video frames and CTbased virtual bronchoscopy images [5–8]. However, such methods suffer heavily from the following: (a) uninformative or problematic bronchoscopic video frames (see Fig. 2) and (b) an optimization procedure that requires a good initial guess and also depends heavily on clearly observed bifurcation or fold information on live bronchoscopic video images. Although electromagnetic tracking (EMT)-based techniques [9–13] have demonstrated their advantages in TBLB interventions, hybrid-based guidance methods that combine image- and EMT-based approaches to localize the bronchoscope tip for the TBLB procedure have outperformed other methods and are a prospective means to accurately track the bronchoscope camera motion during navigated interventions [14,15]. Unfortunately, in EMT systems, two vital bottlenecks remain: (c) erroneous localization due to unavoidable patient movement such as airway deformation or coughing and (d) dynamic error that originates from magnetic field distortion. The former arises from EMT systems
123
Int J CARS (2012) 7:371–387
that measure the position and orientation of the bronchoscope camera involved in a fixed, world coordinate system. This results in asynchronization between EMT measurements and the current bronchoscope camera pose. The latter is highly susceptible to ferrous metals or conductive material within or close to the tracking environment. To correct such erroneous measurements resulting from stationary objects, many calibration techniques have been investigated [16]. They usually acquire some distorted and undistorted measurements to construct a field distortion function on the basis of lookup tables or polynomials. However, such functions cannot be adapted to different setups in operating rooms before bronchoscopic intervention and to dynamic changes of magnetic field distortion due to, for example, relocation of the field generator or movement of instruments during intra-interventions. The current state-of-the-art technique to compensate for erroneous measurements is to integrate EMT with optical tracking [17], which was proven to be more adaptive and convenient. This study introduces sequential Monte Carlo (SMC) sampling to address the above limitations (a), (b), and (c) that occur in image-based and hybrid-based bronchoscope motion tracking approaches during navigated bronchoscopy. Since we use sequential Monte Carlo methods to generate a set of random samples including the camera motion parameters and the similarity between augmenting the virtual bronchoscopic and patient-specific real bronchoscopic images, our proposed SMC-based methods can accurately approximate the posterior probability distribution of the current bronchoscope camera pose even with patient movement or image artifacts (uninformative images). Our experimental results demonstrate that using SMC methods powerfully deals with the restrictions of image-based methods and the disadvantages of EMT systems. Preliminary work of this method was presented at the ACCV (Asian Conference on Computer Vision) conference [18].
Methods Bronchoscope motion tracking aligns a virtual camera pose with a real bronchoscope camera pose in a live or a real bronchoscopy. The virtual camera pose is generated by placing a virtual camera inside a 3D CT-derived virtual bronchoscopy environment. The motion tracking alignment seeks a transformation relationship between the bronchoscope camera and the CT coordinate systems (see Fig. 3). During the proposed image- and hybrid-based guidance bronchoscopy, our strategy first uses SMC samplers to produce random samples in terms of the current bronchoscope camera pose and the current similarity information. Next, these samples are deterministically drifted and stochastically diffused to generate a series of new samples to approximate the distribution space of the current bronchoscope camera pose. Finally,
Int J CARS (2012) 7:371–387
373
Fig. 1 a Composite view of virtual bronchoscopy that enables physicians to fly through the airway tree and observe the bronchial surface by moving a virtual camera to generate a sequence of virtual images (left-side) and simultaneously synchronize its viewing point or localization to CT slices (right-side). b Snapshot of augmented or navigated bronchoscopy system comprised of four windows to provide guidance
information including an 2D camera viewing texture (top-left), a virtual image corresponding to the real camera view point (bottom-left), CT-derived bronchus geometric structure with camera trajectory (topright), and an CT axial view showing current camera position inside CT coordinates (bottom-right)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Fig. 2 Examples of problematic or uninformative video frames of bronchoscopy. a Specular- or inter-reflection. b Electronic noise. c Bubbles. d Collision with bronchial walls. e Poor park region. f Motion blur. g Airway deformation. h Artifact caused by blood
after calculating the weight (similarity) of each sample, the current bronchoscope camera pose in the CT coordinate system can be determined to correspond to the pose of one sample with the maximum weight inside the sample set. In this section, for the sake of completeness, we first briefly review an SMC sampling technique based on a sequential importance sampling (SIS) algorithm in SMC methods. We then parameterize bronchoscopic camera motion
in accordance with a state space model before we develop our SMC sampling algorithm for estimating bronchoscope motion parameters. SMC sampling SMC methods are a set of simulation-based algorithms that collect a cloud of weighted random samples to investi-
123
374
Int J CARS (2012) 7:371–387
Fig. 3 Real or intra-interventional bronchoscopy (left) is usually performed by manipulating a bronchoscope to observe inside a patient’s airway tree and obtain real bronchoscopic video frames. Virtual bronchoscopy (right) is based on pre-interventional information such as CT
and places a virtual camera inside a 3D pre-built anatomical model to generate virtual bronchoscopic video frames. The general challenge is (i) to search for transformation matrix C T QC to align these real and virtual camera poses
gate the problem of optimal state estimation in nonlinear non-Gaussian state space models [19,20]. In such approaches, bronchoscopic camera motion tracking can be considered as a nonlinear discrete dynamic system that is usually described by a state space model involved in state and observation equations as follows:
where q(·) denotes an importance density function that affects the degree of sample degeneracy.
xi = G(xi−1 , ni )
(1)
yi = H(xi , vi )
(2)
where dynamic state xi at time i is determined by transmission function G with respect to previous state xi−1 and process noise ni . Observation or measurement yi relates to xi and measurement noise vi in terms of transform function H. The state Eq. 1 gives Markov transmission probability p(xi |xi−1 ) between times (i −1) and i. The observation Eq. 2 essentially provides the conditional probability p(yi |xi ) of measurement yi given state xi . The goal of SMC sampling is to approximate or estimate posterior probability p(xi |Yi ) of state xi given all available measurements up to i (Yi = {yu : u = 1, . . . , i}). k k k k Suppose we know sample set Si−1 = xi−1 , wi−1 , ci−1 : k = 1, 2, 3, . . . , M at time (i −1), where M is the number of k k k samples and xi−1 , wi−1 , ci−1 denotes a sample including k , important weight w k , and accumulative weight state xi−1 i−1 k ci−1 . To predict p (xi |Yi ), SIS algorithm is first performed by two steps [21]: k k . • Draw samples Sik from Si−1 in terms of p xik |xi−1 k • Calculate “incremental weight” wi by wik
k p yi |xik p xik |xi−1 , = k ,y q xik |xi−1 i
123
(3)
k w k . Finally, We then calculate wik by wik = wi−1 i p (xi |Yi ) is estimated with respect to xik and wik [22]:
p(xi |Yi ) ≈
M
wik δ xi − xik ,
(4)
k=1
where δ(·) is the Dirac delta function. During SIS, an additional resampling step is usually implemented to address the problem of sample impoverishment. For more details on SMC sampling, please refer to [21,22]. Parameterization of bronchoscopic camera motion Since bronchoscopy navigation needs to input bronchoscopic video frames and pre-operative CT data while it outputs a series of bronchoscope camera motion parameters, it involves the bronchoscope camera and the CT coordinate systems (see Fig. 3). We denote the transformation matrix from the bronchoscope camera coordinate system to the CT coordinate system at the ith frame as Qi , which includes the camera position and orientation and is represented by
Ri ti , (5) Qi = 0T 1 where Ri and ti describe a rotation matrix and a translation vector of the bronchoscopic camera in the CT coordinate system. Rotation Ri includes three rotation axes (or vectors) rix , riy , and rzi around the x-, y-, and z-directions [23]: Ri = x ri y ri z ri , (6)
Int J CARS (2012) 7:371–387
375
T where axis ∗ ri = ∗1r i , ∗2 r i , ∗3 r i , and “∗” denotes x, y, or z. To characterize Ri compactly and save its storage space, we use quaternions but not rotation matrices in our implementation. In accordance with quaternion rotation [24,25], an arbitrary vector, i.e., a rotation axis ∗ ri , can be represented by a pure quaternion: ∗ ri ⇔∗ qi (0, ∗ ri ). Given a unit quaternion ∗ qi (φ, e) that describes a rotation change Ri between two consecutive frames, an updated rotation axis ∗ rˆ i after applying Ri to ∗ ri can be calculated by Hart et al. [24]: ∗
rˆ i ⇔∗ qˆ i (φ,∗ rˆ i ) = ∗ qi (φ, e) ×∗ qi (0, ∗ ri ) · (∗ qi (φ, e))−1 ,
(7)
that is, rotating ∗ ri around axis e by angle φ and obtaining new direction or axis ∗ rˆ i : ∗ ri ⇒∗ rˆ i . Based on the above analysis, global motion state xi , which corresponds to Qi of the current camera frame, can be parameterized by a homogeneous matrix:
Ri (x ri , y ri ,z ri ) ti , (8) xi = 0T 1 where rix , riy and rzi are updated by a quaternion method. SMC sampling for bronchoscope motion tracking In our modified bronchoscope motion tracking method, we utilize an SMC sampling method that originates from the work of Isard and Blake [26] to continuously estimate the posterior probability distribution of the bronchoscope camera state on the basis of the set of updating random samples Sik . As described in Sect. 2.1, Sik is drawn or generated from k . In this study, we perform two steps the previous set Si−1 of deterministic drift and stochastic diffusion for samples in k to transmit them to the new state set S k to approximate Si−1 i the posterior probability density of interest. Furthermore, to obtainSik , SMC sampling needs the probability density funck to describe the state transmission probability tion p xik |xi−1 between the consecutive time and function (or an likelihood k , y for observations important density function) q xik |xi−1 i (or measurements) yi . Additionally, to characterize random sample Sik , weight wik also needs to be determined by incre mental importance weight cik that equals p yi |xik . Our proposed bronchoscope camera motion tracking process is mainly implemented by the following steps. (a) Sample selection We require choosing samples to be k is selected by identifying the smallest evolved. Sample xi−1 k k ci−1 that satisfies ci−1 ≥ a, where a is a random number that is produced by a normal random number generator [26]. Samples with large weight are thus sampled most frequently. After selecting samples, some new random samples are added
to avoid getting trapped in local minima and to tackle sudden changes of images. (b) State evolution density We transmit each selected sample or state xik (homogeneous matrix) by performing the following two stages [26]. (i) Deterministic drift During this procedure, according to Eq. 1, an evolved sample xˆ ik can be represented by: k k , A = Axi−1 , (9) xˆ ik = G1 xi−1 where matrix A describes the deterministic drift part and it depends on observations yi and yi−1 during state dynamics. To determine A relative to observation yi , we have two of the following schemes to calculate it: ˆ i denote the • EMT-based Let homogeneous matrix Q EMT result from the bronchoscope camera coordinate ˆ i . We comsystem to the CT coordinate system, yi = Q pute the relative motion predicted by the EMT between ˆ i (Q ˆ i−1 )−1 . In this case, ˆi = Q frames (i − 1) and i: Q ˆ ˆ matrix A equals Qi : A = Qi = yi (yi−1 )−1 . • SIFT-based We use the scale invariant feature transform (SIFT)-based camera motion recovery method to calculate the relative motion between consecutive frames, as described in our previous work [27]. We first detect the SIFT features from the current bronchoscopic video image and find correspondences in the previous image. By these matches, we estimate the relative camera motion up to scale using epipolar geometry analysis. To recover this unknown scale, we additionally apply Kalman filtering to predict the camera translation. We can obtain a full six-degrees-of-freedom inter-frame camera motion ¯ i in the CT coordinates by combining the epimatrix Q polar analysis and the Kalman filtering estimates. Finally, ¯ i. we determine deterministic drift part A as: A = Q At this time, observation yi is determined by: yi = ¯i =Q ¯ i−1 A. ¯ i−1 Q Q (ii) Stochastic diffusion After obtaining xˆ ik by deterministic drift, we apply a stochastic movement B n ik to it: xik = G2 xˆ ik , B n ik , (10) where n ik is the noise coefficient defined as per following. During this procedure, we usually have no prior knowledge of the bronchoscope camera movement inside the airk way tree structure. k k To characterize the noise term B n i and represent p xi |xi−1 for a pointwise state evaluation, we generally use a random walk model. Since bronchoscopic frames are used as image sources, the changes of the motion parameters are usually quite small. For example, in our case, the frame rate of the bronchoscope camera is 30 frames per
123
376
Int J CARS (2012) 7:371–387
second; however, the typical moving speed of the camera is around 10 mm per second, so the magnitude of inter-frame motion changes are 0.33 mm per second. Therefore, we used a random walk on the basis of normal density with respect to noise coefficient n ik : n ik ∼ N μ, σ 2 to predict the state dynamic [26]: 1 k k k p(xi |xi−1 ) ∝ √ exp − B−1 xik − Axi−1 2π σ
2 2 (11) −μ /2σ . For Eq. 10, we must calculate uncertain inter-frame camera motion or stochastic diffusion component B n ik . We use two small constants, σt and σθ , to respectively describe the relative translation and rotation motion since the bronchoscope usually undergoes a small motion (translation and rotation) between two continuous frames, as aforementioned. According to a random number generator based on normal density k (see Eq. 11) to produce n i ∈ [0, 1) for the kth samrandom coefficient n ik ple at time i, we compute relative motion with translaT (i) tion vector tik : tk = n ik σt , n ik σt , n ik σt , and three quaternion rotation parts around the x-, y-, and z-axes: x qik n ik σ θ , ex , y qik n ik σ θ , e y , and z qik n ik σ θ , ez , where ex = (1 0 0)T , e y = (0 1 0)T , and ez = (0 0 1)T . Hence, stochastic diffusion component Bn ik can be represented by: T (12) B n ik = x qik y qik z qik tik . Applying B n ik to xˆ ik , we can obtain translation tik of xik by: ˆ k x rˆ k , y rˆ k , z rˆ k tk , (13) tik = tˆik + R i i i i i ˆ k are translation and rotation of xˆ k (see Eq. 8). where tˆik and R i i ˆ k to Rk , we first rotate x rˆ k and y rˆ k by quaTo update R i i i i ternion z qik according to Eq. 7 and obtain x r˜ ik and y r˜ ik . We then update z rˆ ik by quaternion y qik and get z r˜ ik . Eventually, we apply x qik to y r˜ ik and z r˜ ik and determine y rik and z rik by: y rk ⇔ y qk n k σ , y rk = x qk · y qk 0, y r˜ k · x qk −1 , i i i i i θ i i i z rk ⇔ z qk n k σ , z rk = x qk · z qk 0, z r˜ k · x qk −1 , i i i i i θ i i i (14)
(c) Observation density After state propagation, we must determine observation density p (yi |xi ). As described in Arulampalam [22], an unavoidable problem during SMC sampling is the sample degeneracy phenomenon, which results in an overwhelming majority of samples having negligible weight after a few iterations. However, we can alleviate such problemsby carefully choosing the impork , y . The most appropriate tance density function q xik |xi−1 i k , y is prior density p xk |xk choice for q xik |xi−1 i i i−1 since it is intuitive and simple to implement [22]. Hence, according to [22,26], observation density p (yi |xi ) can be calculated by ⎛ ⎞−1 M j wi ⎠ . (16) p yi |xi = xik ∝ wik ⎝ j=1
Additionally, the accumulative weight cik of each sample can be determined by Isard and Blake [26] (17) cik = cik−1 + p yi |xi = xik . (d) Calculation of sample weight To evaluate the sample performance, sample weight wik must be calculated during the state evolution and observation density determination. As aforementioned, we define sample weight wik as the (i) similarity between current real bronchoscopic image I R and virtual bronchoscopic image IV generated from estimated camera parameters xik using volume rendering techniques. By (i) the selective image similarity measure [8], where I R and IV are divided into subblocks that are chosen for the similarity computation if they include any specific bronchial information (e.g., bifurcations or folds), we utilize a modified mean squared error (MoMSE) to compute the similarity:
1 (i) 1 (i) (i) D IR − IR MoM S E I R , IV = (i) |D| |H | D D∈H (i)
2 D − IV − IV , (18) where |H (i) | denotes the number of chosen subblocks in the (i)
set of chosen subblocks A(i) and D I R and V D indicate the (i) average intensities of all subblocks D of I R and IV , respec(i) tively. The average intensities of I R and IV may be different in actual bronchoscopic images due to the different powers of the light sources. To alleviate such effects, all subblocks
Finally, after state transmission, in terms of Eqs. 13 and 14, propagated state xik can be represented by: Rik z rik × y rik , y rik , z rik tik k xi = . (15) 0T 1
are normalized by subtracting D I R and D IV from each pixel. Hence, weight wik can be represented by (i) (19) wik = MoMSE I R , IV xik .
After being transmitted through a random walk based on normal density, drifted and diffused state xik has a probabilistic distribution in accordance with Eq. 11.
Finally, we clarify that the output of SMC sampling used to predict the current bronchoscope camera motion state (parameters) can be determined with respect to wik :
123
(i)
Int J CARS (2012) 7:371–387
377
Algorithm 1: SMC Sampling for Motion Prediction [26] I(i) R ,
input : Bronchoscopic video images CT slices for generating virtual images IV using volume rendering techniques [28] ˜ i of camera motion poses output: series of estimates Q Initialization: at i = 0, initial parameters are decided by: k k Q0 ⇔ x0 , Q0 ⇔ y0 ; w0k = MoMSE(I(0) R , IV (x0 )) ; p(x0 ) =
Validation setups
1 M;
Start SMC sampling: for each frame i, generate M samples M and compute the weights (similarities) Sik = {(xik , wik , cik )}k=1
˜ i of SMC sampling: wik and finally determine output Q for i = 1 to N do 1. Construct new sample set Sik at frame i by updating sample k set Si−1 from the factored sampling scheme [26]: for k = 1 to M do k (a). Sample selection: compare ci−1 to random number a; k (b). State evolution: draw new xi from selected samples; (c). Weight wik computation by Eq. 19; (d). Compute incremental weight cik by Eq. 17; end M 2. Compute total weights: Wi = k=1 wik ; 3. Calculate observation density p(yi |xi ) by Eq. 16; 4. Weight normalization: wik = Wi−1 wik ; 5. Store updated sample set Sik for next loop; 6. The current estimated state x˜ i by Eq. 20 from set Sik ; ˜ i; 7. Return the current camera pose: x˜ i ⇐⇒ Q 8. Go to next iteration; end
x˜ i = max w˜ ik
xik , wik ,
correspond to combine SMC sampling with the EMT- and SIFT-based schemes, respectively.
(20)
that is, sample x˜ i with maximal weight w˜ i corresponds to the maximal similarity between the current bronchoscope camera frame and the generated virtual frame. As a brief summary of the above process, our modified bronchoscope motion tracking on the basis of SMC sampling is described in Algorithm 1.
Experiments As described in Step (b) in section “SMC sampling for bronchoscope motion tracking”, we have two ways to determine deterministic drift element A in Eq. 9 during SMC sampling. By the EMT-based approach, we deal with limitation (c) described in Sect. 1 and improve our hybrid methods proposed in Mori et al. [14] and Luo et al. [29]. According to the SIFT-based method, we expect to address limitations (a) and (b), which are also depicted in Sect. 1, and augment our previous image-based methods in Deguchi et al. [8] and Luo et al. [27]. Hence, we actually present two approaches named EMT–SMC and SIFT–SMC sampling methods that
Since we have no patient datasets with EMT measurements, we assessed the EMT–SMC sampling approach by a dynamic bronchial phantom that imitates respiratory motion (see Fig. 4). Its simulated respiratory rate is 0–10 “breaths” per minute, which corresponds to maximum breathing motion of 0–24 mm. Note that such a simulated breath rate is quite slow and impractical, which is one main drawback of our dynamic phantom, compared to human breathing rates of 14–30. For more details on the motion phantom, please refer to our previous work [29]. Bronchoscopic video images of size 362 × 370 pixels were recorded at 30 frames per second using a bronchoscope (BF-P260F, Olympus, Tokyo). In phantom experiments, a 3D Guidance medSAFE tracker (Ascension Technology Corporation,1 USA) was used as an EMT system, which includes a 9-coil flat type transmitter as a magnetic field generator. Additionally, we had to perform camera and hand-eye calibrations and CT-to-physical space registration to calculate the transformation between the EMT and the CT coordinate systems, as shown in Luo et al. [29]. To evaluate the SIFT–SMC sampling method, we applied it to ten patient datasets that include bronchoscopic video frames and their corresponding 3D chest CT images. CT acquisition parameters were (512 pixels)2 × (72–361 slices), voxel resolution of (0.547 ∼ 0.723 mm)2 × (1.0–2.0 mm), and 2.0–5.0 mm thick slices. The dimensions of the bronchoscopic video images were 362 × 370 and 256 × 263 pixels. For the lateral comparison of our SMC-based methods, we compared the proposed methods to our previous published tracking approaches [8,14,27,29]. As for dynamic phantom evaluation of the EMT–SMC method, we only investigated three hybrid-based tracking approaches: (1) Mori et al. [14], who used the EMT results to directly initialize the intensity-based image registration method, (2) Luo et al. [29], who combined a respiratory motion compensation method that utilizes a surrogate sensor to measure breathing movements, and (3) the EMT–SMC sampling method, presented above (Sect. “SMC sampling for bronchoscope motion tracking”). On the other hand, we only performed patient data validation on the SIFT–SMC method by comparing the following three image-based methods: (1) Deguchi et al. [8], a selective image similarity measure-based registration method, (2) Luo et al. [27], who utilized SIFTbased motion estimation and Kalman filtering to improve the initialization of the registration optimization and enhanced its tracking performance [8], and (3) the SIFT–SMC sampling method, as described in section “SMC sampling for bron1
http://www.ascension-tech.com/.
123
378
Int J CARS (2012) 7:371–387
Fig. 4 Dynamic motion phantom: a real phantom and b drawing of phantom movement
choscope motion tracking”. Additionally, we also compare only image- and hybrid-based methods on dynamic phantom; however, we cannot evaluate them on the patient data due to shortages of patient data with EMT measurements. We further clarify that we did not evaluate or compare EMT– and SIFT–SMC methods on patient data in this study. We implemented all tracking procedures off-line outside the operating rooms since our methods cannot track the bronchoscope tip and its integrated camera in real time; we also initialized those tracking processes manually. We have done all implementations on a Microsoft Visual C++ platform and ran it on a conventional PC (CPU: Intel(R) Xeon(R) X5355 ×2 processors, 16-GByte main memory).
Evaluation criterion To quantitatively analyze our tracking results, we introduce the following three evaluation criterion. First, we compute the position error by δ = t − tG ,
(21)
Ground truth data
where δ denotes the Euclidean distance between reference (ground truth) position tG and predicted position t obtained by our proposed tracking methods. Next, for the orientation error, there is no generally used measure. We calculate orientation error by the rotation error about the invariant Euler axis in accordance with the work of Schneider and Stevens [30]: (22) φ = arccos trace RRG T − 1 /2 ,
To investigate the accuracy (clinical relevance) of the tracking results of our SMC-based methods, we collected four phantom and five patient ground truth datasets (GTDs) by updating the position and orientation of the virtual camera to qualitatively align the RB and VB viewing points by hand; this involved an extreme amount of manual labor. These GTDs were independently and repeatedly generated in multiple sessions by three observers, one bronchoscopist and two scientists. We clarify that intra-observer consistency (standard deviation) was 1.71 mm and 4.90◦ , 1.56 mm and 5.9◦ , and 1.63 mm and 3.80◦ from three observers, respectively; inter-observer consistency was 1.32 mm and 4.20◦ .
where RG and R are the reference (ground truth) and predicted orientation matrices. Finally, it is crucial to define a visual assessment to determine whether a method is more accurate or robust than another, because physicians eventually expect to easily and directly visualize the tracking results (the estimated camera position and orientation) during intra-interventional bronchoscopy in examination rooms. Since the tracking results are usually used to generate virtual bronchoscopic (VB) images using volume rendering techniques [28], a visual assessment standard can be defined when a real bronchoscopic (RB) video frame is tracked successfully if a VB image resembles
123
Int J CARS (2012) 7:371–387 Table 1 Examples of position error (mm) and orientation error (degrees) calculated by Eqs. 21 and 22 during phantom validation, in terms of tracking results using the methods of Mori et al. [14], Luo et al. [29], and EMT–SMC
Table 2 Quantitative comparison of registered results for our phantom studies according to the visual inspection criterion
379
Four GTDs Position error (mm) and orientation error (degrees) for experiments Mori et al. [14] Luo et al. [29]
EMT–SMC
Position
Orientation
Position
Orientation
Position
Mean SD
Mean SD
Mean SD
Mean SD
Mean SD
Orientation Mean SD
A
3.04
3.36 10.9
9.42
1.83
3.63 7.46
7.54
0.54
0.57 4.38
4.22
B
4.12
4.13 11.4
9.90
3.95
3.16 8.14
7.69
1.26
2.66 5.73
5.57
C
4.96
4.11 11.6
10.1
4.71
3.20 8.25
8.02
1.71
2.44 5.77
5.86
D
6.02
6.10 12.1
10.4
5.62
3.23 8.47
7.88
2.52
3.06 5.87
5.80
Average
4.54
4.43 11.5
3.23
3.31 8.08
7.78
1.51
2.18 5.44
5.36
Experiments (frames)
9.96
Maximum motion (mm)
Number (%) of successfully processed frames Mori et al. [14]
Luo et al. [29]
EMT–SMC
A (1,498)
0.00
1,150 (76.8%)
1,334 (89.1%)
1,476 (98.5%)
B (1,245)
7.25
918 (73.7%)
1,076 (86.4%)
1,195 (96.0%)
C (1,274)
12.72
811 (63.7%)
1,042 (81.8%)
1,193 (93.6%)
D (1,522)
19.42
Total (5,539)
921 (60.5%)
1,164 (76.5%)
1,378 (90.5%)
3,800 (68.6%)
4,616 (83.3%)
5,242 (94.6%)
Fig. 5 Absolute distance between tracked camera position and CT origin from tracking results estimated by methods of Mori et al. [14] (cyan), Luo et al. [29] (blue), and EMT–SMC (green) plotted against the ground
truth (red) of Experiment D. The green curve mostly overlaps the red curve in contrast to the cyan or blue curves
its corresponding RB image. Such a measure can compare the current frame to the previous and successive frames to check whether the predicted result follows the actual motion of the RB camera. Moreover, based on this standard, we manually inspected and counted the number of successfully registered frames to quantify our tracking results.
GTDs to the predicted results from different tracking methods. The average position error was 4.54, 3.23, and 1.51 mm for the methods of Mori et al. [14], Luo et al. [29], and EMT– SMC, respectively. The average orientation error of the three methods was 11.5◦ , 8.08◦ , and 5.44◦ . From Experiments A, B, C, and D in Table 1, the average tracking errors were heavily affected by airway deformation (as shown in Table 2). Figure 5 plots Euclidean distances between the CT coordinate origin and the tracked camera positions or ground truth positions. As shown in Fig. 5, our algorithm performed best since its distance curve (green) overlaps the ground truth curve (red) more closely, compared with the other two (cyan and blue curve). Finally, based on visual inspection, all successfully registered RB frames are quantitatively summarized
Results During phantom assessment of EMT-based methods, we validated our proposed EMT–SMC method on four video sequences acquired from our dynamic phantom. Table 1 quantifies the position and orientation error by comparing the
123
380
Int J CARS (2012) 7:371–387 Frame number
0062
0170
0241
0401
0675
0773
0820
0923
1160
1326
1463
1510
0731
0825
1132
1182
1237
1333
RB images
Mori et al. [14]
Luo et al. [29]
EMT-SMC
(a) Frame number
0063
0139
0193
0332
0433
0517
RB images
Mori et al. [14]
Luo et al. [29]
EMT-SMC
(b) Fig. 6 Visual comparison of tracking results of Experiments C and D for different methods during dynamic phantom validation. Top row shows selected frame numbers, and second row shows their corresponding phantom RB images. Other rows display virtual bronchoscopic
images generated from tracking results using the methods of Mori et al. [14], Luo et al. [29], and EMT–SMC. Our proposed EMT–SMC method shows the best performance. a Examples of Experiment C. b Examples of Experiment D
Table 3 Quantitative comparison of registered results for our phantom studies using only image-based methods according to the visual inspection criterion and their position and orientation errors Experiments (frames)
Maximum motion (mm)
Number (%) of successfully processed frames Deguchi et al. [8]
Luo et al. [27]
SIFT–SMC
A (1,498)
0.00
595 (39.7%)
653 (43.6%)
867(57.9%)
B (1,245)
7.25
987 (79.3%)
1,135 (91.2%)
1,214 (97.5%)
C (1,274)
12.72
916 (71.9%)
1,107 (86.9%)
1,208 (94.8%)
D (1,522)
19.42
516 (33.9%)
621 (40.8%)
843 (55.4%)
Total (5,539)
3,014 (54.4%)
Average errors
Position
Orientation
3,516 (63.5%) Position
Orientation
Position
Orientation
(mm, degrees)
8.32
22.3
6.04
19.6
4.88
14.3
in Table 2. Figure 6 visually inspects the successfully registered RB and VB images at the chosen frames by all three methods. Both further demonstrate the better accuracy and robustness of our proposed EMT–SMC method. For validation of the SIFT–SMC method, we first use the same four video sequences used in the dynamic phantom evaluation of the EMT–SMC method for the comparison of only image- and hybrid-based methods. During patient
123
4,132 (74.6%)
studies, we evaluated our SIFT–SMC method on ten patient cases including five paths of ground truth datasets generated by manual registration. Table 3 quantitatively compares the tracking results and gives the position and orientation errors of each image-based method during phantom validation. It shows that the average errors of the SIFT–SMC method are 4.88 mm and 14.3◦ and lower than those of the other two methods. Table 4 illustrates the position and orientation errors
Int J CARS (2012) 7:371–387
381
Table 4 Examples of position error (mm) and orientation error (degrees) computed by Eqs. 21 and 22 during patient assessment in accordance with tracking results from methods of Deguchi et al. [8], Luo et al. [27], and SIFT–SMC Five GTD cases
Position error (mm) and orientation error (degrees) Deguchi et al. [8] Position Mean
6 (C) 7 8 9
32.8 6.93 4.34 15.3
SD 26.2
Luo et al. [27] Orientation
Position
Mean
Mean
SD
31.9
28.5
7.63
23.4
22.1
2.77
10.1
14.3
45.6
5.91
16.9
SD 13.1
5.07
5.10
SIFT–SMC Orientation
Position
Mean
SD
Mean
29.3
19.9
16.5
14.5
9.29
4.51
Orientation SD
Mean
7.70
8.28
9.78
7.12
1.55
0.30
3.65
0.95
4.07
2.28
3.85
2.03
28.5
7.75
9.09
24.1
30.9
3.39
3.61
8.73 15.0
SD
4.38 15.3
10
13.8
11.6
23.8
18.6
5.06
4.58
17.1
15.9
2.10
3.19
13.8
25.3
Average
14.6
12.5
27.0
20.7
7.77
6.83
19.3
17.1
3.72
3.48
10.2
10.6
of the application of our method to various patient cases. The average position error was reduced from 14.6 and 7.77 to 3.72 mm by utilizing the methods of Deguchi et al. [8], Luo et al. [27], and SIFT–SMC. The average orientation error of the three methods also decreased from 27.0◦ and 19.3◦ – 10.2◦ . Figure 7 compares the estimated position and orientation of the three methods (cyan, blue, and green curves) to their ground data poses in x-, y-, and z-directions. We can see the green curve of the SIFT–SMC method that is closer to the red curve in six dimensions. Figure 8 plots the position and orientation errors of Case 9 from the three methods. As Table 4 and Figs. 7 and 8 show, the SIFT–SMC method more precisely approximates the motion of the bronchoscope camera in contrast to our previous methods [8,27]. Table 5 displays the quantitative tracking results of each method performance. Figure 9 shows the selected RB frames aligned to VB images to compare the performance of three methods based on the visual assessment criterion. Our method in general successfully tracked long sequence of RB frames. Figure 10 plots the estimated camera trajectories of each approach on pre-built 3D airway anatomies. Our method can still follow the longer paths of the bronchoscope camera movements better than the other two methods.
Discussion Our proposed SMC-based method deals with the disadvantages of image-, EMT-, or hybrid-based bronchoscope motion tracking approaches during navigated bronchoscopy. Those disadvantages are usually caused by the shortage of characteristic bronchial information (e.g., bifurcations or folds), airway deformation, problematic or uninformative video frames, and so on. According to dynamic phantom and patient validation, our experimental results in general
demonstrated that our method enables a more accurate and robust means to navigate a bronchoscope inside a pre-built 3D airway tree model during intra-bronchoscopy.
Dynamic phantom and patient study During navigated bronchoscopy, no matter what kind of tracking methods (e.g., image-, EMT-, or hybrid-based) we use, tracking a bronchoscope tip and its integrated camera is an inherently ambiguous and accumulatively erroneous procedure that unavoidably suffers from the following study limitations. First, in regard to an EMT system, calibration and registration errors are inevitably introduced when we perform camera and hand-eye calibration to determine the relationship between an EMT sensor and a bronchoscopic camera and marker-based CT-to-physical space registration to calculate the transformation between EMT and CT coordinate systems. Next, an entire airway tree structure sometimes may be recorded incompletely by a CT scanner due to its systematic disadvantages. In such cases, it is difficult for image-based methods to work well. Again, during intra-bronchoscopy, unpredictable bronchoscopic video scenes (see Fig. 2 in Sect. 1) often occur and cause ambiguities or multi-modal (not unimodal) distribution of camera motion parameters in bronchoscope motion tracking when physicians navigate a bronchoscope. Finally, the bronchial tree shape definitely changes between real-time bronchoscopic video and CT images usually acquired when patients are asked to fully inhale or exhale and hold their breath. This is a challenge for all tracking methods since patients are breathing regularly and constantly deforming the airways during intra-bronchoscopy; it is particularly more challenging to track a bronchoscope inside a small bronchial branch with a large respiratory motion. These limitations result in ambiguities and uncertainties that must be addressed during
123
382
Int J CARS (2012) 7:371–387
Fig. 7 Tracked position and orientation from methods of Deguchi et al. [8] (cyan), Luo et al. [27] (blue), and SIFT–SMC (green) drawn against the ground truth (red) of Case 9 in total 6◦ of freedom. a Orientation comparison of Case 9. b Position comparison of Case 9
intra-bronchoscopy. An ambiguous or multi-modal model is required to characterize such stochastic errors in bronchoscope motion tracking. The SMC sampling (a multi-modal model) has the ability to approximate the posterior density distribution of bronchoscope camera motion parameters and provides a more stable and accurate scheme to estimate a bronchoscope tip and its combined camera motion, particularly in ambiguity occurrence due to problematic bronchoscopic video frames and airway deformation during bronchoscopy navigation. Several key advantages are clarified when applying our proposed SMC-based approach to endoscope tracking. First, SIFT features were employed in SMC methods, which introduce texture unrelated to the characteristic bronchial information and to improve the tracking performance, for instance, by making the registration stage of an image-based method less susceptible to visible bifurcations or folds. Furthermore, our developed SMC sampling approach can recover from a failed tracking procedure by itself. During the registration step (an optimization procedure) of the image- or hybrid-based method, the optimizer can get trapped in local minima due to image artifacts, which results in failure to follow the tracking
123
procedure. The SMC method described, however, is a multimodal probability distribution, approximating the posterior densities of the state parameters by collecting a set of random samples, and sequentially predicts the state vector on the basis of the factored (or importance) sampling. It provides the ability to maintain potential importance modes that are either confirmed or moved to be subsequent observations. This contributes significantly to the avoidance of tracking failure or to the automatic retrieval of tracking loss even in case of image artifacts or airway deformation. Hence, our proposed method can successfully register long sequences of video and follow long camera trajectories of the bronchoscope motion, as shown in Tables 2 and 5 and Figs. 6, 9, and 10; furthermore, our method can obtain better accuracy as shown in Tables 1 and 4 and Figs. 5, 7, and 8. Lastly, although we focused on developing bronchoscope motion tracking algorithm to construct intra-bronchoscopy navigation to guide physicians for diagnosis and therapy of bronchus and lung cancer, we believe that our SMC-based tracking methods are also appropriate for other surgical endoscopic tracking environments, such as colonoscopic and angioscopic intrainterventions.
Int J CARS (2012) 7:371–387
383
Fig. 8 Position and orientation error of Case 9 from different methods including Deguchi et al. [8] (cyan), Luo et al. [27] (blue), and SIFT–SMC (green). a Orientation error of Case 9. b Position error of Case 9 Table 5 Quantitative comparison of tracking results for our patient studies in terms of visual inspection criterion Cases (path)
Number of frames
Moving trajectory
Number (%) of frames successfully tracked Deguchi et al. [8]
Luo et al. [27]
SIFT–SMC
1
750
TR
144 (19.2%)
447 (59.6%)
610 (81.3%)
2
1,000
TR → RM → RT
456 (45.6%)
468 (46.8%)
929 (92.9%)
3
1,000
TR → RM
591 (59.1%)
685 (68.5%)
815 (81.5%)
4 (A)
2,200
TR → LM
896 (40.7%)
1,660 (75.5%)
2,100 (95.4%)
4 (B)
1,500
LM → LU
152 (10.1%)
1,027 (68.5%)
1,218 (81.2%)
5
1,000
RM → RU
130 (13.0%)
263 (26.3%)
816 (81.6%)
6 (A)
1,500
TR → RM → RU
335 (22.3%)
483 (32.2%)
1,107 (73.8%)
6 (B)
800
TR → LM → LL1 → LL2
583 (72.9%)
677 (84.6%)
690 (86.3%)
379
LL → LM → TR
6 (C) 7
1,120
85 (22.4%)
208 (54.9%)
293 (77.3%)
TR → RM → TR → LM
603 (53.8%)
926 (82.7%)
1,105 (98.7%)
8
450
TR → LM → LL
416 (92.4%)
418 (92.8%)
423 (94.0%)
9
1,750
TR → LM
505 (28.9%)
1,197 (68.4%)
1,603 (91.6%)
450
TR → LM
10 Total
13,899
245 (54.4%)
389 (86.4%)
424 (94.2%)
5,141 (37.0%)
8,848 (63.7%)
12,133 (87.3%)
TR trachea, LM left main bronchus, RM right main bronchus, LL1 left lower lobe bronchus 1, LL2 left lower lobe bronchus 2, LU left upper lobe bronchus, RU right upper lobe bronchus, RT right trunchus intermedius
Additionally, we evaluated image-based methods on our dynamic phantom to compare them to hybrid-based methods. Tables 1, 2, and 3 demonstrate that the tracking performance of image-based methods is generally worse than that of hybrid-based methods, which was also proven in the work of Soper et al. [15]. However, the tracking results of Experiments B and C using image-based methods show better results than those of hybrid-based methods. This is because these two video sequences have little dark (uninformative) bronchoscopic images during their acquisitions. Conversely, the bronchoscope often collided with the bronchial walls and the illumination of the bronchoscope changed nonlinearly and discontinuously since we moved the bronchoscope very fast during the data acquisition of Experiments A and D. This causes incorrect SIFT feature detection and
difficulties of converging to a good estimate of the bronchoscope motion during optimization processes, as previously discussed [27]. We may conclude that image-based methods depend on image quality and are hardly sensitive to airway deformation or other dynamic errors (good local localizers) while hybrid-based (or EMT-based) approaches are good global localizers (they can always follow general movements of a bronchoscope) but susceptible to patient movements and magnetic field distortion. Runtime Our EMT–SMC method takes approximately 0.9 s to process one frame, while the methods of Mori et al. [14] and Luo et al. [29] need 0.3 and 0.5 s per frame. Compared
123
384
Int J CARS (2012) 7:371–387 Frame number
6855
6925
7035
7113
7218
7328
7365
7435
7464
7522
7563
7616
12669
12836
12951
13025
13436
13475
13964
14020
14168
14227
14302
14373
RB images
Deguchi et al. [8]
Luo et al. [27]
SIFT-SMC
(a) Frame number
12367
12392
12419
12484
12511
12613
RB images
Deguchi et al. [8]
Luo et al. [27]
SIFT-SMC
(b) Frame number
13585
13677
13775
13876
13918
13932
RB images
Deguchi et al. [8]
Luo et al. [27]
SIFT-SMC
(c) Fig. 9 Visual comparison of tracking results of Cases 2, 4(B), and 5 for different methods during patient validation. Top row shows selected frame numbers, and second row shows their corresponding patient RB images. Other rows display virtual bronchoscopic images generated
from tracking results using methods of Deguchi et al. [8], Luo et al. [27], and SIFT–SMC. Our proposed SIFT–SMC method shows the best performance. a Examples of Case 2. b Examples of Case 4 (B). c Examples of Case 5
with the computation times of 0.2 and 0.8 s per frame of Deguchi et al. [8] and Luo et al. [27], the SIFT– SMC method requires 1.2 s to process one frame. This is because the SMC method must determine each sample weight calculated from the similarities between real and virtual images, which is very time-consuming. Moreover, SIFT-based motion tracking processing is also timeconsuming in terms of the detection of points to find correspondences and perform epipolar geometry analysis. Figure 11 gives examples of the computation times by each method. Additionally, we clarify that we use 500
samples (M = 500) in our SMC-based methods, which was chosen as a compromise between accuracy and computational time. In our experiments, we changed the sample number M from 200 to 900 in increasing of 100 and found that the tracking performance of our methods was very slightly improved after 500.
123
Other challenges During our dynamic phantom and patient assessment of image- and hybrid-based methods, our SMC sampling
Int J CARS (2012) 7:371–387
385
Fig. 10 Examples of tracking results plotted as camera trajectories on pre-built 3D airway tree models in terms of methods of Deguchi et al. [8] (cyan), Luo et al. [27] (blue), and SIFT–SMC (green). These results
clearly demonstrate that our method can follow long paths of bronchoscope camera motion (red dots denote ground truth)
Fig. 11 Examples of computation time by each method: a the EMT– SMC method (green) needs about 0.9 s per frame, which is much higher than the approaches of Mori et al. [14] (cyan) and Luo et al. [29] (blue),
b the SIFT–SMC method (green) average computational time with about 1.2 s per frame is much slower than the methods of Deguchi et al. [8] (cyan) and Luo et al. [27] (blue)
methods still misaligned the RB and VB frames in some phantom cases (e.g., Case E) when continuously tracking the bronchoscope. Several challenges remain to solve this.
The general challenge of the above EMT system in Sect. 1 is the dynamic error of EMT due to ferrous material contained inside the bronchoscope, resulting in magnetic field
123
386
distortion of the guidance transmitter, sharply deteriorating the tracking accuracy. For image-based tracking approaches, it remains challenging to deal with situations where the information on the insertion depth and rotation around the viewing (running) direction of the bronchoscope camera are absent. Since such information usually presents the predominate motion of the bronchoscope, an optimization process easily gets trapped in local minima without such information. New external tracking methods to directly measure such information are needed in future work. Another problem in all methods is that, when acquiring pre-operative CT data of patients, physicians usually require patients to hold their breath in full inhalation or exhalation; however, during bronchoscopic intervention, patients are breathing regularly, deforming the airways, resulting in inaccurate virtual rendering due to static CT data used in similarity computation procedures. This also needs to be addressed to construct more practical virtual bronchoscopy in the future. Additionally, when using the EMT system, an EMT sensor is usually integrated into ultra-thin bronchoscopes (i.e., 2.8 mm in external diameter). However, after attaching the EMT sensor with a 1.3 mm diameter at the bronchoscope tip, an unavoidable problem appeared: the bronchoscope cannot fly through most of the bronchial tree because of the enlarged size of the bronchoscope tip with an additional sensor. Finally, we still clarify that the drawback of our approach is its computational efficiency since each random sample must compute its weight based on the similarities between real and virtual images during the SMC sampling; this is a time-consuming process. Currently, we did neither code nor speed optimizations; nor did we use multi-threading. We can also employ graphics processing unit (GPU) techniques to accelerate our implementations and approximate real time.
Conclusions We proposed an SMC sampling means of modeling ambiguities or uncertainties that occur in bronchoscope motion tracking and demonstrated its effectiveness (more accurate and robust) to predict bronchoscope motion during intrabronchoscopy navigation. We believe that it is a better choice to use the SMC sampling to fuse the tracking sources (e.g., EMT outputs or SIFT-based estimates) to predict the bronchoscope motion in the image- or hybrid-based method than directly using either the EMT outputs or the SIFT-based estimates to initialize the optimization process of the registration step. Our future work includes experiments on patient datasets using the EMT–SMC method in operating rooms, comparisons of EMT–SMC and SIFT–SMC approaches on patient data, reduction in computational time, and further improvements of the tracking performance of the image- and hybrid-based methods during bronchoscopy navigation.
123
Int J CARS (2012) 7:371–387 Conflict of interest None.
References 1. American Cancer Society (2010) Global cancer facts & figures 2010, American Cancer Society, Atlanta. http://www.cancer.org/ Research/CancerFactsFigures 2. Mori K, Urano A, Hasegawa J, Toriwaki J, Anno H, Katada K (1996) Virtualized endoscope system—an application of virtual reality technology to diagnostic aid. IEICE Trans Inf Syst E 79D(6):809–819 3. Kiraly AP, Higgins WE, McLennan G, Hoffman EA, Reinhardt JM (2002) Three-dimensional human airway segmentation methods for cinical virtual bronchoscopy. Acad Radiol 9:1153–1168 4. Mayer D, Bartz D, Fischer J, Ley S, del Rio A, Thust S, Kauczor H-U, Heussel CP (2004) Hybrid segmentation and virtual bronchoscopy based on CT images. Acad Radiol 11:551–565 5. Deligianni F, Chung AJ, Yang GZ (2006) Nonrigid 2-D/3-D registration for patient specific bronchoscopy simulation with statistical shape modeling: phantom validation. IEEE Trans Med Imaging 25(11):1462–1471 6. Helferty J, Sherbondy A, Kiraly A, Higgins W (2007) Computerbased system for the virtual-endoscopic guidance of bronchoscopy. Comput Vis Image Underst 108:171–187 7. Suter MJ, Reinhardt JM, McLennan G (2008) Integrated CT/bronchoscopy in the central airways: preliminary results. Acad Radiol 15:786–798 8. Deguchi D, Mori K, Feuerstein M, Kitasaka T, Maurer CR Jr, Suenaga Y, Takabatake H, Mori M, Natori H (2009) Selective image similarity measure for bronchoscope tracking based on image registration. Med Image Anal 13(4):621–633 9. Solomon SB, White JP, Wiener CM, Orens JB, Wang KP (2000) Three-dimensionsal CT-guided bronchoscopy with a realtime electromagnetic position sensor: a comparison of two image registration methods. Chest 118(6):1783–1787 10. Schwarz Y, Greif J, Becker HD, Ernst A, Mehta A (2006) Realtime electromagnetic navigation bronchoscopy to peripheral lung lesions using overlaid CT images: the first human study. Chest 129(4):988–994 11. Levy EB, Tang J, Lindisch D, Glossop N, Banovac F, Cleary K (2007) Implementation of an electromagnetic tracking system for accurate intrahepatic puncture needle guidance: accuracy results in an in vitro model. Acad Radiol 14:344–354 12. Yaniv Z, Cheng P, Wilson E, Popa T, Lindisch D, Campos-Nanez E, Abeledo H, Watson V, Cleary K, Banovac F (2010) Needle-based interventions with the image-guided surgery toolkit (IGSTK): from phantoms to clinical trials. IEEE Trans Biomed Eng 57(4): 922–933 13. Muller SA, Maier-Hein L, Tekbas A, Seitel A, Ramsauer S, Radeleff B, Franz AM, Tetzlaff R, Mehrabi A, Wolf I, Kauczor H-U, Meinzer H-P, Schmied BM (2010) Navigated liver biopsy using a novel soft tissue navigation system versus CT-guided liver biopsy in a porcine model: a prospective randomized trial. Acad Radiol 17:1282–1287 14. Mori K, Deguchi D, Akiyama K, Kitasaka T, Maurer CR Jr, Suenaga Y, Takabatake H, Mori M, Natori H (2005) Hybrid bronchoscope tracking using a magnetic tracking sensor and image registration. In: Proceedings of MICCAI 2005, vol LNCS 3750. pp 543–550 15. Soper TD, Haynor DR, Glenny RW, Seibel EJ (2010) In vivo validation of a hybrid tracking system for navigation of an ultrathin bronchoscope within peripheral airways. IEEE Trans Biomed Eng 57(3):736–745 16. Kindratenko VV (2000) A survey of electromagnetic position tracker calibration techniques. Virtual Real 5(3):169–182
Int J CARS (2012) 7:371–387 17. Feuerstein M, Reichl T, Vogel J, Traub J, Navab N (2009) Magneto-optical tracking of flexible laparoscopic ultrasound: model-based online detection and correction of magnetic tracking errors. IEEE Trans Med Imaging 28(6):951–967 18. Luo X, Reichl T, Feuerstein M, Kitasaka T, Mori K (2010) Modified hybrid bronchoscope tracking based on sequential monte carlo sampler: dynamic phantom validation. In: Proceedings of ACCV 2010, vol 3. pp 1722–1733 19. Arnaud Doucet CA, Godsill S (2000) On sequential Monte Carlo sampling methods for Bayesian filtering. Stat Comput 10(3): 197–208 20. Doucet A, de Freitas N, Gordon N (2001) Sequential Monte Carlo methods in practice. Springer, Berlin 21. Liu JS, Chen R (1998) Sequential Monte Carlo methods for dynamic systems. J Am Stat Assoc 93(443):1032–1044 22. Arulampalam M et al (2002) A tutorial on particle filters for nonlinear/non-gaussian Bayesian tracking. IEEE Trans Signal Process 50:174–188 23. Hartley R, Zisserman A (2004) Multiple view geometry in computer vision. Cambridge University Press, Cambridge
387 24. Hart JC, Francis GK, Kauffman LH (1994) Visualizing quaternion rotation. ACM Trans Graph 13(3):256–276 25. Hanson AJ (2006) Visualizing quaternions. Morgan-Kaufmann/ Elsevier, Los Altos 26. Isard M, Blake A (1998) Condensation—conditional density propagation for visual tracking. Int J Comput Vis 29(1):5–28 27. Luo X, Feuerstein M, Deguchi D, Kitasaka T, Takabatake H, Mori K Development and comparison of new hybrid motion tracking for bronchoscopic navigation. Med Image Anal (in press) 28. Mori K, Suenaga Y, Toriwaki J (2003) Fast software-based volume rendering using multimedia instructions on PC platforms and its application to virtual endoscopy. In: Proceedings of SPIE, vol 5031. pp 111–122 29. Luo X, Feuerstein M, Sugiura T, Kitasaka T, Imaizumi K, Hasegawa Y, Mori K (2010) Towards hybrid bronchoscope tracking under respiratory motion: evaluation on a dynamic motion phantom. In: Proceedings of SPIE, vol 7625. p 76251B 30. Schneider M, Stevens C (2007) Development and testing of a new magnetic-tracking device for image guidance. In: Proceedings of SPIE, vol 6509. p 65090I
123
