Optical Flow-Based Motion Estimation of Ultrasonic

0 downloads 0 Views 2MB Size Report
called time-domain cross-correlation with prior estimates ... (TOPE) which is based on the standard time-domain cross ..... I.Z. Zhu, "Solid mechanics", fifth edition ...
Optical Flow-Based Motion Estimation of Ultrasonic Images for Force Estimation in Percutaneous Procedures: Theory and Experimental Validation

Arash Maghsoudil, Mehran Jahed2 I Robotics and Machine Vision lab, School of Electrical Engineering, Sharif University of Technology, Tehran, Iran. 2 School of Electrical Engineering, Sharif University of Technology, Tehran, Iran. Email: la [email protected] Email: [email protected] Ahstract- In recent years, there hass been a pronounced emphasis Langevin et al. [12] proposed an in vivo ultrasound (US)-based on percutaneous needle steering with the aid of advanced soft technique to estimate the tissue displacement during tiss�e mo� elin� techniques In this wo�k an optical flow �ased acupuncture manipulation based on the cross-correlation : motIOn estImatIOn method IS used to estImate the force apphed to techniques. Their method used cross correlation on the RF. . . . the needle by the soft tissue during percutaneous applications. The Imes m pre-co�pr�ssed and post-compre:sed ultrasonIc sIgnal study considers Finite Element Model (FEM) of the tissue evaluated by the deformation data acquired through the optical to fi� ? the motIon m the post-compressed Image. . flow method. To represent the soft tissue behavior, dynamic FEM Zahm-Azar et al. [13] proposed a speckle trackmg method with Rayleigh damping and viscoelastic models are used. The called time-domain cross-correlation with prior estimates method is validated experimentally through offline evaluation of (TOPE) which is based on the standard time-domain cross the ultrasonic images of the chicken breast punctured by a needle. correlation strain estimation (TDE) for the motion estimation in The force applied to the needle is measured through a force sensor ultrasonic images. Their method improved the computation situated externally at the dist�1 end of the ne�dle shaft. Measured speed due to the usage of the prior estimates. data from the force sensor IS compared wIth the result of the Wang et al . [14] used the theory of optical flow to present a . . . . proposed method. It is shown that both tissue models are framework for robust non-ngld regIstratIOn. They also e ftiect·IveIy able t0 est'Imate the fiorce. . . mtroduced a robust M-estImator to decrease the effect of Keywords: needle insertion, optical flow, force estimation, outliers in global optimization. viscoelastic model, dynamic FEM with Rayleigh damping. Marti' et al. [15] evaluated the applicability of the deformation I. INTRODUCTION field obtained by optical flow and image registration algorithms In many practical medical procedures, it is required to reach a for elastic modulus imaging. Their results showed that optical target located deep inside a soft tissue. Often flexible needles flow method provides a slightly better reconstruction. are used in procedures such as brachytherapy [1] or biopsy [2]. In this paper, optical flow is used to estimate the tissue Complex mechanical behavior of the tissue along with needle deformation during needle insertion. Next, this deformation flexibility and beveled structure of its tip are some of the data is used to evaluate two FEM models for the soft tissue namely, dynamic FEM with Rayleigh damping and the challenges that are to be dealt with in such problems. Force estimation during needle insertion is crucial as it can be viscoelastic model. The method is implemented on the exploited either for haptic purposes [3] or model-based control ultrasonic images captured during needle insertion into the procedures [4]. In haptic applications it is essential to give a chicken breast tissue and the estimated force by this method is sensory data to the physician during insertion, and in the compared to the force data acquired by a force sensor devised model-based control, the force can be used as a feedback for at the base of the needle. II. METHODS the controller to improve the performance of the automatic A. Rayleigh damping model for soft tissue insertion system [5]. The vital element of a force estimation system is the soft tissue In general, linear static problems in elasticity require solving modeling. Finite Element Model (FEM) as a powerful solution the following equations [16], 1 to solid mechanics problems has been widely used for the E = '2 (Vw + (VW)T) simulation of the soft tissue behavior [6]. DiMaio et al. [7] V. (JT + F = 0 used a 2-D, static and linear FEM model to represent a linearly­ _

elastic soft tissue. Alterovitz et al. [8] used a dynamic and linear FEM to model the soft tissue while considering the mass of each element and velocity dependent forces in the FEM formulation. Goksel et al. [9], [10] used a 3-D FEM model of the prostate for the simulation of the prostate brachytherapy. Dehghan et al. [11] simulated the prostate with a 3-D FEM with the Neo-Hookean constitutive equation. Beside tissue model, the tissue deformation during needle insertion is also required for efficient force estimation.

978-1-4673-2421-2/12/$31.00 ©2012

IEEE

(1) (J = 2f1E + A(tr(c))I where w is the displacement vector field, c is the strain tensor field, (J is the stress tensor field, � and Ie are the Lame material constants (which have relation to Young modulus and Poisson ratio of the tissue), I is the identity matrix, tr(.) denotes the trace of matrix and F is the field of body force per unit volume. Finding solutions to the equations of elasticity can be restated in terms of either finding a displacement field w that satisfies the Lame-Navier equations or finding a stress field (J that

1557

satisfies the equations of equilibrium and the Beltrami-Michell compatibility equations [17]. The Lame-Navier equations can be written as (A + fl)V(V.w) +flV2W + F= 0 (2) However in dynamic problems of elasticity, the Lame-Navier equation can be written as [IS]

(A + fl)V(V.w) +flV2W + F= P

a 2t: + � �� a

(3)

where p is the density of the material and � is the damping constant of the material. The solution of (3) using FEM [IS] results in the following equation M�+ Ct:\!+Kw = F (4) where M is the system mass matrix, C is the system damping matrix, K is the global stiffness matrix depending on the tissue mechanical properties (Young modulus and Poisson ratio); and w represent the values ofw in the discrete space. It is sometimes assumed, that the damping matrix is a linear combination of mass and stiffness matrices [19] C = aK + 13M (5) where the parameters a and 13 are determined experimentally. Such damping is known as 'Rayleigh damping'. B. Viscoelastic model for soft tissue The stress-strain relationship for a viscoelastic material using the Voigt model can be written as [20] (J = 2flE + Atr(E)1 + A'tr(E)1 + 2fl'£ (6) where A'and fl' are the viscosity characteristic parameters. The viscosity parameters of the tissue are defined as [21], ' fl'= A =(7) where rJ is the coefficient of viscosity of the medium. Using the new expression for (J from equation (6) along with other two equations in (1) and exploiting the FEM results in the following [22], M�+ Cvet:\!+Kw = F (S) where Cve is the damping matrix which depends on the coefficient of viscosity (11) and the K and M matrices are as defined in (4). C. Optical flow The main assumption of the optical flow method for motion estimation is that the luminance of the objects in the image does not vary much between two images in a video sequence. Considering this assumption, it can be written (9) I(X, t) == I(X + oX, t + Ot) where I(X, t) is the image intensity function at time t and oX is the displacement field of the image after time Ot. Using the Taylor series, left side of (9) can be written as, (10) I(X, t) == I(X, t) + VI. oX + otIt where VI= (Ix,ly) and It are the first derivatives of I with respect to variable t. Dividing (10) by Ot results in (11) VI. v + It = 0



where v=

oX ot



is the image velocity (flow) filed.

Equation (II) which is called optical flow constraint equation is not a well-posed problem. Often regularization methods are required to make the problem well-posed. Tikhonov-like regularizers [23] result in the Horn & Schunck method [24].

This regularizer along with optical flow constraint equation introduce constraints on the image which can be used to define this error functional [25], JD ((VI. v + It)2 + y2tr((VV)T (Vv))) dX (12)

where D denotes the domain of interest and y is the weight of the regularizer. Minimization of the functional in (12) with respect to v, results in the velocity of the motion of the objects in the image, hence the optical flow. EXPERIMENTAL SETUP III. To experimentally validate the proposed method, an IS gauge biopsy BARD needle (MCIS20) was driven by a Nanomotion linear servo motor with AB5 driver and an optical encoder. A PO controller was exploited to insert the needle at constant velocity into the chicken breast tissue. A Honeywell force sensor (Honeywell FSS series) which measures the force in range of 0-15 Newtons was placed at the needle base to measure the force applied to the needle during insertion. The force data was acquired via a dSPACE (OS1104) data acquisition device. To capture the ultrasound images, a Sonix RP unit was used in conjunction with the Ulterius SDK for image acquisition via TCP/IP connection using Microsoft® Visual C++ which enabled us to capture the frames in a real-time manner. The linear probe utilized was an Ultrasonix Medical Corp 3S-mm L12-5. Fig. 1, shows the experimental set-up used to insert the needle into the tissue along with the ultrasonic probe and the force sensor at the base of the needle. EXPERIMENTAL RESULTS IV. Position of the needle in video images was fused with the position data from linear motor encoders to synchronize the force and the motion frame information. Each frame was segmented manually to localize the tissue boundary and a triangular mesh was built for this extracted area. The needle tract was also extracted manually. Then the optical flow was calculated inside the tissue boundary for each frame and consequently the displacement of the mesh nodes juxtaposed to the needle tract was used to evaluate equations (4) and (S) to compute the forces. The synchronicity of the calculated force and that of the force sensor was achieved via accordance of the needle tract in each frame with the position data acquired through the motor encoders. Fig. 2, shows the initial tissue mesh with corresponding extracted needle tract, two selected deformed mesh and the motion estimated using the optical flow. Fig. 3, presents a comparison of the force calculated through the evaluation of equations (4) and (S) and the motion estimation data; and the force acquired by the force sensor. V. CONCLUSION Force estimation techniques in needle insertion frequently use the correlation of the ultrasound RF-lines (1-D information of the image). In contrast, in this paper an optical flow method which uses the 2-D information of the image to find the motion in the ultrasound image was exploited. To estimate the force, the motion extracted by optical flow is used to evaluate the tissue models.

1558

(a)

(b)

Fig. 1. a) Experimental set up: 1) Needle, 2) Ultrasound probe, 3) Tissue (Chicken breast), 4) Force sensor. The needle, tissue holder and the linear motor actuator apparatus are also visible. b) Experimental schematic.

(a)

(c) Fig. 2. a) The pre-needle insertion tissue (Left) with corresponding initial mesh (Right), b) and c) the tissue boundary and needle tract (Left) extracted manually, the defonned mesh (Middle) and the estimated motion using optical ±low (Right) for two selected frames

1559

r-------r--r--..---.-,

ir: ..-. :C ',' � � , 1

0.2 Z

I ut--�-"'�.J...�""""-�

. --= --,-,-=---.---.- --= ------------�������� -r- � nr-: == , ---.-� : J f :,ll-�r ������:��� , .0 20

0 .5

1

2

1.5

2.5

-020

0'5

2

2.5

2

2.5

------



t



°0�==== ��======-7------��------�------� 1. 5 U � 1

�-O.10:---

---=-'=-

---

0. 5

t

--....

Of----' ----'-

---

Time(s)

-' ----,,-

---

(a)

1.5

;;

-' -----

---

2

-----l

---

2.5

1

a

0.5

1

1.5

-'-1 Time(S)

-'- _ -1-_ 1. 5 2

a

8 �-01

0

-'0.5

(b)

2.5

Fig. 3. The comparison of the force estimated using the optical flow and the force acquired by sensor and the corresponding needle position and velocity for a) Viscoelastic model, b) Rayleigh Damping model. [10] Goksel, 0., Salcudean, S. E, DiMaio, S. P , Rohling, R, and Morris, J, Two models namely, Rayleigh damping and viscoelastic '3D needle-tissue interaction simulation tor prostate brachytherapy', in Proc. Int. ConfMed. Image Comput. Comput. Assist. Interv, 2005, pp. models were used in this work. Based on the estimated force 827-834. samples, there is a close accordance between the date acquired [11] Dehghan, E, and Salcudean, S E, 'Needle Insertion Point and through measurement and FEM model which proves the Orientation Optimization in Non-linear Tissue with Application to efficacy of the motion estimation technique and the tissue Brachytherapy', IEEE International Conference on Robotics and Automation, APRIL 2007, pp. 2267 - 2272. models. The comparison of the estimated force shows that viscoelastic model performance is almost identical to that of [12] HELENE M. LANGEVIN, ELISA E KONOFAGOU, GARY J BADGER, DAVID L. CHURCHILL, JAMES R. FOX, JONATHAN the Rayleigh damping model. OPHIR and BRIAN S. GARRA, "TISSUE DISPLACEMENTS USING ULTRASOUND DURING ACUPUNCTURE ACKNOWLEDGMENT ELASTOGRAPHY TECHNIQUES", Ultrasound in Med. & Biol, Vol 30, No. 9, pp. 1173-1183, 2004. The authors wish to thank Prof. Kay Soon Low for providing [13] Reza Zahiri-Azar, and Septimiu E. Salcudean, "Motion Estimation in the required resources in Aerospace Electronics Laboratory, Ultrasound Images Using Time Domain Cross Correlation With Prior School of Electrical and Electronic Engineering, Nanyang Estimates", IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 53, NO. 10, OCTOBER 2006. Technological University, Singapore. The authors also would like to thank Mr. Human M. Esmaeili for his generous help in [14] Minyou Wang, Haibo Hu, Binjie Qin, "Non-Rigid Medical Image Registration Using Optical Flow and Locally-Refined Multilevel Free designing and rendering the 3D images of the experimental set Form Deformation", IEEE Nuclear Science Symposium Conference up. Record, 2007. [15] Robert Mart', Alison Noble, "ELASTIC MODULUS IMAGING USING REFERENCES OPTICAL FLOW AND IMAGE REGISTRATION", Proceedings of [1] Wei, Z , Wan, G , Gardi, L, Mills, G., and Downey, D, 'A Fenster, IEEE 17th International Conference on Image Processing, Hong Kong, Robot-assisted 3D-TRUS guided prostate brachytherapy system September, 2010. integration and validation'. Med Phys 2004; 31(3), pp 539-48. [2] Danese, D., Sciacchitano, S , Farsetti, A, Andreoli, M., and Pontecorvi, [16] Petr Jord'an, Image-Based Mechanical Characterization of Soft Tissue using Three Dimensional Ultrasound, Harvard University, PhD thesis, A, 'Diagnostic accuracy of conventional versus sonography-guided fine­ 2008. needle aspiration biopsy of thyroid nodules', Thyroid. 1998 Jan; 8(1), pp [17] W S. Slaughter. The Linearized Theory of Elasticity. Birkhauser, 2002. 15-21. [3] Orcun Goksel, Kirill Sapchuk, and Septimiu E. Salcudean, "Haptic [18] o.C Zienkiewicz, RL Taylor, J,Z, Zhu, The Finite Element Method: Its Basis and Fundamentals, Sixth edition, Elsevier Butterworth-Heinemann, Simulator tor Prostate Brachytherapy with Simulated Needle and Probe 2005. Interaction", IEEE TRANSACTIONS ON HAPTICS, VOL. 4, NO. 3, [19] M. D. J McGarry, Rayleigh Damped Magnetic Resonance Elastography, 2011. University of Canterbury, Msc thesis, 2008. [4] A Maghsoudi, M. Jahed, "Inverse Dynamics Control of Needle III Prostate Brachytherapy", Industrial Technology (ICIT), IEEE [20] D. C Hammerand, "GEOMETRICALLY-LINEAR AND NONLINEAR ANALYSIS OF LINEAR VISCOELASTIC COMPOSITES USING International Conference on, Athens, Greece. March 2012, pp. 510-515. THE FINITE ELEMENT METHOD ", VIRGINIA POLYTECHNIC [5] A Maghsoudi, M. Jahed, "A Comparison of PD and Model-based INSTITUTE AND STATE UNIVERSITY, 1999. Control in Prostate Brachytherapy", Industrial Electronics (ISlE), IEEE [21] Hani Eskandari, Septimiu E Salcudean, Robert Rohling and Jacques International Symposium on, China, May 2012, pp. 780-785. Ohayon, "Viscoelastic characterization of soft tissue trom dynamic finite [6] Alterovitz, R., Goldberg, K., Pouliot, .I., Taschereau, R., and Hsu, 1. element models", Phys. Med. Biol 53 (2008) 6569-6590. C,'Sensorless planning tor medical needle insertion procedures' in Froc. [22] O.C Zienkiewicz, R.L. Taylor, .I.Z. Zhu, "Solid mechanics", fifth IEEEIRSJ Int. Conf Intell. Robots Syst., 2003, vol 3, pp. 3337-3343. edition, Elsevier Butterworth-Heinemann, 2000. [7] DiMaio, S. P., and Sa1cudean, S. E, 'Needle insertion modeling and simulation', IEEE Trans. Robot. Autom Spec. Issue Med. Robot, 2003, [23] N. Papenberg, A Bruhn, T. Brox, S Didas, and J Weickert. Highly accurate optic ±low computation with theoretically justified warping. 19 (5), pp. 864--875. International Journal of Computer Vision, 67(2):141-158, 2006. [8] Alterovitz, R, Pouliot, J, Taschereau, R, Hsu, 1. C, and Goldberg, K , 'Simulating needle insertion and radioactive seed implantation for [24] B. K. P. Hom and B. G. Schunck. Determining Optical Flow. Artificial Intel- ligence, 16(1-3):186-203, 1981. prostate brachytherapy', in Medicine Meets Virtual Reality, Eds. lOS [25] .I. L. Barron, D. .I. Fleet, and S. S. Beauchemin. Performance of optical Press, 2003, pp. 19-25. flow techniques. International Journal of Computer Vision, 12(1 ):43-77, [9] Goksel, 0., Salcudean, S. E., and DiMaio, S. P., '3D simulation of 1994. needle-tissue interaction with application to prostate brachytherapy', Comput. Aided Surg., 2006, vol II, no. 6, pp. 279-288.

1560