work with in my master's; you were very supportive and discerning. Carolyn ...... must communicate this view to the radiation technologist (rad tech) operating the ...
UNIVERSITY OF CALGARY
Photogrammetric Advances to C-arm Use in Surgery
by
Hooman Esfandiari
A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENECE
GRADUATE PROGRAM IN GEOMATICS ENGINEERING
CALGARY, ALBERTA
DECEMBER, 2014
© Hooman Esfandiari 2014
Abstract C-arms are commonly used during orthopaedic surgeries to provide real-time static or dynamic fluoroscopic imaging. These devices are mostly utilized for qualitative assessment during operations; several advancements, such as C-arm tracking, must be accomplished to make them capable of providing quantitative measurements. This thesis presents in two major contributions to C-arm quantification: (1) development and testing of a monocular visual odometry method to track the C-arm base and (2) development and testing of a particular application, estimating the pose of an intramedullary (IM) nail for fracture surgery. The proposed base-tracking system can either be integrated with a C-arm joint tracking module or employed on its own, e.g. for steering. An IM-nail pose estimation method is proposed in this research that is capable of reporting the position and orientation of an inserted IM-nail. The offered IM-nail pose estimation method can help reduce both radiation exposure and time during surgery.
ii
Acknowledgements I would like to express my deepest gratitude to my supervisors Dr. Carolyn Anglin and Dr. Derek Lichti. This work would not have been completed without their valuable guidance and consideration. When I think of all the knowledge I have gained during the time I was working with you, it becomes obvious to me that you have been the best professors I could have ever asked to work with in my master’s; you were very supportive and discerning. Carolyn, I want to thank you for all your help and excellent support in my academic and personal life. I have learned a lot from you and you always made me enjoy working in such a friendly environment. Derek, your way of teaching and problem solving was so exciting to me that I undoubtedly think of you as one of the strongest motivations for me to stay in academia in the future. I am sure this is the right decision for me, so thanks Derek. I also want to acknowledge Dr. Shahram Amiri for his help with the IM-nail study. Your time and effort for knowledge transfer as well as conducting the experiments are much appreciated. I am grateful to John Person and Dr. Josh Rosvold at Tangent Design Engineering for letting me experience the industry environment and for their brilliant feedback on the C-arm basetracking study. A lot of thanks go to my colleagues especially Ting, Hervé, Jeremy and Mohsen for all the constructive discussions we had and for letting me gain benefit from their precious experiences. I could not have succeeded throughout my life without the love and support from my family. I want to thank my Mom and Dad for always believing in me and for teaching me all the life skills that I needed along the way. I will do my best to be a Son who you are always proud of. Thank you Nashmil for being such an excellent sister who has always encouraged me to move forward.
iii
Sarah, words are helplessly unable to express my love and appreciation to you. You were always there to cheer me up and motivate me through the hard times. I hope I can give you back this kindness in the future. I acknowledge all the founding resources that made this research happen: Alberta Innovates – Technology futures, National Science and Engineering Research Council of Canada (NSERC) and Tangent Design Engineering.
iv
List of Publications Conference proceedings and presentations Esfandiari, H., Amiri, S., Lichti, D., Anglin, C. (2014). Fast, Accurate Procedure for Identifying the Current Pose of an Intramedullary Nail. The International Society of Photogrammetry and Remote Sensing (ISPRS) Technical Commission V Symposium, Riva del Garda, Italy, June 2014, http://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XL-5/217/2014/isprsarchivesXL-5-217-2014.pdf
Esfandiari, H., Amiri, S., Lichti, D., Anglin, C. (2014). Fast, Accurate Procedure for Identifying the Current Pose of an Intramedullary Nail. 14th Annual meeting of the International Society for Computer Assisted Orthopaedic Surgery (CAOS), Milan, Italy, June 2014.
Esfandiari, H., Amiri, S., Lichti, D., Anglin, C. (2014). Fast Procedure for Identifying the Current and Desired C-arm Views of an Intramedullary Nail. 60th Annual Meeting of the ORS (Orthopaedic Research Society), New Orleans, Louisiana, USA, March 15-18, 2014, http://www.ors.org/Transactions/60/1130.pdf
Esfandiari, H., Amiri, S., Lichti, D., and Anglin, C. (2013). Fast Procedure for Pose Estimation of an Intramedullary Nail. 14th Annual Meeting of the Alberta Biomedical Engineering Conference, Banff, Canada, October 25-27, 2013.
v
Manuscripts to be submitted Esfandiari, H., Lichti, D., Anglin, C. Single-Camera Visual Odometry to Track a Surgical X-ray C-arm Base: Technique and Validation. [See Chapter Two] Proposed Journal: Journal of Computer Assisted Radiology and Surgery
Esfandiari, H., Amiri, S., Lichti, D., Anglin, C. A Fast, Accurate and Closed-Form Method for Pose Recognition of an Intramedullary Nail using a Tracked C-Arm. [See Chapter Three] Proposed Journal: Journal of Computer Assisted Radiology and Surgery
vi
Dedication
To my great Mom, Parvin, who is my immortal source of inspiration and to my Dad, Hormoz, who is my strongest hero of all time.
vii
Table of Contents Abstract ............................................................................................................................... ii Acknowledgements ............................................................................................................ iii List of Publications ..............................................................................................................v Dedication ......................................................................................................................... vii Table of Contents ............................................................................................................. viii List of Tables ................................................................................................................... xiii List of Symbols, Abbreviations and Nomenclature ........................................................ xvii INTRODUCTION ..................................................................................1 1.1 Motivation and Background ......................................................................................1 1.2 Smart-C Project and Thesis Objectives .....................................................................6 1.3 C-arm Tracking ..........................................................................................................7 1.3.1 Image Distortion ................................................................................................7 1.3.2 Pose Estimation: Concepts and Challenges .......................................................8 1.3.3 TC-arm ............................................................................................................10 1.3.4 Pose Estimation of Base: Need for Further Work ...........................................11 1.4 Base Tracking Technologies ....................................................................................12 1.4.1 Optical Flow Sensors .......................................................................................13 1.4.2 Visual Odometry .............................................................................................15 1.4.3 C-Pilot Power Assist Base ...............................................................................16 1.5 Intramedullary Nail Fixation ...................................................................................16 1.6 Affiliation and contribution of the manuscripts .......................................................19 1.7 Thesis Outline ..........................................................................................................20 viii
SINGLE-CAMERA VISUAL ODOMETRY TO TRACK A SURGICAL X-RAY C-ARM BASE.............................................................................................21 2.1 Introduction ..............................................................................................................21 2.2 Methods ...................................................................................................................26 2.2.1 Base-Tracking Algorithm ................................................................................26 2.2.1.1 Concepts.................................................................................................26 2.2.1.2 Undistortion ...........................................................................................28 2.2.1.3 Frame-to-Frame Pose Estimation ..........................................................28 2.2.1.4 Perspective Rectification (Frame-to-Floor) ...........................................31 2.2.1.5 Reference (Feature) Extraction ..............................................................32 2.2.1.6 Pose Recovery........................................................................................33 2.2.1.7 Optical Flow ..........................................................................................37 2.2.1.8 Odometry Loop Closure (Absolute Tracking) .......................................39 2.2.2 Error Propagation Analysis and Simulation ....................................................40 2.2.2.1 Error Propagation Analysis ....................................................................40 2.2.2.2 Error Propagation Simulation ................................................................43 2.2.3 Experimental Evaluation .................................................................................44 2.2.3.1 Camera & Computer ..............................................................................44 2.2.3.2 Experimental Setup ................................................................................44 2.2.3.3 Camera Heights (Ranges) ......................................................................46 2.2.3.4 Camera Speed ........................................................................................47 2.2.3.5 Floor Textures ........................................................................................47 2.2.3.6 Undistortion ...........................................................................................47
ix
2.2.3.7 Perspective Rectification .......................................................................49 2.2.3.8 Optical Flow ..........................................................................................49 2.2.3.9 Error Determination ...............................................................................50 2.2.3.10 Odometry Loop Closure (Absolute Tracking) .....................................50 2.3 Results ......................................................................................................................50 2.3.1 Undistortion evaluation ...................................................................................50 2.3.2 Error Propagation Simulation ..........................................................................52 2.3.3 Camera Height (Range) ...................................................................................54 2.3.4 Repeatability Analysis .....................................................................................56 2.3.5 Camera Velocity ..............................................................................................57 2.3.6 Floor Texture (Standardized + Real OR) ........................................................58 2.3.7 Loop Closure & Absolute Tracking ................................................................59 2.4 Discussion ................................................................................................................59 A FAST, ACCURATE AND CLOSED-FORM METHOD FOR POSE RECOGNITION OF AN INTRAMEDULLARY NAIL USING A TRACKED C-ARM ...................................................................................................................................64 3.1 Introduction ..............................................................................................................64 3.2 Methods ...................................................................................................................67 3.2.1 Coordinate System Definition .........................................................................68 3.2.2 Tracking System ..............................................................................................69 3.2.3 Biplane X-ray Acquisition ...............................................................................70 3.2.4 Feature Extraction ...........................................................................................71 3.2.5 Pose Recognition .............................................................................................72 x
3.2.6 Acceptable Error Tolerance .............................................................................77 3.2.7 Experimental Methods.....................................................................................79 3.3 Results ......................................................................................................................84 3.4 Discussion ................................................................................................................88 DISCUSSION AND CONCLUSIONS .............................................91 4.1 Summary and Overview ..........................................................................................91 4.2 Novelty and Contributions .......................................................................................95 4.3 Limitations ...............................................................................................................97 4.4 Future Directions .....................................................................................................98 4.5 Conclusions ..............................................................................................................99 APPENDIX A: OPTICAL FLOW BASED ON STANDARD COMPUTER MOUSE SENSORS AS A BASE TRACKING SOLUTION FOR A SURGICAL C-ARM: PRACTICAL APPROACH AND VALIDATION ................................................101 Introduction ...........................................................................................................101 Error Sources of Optical Flow Sensors ........................................................103 Texture ................................................................................................103 Sensor Distance to the Floor ...............................................................104 Methods.................................................................................................................104 Texture ..........................................................................................................105 Sensor Distance to the Floor .........................................................................105 Odometry Model...........................................................................................105 Setup Calibration ..........................................................................................110 Optimal Configuration ..................................................................................111 xi
Experimental Methods ..................................................................................112 Results ...................................................................................................................113 Discussion .............................................................................................................120 REFERENCES ................................................................................................................123
xii
List of Tables Table 1-1: Estimated radiation exposure to the patient ................................................................. 3 Table 1-2: Fluoroscopy time and dose for different orthopaedic procedures ................................ 4 Table 1-3: Comparison between internal and external tracking .................................................. 13 Table 2-1: Acceptable translational base tracking accuracies ...................................................... 23 Table 2-2: Technical characteristics of the imaging sensor and lens assembly ............................ 44 Table 2-3: Recovered distortion coefficients ................................................................................ 48 Table 2-4: Tracking results for different velocities ...................................................................... 57 Table 3-1: Coordinate system definitions .................................................................................... 69 Table 3-2: Pose recovery results .................................................................................................. 86 Table 3-3: Pose recovery results for the second nail (B) geometry .............................................. 87 Table 3-4: Repeatability analysis results ...................................................................................... 87 Table A-1: Sensor specifications ................................................................................................ 112 Table A-2: Repeatability (mean +/- standard deviation) of recovered baseline length ............. 114 Table A-3: Optimum configuration of 3 sensors ....................................................................... 115 Table A-4: Recovered calibration parameters ........................................................................... 117 Table A-5: Accuracy assessment results..................................................................................... 119 xiii
List of Figures and Illustrations Figure 1-1: Arcadic Orbic Iso-C C-arm .......................................................................................... 1 Figure 1-2: C-arm X-ray generation mechanism ............................................................................ 8 Figure 1-3: C-arm degrees of freedom; dashed lines = 2D base movements ............................... 11 Figure 1-4: Schematic view of a standard optical mouse ............................................................. 14 Figure 1-5: Distal part of two different IM-nail models ............................................................... 17 Figure 1-6: IM-nail fixation procedure ......................................................................................... 18 Figure 1-7: Overall work flow of the research ............................................................................. 19 Figure 2-1: Feature extraction for perspective rectification ........................................................ 33 Figure 2-2: Schematic workflow of the base-tracking system...................................................... 34 Figure 2-3: Flowchart of the base-tracking algorithm .................................................................. 36 Figure 2-4 A: Rotating platform ................................................................................................... 45 Figure 2-5: Sliding platform - left: H1 (70 cm), right: H2 (16.5 cm) ........................................... 46 Figure 2-6: Tested floor textures................................................................................................... 47 Figure 2-7: Undistortion procedure ............................................................................................. 48 Figure 2-8: Undistortion results .................................................................................................... 51 Figure 2-9: Error propagation simulation ..................................................................................... 53 xiv
Figure 2-10: Errors for ~6 m translation tests for H1 ................................................................... 55 Figure 2-11: Errors for ~3.6 meter translation tests for H2 .......................................................... 55 Figure 2-12: Accuracy of orientation recovery............................................................................. 56 Figure 2-13: Repeatability results, error bars=2% of the truth ..................................................... 57 Figure 2-14: Translational tracking errors for different floor textures ......................................... 58 Figure 2-15: Feature detection on real OR floor textures ............................................................. 59 Figure 3-1: Distal part of the IM-nail and the local object coordinate system ............................. 66 Figure 3-2: Overall work flow. Red: Inputs; Blue: Computations; Green: Outputs .................... 68 Figure 3-3: Configuration of the different coordinate systems ..................................................... 69 Figure 3-4: Calibration parameters ............................................................................................... 70 Figure 3-5: A) low-pass image, B) edge image, C) Hough image, D) detected circles .............. 72 Figure 3-6: Distal-hole centre tracking ......................................................................................... 75 Figure 3-7: Intersection geometry................................................................................................. 75 Figure 3-8: Schematic cross-section view of an IM-nail and drill bit . ........................................ 78 Figure 3-9: Acceptable error tolerance ......................................................................................... 79 Figure 3-10: Experimental setup ................................................................................................... 80 Figure 3-11: Simulation of soft tissue artifacts ............................................................................. 81 xv
Figure 3-12: User interface ........................................................................................................... 82 Figure 3-13: Bone and soft tissue artifact scenario ....................................................................... 84 Figure 3-14: Surgical instrumentation scenario ............................................................................ 85 Figure A-1: Odometry Principle ................................................................................................. 106 Figure A-2: Relative configuration of the main coordinate frames ............................................ 107 Figure A-3: Calibration Setup ..................................................................................................... 113 Figure A-4: The effect of ri on planimetric and angular precision ............................................. 115 Figure A-5: The effect of θi on planimetric and angular precision ............................................ 116 Figure A-6: The effect of φi on planimetric and angular precision . ......................................... 116 Figure A-7: Experimental setup .................................................................................................. 117
xvi
List of Symbols, Abbreviations and Nomenclature
Abbreviations
Definition
2D
Two-dimensional
3D
Three-dimensional
ACL
Anterior cruciate ligament
CAD
Computer-aided design
CCD
Charged coupled device
CIHR
Canadian Institutes of Health Research
CMM
Coordinate measuring machine
CMOS
Complementary metal oxide semiconductor
CT
Computed tomography
DLT
Direct linear transform
DOF
Degree(s) of freedom
DRR
Digitally reconstructed radiograph
EOP
Exterior orientation parameter
ESD
Entrance surface dose
FPS
Frames per second
GigE
Gigabit Ethernet
GNSS
Global navigation satellite system
IDE
Integrated development environment
IM-nail
Intramedullary nail
IMU
Inertial measurement unit
xvii
IOP
Interior orientation parameter
Iso-C
Iso-centric
KLT
Lucas Kanade
LED
Light emitting diode
MRI
Magnetic resonance imaging
NSERC
National Sciences and Engineering Research Council of Canada
OR
Operating room
ORIF
Open reduction and internal fixation
Rad tech
Radiation technologist
SD
Standard deviation
SDK
Software development kit
SFM
Structure from motion
SLAM
Simultaneous localization and mapping
VO
Visual odometry
xviii
Introduction
1.1 Motivation and Background A C-arm is a mobile X-ray device that is frequently used during orthopaedic surgeries. It consists of a semi-circular, arc-shaped arm that holds an X-ray transmitter at one end and an Xray detector at the other (Figure 1-1). The use of C-arm devices for X-ray imaging in orthopaedic surgery is widespread to visualize bone and implant components. The device aids with visual delineation of the desired elements in the patient’s body and helps the surgeon perform the surgery more precisely. Surgeries in which C-arm imaging is common include fracture fixation surgery (e.g. femur, tibia, and humerus), pedicle screw placement in spine surgery, and sacroiliac screw placement in pelvis surgery.
Control Unit
Detector
Monitoring Unit
Base
Source
Figure 1-1: Arcadic Orbic Iso-C C-arm (Siemens AG, Munich, Germany) at the University of British Columbia. The device was used for the IM-nail experiments in this thesis.
1
C-arms have several advantages over conventional X-ray imaging systems that make them very popular in different orthopaedic routines. The C-arm can be moved and rotated around the surgical table, enabling the surgeon to acquire different X-ray views from different viewing angles. C-arms utilize less radiation dose during fluoroscopic imaging in comparison to other X-ray devices, thus this device can be categorized as a low-dose X-ray modality, which is a critical fact with respect to patient and clinical staff health. Real-time fluoroscopy is one of the main reasons C-arms are used in orthopaedic procedures; constant (video) mode enables the surgeon to get ongoing feedback whereas the single buffered or pulsed (X-ray image) mode is used for diagnosis and verification. Earlier systems that caused the detector to fluoresce, thereby creating the image and giving them their name, have now largely been replaced by digital systems. C-arms are also used for non-orthopaedic procedures, such as when placing various types of catheters. Our main focus, however, is for orthopaedic procedures. Whereas C-arms have commonly been used as a qualitative assessment tool, their quantitative performance is limited due to several different factors. One of the main challenges of C-arm usage in orthopaedic surgery is to reach the desired view, i.e. the X-ray shot in which a specific component or bone appears in a predefined manner. The challenge is partly because it is not clear how to reorient the C-arm to achieve the desired view and partly because the surgeon must communicate this view to the radiation technologist (rad tech) operating the equipment, resulting in a trial-and-error process. The experience of the rad tech and surgeon can therefore affect both the result and duration of the surgery. The more images taken to achieve the desired view, the higher the radiation dose to the patient and surgical staff. Over the long term, this can lead to cancer or cataracts in the surgical staff (Harstall et al. 2005). The amount of risk related to fluoroscopy exposure depends on intraoperative radiation time, cumulative career exposure and 2
the protective measures used. Surgeon dose level varies 10 to 12-fold (from lowest to highest dose) depending on the orthopaedic procedure (Moore et al. 2011). It is important to note that even a slight reduction in radiation dose would be beneficial since small amounts of radiation can cause cumulative tissue damage. Hence it has been recommended to minimize the radiation exposure whenever possible (Mechlenburg et al. 2009). According to a study of different surgical procedures (Singer 2005), considerable radiation is being emitted in C-arm related fluoroscopic imaging (Table 1-1).
Table 1-1: Estimated radiation exposure to the patient during different radiographic imaging routines (Singer 2005). In the last two, fluoroscopic imaging, the surgical staff also receive a portion of the radiation, through X-ray scattering. Procedure
Radiation (mGy)
Chest radiograph
0.25
Dental survey
1.5 (per view) * 3 = 4.5
Hip radiograph
5 to 6
Computed tomography (wrist)
7
Computed tomography (hip)
10
Fluoroscopic imaging (regular C-arm)
12 to 40 per min*
Fluoroscopic imaging (mini C-arm)
1.2 to 4 per min*
*
See Table 1-2 for typical times.
Duration of radiation exposure (fluoroscopy time) as well as radiographic dose have been investigated in a separate study (Tsalafoutas et al. 2007; Table 1-2)
3
Table 1-2: Fluoroscopy time and dose for different orthopaedic procedures (Tsalafoutas et al. 2007). Fluoroscopy Time (min)
𝑘𝑉𝑝
𝑚𝐴
Mean±SD
Mean±SD
Mean±SD
Range
Mean±SD
1.1-10.2
3.2±1.7
90±16
2.1±0.3
15-772
183±138
16
0.2-4.6
1.5±1.2
58±7
1.3±0.4
0.9-117
21±27
15
3.0-11.6
6.3±2.7
88±17
2.1±0.3
70-807
331±21
13
0.3-2.0
0.9±0.7
64±3
1.5±0.4
4.0-63
19±20
13
0.3-2.8
1.2±1.0
67±7
1.7±0.4
4.1-120
35±36
11
0.2-2.0
0.8±0.6
103±15
2.2±0.2
18-118
46±32
8
2.0-12.2
5.7±3.5
67±8
1.7±0.3
41-378
137±111
7
0.3-3.0
1.8±0.9
50±3
1.4±0.5
3.3-26
17±162
4
0.2-7.7
4.2±3.0
76±12
2.0±0.3
5.2-407
173±162
Vertebroplasty
4
3.3-7.0
5.1±1.3
100±8
2.3±0.1
261-378
323±51
All procedures
204
0.2-12.2
3.0±2.3
83±20
2.0±0.4
0.9-807
149±152
Surgical procedure Intramedullary nailing of peritrochanteric fractures ORIF2 of malleolar fracture Intramedullary nailing of diaphyseal femoral fracture Arthroscopy for ACL3 reconstruction with artificial ligament Tibial plateau (plate and medullary screws) Bilateral pedicle screw placement in the lumbar spine Tibial intramedullary nailing Fracture of the distal radius fixed with a plate Bilateral pedicle screw placement in the cervical spine
Number of patients
Range
113
ESD1 (𝑚𝐺𝑦)
The other important issue is the time of surgery; as it becomes longer, the risk of infection and the cost of the operation increase (Cizik et al. 2012).
1
Entrance surface dose: the energy per unit mass absorbed by the skin of the patient Open reduction and internal fixation 3 Anterior cruciate ligament 2
4
Depending on the application, it is important to be able to reconstruct the pose (position and orientation) of a specific component (patient’s bone or implant) in real time. This is often referred to as fluoroscopy-based navigation. Fluoroscopy-based navigation is the procedure of calculating the pose of a specific surgical component intraoperatively, based on the images captured from an X-ray acquisition device. The main challenge in fluoroscopy-based navigation is to recover the 3-dimensional (3D) information of objects of interest based on the acquired 2D X-rays. This makes it difficult to manipulate the full 3D information and illustrates the necessity of taking several X-rays from different viewing angles (Hofstetter et al. 1999). Computed tomography (CT) data can also provide the required spatial information for the surgical components of interest, however the intraoperative performance of such systems is limited due to setup configuration and time consumption, and adds radiation dose to the patient in the acquisition of the CT. Although C-arm fluoroscopy has been visually (i.e. qualitatively) used to place guide wires and screws in orthopaedic trauma procedures (e.g. distal locking of intramedullary nails in IM-nail fixation surgeries or pedicle screw placement in spinal fusion surgeries) the main challenge to employing this device for providing navigation solutions (on its own and without external tracking systems) is that it can only provide imaging in one 2D plane at a time. The surgeon needs to adjust the position of the implant appearing in the first shot and then acquire additional shots for further adjustments. Consequently, considerable operating time is occupied while the rad tech repositions the C-arm to obtain the views in multiple planes (Kahler 2004). In this thesis, two aspects were developed to aid the quantitative use of C-arm devices for fluoroscopy-based navigation: 1) tracking the C-arm base for precise positioning, and 2) creating an improved method for identifying the current and desired pose of a particular orthopaedic implant, an intramedullary (IM) nail for fracture fixation, as described below. These projects 5
contribute to a larger collaborative Smart-C project, in conjunction with researchers at the University of British Columbia as well as three industry partners, in particular Tangent Design Engineering of Calgary.
1.2 Smart-C Project and Thesis Objectives The main goal of the NSERC-CIHR-funded Smart-C project is to create a retrofittable C-arm system capable of quantitative estimation in different orthopaedic procedures. This project has been divided into three main categories as follows: Sim-C: A training tool for radiation technologists consisting of a C-arm shell with tracking technology paired with a tracked patient dummy and a computed-tomography (CT) morphing algorithm, with fast digitally reconstructed radiographs (DRRs), to generate synthetic fluoroscopy images; Accu-C (including IM-nail pose-recovery): Builds on Sim-C for operative use by: accurately locating 3D anatomical and tool locations; registering statistical shape models to individual patients to produce live predictive images; and stitching together separate shots to create accurate panoramic models and measurements; and Navi-C (including base-tracking plus IM-nail view guidance): Adds software to predict optimal C-arm positioning from generic and patient-specific scout images resulting in navigation displays to guide the rad tech to correctly position the C-arm, plus a powerassist base and base-tracking to improve the efficiency of acquiring the desired view. The fundamental precursor for both the base-tracking and IM-nail projects is to track the degrees of freedom of the C-arm. Therefore this aspect is described in detail in the following section. 6
1.3 C-arm Tracking It is vital to track the pose (position and orientation) of the C-arm’s transmitter-detector set in order to be able to relocate it to any desired position and compute the location of the object of interest. C-arm pose estimation can be used in different applications ranging from computerassisted surgery and 2D-3D registration to panoramic X-ray imagery (Amiri et al., 2013). C-arms are commonly used due to their relatively low cost and ease of use, and the surgical need to see inside the body; however they currently lack accurate and easy quantitative guidance. The following issues must be solved to utilize a C-arm for quantitative 3D measures during orthopaedic surgeries: 1) calibration of the image distortions and intrinsic image parameters (principal distance, principal point, vertical and horizontal pixel scales) and 2) pose estimation (Chintalapani et al. 2008). Brief descriptions about the first two issues are provided below followed by a comprehensive description of the last issue.
1.3.1 Image Distortion As seen from the schematic view of an X-ray detector mechanism (Figure 1-2), an optical lens assembly projects the acquired X-ray onto a charged-coupled device (CCD) sensor in order to provide the digital output. The lens assembly is subject to different aberration effects (radial, decentring etc.). Different distortion rectification techniques have therefore been introduced to address the lens aberration problem of the optical assembly in C-arm X-ray detectors.
7
Figure 1-2: C-arm X-ray generation mechanism.
Some studies utilize a calibration phantom including small metallic beads in a specific configuration (Fahrig et al. 1997) while more recent studies make use of a priori patient-specific CT scans as the reference for calibration (Chintalapani et al. 2007). Fifth-order polynomials and Bernstein polynomials were used, respectively, to model the distortion behaviour in these studies. These techniques are capable of recovering the internal parameters of the detector set (principal distance, principal point, vertical and horizontal pixel scales) as they were at the time of exposure to be able to use the acquired X-rays for quantitative purposes. Some methods solve for the calibration (estimating distortion and intrinsic parameters) and pose problem simultaneously since it is known that the intrinsic parameters depend on the actual pose of the device (Siewerdsen et al. 2007).
1.3.2 Pose Estimation: Concepts and Challenges Different C-arm pose recovery techniques exist, each trying to provide more accurate, easy to use and inexpensive solutions. Pose recovery algorithms can be divided into two main categories (Livyatan et al. 2002): 1) on-line methods; and 2) off-line methods. 8
On-line methods generally rely on the visibility of fiducial markers in the X-rays of interest. The calibration (pose) parameters are recovered at each C-arm pose and for each X-ray shot. This method requires a physical reference to be present in each X-ray for the purpose of calibration. Extraction and matching of the projected fiducial markers are challenging since it is hard to separate the anatomical and fiducial features. Additional limitations of on-line methods are the imaging volume occupied by the designed phantom and the number of steps added to the current imaging procedures. Off-line methods make use of several a priori X-rays being acquired from a custom calibration phantom at pre-defined intervals. The real-time corresponding pose parameters for each X-ray are then extracted based on the previously established off-line calibration. In other words, the C-arm tracking is performed after accomplishing a set of calibration procedures. In off-line calibration approaches, the calibration parameters of the C-arm (intrinsic and extrinsic parameters) are computed at the preoperative stage. This means that the calibration parameters are recovered, for a set of different poses, before the surgery. As in any other off-line calibration procedure, it is important to maintain a sufficiently-small interval distance to satisfy the required accuracies, while still covering the whole imaging range. Both of these pose estimation modalities require accurate information about C-arm pose (or the calibration phantom depending on the method) at each imaging location. Different methods exist in the literature, utilizing different tracking options as a means to achieve C-arm localization. External tracking devices have been employed in different experimental and clinical situations as the sensory units for pose estimation of C-arm devices: optical tracking systems have been developed and proposed as a tracking solution for the C-arms in some studies (Reaungamornrat et al. 2012; Amiri et al. 2011) but they are mostly limited by the requirement of line of sight; magnetic 9
tracking methods do not satisfy the sub-millimetre accuracies required in many applications; and finally, another set of studies try to solve for the pose estimation problem by using precise calibration phantoms including shapes with defined geometric properties ranging from beads and lines to conics (Hofstetter et al. 1999; Jain et al. 2005; Yaniv et al. 2005). The main challenges for these methods are the need for the fiducial markers to appear in the X-rays, the need for accurate fabrication of the calibration phantoms, segmentation problems and visibility of the markers in the presence of image artifacts. A camera-augmented C-arm (Navad et al. 2010) is another C-arm pose recovery method. This method fuses the commonly used C-arm device with a standard video camera in such a way that the perspective centre of the camera is pointed at the X-ray source. A calibration procedure is then undertaken to register the acquired X-rays with the video frames. Line-of-sight is the main limitation of this method.
1.3.3 TC-arm An inertial measurement system for C-arm tracking (TC-arm) has been developed by our research group (Amiri et al. 2013) that utilizes two inertial measurement units (IMUs) attached to the gantry of the C-arm to measure its orbit, tilt and wig-wag, as well as two laser beam sensors to measure up-down and in-out movements of the gantry (Figure 1-3). For any configuration of the C-arm, the pose of the transmitter-detector set is calculated by performing an interpolation procedure in off-line mode. The comprehensive calibration protocol and phantoms provide full three-dimensional spatial information of the camera and image intensifier for any arbitrary image acquired by the TC-arm system. This low-cost system is capable of determining an accurate estimate of the extrinsic and intrinsic parameters of the C-arm for any acquired fluoroscopic image
10
and can track the centre point of the gantry with better than 1.5±1.2 mm accuracy. A detailed description of this method is provided in Chapter Three.
Figure 1-3: C-arm degrees of freedom; dashed lines = 2D base movements.
1.3.4 Pose Estimation of Base: Need for Further Work An important limitation to many of the C-arm pose estimation methods (including the TCarm system) is the assumption that the C-arm base is kept stationary during the calibration and movement. This may not be acceptable in many applications (e. g. panoramic X-ray registration) where there is a necessity to have longer-range movements that can only take place by moving the base. In order to address this issue, the performance of a dead-reckoning system has been investigated that can act as a supplementary module to provide comprehensive localization outcomes for the C-arm base. These data will augment the TC-arm five degree-of-freedom tracking to provide full range, 3D pose of the whole unit. The desired output of this system is the 11
𝐺 𝐺 𝐺 absolute position (𝑋𝐶−𝑎𝑟𝑚 , 𝑌𝐶−𝑎𝑟𝑚 ) and orientation (𝜃𝐶−𝑎𝑟𝑚 ) of the C-arm base in the operating
room relative to any arbitrary origin (global coordinate system). Additional specifications are provided in Chapter Two.
1.4 Base Tracking Technologies For the purpose of base tracking, the selection of appropriate sensors to measure each of the individual movement parameters associated with the C-arm base is critical since each tracking technology has its own limitations. There are two main categories of tracking modalities for any moving platform (robot): 1) Absolute tracking (reference based): In this method the current position of the robot is localized with respect to an absolute reference using signal generating beacons (GNSS: global navigation satellite system, ultrasonic and etc.) or vision-based feature extraction and analysis (Borenstein et al. 1997). 2) Internal sensors (dead-reckoning): The location of the robot is inferred based on an accumulation of movements from an initial position using inertial sensors or odometry (Ross et al. 2012). This method is usually known as dead-reckoning in robotic communities. External (reference based) tracking systems usually provide more accurate, absolute and insensitive results but they are mostly subject to communication problems with the sensory unit. For instance, optical-based tracking systems rely on the visibility of targets attached to the moving device by the cameras pointing towards them from a distance (line-of-sight). This is challenging in operating rooms due to the movements of surgical staff and clinical devices. Internal sensors,
12
however, are attached to the body of the moving device and do not suffer from these problems. Table 1-3 shows a comparison between external and internal tracking sensors. Table 1-3: Comparison between internal and external tracking.
External
Markers*
Line of sight*
Absolute measurement
✔
✔
✔
Relative measurement
Error accumulation
High Cost
High Frequency
✔ ✔
Internal
✔
✔
*
optical-based systems.
In this study, standard optical mouse sensors (optical flow) were first chosen for the purpose of C-arm base tracking (Appendix A); these were later replaced by camera sensors (optical odometry) for better accuracy (Chapter Two). Descriptions about both of the systems are provided in the following.
1.4.1 Optical Flow Sensors The first candidate method for the C-arm dead-reckoning system was wheel odometry. Typically, odometry relies on the measurement of the covered distance by using shaft-mounted encoders to calculate the 2D movement (as well as orientation) of the moving device relative to a starting coordinate system (Bonarini at al., 2004). This is done by installing a set of shaft encoders mounted to the robot’s wheel to measure speed and distance (Bell 2011). It is well known that wheel-based odometry is subject to 1) systematic errors regarding the factors such as unequal wheel diameters, incorrectly measured wheel diameters, and/or wheel distance (Borenstein et al. 1997) and 2) random (or not modeled systematic) errors caused by irregularities of the floor, bumps, cracks or wheel slippage (Bonarini et al. 2005). 13
In order to address these issues, optical flow sensors (e.g. a standard optical computer mouse sensor) were recently introduced. The main idea of developing optical flow sensors is to measure the movement of the platform without the need to be in direct contact with the surface underneath. This is a revolutionary advantage over previous mechanics-based encoders (e.g. conventional trackball computer mice). Figure 1-4 demonstrates different parts of an optical mouse in a schematic view. For the purpose of illumination, a light emitting diode (LED) is used to generate an adequate and suitable amount of light. In order to avoid blurry images, and by using mirrors, the light is reflected to the floor in a such a way that the lowest amount of multi-pathed light is guided to the sensor and the largest possible area is illuminated on the floor. Then all the light beams go through a lens/prism set-up to be concentrated on the charge-coupled device (CCD) or complementary-metal–oxide– semiconductor (CMOS) sensor, which is usually mounted on the processing chip itself (e.g. ADNSD 2051, AVAGO Technologies).
Figure 1-4: Schematic view of a standard optical mouse.
In Appendix A, the performance of optical flow odometry (more specifically, an optical mouse sensor) as a solution for C-arm base tracking system was investigated. The desired configuration 14
includes a number of mouse sensors attached to the body of the C-arm, looking downward to the operating room floor underneath.
1.4.2 Visual Odometry Visual (optical) odometry has been investigated in the computer vision and robotics communities for years and refers to the problem of determining the pose of the camera-carrying (here C-arm base) platform after processing the frames being acquired by the camera (Kneip et al. 2011). The recovered trajectory of the camera can represent the equivalent trajectory of the platform after a simple registration procedure. The term “odometry” was chosen to replicate the similarity of wheel odometry, which incrementally measures the movement of any platform by incorporating the number of turns over time. The same principle is applied in visual odometry where the pose of the platform is incrementally extracted based on examining the evolutions within the images acquired by the mounted camera. Due to the vision-based concept of the visual odometry (VO) methods, there must exist sufficient illumination and texture in the environment (field of view of the mounted camera), and successive frames must ensure reasonable overlap (Scaramuzza et al. 2011). Visual odometry techniques can be divided into two main categories: 1) monocular odometry and 2) stereo (binocular) odometry. In stereo (binocular) methods, the pose of the platform is derived after performing a triangulation procedure for the corresponding features appearing in consecutive frames. These methods can provide the full 6 degree-of-freedom (DOF) pose of the vehicle with a meaningful, real scale. Stereo methods are mostly associated with computational complexity and time. 15
Monocular VO makes use of single camera data and is preferred over the stereo techniques in applications in which on-line solutions are of interest (as in C-arm base tracking) or the distance to the scene is much larger than the stereo baseline. Because of the well-known low redundancy of 2D (image)-based 3D reconstruction, the monocular camera cannot ultimately provide the full 6 DOF pose information. Only bearing information can be extracted from monocular data (which is the exact case needed in C-arm base tracking) and the motion can only be recovered up to a scale factor. The absolute scale must be extracted from direct measurements, motion constraints (e.g. homography) or from integration with other sensors (Scaramuzza et al. 2011). Principles of monocular odometry are provided in Chapter Two.
1.4.3 C-Pilot Power Assist Base C-Pilot is a C-arm power-assist system, developed by Tangent Design Engineering, designed to help the operator in propelling the (typically hefty) C-arm. This is part of the overall Smart-C (Navi-C) project. The steering and actuating procedures of the current beta prototype are open-loop in concept and rely on occasional corrections by the operator. The operator has the ability to control the amount of movement. Base-tracking could act as a monitoring and control module for the C-Pilot system that can report the on-line position of the platform so that the deviations from the desired path can be solved in real time. This provided further motivation for developing the base-tracking system.
1.5 Intramedullary Nail Fixation One potential application of the fully tracked (calibrated) C-arm is to use it in intramedullary nail fixation surgeries to provide positional information about the implant. An 16
intramedullary nail (IM-nail, also known as IM-rod) is a metallic rod placed into the medullary cavity of a bone for the purpose of fracture fixation (Figure 1-5). These implants are mostly used to treat the fractures of the long bones (e.g. femur, tibia, humerus) by stabilizing the fractured components (Figure 1-6). IM-nails share the load with the bone itself, which helps the patient to use the fractured extremity more quickly. If the patient experiences long-term pain, there might be a need for a removal operation; otherwise the rod is left in. From the earliest records of IM-nail usage (in the 16th century) to the current implementations, there have been evolutions in design, materials and operating techniques, which has resulted in wide acceptance and a high level of success of this procedure (Bong et al. 2006).
Figure 1-5: Distal part of two different IM-nail models.
17
Figure 1-6: IM-nail fixation procedure with single distal and proximal screws for tibial fracture reduction; Illustration used by permission from Amicus Visual Solutions.
Distal locking refers to the action of fixing the IM-nail to the fractured bone by inserting several screws (depending on the design) into distal holes embedded in the nail (Figure 1-5). This is commonly done by relying on 2D radiographs to guide what is essentially a 3D procedure. Although there have been many improvements in IM-nail design and material, distal locking remains a difficult part of the procedure, leading to a prolonged operating time and radiation exposure (Grewal et al. 2012; Table 1-2). Current IM-nail distal locking techniques are described and compared in detail in Chapter Three. The strong motivation for this part of the study is to improve the current surgical routine of distal locking by introducing an autonomous IM-nail tracking system that can help to limit the 18
radiation usage and operation (and consequently anaesthetic) time. There is still is a high demand for an accurate, fast and clinically accessible IM-nail localization method that is capable of reporting the real-time pose (position and orientation) of the implant. A detailed description of this system is provided in Chapter Three.
1.6 Affiliation and contribution of the manuscripts Although each of the provided manuscripts of this thesis is written in stand-alone format and will be submitted as a journal paper in future, this thesis delivers logical transitions between individual aspects of the research (Figure 1-7).
Figure 1-7: Overall work flow of the research. Primary areas of focus for this thesis are highlighted in grey.
19
This first manuscript (Chapter Two) along with the appendix (Appendix A) of this thesis provide a frame work for C-arm base tracking. The proposed system meets the clinical application criteria and is capable of providing robust tracking outcomes. A particular field of application for a fully tracked C-arm is investigated in the second manuscript (Chapter Three). This research introduces a fast and accurate IM-nail pose estimation solution (base on the tracked C-arm).
1.7 Thesis Outline This thesis is written in manuscript-based format, consisting of an overall introduction (Chapter One), two chapters representing the core of the thesis (Chapter Two: base-tracking system, and Chapter Three: IM nail pose identification), and an overall discussion (Chapter Four), plus Appendix A describing a substantive study on optical flow sensors that ultimately led to the approach taken in Chapter Two. While the manuscript format necessitates some repetition, this format was chosen as the best approach due to the diversity of topics. Both thesis topics are directed at improving the use of C-arm devices during orthopaedic surgeries.
20
Single-camera visual odometry to track a surgical X-ray C-arm base
2.1 Introduction Different factors must be addressed to allow C-arms to provide quantitative (positional) information; the most important of these is to track the pose (position and orientation) of the device itself. The X-ray images provided by the C-arm can be used for image-based procedures, such as image-based navigation, panoramic X-ray generation, X-ray mosaics and X-ray image processing, if and only if the pose of the source-detector set (corresponding to the time of exposure) is accurately known. This illustrates the necessity of an accurate tracking system that can provide the real-time pose of the device for each X-ray shot. Although different off-line C-arm joint-tracking systems have been reported in the literature (Amiri et al. 2011, 2013; Reaungamornrat et al. 2012; Grzeda et al. 2011), optical-based systems are costly and require a large field of view if the C-arm is moved in and out of the surgical field, and non-optical (e.g. accelerometer-based) systems to date have all required a fixed (stationary) C-arm base, to our knowledge. Our group has developed both types of joint-tracking systems (Amiri et al. 2011, 2013), but prefers the accelerometer-based method due to its low cost, retrofittability, and no need for line-of-sight in the operating room. Since there are cases where images need to be acquired over a longer distance, as in panoramic registration of the images (Amiri et al. 2013), or over a larger working volume, our goal was to design a low-cost basetracking system capable of reporting the real-time two-dimensional (2D) position and orientation of the C-arm base. These data can be integrated with any joint tracking system (such as those developed by other researchers) to recover the full six degree-of-freedom (DOF) pose of the device for any X-ray of interest. This has two purposes: 1) to report the current base pose, and 2) to return 21
to a defined position. These can be valuable with or without C-arm joint tracking, especially if precise and repeatable positioning of the base is desired, as in the C-Pilot project (Section 1.4.3). The base-tracking system can also benefit optically-based joint-tracking systems by limiting the required field of view to the operating field, tracking the base as it is moved in and out of the field of view. Different solutions were considered (see Appendix A); the optimal solution for this tracking application was determined to be an internal (dead-reckoning) system that can be attached to the C-arm’s base and report the amount of movement made by the platform in real time (considering the clinical situations). The desired system must be accurate, robust, retrofittable, inexpensive and as dimensionally small as possible. Amongst all available dead-reckoning systems (e.g. encoder-based systems, optical flow sensors), visual odometry (VO) appears to satisfy these expectations to the greatest extent. The proposed structure of the C-arm-specific visual odometry system consists of a downward-looking camera, rigidly attached to the base platform of the device and a processing (software) module, which is capable of estimating the 2D real-time movement of the cameracarrying platform (C-arm base). The only required sensory unit for this odometry method is a single consumer-grade gray-scale camera. The base tracking is assumed to be integrated with the current C-arm joint tracking system, with the goal of guaranteeing the appearance (within the Xray field of view) of a desired object (e.g. bone or implant) after practical movement of the C-arm. Before developing the system, or investigating existing systems, the required accuracy needs to be established. Considering a typical imaging plane with an approximate size of 20 cm × 20 cm we defined successful base tracking to be achieved when the odometry results can locate the C-arm within a range of half the size of the imaging plane 10 cm × 10 cm to ensure that the 22
desired object appears in the acquired fluoroscopic imagery. For 1 m translational movement of the base, 10% accuracy would be sufficient to achieve this goal whereas for 6 m base movement, 1.6% accuracy would be required (Table 2-1). Based on observations at surgeries attended, moving the C-arm in and out of the surgical field requires approximately 3 to 4 m of total travel. For this distance, accuracy better than 3% of the total absolute cumulative distance changes would be needed to return the base with 10 cm of the original location.
Table 2-1: Acceptable translational base tracking accuracies to return the base to within a 10 cm x 10 cm window (within a 20 cm x 20 cm field of view) Movement Range (m) Acceptable Tracking*
1
2
3
4
5
6
10%
5%
3.3%
2.5%
2%
1.6%
*
Per cent of the total traveled distance.
An orientation accuracy goal is also required. Most commonly, the C-arm base remains perpendicular to the surgical table (i.e. minimal angular change), with changes in orientation being executed with the other C-arm degrees of freedom. Nonetheless changes in orientation do occur when repositioning the C-arm. Furthermore, changes in orientation will affect the translational positioning. Given an approximate base-centre to image-centre distance of 1.5 m, and a desire to add no more than 5 cm error to the translation, acceptable orientation accuracy was defined as 5
better than 3o (arctan(150)) of the total cumulative orientation changes. Note that, given the constraints of the operating room, the C-arm would rarely be rotated more than 30o, resulting in a
23
total excursion of 60º. Under these circumstances, an orientation accuracy within 5% would be sufficient. Visual (optical) odometry (VO) has been investigated in the computer vision and robotics communities for numerous applications, although we are unaware of its use for C-arm base tracking. VO refers to the problem of determining the pose of a camera-carrying platform (here the C-arm base) after processing the frames acquired by the camera (Kneip et al. 2011). The final pose can then be derived by incrementing the estimated motions for each epoch. Different visual odometry techniques have been introduced recently, each trying to provide an optimal solution to address a specific criterion, based on the measurement environment and the desired end product. Simultaneous localization and mapping (SLAM) and structure from motion (SFM) approaches aim to solve for the moving platform pose while generating a real-time map (or a set of object space coordinates) of the surrounding environment (Davison et al. 2007). These techniques have a considerable dependency on accurate frame-to-frame camera motion estimation to provide accurate final results (Chli 2009). Visual odometry techniques can be divided into two main categories: monocular and stereo odometry. In stereo (binocular) methods, the pose of the platform is derived after performing a triangulation procedure for the corresponding features appearing in consecutive frames. These methods can provide the full 6 DOF pose of the vehicle up to a meaningful scale. Stereo methods are computationally complex and time consuming. Monocular VO makes use of single-camera data and is preferred over the stereo techniques in applications for which on-line solutions are desired (as in C-arm base tracking) or the distance to the scene is much larger than the stereo baseline. However, the monocular camera cannot provide the full 6 DOF pose information (the solutions are valid up to an arbitrary scale factor). Since in this study the main focus was to provide 24
an inexpensive, accurate and retrofittable tracking solution, a monocular modality was chosen. Although there exist different monocular tracking algorithms, they have rarely been integrated into motion estimation systems for mobile devices (Scaramuzza et al. 2009) for the following reasons: a) several algorithms are only capable of working in off-line mode and at low frame-rates; b) others need high processing power with expensive dedicated processors; and c) the majority of the methods are designed for specific cameras and particular environments. This is due to the fact that SFM and SLAM methods as well as bundle-adjustment-based solutions solve for the sensor pose and the object-space 3D structure at the same time, leading to the high computational resource requirement. Monocular (and even stereo) odometry systems require quantifiable data about the amount and the rates of evolution in image space. This can be achieved by incorporating optical flow estimators to feed the dead-reckoning algorithm. Optical flow estimation is the action of computing the displacement field between two image frames. Different optical flow estimation strategies exist (Horn et al. 1981; Lucas et al. 1981; Deriche et al. 2004; Alvarez et al. 2000; Anandan 1989) each trying to address an implementation requirement. Feature correspondence can then be easily inferred from the optical flow outputs. A monocular visual odometry system has been proposed (Campbell et al. 2005) that has an optimal error range of 3.3% (equivalent to 13.2 cm error over 4 m of displacement for our scenario). This system requires a forward-looking camera (having upper and lower parts of the horizon in the field of view), which is impractical for our application. Another monocular system (Forster et al. 2014) was shown to have 0.0051 m/s error in the ideal case (equivalent to 45.9 cm error over an estimated 90 seconds of movement). A forward-looking camera was investigated as a VO system (Silva et al. 2012) for indoor vehicles having a maximum error range of ~6% of total 25
traveled distance. Due to the presence of various moving objects in the operating room (surgical staff feet and other moving surgical instruments) within a short range from the VO camera, a forward-looking camera based system (e.g. Campbell et al. 2005) is not appropriate in this application. Lack of sufficient accuracy for other systems (e.g. Silva et al. 2012; Forster et al. 2014) limits their performance in a C-arm base-tracking application. The purpose of this study was therefore to design and test a custom VO method that takes advantage of a downward-looking camera attached to the C-arm base. An error propagation analysis was undertaken to clarify the effect of different sources of error on the dead-reckoning results. Accuracy was then tested experimentally under varying conditions of camera height, camera velocity, floor texture, and relative vs. absolute positioning. The performance was verified on four operating room (OR) floors.
2.2 Methods The methods description is divided into three sections: (1) base-tracking algorithm, (2) error propagation analysis and simulation; and (3) experimental evaluation.
2.2.1 Base-Tracking Algorithm 2.2.1.1 Concepts The basic image formation model used in this study is the camera matrix. The image coordinates and their corresponding camera coordinates are represented in homogeneous forms as follows (Hartley et al. 2003):
26
1⁄ 𝜌𝑥 𝑢̃ [ 𝑣̃ ] = 𝜆 [ 0 𝑤 ̃ 0
0
𝑢0
1⁄ 𝜌𝑦 0
𝑓 ] [ 𝑣0 0 0 1
0 0 0 𝑅 3×3 𝑓 0 0] [ 0 1×3 0 1 0
𝑋 𝑇3×1 𝑌 ][ ] 1 𝑍 1
(1)
where (𝑢̃, 𝑣̃, 𝑤 ̃) are the homogeneous image coordinates, (𝑋, 𝑌, 𝑍) are the corresponding object coordinates in meters, 𝜆 is the scale factor, (𝜌𝑥 , 𝜌𝑦 ) are the pixel dimensions along the two axes of the image coordinate frame in meters, 𝑓 is the focal length in meters, 𝑅 is the rotation matrix, and 𝑇 is the translation vector between the image and ground-truth coordinate systems. The first two matrices on the right hand side of the equation comprise the interior orientation parameters; the third one comprises the exterior orientation parameters. After multiplication of these matrices, the camera matrix, 𝐶, is obtained as: 𝐶11 𝑢̃ [ 𝑣̃ ] = 𝜆 [𝐶21 𝑤 ̃ 𝐶31
𝐶12 𝐶22 𝐶32
𝐶13 𝐶23 𝐶33
𝐶14 𝑋 𝐶24 ] [𝑌 ] 𝑍 𝐶34 1
(2)
The Cartesian camera coordinates (𝑢′ , 𝑣 ′ ) can be obtained from the above homogeneous coordinates as follows: 𝐶11 𝑢̃ [ 𝑣̃ ] = 𝜆 [𝐶21 𝑤 ̃ 𝐶31
𝑢̃ 𝐶12 𝐶13 𝐶14 𝑋 𝑢′ = 𝑤 ̃ 𝐶22 𝐶23 𝐶24 ] [𝑌 ] → { 𝑍 𝑣 ̃ ′ 𝐶32 𝐶33 1 𝑣 = 1 𝑤 ̃
(3)
It is important to note that this representation of image formation is insensitive to the scale factor (𝜆), since the scale is eliminated in the Cartesian coordinate derivation.
27
2.2.1.2 Undistortion The lens assembly for every camera setup can suffer from different sources of distortion (mostly radial and to a lesser extent decentring distortion), which have not been taken into account up to this stage. The calibrated Cartesian image coordinates (𝑢, 𝑣) can be estimated as: 2
𝑢 = 𝑢′ (𝑘1 𝑟 2 + 𝑘2 𝑟 4 + 𝑘3 𝑟 6 ) + 2𝑝1 𝑢′ 𝑣 ′ + 𝑝2 (𝑟 2 + 2𝑢′ ) + 𝑢′ 𝑣=
𝑣 ′ (𝑘1 𝑟 2
4
+ 𝑘2 𝑟 + 𝑘3 𝑟
6)
′ ′
2
′2
+ 2𝑝2 𝑢 𝑣 + 𝑝1 (𝑟 + 2𝑣 ) + 𝑣
(4) ′
in which, (𝑘1 , 𝑘2 , 𝑘3 ) are the radial distortion coefficients, (𝑝1 , 𝑝2 ) are the decentring distortion coefficients, and 𝑟 is the radial coordinate of the image point in the image polar coordinate system (with the origin at the principal point). The coefficients can be estimated by utilizing a specific geometric pattern (e.g. a chessboard pattern, or symmetrical circular patterns on a planar surface) or a 3D object point network from which one can measure the coordinates of the projected image features and the corresponding object coordinates. There is a wealth of literature dedicated to different aspects and concepts of camera calibration dealing with topics ranging from parameter stability to accuracy issues (Remondino et al. 2006). In this study, a flexible closed-form solution (Zhang 2000) was employed, which can estimate the distortion parameters for a camera after observing a planar pattern from at least two different orientations. This method does not require absolute object coordinates and the camera or the pattern can be freely moved. This allows an easy to implement undistortion solution that can be easily incorporated by the end-user.
2.2.1.3 Frame-to-Frame Pose Estimation The camera matrix consists of 11 unknowns. When dealing with a planar object (in which the condition 𝑍 = 0 is always satisfied), the camera matrix is:
28
1⁄ 𝜌𝑥 𝑢̃ [ 𝑣̃ ] = 𝜆 [ 0 𝑤 ̃ 0
0 1⁄ 𝜌𝑦 0
𝐻11 𝑢̃ 𝐻 𝑣 ̃ [ ] = 𝜆 [ 21 𝑤 ̃ 𝐻31
𝑢0
𝑅11 𝑅 ] [ 21 𝑣0 𝑅31 1 𝐻12 𝐻22 𝐻32
𝑅12 𝑅22 𝑅32
𝑇1 𝑋 𝑇2 ] [𝑌 ] 1 1
𝐻13 𝑋 𝐻23 ] [𝑌 ] 1 1
(5)
(6)
The eight unknowns (𝐻11 , 𝐻12 , … , 𝐻32 ) of the homography matrix can be estimated from the coordinates of at least four homologous points (one set lying on the image plane and the other set on the planar object). The Cartesian image coordinates of the planar features can be calculated by utilizing the estimated homogeneous coordinates as follows: 𝑢̃ 𝐻11 𝑋 + 𝐻12 𝑌 + 𝐻13 →𝑢= 𝑤 ̃ 𝐻31 𝑋 + 𝐻32 𝑌 + 1 𝑣̃ 𝐻21 𝑋 + 𝐻22 𝑌 + 𝐻23 𝑣= →𝑣= { 𝑤 ̃ 𝐻31 𝑋 + 𝐻32 𝑌 + 1 𝑢=
(7)
This set of equations is also known as the 2D projective transformation or 2D direct linear transformation (DLT) (Abdel-Aziz et al., 1971; Shapiro 1978). Having a calibrated camera implies that the image is already corrected for the lens distortions and the interior parameters of the imaging cone are known. In this method each acquired frame is undistorted using the pre-computed distortion parameters in real time. Assuming that there are sufficient (four or more) conjugate features in successive frames, the 3D frame-to-frame transformation between them can be estimated as: 𝑅11 𝑋 𝑇 𝑌 𝑅 ] [ ] = 𝜆 [ 21 1 𝑍 𝑅31 1 0
𝑢̃ 𝑅 [ 𝑣̃ ] = 𝜆 [ 01∗3 𝑤 ̃ in which [𝑢̃
𝑣̃
𝑤 ̃ ]𝑇 and [𝑋
𝑌
𝑍
𝑅12 𝑅22 𝑅32 0
𝑅13 𝑅23 𝑅33 0
𝑇1 𝑋 𝑇2 𝑌 ][ ] 𝑇3 𝑍 1 1
(8)
1]𝑇 are the calibrated homogeneous image coordinates of
the corresponding features. Note that the interior parameters of the camera do not appear in Eq. 8 29
because the homography is only established between two corresponding image planes (not the image and the object plane). The homography equivalent (𝑍 = 0) is: 𝑅11 𝑢̃ [ 𝑣̃ ] = 𝜆 [𝑅21 𝑤 ̃ 𝑅31
𝑅12 𝑅22 𝑅32
𝑢̃ 𝑅11 𝑋 + 𝑅12 𝑌 + 𝑇1 𝑢= →𝑢= 𝑇1 𝑋 𝑤 ̃ 𝑅31 𝑋 + 𝑅32 𝑌 + 1 𝑇2 ] [𝑌 ] → 𝑣̃ 𝑅21 𝑋 + 𝑅22 𝑌 + 𝑇2 1 1 𝑣= →𝑣= { 𝑤 ̃ 𝑅31 𝑋 + 𝑅32 𝑌 + 1
(9)
By solving for the frame-to-frame homography coefficients between corresponding frames (which will be described in Section 2.2.1.6) and according to Eq. 9, the homography estimates are valid up to a scale factor. In the particular application of this study, the relative pose of the camera assembly is nadir looking and kept constant; thus a scale factor can be extracted from the first frame and applied to any estimates later. This is an important factor simplifying the overall problem. One major problem is the non-symmetric effect of the scale in different areas of the image due to its non-parallel pose relative to the floor. In other words, if the imaging plane were completely perpendicular to the normal vector of the floor, one could recover a single scalar as the scale factor and apply it to all the recovered estimates; however, this might not be a valid assumption and must be addressed. In order to solve this issue (and remove the tilt effects) and bring the homography estimates to meaningful units, a perspective rectification procedure is performed for the first frame based on the projection of a physical reference. In this research, the estimated projective parameters are not decomposed into exterior orientation parameters (EOPs) and interior orientation parameters (IOPs) since they are not used for the porpose of odometry.
30
2.2.1.4 Perspective Rectification (Frame-to-Floor) To solve for the 8 parameters of frame-to-floor perspective transformation (using Eq. 7) we have: 𝑢𝑟 =
𝐿1 𝑢 + 𝐿2 𝑣 + 𝐿3 → 𝑢𝑟 = 𝐿1 𝑢 + 𝐿2 𝑣 + 𝐿3 − 𝐿7 𝑢𝑟 𝑢 − 𝐿8 𝑢𝑟 𝑣 𝐿7 𝑢 + 𝐿8 𝑣 + 1 (10)
𝐿4 𝑢 + 𝐿5 𝑣 + 𝐿3 𝑣𝑟 = → 𝑣 𝑟 = 𝐿4 𝑢 + 𝐿5 𝑣 + 𝐿6 − 𝐿7 𝑣 𝑟 𝑢 − 𝐿8 𝑣 𝑟 𝑣 𝐿7 𝑢 + 𝐿8 𝑣 + 1
in which, the homography coefficients 𝐻𝑖𝑗 have been denoted as 𝐿𝑖 for the matter of simplicity (reducing the number of subscripts to one), and (𝑢𝑟 , 𝑣 𝑟 ) are the rectified image coordinates (the object space equivalent of image points). The matrix form of the above system is: 𝑣 𝑟 ]𝑡
[𝑢 𝑟 =[
𝑢 0
𝑣 0
1 0 0 𝑢
0 0 −𝑢𝑟 𝑢 −𝑢𝑟 𝑣 ] [𝐿 𝑣 1 −𝑣 𝑟 𝑢 −𝑣 𝑟 𝑣 1
𝐿3
𝐿2
𝐿4
𝐿5
𝐿6
𝐿7
𝐿8 ]𝑡
(11)
The unknown 2D projective parameters can be determined by least-squares solution of a Gauss-Markov model using at least four conjugate points, as follows. The number of corresponding points is represented by the scalar n. 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠: 𝑂 = [𝑢1𝑟 𝑢1 0 𝐷𝑒𝑠𝑖𝑔𝑛 𝑚𝑎𝑡𝑟𝑖𝑥: 𝐴 = ⋮ 𝑢𝑛 [0 𝑈𝑛𝑘𝑛𝑜𝑤𝑛𝑠: 𝐿 = [𝐿1
𝑣1 0 ⋮ 𝑣𝑛 0 𝐿2
𝑣1𝑟
𝑢2𝑟
1 0 0 𝑢1 ⋮ ⋮ 1 0 0 𝑢𝑛 𝐿3
𝐿4
𝑣2𝑟 0 𝑣1 ⋮ 0 𝑣𝑛 𝐿5
⋯
𝑢𝑛𝑟
𝑟 0 −𝑢1 𝑢1 𝑟 1 −𝑣1 𝑢1 ⋮ ⋮ 0 −𝑢1𝑟 𝑢𝑛 1 −𝑣1𝑟 𝑢𝑛
𝐿6
𝐿7
𝑣𝑛𝑟 ]𝑡 −𝑢1𝑟 𝑣1 −𝑣1𝑟 𝑣1 ⋮ −𝑢𝑛𝑟 𝑣𝑛 −𝑣𝑛𝑟 𝑣𝑛 ]
(12)
𝐿8 ]𝑡
If we consider the weight matrix of the observations as 𝑃 = 𝜎0−2 𝐶𝑂−1 and the observation errors as 𝑣~ (∅, 𝜎02 𝐶𝑂 ) (zero mean with variance factor of 𝜎02 ) the Gauss-Markov model can be formed as: 31
{
𝐴 𝐿 = 𝑣 + 𝑂 (𝑣𝑡𝑃𝑣=min) ̂ → 𝐿 = (𝐴𝑡 𝑃𝐴)−1 𝐴𝑡 𝑃𝑂 𝐸(𝑣𝑣 𝑡 ) = 𝜎02 𝐶𝑂
(13)
A square pattern with known dimensions is utilized as the reference just in the first frame to recover the relative pose of the camera to the floor, as described subsequently.
2.2.1.5 Reference (Feature) Extraction The object space features must be accurately extracted from the image in order to be able to perform the perspective rectification process. For this purpose, a custom physical reference was designed and attached to the location on the floor to which the camera is pointing at the beginning of the movement. In order to autonomously detect the projected ground point in the captured frames, the following procedure is applied to the first frame in which the physical pattern is projected (Figure 2-1): 1) a Gaussian low pass filter is applied to reduce the chance of false circle detection; 2) a circular Hough transform is applied to detect the circular boundaries of the desired object points; 3) an individual mask frame is created with the same size as the original image in which all the pixel values are zero except the ones inside one of the detected circles (number of mask frames = number of detected circles); 4) each mask is applied on the original grabbed frame and saved separately; 5) a Canny edge detector is applied on each of the masked frames followed by a linear Hough transform to extract the two internal line segments inside each circle; 6) intersection points between the two line segments are extracted as the projection of the object features (it is important to note that these points might not correspond to the centre of the detected circles due to perspective artifact caused by non-parallel relative pose of the imaging plane and the floor); and 7) the recovered points are then reordered according to the centroid point. This method
32
provides robust landmark extraction results and is insensitive to different imaging conditions such as landmark size, sensor tilt and brightness.
Figure 2-1: Feature extraction for perspective rectification. Left: source frame; middle: masked frame to isolate the circular patterns; right: extracted features.
2.2.1.6 Pose Recovery Consecutive frames are first rectified (for possible tilt effect) based on the projection of physical ground landmarks on acquired frames. In order to recover the rectified frame-to-frame homography parameters, the same least squares procedure is applied as mentioned in Section 2.2.1.4. The corresponding features are extracted (described in Section 2.2.1.7) and utilized to perform the rectified frame-to-frame homography estimation. As seen in Eq. 9, the recovered parameters (𝐿3 , 𝐿6 , 𝐿4 , 𝐿1 ) represent (𝑇𝑥 , 𝑇𝑦 , 𝑅21 , 𝑅11 ) respectively. Decomposition of the yaw rotation (incremental changes in the platform’s orientation) can be performed as: 𝑅
𝑦𝑎𝑤 = atan(𝑅21) 11
33
(14)
All the estimated incremental positional movements (𝑇𝑥𝑖 , 𝑇𝑦𝑖 ) are therefore transformed back into the coordinate system of the frame that was captured at the beginning of the movement (Eq. 15). The recovered rotation (𝑦𝑎𝑤 𝑖 ) at each epoch can be simply accumulated to achieve the real time pose of the system (𝜃 𝑖 = ∑𝑖𝑛=1 𝑦𝑎𝑤 𝑛 ). 𝑖 𝑖 [∆𝑋𝑖 ] = [cos 𝜃𝑖 ∆𝑌 𝑠𝑖𝑛𝜃
𝑖 𝑖 𝑖 𝑖 𝑖 𝑖 −𝑠𝑖𝑛𝜃 𝑖 ] [𝑇𝑥 ] → { ∆𝑋 = 𝑇𝑥 𝑐𝑜𝑠𝜃 − 𝑇𝑦 𝑠𝑖𝑛𝜃 𝑐𝑜𝑠𝜃 𝑖 𝑇𝑦𝑖 ∆𝑌 𝑖 = 𝑇𝑥𝑖 𝑠𝑖𝑛𝜃 𝑖 + 𝑇𝑦𝑖 𝑐𝑜𝑠𝜃 𝑖
(15)
Figure 2-2 shows the overall workflow of the described visual odometry method.
Figure 2-2: Schematic workflow of the base-tracking system.
As seen in the more detailed flowchart of Figure 2-2 (Figure 2-3), the method can be summarized as follows: the frame (𝑭) at epoch 𝒊 is captured and undistorted based on the distortion coefficients (𝒌𝟏 , 𝒌𝟐 , … , 𝒑𝟐 ) recovered during initial camera calibration. The perspective rectification procedure is undertaken for the first frame and the 2D projective coefficients stored into the 𝒑𝒆𝒓𝒔 (perspective) vector. The trackable features are extracted by a Harris corner detector and imported into a two-way KLT (Lucas-Kanade, will be described in 2.2.1.7) visual odometry estimator (this function accepts the old frame 𝑰𝒊 and the old point set 𝑷𝒊 as well as new image 𝑰𝒊+𝟏 as inputs and 34
reports the new point set 𝑷𝒊+𝟏 as the tracked points Th stands for Threshold); the number of successfully tracked features is checked at each epoch and if they are less than a threshold 𝒌 (typically when the old features move out of the image periphery), a new set of trackable features is extracted; the corresponding point sets in the new and old frames are rectified based on the 𝒑𝒆𝒓𝒔 vector to achieve their equivalent object-space coordinates; and the frame-to-frame homography is then estimated followed by a decomposition procedure to determine the incremental translational and angular transformations. The real time pose of the platform (𝑿, 𝒀, 𝒀𝒂𝒘) is calculated by accumulating the incremental parameters. The new frame and its corresponding points are stored as the old frame and old point set of the next iteration.
35
Figure 2-3: Flowchart of the base-tracking algorithm.
36
2.2.1.7 Optical Flow As described earlier, it is essential to detect corresponding features between successive frames to have an accurate estimate of the platform’s pose. This study employs the optical flow algorithm developed by Lucas and Kanade (Lucas et al., 1981) due to its robustness, flexibility and closed-form notion. The term “Optical Flow” is “the distribution of apparent brightness patterns in an image” (Horn et al. 1981). Optical flow gives valuable information about the pattern, rate and amount of displacement of the moving features. The principal idea of the Lucas Kanade algorithm is provided in the following section based on the original paper (Lucas et al. 1981) and a later revision (Baker et al. 2004). The objective of the Lucas Kanade (KLT) method is to minimize the sum of squared errors between the original pixel in the source frame and the warped pixel in the destination frame. In other words, this method tries to minimize the function 𝐹: 𝐹 = ∑[𝐼(𝑊(𝑥; 𝑝)) − 𝑇(𝑥)]
2
(16)
𝑥
in which 𝑇(𝑥) represents the gray scale value of the point 𝑥(𝑢, 𝑣) within the source frame, 𝐼(𝑥) is the same parameter in the destination frame and 𝑊(𝑥; 𝑝) is the warping function that maps the pixel 𝑥 to the destination image with a transformation based on parameters comprised in the vector 𝑝. By minimizing the function 𝐹, this method looks for the best set of transformation parameters between the source and the destination frame. If an initial estimate of the transformation parameters is available at this stage, the function 𝐹 can be revised as: 𝐹 = ∑[𝐼(𝑊(𝑥; 𝑝 + ∆𝑝)) − 𝑇(𝑥)] 𝑥
37
2
(17)
For the sake of convenience, the target function can be approximated based on a Taylor series expansion as in the following (after dismissing the higher order terms): 2 𝛿𝑊 𝐹 = ∑ [𝐼(𝑊(𝑥; 𝑝)) + ∇𝐼 Δ𝑝 − 𝑇(𝑥)] 𝛿𝑝
(18)
𝑥
The vector ∇𝐼 is the gradient operator applied to the image 𝐼 (at each pixel) and
𝛿𝑊 𝛿𝑝
represents the Jacobian operator on the desired transformation. For instance, if the desire is to determine the rigid transformation parameters (that minimize the function 𝐹) between the original frame 𝑇, and the destination frame 𝐼, then: 𝑊(𝑥; 𝑝) = (𝑢𝑐𝑜𝑠(𝛼) − 𝑣𝑠𝑖𝑛(𝛼) + 𝑏1 , 𝑢𝑠𝑖𝑛(𝛼) + 𝑣𝑐𝑜𝑠(𝛼) + 𝑏2 ) 𝑐𝑜𝑠𝛼 → 𝑊(𝑥; 𝑝) = [ 𝑠𝑖𝑛𝛼
−𝑠𝑖𝑛𝛼 𝑐𝑜𝑠𝛼
𝑢 𝛿𝑊 𝑏1 −𝑢𝑠𝑖𝑛𝛼 − 𝑣𝑐𝑜𝑠𝛼 ] [𝑣 ] → =[ 𝑏2 𝑢𝑐𝑜𝑠𝛼 − 𝑣𝑠𝑖𝑛𝛼 𝛿𝑝 1
1 0 ] 0 1
(19) (20)
in which the vector 𝑝 includes the rigid transformation parameters (𝑏1 , 𝑏2 , 𝛼). In order to minimize the function 𝐹, the partial derivative of Eq. 18 with respect to the desired parameter ∆𝑝 is set to zero: 2 ∑ [∇𝐼 𝑥
𝛿𝑊 𝑡 𝛿𝑊 ] [𝐼(𝑊(𝑥; 𝑝)) + ∇𝐼 ∆𝑝 − 𝑇(𝑥)] = 0 𝛿𝑝 𝛿𝑝
(21)
Reordering Eq. 21 will result in: ∆𝑝 = 𝐻
−1
𝛿𝑊 𝑡 ∑ [∇𝐼 ] [𝑇(𝑥) − 𝐼(𝑊(𝑥; 𝑝))] 𝛿𝑝 𝑥
𝑡
𝑖𝑛 𝑤ℎ𝑖𝑐ℎ: 𝐻 −1 = ∑ [∇𝐼 𝑥
(22)
𝛿𝑤 𝛿𝑤 ] [∇𝐼 ] 𝛿𝑝 𝛿𝑝
The ideal transformation parameter is acquired by iteratively solving Eq. 22 and adding the results to the initial estimates: ∆𝑝 + 𝑝 → 𝑝. 38
2.2.1.7.1 Implementation The well-known Harris corner detection operator (Harris et al., 1988) is applied on the first frame to recover the proper (trackable) features as the input for the KLT optical flow estimator. This gives a set of image points with their corresponding intensity values. After acquiring the second frame, the corresponding points are detected by utilizing the optical flow algorithm. The KLT method must provide the same results if it is applied in the reverse order. This fact is used as a cue for error elimination in correspondence detection. If the difference between the original and the reverse implementation of the KLT method is less than a threshold for a specific pixel (0.5 pixels in this study) the recovered matches are considered as true. This process is repeated until the number of successfully tracked features reaches below a threshold (50 in this study). Refer to Figure 2-3 for further details about the optical flow module.
2.2.1.8 Odometry Loop Closure (Absolute Tracking) The same physical reference used for the purpose of perspective rectification can be used as a cue for loop closure. This can be done in a way that the accumulated odometry estimates are forced to reset to the object-space coordinates of the projected reference as soon as it appears entirely in the camera`s field of view. In other words, first the extracted object points (from the early frames) are stored in the memory, then after approximately returning to the starting point, the reference should appear somewhere in the grabbed frame. The object point coordinates are again extracted at this stage (based on the established perspective rectification). Odometry alone may not return the base to the origin of the movement due to accumulated errors. This bias can be solved by performing a frame-to-frame homography (refer to Section 2.2.1.3) between the stored
39
object points and the extracted equivalents at any time the reference is visible in the image, allowing the base either to return to the starting point or to recognize another defined destination.
2.2.2 Error Propagation Analysis and Simulation 2.2.2.1 Error Propagation Analysis Solving for the frame-to-floor homography coefficients means that the image is rectified as if the normal vector of the imaging plane is perfectly perpendicular to the floor. The correctness of matching between object and image points will affect the accuracy of rectification, which can dilute the pose recovery accuracy. In this section, the influences of different sources of error on the final estimated pose are investigated. As mentioned in Section 2.2.1.4 a perspective transformation is applied to relate the image coordinate system to the ground-truth coordinate system (Eq. 10). Due to the rigidity of the camera platform, an assumption can be made at this stage that the 3D camera pose is constant relative to the ground-truth coordinate system; therefore the perspective transformation parameters are estimated in an a priori stage and applied to each of the grabbed frames. Depending on the location of each feature of interest (in each frame) the rectification error can be estimated by applying the error propagation law to the 2D perspective equations. The variance-covariance matrix of the desired unknowns (perspective coefficients) can be estimated as: 𝐶̂𝐿 = (𝐴𝑡 𝑃𝐴)−1
,
𝑃 = 𝜎𝑔−2 𝐶𝑂−1
(23)
in which 𝜎𝑔2 is the precision of the ground-truth point (landmark) coordinate measurement. This matrix contains the estimated accuracies of perspective coefficients that propagate into the results of image rectification. In order to assess the precision of each image point in the rectified frame (based on its image coordinates): 40
𝑟
𝑢 𝐹⃗ = [ 𝑟 ] 𝑣
→
𝐶𝑅𝑒𝑐1(2×2) =
𝛿𝐹⃗ 𝛿𝐿 (2×8)
𝐶̂𝐿 (8,8)
𝑇 𝛿𝐹⃗ 𝛿𝐿 (8×2)
𝜎𝑢2𝑟 =[ 𝜎 𝑢𝑟 𝑣 𝑟
𝜎𝑢𝑟 𝑣 𝑟 ] 𝜎𝑣2𝑟
(24)
in which (𝑢𝑟 , 𝑣 𝑟 ) are the point coordinates in rectified frame and parameters (𝜎𝐿21 , 𝜎𝐿22 , … , 𝜎𝐿28 ) are obtained from the variance-covariance matrix of the unknowns 𝐶̂𝐷 (as a by-product of the previously established least-squares-based frame-to-floor homography). Eq. 23 and Eq. 24 illustrate the effect of the ground-truth point precision (𝜎𝑔2 ) on the correctness of the rectified image points. Another important source of error that influences the image point precision in the 2 rectified image is the error in the original image point measurement 𝜎𝑃𝑛 (the error in feature
extraction of the projected physical landmark) that is investigated in the following equation: 2 𝐶𝐼𝑚(2×2) = 𝜎𝑃𝑛 ∗ 𝐼(2×2) & 𝐸𝑥 = (𝑢, 𝑣) → 𝑇
𝐶𝑅𝑒𝑐2(2×2)
2 𝛿𝐹⃗ 𝛿𝐹⃗ 𝜎𝑢′ 𝑟 = 𝐶 =[ ′ 𝛿𝐸𝑥(2×2) 𝐼𝑚(2×2) 𝛿𝐸𝑥 (2,2) 𝜎𝑢𝑟 𝑣𝑟
𝜎𝑢′ 𝑟 𝑣𝑟 2
𝜎𝑣′ 𝑟
(25) ]
The 2D extracted landmark coordinates are assumed to be uncorrelated and 𝐸𝑥 = (𝑢, 𝑣) is the vector containing the image coordinates. The final precision of each point 𝑅(𝑢𝑟 , 𝑣 𝑟 ) in the rectified image is estimated as: 2
2
2 𝐶𝑅𝑒𝑐(2×2) = 𝐶𝑅𝑒𝑐1(2×2) + 𝐶𝑅𝑒𝑐2(2×2) → 𝜎𝑅𝑒𝑐 = (𝜎𝑢2𝑟 + 𝜎𝑢′ 𝑟 , 𝜎𝑣2𝑟 + 𝜎𝑣′ 𝑟 )
(26)
It is important to note that 𝜎𝑅2 depends on the location of the points in the original image 𝐸𝑥(𝑢, 𝑣), the measurement precision of ground-truth points, 𝜎𝑔2 , position and orientation of perspective centre of the image relative to the ground-truth coordinate system (𝐿1 , 𝐿2 , … , 𝐿8 ) and 2 the precision of feature extraction in the perspective rectification stage 𝜎𝑃𝑛 .
The frame-to-frame homography coefficient between two rectified image planes are computed at this stage. The variance-covariance matrix of the observations is formed as:
41
2 𝐶𝑙 (𝑢𝑟 , 𝑣 𝑟 ) = 𝜎𝐾𝐿𝑇 ∗ 𝜎𝑅2 (𝑢, 𝑣)
(27)
in which 𝜎𝐾𝐿𝑇 is the standard deviation of the optical flow procedure in finding the corresponding features within the successive frames and 𝜎𝑅 is the rectification error for any original image 2 point (𝑢, 𝑣) that comprises the effects of 𝜎𝑔2 (object space measurement error) and 𝜎𝑃𝑛 (feature
extraction error). After solving the rectified frame-to-frame homography the final accuracy of the desired parameters (𝑇𝑥 , 𝑇𝑦 , 𝑦𝑎𝑤) can be derived as: 𝑇𝑥 = 𝐿(3,3) → 𝜎𝑇2𝑥 = 𝐶̂𝐿 (3,3) , 𝑇𝑦 = 𝐿(6,6) → 𝜎𝑇2𝑦 = 𝐶̂𝐿 (6,6) 𝛿𝑦𝑎𝑤 𝐿4 2 𝑦𝑎𝑤 = atan ( ) → 𝜎𝑦𝑎𝑤 =[ 𝛿𝐿1 𝐿1 𝛿𝑦𝑎𝑤 2 𝜎𝑦𝑎𝑤 =[ 𝛿𝐿1
𝛿𝑦𝑎𝑤 𝜎𝐿21 ][ 𝛿𝐿4 𝜎𝐿4 𝐿1
𝜎𝐿1𝐿4 𝛿𝑦𝑎𝑤 ][ 𝛿𝐿1 𝜎𝐿24
̂ (1,1) 𝐶 ̂ 𝐿 (1,4) 𝛿𝑦𝑎𝑤 𝛿𝑦𝑎𝑤 𝐶 ][ 𝐿 ][ ̂ 𝐿 (4,1) 𝐶 ̂ 𝐿 (4,4) 𝛿𝐿1 𝛿𝐿4 𝐶
𝛿𝑦𝑎𝑤 𝑇 ] → 𝛿𝐿4
(28)
𝛿𝑦𝑎𝑤 𝑇 ] 𝛿𝐿4
Since the camera frame keeps its relative pose to the floor constant during the movement, the real time pose of the camera can be estimated by accumulating the translational and angular parameters (extracted by performing a 2D frame-to-frame homography estimation between successive frames). This method assumes no correlation between consecutive frames. 2 𝜃 𝑖 = 𝜃 𝑖−1 + 𝑦𝑎𝑤 𝑖 → 𝜎𝜃2𝑖 = 𝜎𝜃2𝑖−1 + 𝜎𝑦𝑎𝑤 𝑖
{
𝑋 𝑖 = 𝑋 𝑖−1 + ∆𝑋 𝑖 → 𝑋 𝑖 = 𝑋 𝑖−1 + 𝑇𝑥 𝑖 𝑐𝑜𝑠𝜃 𝑖 − 𝑇𝑦 𝑖 𝑠𝑖𝑛𝜃 𝑖 𝑌 𝑖 = 𝑌 𝑖−1 + ∆𝑌 𝑖 → 𝑌 𝑖 = 𝑌 𝑖−1 + 𝑇𝑥 𝑖 𝑠𝑖𝑛𝜃 𝑖 + 𝑇𝑦 𝑖 𝑐𝑜𝑠𝜃 𝑖
Using the following notation for simplicity,
42
(29)
𝑋 𝑖−1 + 𝑇𝑥 𝑖 𝑐𝑜𝑠𝜃 − 𝑇𝑦 𝑖 𝑠𝑖𝑛𝜃 ⃗⃗⃗⃗⃗⃗ = [ 𝑇𝑅 ] 𝑌 𝑖−1 + 𝑇𝑥 𝑖 𝑠𝑖𝑛𝜃 + 𝑇𝑦 𝑖 𝑐𝑜𝑠𝜃 𝐶𝐾 = 𝑑𝑖𝑎𝑔 [𝜎𝑋2𝑖−1
, 𝐾 = [𝑋 𝑖−1
𝑇𝑥 𝑖
𝑌 𝑖−1
𝑇𝑦 𝑖
𝜃𝑖] (30)
𝜎𝑇2 𝑖
𝜎𝑌2𝑖−1
𝑥
𝜎𝑇2 𝑖 𝑦
𝜎𝜃2𝑖 ]
𝑇
the final translational precisions can be estimated as: 𝑇
𝐶 ⃗⃗⃗⃗⃗⃗ 𝑇𝑅
⃗⃗⃗⃗⃗⃗ ⃗⃗⃗⃗⃗⃗ 𝜎𝑋2𝑖 𝛿𝑇𝑅 𝛿𝑇𝑅 = 𝐶 =[ 𝛿𝐾 (2,5) 𝐾 (5,5) 𝛿𝐾 (5,2) 𝜎𝑋 𝑖 𝑌 𝑖
𝜎𝑋 𝑖 𝑌 𝑖 𝜎𝑌2𝑖
]
(31)
2.2.2.2 Error Propagation Simulation The error propagation principles (described in Section 2.2.2.1) were employed to simulate the expected performance of the proposed method under the influence of different error sources. In order to simulate the actual setup as closely as possible, all the required parameters (object points coordinates and their projection in image space, camera height, ranges of error, image size and resolution and etc.) were borrowed from the experimental findings for the first height configuration (see below). Four image points were considered in a 1288 × 728 pixel image (near the corners of a window with the dimensions of 75% of the image size) with their corresponding object space coordinates, to perform the perspective rectification procedure. The image points were then moved and tracked (in a known interval) to replicate the real movement of the platform. In this simulation study, all four points are considered trackable over 100 frames and they are replaced with a new set of points after this period. The effects of precision of object point 2 2 measurement (𝜎𝑔2 ), landmark extraction (𝜎𝑝𝑛 ) and optical flow estimation (𝜎𝐾𝐿𝑇 ) have been
evaluated on the final tracking results.
43
2.2.3 Experimental Evaluation 2.2.3.1 Camera & Computer For the purpose of accuracy assessment and reliability measurement, a GigE vision4 grayscale camera was exploited in this study (BFLY-PGE-09S2M-CS, Point Grey Research Inc., Vancouver, Canada; Table 2-2). Data acquisition was based on the product specific software development kit (FlyCapture2) over a standard Ethernet cable. Processing computer specifications are: Intel® core™ i5 processor, 7.87 GB of random access memory and 64-bit Windows 7 as the operating system.
Table 2-2: Technical characteristics of the imaging sensor and lens assembly. Manufacturer & model
Imaging sensor
Imaging architecture
Resolution
Frame rate
Machine Vision Standard
Focal length
Iris range
1/3”, 4.08 µm
Global shutter CCD
1288x 728 pixels
30 FPS
GigE Vision v1.2
2.8～8 mm
F 1.2 ~ Close
(*) Point
Grey BFLY-PGE-09S2MCS (**)
Fujinon YV2.8x2.8SA-2
(*)
Imaging Sensor. (**) Lens Assembly.
2.2.3.2 Experimental Setup In order to ensure smooth, continuous and easy to handle movement of the camera, two custom-made platforms were fabricated to replicate the rotational and translational movements of the C-arm. The rotational platform consists of a caster-based trolley that can freely move inside a reference frame, for which the dimensions and angles were measured in advance (Figure 2-4-A).
4
Global camera interface standard developed using the Gigabit Ethernet communication protocol
44
The translational platform was built on the concept of DSLR camera sliders in videography applications. It contains two tripod-connectable bases at each end and a movable housing unit consisting of three camera mountable shafts (Figure 2-4-B). The translational movements along the 𝑋 and 𝑌 axes were simulated by mounting the camera on each of these shafts. The distance between the camera and the floor underneath can be tuned by changing the height of the tripods. The translational ranges of movement were matched to the final product expectations in that, in a normal orthopaedic surgery (after propelling the C-arm into the operating room), it is most likely moved a maximum 4 m to and from the operating table. In each of the experimental surveys, the camera was slid several times (to achieve the 4 m range) back and forth along a 593 mm base line (this distance was measured with a tape measure). The ability of this method to estimate the orientation of the camera-carrying platform was studied by changing the angle of the platform in intervals of 90o (Figure 2-4-A).
A
B
Figure 2-4 A: Rotating platform with a right-angled reference; B: Sliding platform with three camera-mountable shafts shown with red arrows.
45
2.2.3.3 Camera Heights (Ranges) All of the validation studies were conducted for two different camera heights (70 cm and 16.5 cm as measured with a tape measure), each having a particular lens assembly configuration (implicitly included in the camera matrix obtained from the calibration procedure) to have the floor texture in sharp focus (visually evaluated). The 16.5 cm is the actual base height for the C-arm available in this study; the other height was chosen to evaluate the efficiency of the proposed system for higher camera heights. The lens configurations for the camera heights of 70 cm and 16.5 cm will be referred to as Configuration 1 (H1) and Configuration 2 (H2), respectively (Figure 2-5-A, Figure 2-5-B). The program for computation and analysis was developed with the OpenCV (Version 2.7) library in Visual C++ (X86, 2010 interface; Microsoft Corporation). The error propagation simulation was modeled in MATLAB R2014a (MathWorks®, Natick, Massachusetts, US). Different aspects of the experimental study are described in the following.
Figure 2-5: Sliding platform - left: H1 (70 cm), right: H2 (16.5 cm).
46
2.2.3.4 Camera Speed The method was tested for three different movement velocities and for both camera heights. A velocity of 0.03 m. s−1 can be considered as the mean displacement velocity of a C-arm device (best guess based on the observations). The velocity of the platform was inferred from the frame rate of image acquisition (30 fps) and the total number of frames for each dataset.
2.2.3.5 Floor Textures The performance of the proposed method was evaluated for different floor textures. This scenario was replicated by attaching four artificial (printed in grayscale) textures: Office floor, parquet, tile and carpet on the test floor (Figure 2-6). In order to assess the ability of this method to identify and track the features on a real OR floor, sets of experimental data were collected at the Foothills Medical Centre (Calgary, Alberta) from four different OR floors.
A
C
B
D
Figure 2-6: Tested floor textures. A: Parquet, B: Office Floor, C: Carpet, D: Tile. These were printed out on flat paper. 2.2.3.6 Undistortion As mentioned in Section 2.2, the undistortion was done using a previously-described method (Zhang 2000). Fifty frames were captured from a custom-made manually-oriented 47
chessboard pattern configuration (Figure 2-7) as the input of the calibration module. The frames were convergent, rolled and from the target locations that occupy the whole imaging volume. The employed model requires only a minimum number of 2 frames, however, in this study we utilized fifty calibration frames to increase the redundancy and consequently improve the robustness of undistortion. The recovered distortion coefficients for each setup (Table 2-3) were subsequently used to undistort the images prior to rectification.
Table 2-3: Recovered distortion coefficients. Configuration
𝑘1 (𝑝𝑖𝑥𝑒𝑙 −2 )*
𝑘2 (𝑝𝑖𝑥𝑒𝑙 −4 )
𝑘3 (𝑝𝑖𝑥𝑒𝑙 −6 )
𝑃1 (𝑝𝑖𝑥𝑒𝑙 −1 )
𝑃2 (𝑝𝑖𝑥𝑒𝑙 −1 )
Error**
H1
-3.307 e-1
2.005 e-1
-2.249 e-1
-1.14 e-3
-7.70 e-4
0.41
H2
-3.519 e-1
2.003 e-1
-8.997 e-2
5.74 e-4
1.12 e-3
0.35
(*)
(**)
The reported coefficients are scaled. Average reprojection error (RMS) for the identified corners (in pixels).
Figure 2-7: Undistortion procedure - left: setup; middle: distorted frame; right: undistorted frame. In order to calculate the correctness of the undistrotion procedure, an evaluation study was performed by measuring the straightness of a projected linear object. A linear object (independent 48
of the ones used for the undistortion procedure) was first captured by the camera and the recovered distortion parameters were incorporated to the original frames to achieve the distortion-less frames. After extracting the feature, a best fit line was calculated to the image points (both for the original and undistorted frames) by applying a linear regression model. The quality of fit for the regression was estimated by the root mean square error (RMSE) between the extracted lines and the interpolated lines. The higher estimated RMSE, the more distortion exist in the investigated frame. Results of this evaluation study are reported in 2.3.1.
2.2.3.7 Perspective Rectification The object space coordinates of the desired landmarks were measured by a Vernier caliper (with a manufacturer’s specified accuracy of 0.02 mm). The perspective rectification stage was then performed to relate the image and object space coordinate in the early frames (the frames in which the reference is visible), after which these data were used for the purpose of perspective rectification on the acquired frames at each epoch.
2.2.3.8 Optical Flow A two-way KLT estimator was utilized to calculate the optical flow between successive frames with a window size of (31 × 31) pixels and with a maximum error threshold of 0.5 pixels. The window size and maximum error threshold parameters were tuned in a trial-and-error procedure.
49
2.2.3.9 Error Determination The odometry error at each tracking scenario was calculated based on a comparison between tracking results and the reference traveled distance (or overall changes in orientation in case of angular accuracy assessment). The reference distance for each set of movements was inferred from the baseline of the translational experimental setup (Figure 2-4-B) and the reference angle measured from the rotational experimental setup (Figure 2-4-A). Two sets of translation studies were performed, one for a camera that was displaced mostly along the X-axis of the image coordinate system and other for a camera that was displaced mostly along the Y-axis of the same frame.
2.2.3.10 Odometry Loop Closure (Absolute Tracking) In order to replicate the absolute tracking scenario (described in Section 2.2.1.8), two frames that each included the reference frame at a specific object coordinate (x, y, yaw) were acquired from the same exposure stations, i.e. with the camera in the same location but with the reference in two different locations (equivalent to the reverse situation in the real scenario in which the camera is moving but the reference frame is stationary). The separation between the reference frames was measured and compared to the loop closure results by odometry alone.
2.3 Results 2.3.1 Undistortion evaluation As described in 2.2.3.6, straightness of linear features were used to determine the quality of the undistortion process. Visual inspection of the outcomes, along with the RMSE measures show a
50
significant improvement in image aberration after incorporation of the undistortion procedure (Figure 2-8).
Calibration Results, Top: H2, Bottom: H1 700 690
RMSE: 2.81 Pixels
680
Y Pixels
670 660 650 640 630 620
RMSE: 34.48 Pixels
610 600 200
400
600
800
1000
1200
1400
1165
1180
X Pixels 800 700 RMSE: 3.90
Y Pixels
600 500 400 300 200
RMSE: 20.67
100 0 1090
1105
1120
1135
1150
X Pixels Original
Fit Line
Undistorted
Fit Line
Figure 2-8: Undistortion results for the two lens assemblies used in the experiments of this study measured by straightness of linear features. Numbers in the box show the quality of line fitting. 51
2.3.2 Error Propagation Simulation 2 2 All the investigated parameters (𝜎𝑔2 , 𝜎𝑃𝑛 , 𝜎𝐾𝐿𝑇 ) were shown to have a direct impact on the
odometry outcomes (Figure 2-9), meaning that the higher the precision of each of the parameters, the higher the expected tracking precisions would be. These parameters were tuned as close as possible to the actual testing circumstances and as a result, the final simulated odometry precisions match the real experimental ones (3.75% of the total traveled distance of 1150 mm for 𝜎𝑔 = 0.01 mm, 𝜎𝑃𝑛 = 0.5 pixels, 𝜎𝐾𝐿𝑇 = 0.5 pixels).
52
𝜎 (mm)
A: σPn=0.5 pixel, σKLT=0.5 pixel 140 120 100 80 60 40 20 0
σg=0.01 mm σg=0.02 mm σg=0.04 mm σg=0.06 mm 0
200
400
600
800
1000
1200
Traveled Distance (mm)
𝜎 (mm)
B: σg=0.01 mm, σKLT=0.5 pixel 140 120 100 80 60 40 20 0
σPn=0.25 pixel σPn=0.5 pixel σPn=1 pixel 0
200
400
600
800
1000
1200
σp=1.5 pixel
Traveled Distance (mm)
𝜎 (mm)
C: σg=0.01 mm, σPn=0.5 pixel 140 120 100 80 60 40 20 0
σklt=0.25 pixel σklt=0.5 pixel σklt=0.75 pixel 0
200
400
600
800
1000
1200
σklt=1 pixel
Traveled Distance (mm)
Figure 2-9: Error propagation simulation; A: The effect of object space measurement (𝝈𝒈 ) (for perspective rectification) on the tracking results (planimetric odometry error 𝝈 = √𝝈𝟐𝑿 + 𝝈𝟐𝒀 ); B: The effect of marker extraction error (𝝈𝑷𝒏 ) in image space (for perspective rectification) on the tracking results; C: The effect of optical flow error (𝝈𝑲𝑳𝑻 ) on the tracking results.
53
2.3.3 Camera Height (Range) The recovered displacement accuracies for the movements, conducted mostly along the 𝑋 axis and mostly along the 𝑌 axis of the image for H1 (70 cm) were better than 2% of the total traveled distance (Figure 2-10) and better than 4% for H2 (16.5 cm) (Figure 2-11). The accuracy goal of 10 cm was achieved for the full 6 m cumulative movement for H1 and for up to 3 m for H2. The attitude of the movement (i.e. whether along X or Y axes of the image) did not have a major impact on the tracking outcomes (note: the differences seen in Figure 2-10 and Figure 2-11 are due to the difference in movement velocities caused by the experimental setup, which had different values along the two axes).
54
Translation test for H1 (70 cm), dashed lines=2% of Truth 15
Error (cm)
10 5
1694
847
2431
3144
3776
4527
4196
4908
5890
5384
0 0 -5
100 682
1172
200 1485
-10 -15
300 1776
2035
400
2404
500
2737
3215
600
3744
4238
Traveled Distance-Truth (cm) Error-X
Error-Y
2% Error
Acceptable Error
Figure 2-10: Errors for ~6 m translation tests for H1. Dashed lines=2% of truth; numbers in the boxes: Frame#; Red: movement conducted along the X-axis of the image; Green: movement conducted along the Y-axis of the image.
Translation test for H2 (16.5 cm) , dashed lines=2%-4% of Truth 20 15
647
1057
1784
Error (cm)
10
2355
5
3044
3581
0 -5 0 -10
50
100
150
200
250
300
350
891 1466 2329
1901
-15 -20
2812
3222
Traveled Distance-Truth (cm) Error-X
Error-Y
2% Error
4% Error
Acceptable Error
Figure 2-11: Errors for ~3.6 meter translation tests for H2, dashed lines=2% & 4% of truth. Numbers in the box: Frame#; Red: movement conducted along X-axis of the image; Green: movement conducted along Y-axis of the image. 55
The rotation recovery accuracy was shown to be better than 2.7% of the total accumulated changes in the platform’s orientation for both of the camera heights (Figure 2-12). The accuracy goal of 3º was achieved up to 200º of cumulative rotation for H1 and up to 192 o of cumulative rotation for H2.
A:Orientation test (H1-70 cm) 20
Error (o)
15 10 5 0 -5 0
50
100
150
200
250
300
350
-10 -15 -20
Cumulative Rotation (o) Recovered Error-H1
Recovered Error-H2
5% Error
Acceptable Error
Figure 2-12: Accuracy of orientation recovery.
2.3.4 Repeatability Analysis Repeatability of the proposed method was analysed by recovering the measured odometry results for the setup base line and for the camera height H1 (Figure 2-13).
56
Recovered baseline (mm)
Repeatability Analysis 600 500 400 300 200 100 0
Truth
1
2
3
4
5
6
7
8
9
10
593
593
593
593
593
593
593
593
593
593
590
592.3
588.9
591.1
584.9
582.6
587
590.1
586.1
Repeated Measurments 592.2
Mean: 588.5
Measurment #
SD: 3.09 Truth
Repeated Measurments
Figure 2-13: Repeatability results, error bars=2% of the truth; Table numbers are in mm.
2.3.5 Camera Velocity Increasing the velocity of platform movement caused degradation in accuracy of the odometry results (Table 2-4):
the absolute accuracy of the recovered distance (baseline
of 593 mm) reduced from 3.5 mm to 6.4 mm when the velocity was increased from ~0.02 m. s −1 to ~0.07 m. s−1 for H1, and from 7 mm to 9.9 mm when the velocity was increased from ~0.015 m. s −1 to ~0.038 m. s −1 for H2. Table 2-4: Tracking results for different velocities. Height Speed~ Traveled Distance (mm)
H1 (70 cm) 0.020 𝑚. 𝑠 −1
0.040 𝑚. 𝑠 −1
H2 (16.5 cm) 0.07 𝑚. 𝑠 −1
0.015 𝑚. 𝑠 −1
0.0252 𝑚. 𝑠 −1
0.0389 𝑚. 𝑠 −1
Rec(#)
Er(##)
Rec
Er
Rec
Er
Rec
Er
Rec
Er
Rec
Er
1182.4
3.5
1180.0
5.9
1179.6
6.4
1178.9
7.0
1177.8
8.2
1176.1
9.9
(#)
(##)
Recovered distance. Error (difference between the recovered and actual distances).
57
2.3.6 Floor Texture (Standardized + Real OR) The system’s accuracy in all of the four floor textures was better than 2% of the traveled distance for both of the camera heights (Figure 2-14). The number of trackable features ranged from 479 to 1000 (depending on the floor type) for the real OR floors (Figure 2-15).
A:Texture test H1 (70 cm) 6
Error (cm)
4 2 0 -2
0
50
100
150
200
250
200
250
-4 -6
Traveled Distance (Along X-axis) -Truth (cm)
B:Texture test H2 (16.5 cm) 6
Error (cm)
4 2 0 -2
0
50
100
150
-4 -6
Traveled Distance-Truth (cm) Error-Tile
Error-Parquet
Error-Rug
Error-Mosaic
2% Error
Figure 2-14: Translational tracking errors for different floor textures and two camera heights (A: H1, B: H2).
58
Figure 2-15: Feature detection on real OR floor textures. Top-left box in each image: total number of successfully tracked features.
2.3.7 Loop Closure & Absolute Tracking The recovered distance between the original reference frame and the moved reference frame was 30.9 mm compared to the measured displacement of 30.0 mm. This means that the performance of the translational loop closure was better than 1 mm over 3 cm (0.33%).
2.4 Discussion This study presented an algorithm and experimental testing results of a C-arm base-tracking system. The method offers a robust, accurate, retrofittable and inexpensive C-arm base tracking solution that can be easily added to any C-arm (together with an interface). The only employed sensory unit in this system is a consumer-grade monocular camera, which makes the setup capable 59
of being integrated with existing devices. The performance of this system was examined under different situations to evaluate the final tracking results for actual manipulation. As shown by the experimental results, one can expect the system to provide final translational and rotational accuracies of less than 2% (with respect to the total travelled distance and rotation) in most of the cases studied. The experimental analysis revealed that the higher the distance of the camera to the underlying floor, the better the accuracies of the system. This is due to the fact that, as the camera increases its distance from the underlying floor, its field of view will increase resulting in observing more trackable features and consequently higher redundancy in estimation. As the camera increases its distance from the floor, the observation scale decrease, resulting in less sensitivity of the recovered pose to the odometry errors. However even for the camera distance of 16.5 cm, the system performed with external precision of better than 2% for most of the cases and less than 4% for all the cases (almost 4% for H2 and for displacement along the y axis). The movement velocity was shown to have a direct impact on the tracking errors, meaning that the correctness of the recovered platform pose was higher for lower movement velocities. This is due to the significant motion blur appearing in the acquired frames at higher speeds. Therefore, higher temporal resolutions (frame rates) are highly recommended for future implementations. Thanks to the robustness of the KLT optical-flow estimator after tuning the parameters of the feature extractor, this method reached the acceptable ranges of accuracy for different floor textures. The proposed algorithm requires substantial texture to be present in the floor underneath and might underperform on homogenous surfaces. It can be claimed, however, based on Section 2.3.6, in which considerable texture was found on real operating room floors, that this system is capable of providing robust base tracking results in real clinical applications. 60
The error propagation analysis indicated that the expectable accuracies of the system depend highly on the precision of the object point measurement (𝜎𝑔 ), marker extraction (𝜎𝑝 ) and optical flow estimation (𝜎𝐾𝐿𝑇 ). Periodical non-linearities in error curves (Figure 2-9) are attributable to the lower rectification precision for the points located out of the valid image periphery. This implicitly demonstrates the significance of extracting new features when the old points start to move out of the valid margins (4/3 of the margin for which the perspective rectification is performed, simulated imaging area). A 10 cm error range over 6 m displacement conducted in 150 s was defined as the performance norm to be compared with the existing systems found in the literature. Comparing the accuracy of the proposed method with the current VO systems found in the literature (19.6 cm error over 6 m of displacement (Campbell, et al. 2005); 76.5 cm error over 150 s of movement (Forster et al. 2014) and maximum error range of ~6% (Silva et al. 2012)) and considering the application-specific criteria (that limits the performance of a forward-looking camera) it can be concluded that this method is capable of providing either more accurate or more appropriate solutions for the C-arm base tracking. According to Table 2-1 and based on the results reported in Figure 2-10 and Figure 2-11, it can be concluded that the relative odometry results can be utilized individually to satisfy the expected accuracies when the total movement of the device is less than 3 m. This is enough for most of the clinical cases (as mentioned before a C-arm is moved a maximum of 4 m during a standard orthopaedic procedure); however, for longer movements, a loop closure procedure (described in Section 2.2.1.8) is recommended by incorporating ground-truth fiducials (landmarks) at 3 m intervals within the relevant areas (e.g. in the desired position at the surgical position, and
61
in the retracted position away from the table, or located field-of-view widths apart along the surgical table). Each of the steps of the algorithm is based on well-established techniques, but the combination of techniques is unique to our knowledge. The proposed odometry package is capable of providing real-time and robust localization estimates with minimum practical and computational requirements. The key concept of this method is to take advantage of the application-specific practical susceptibilities: the ability to have a downward-looking camera rigidly attached to the body of the platform that has a constant distance to the floor as well as the need only to compute 2D location parameters instead of the full 6 DOF pose. This avoids SFM and SLAM techniques for the purpose of VO. Instead a homography-based framework is provided that best fits the conditions and requirements of the confronted dead-reckoning scenario. The final positioning results are in meaningful (e.g. metric) units and have higher accuracies compared to current techniques. For the reasons explained in the Introduction, compared to existing VO, SFM and SLAM methods, this system requires considerably lower computational resources and can easily be coded with a micro-controller. This is an important factor to consider when integrating the base-tracking system with a joint-tracking system such as the TC-arm. The current system and evaluation study have several limitations. The system currently operates in stop-and-go off-line mode (due to limitations in the camera communication SDK); development of an on-line version is scheduled for future phases of the research. Parallel processing methods will be of interest in future implementations of the software to allow for realtime processing. Another main limitation for the proposed method is the non-intelligent trackablefeature selection across the acquired frames. This can be developed and replaced with selective 62
approaches to guarantee the diverse presence of features all over the successive images. The performance of the feature detection method was evaluated on real OR room floors, the overall performance of the tracking system has not been tested in a real clinical situation. By looking at the odometry results it can be hypothesised that there is a linear trend in the tracking errors. A more comprehensive repeatability analysis has to be performed to confirm this hypothesis (and to improve the accuracy of the proposed method consequently by introducing the inverse of the modeled linear trend to the odometry solutions). Once the on-line development of the tracking software is accomplished. As shown by undistortion evaluation analysis (described in section 2.3.1), there is a considerable improvement in the image quality (in terms of presence of optical-based distortions), however, usage of more robust undistortion procedure is aimed in the future phases of the research to eliminate the effect of more distortion and consequently higher odometry results. High localization accuracy of this method can be of interest for any application in which real-time location of a moving platform is needed. This method satisfies the required clinical base tracking accuracies as long as either (a) the excursion distance is kept within 3 m, and a reasonably slow velocity is used, or (b) loop closure (absolute tracking) is used to reset the location once the reference is within view. This system is expected to be integrated with the TC-arm system and other C-arm technologies, including the C-Pilot, in the near future to deliver a full degree-offreedom C-arm tracking system and quantitative analysis capability to improve the surgical outcome.
63
A Fast, Accurate and Closed-Form Method for Pose Recognition of an Intramedullary Nail using a Tracked C-Arm In a separate development from the base-tracking, a specific application of a navigated Carm, for intramedullary (IM) nail fixation, was developed and tested.
3.1 Introduction Intramedullary nail fixation is a common technique in orthopaedic trauma surgeries. A rod is placed into the canal of the patient’s fractured bone and fixed using metal screws (Figure 1-6). The action of fixing the IM nail is challenging since the surgeon cannot see the distal holes in the rod after insertion. During a typical IM-nail insertion surgery, both the surgical staff and the patient are exposed to considerable radiation (Neatpisarnvanit et al. 2006). One study reports an average radiation time of 4.6 minutes resulting in average radiation doses of 1.27 mSv to the surgeon and 1.19 mSv to the first assistant (Muller et al. 1998). This is a particular concern since radiation dose is cumulative over each surgery. The purpose of this study is to propose a navigation system for IM-nail distal locking surgery that helps to perform precise operation with low time and radiation consumption. Different surgical techniques currently exist to reduce the amount of radiation dose and operation time, each having advantages and disadvantages. The most prevalent methods are: (1) nail-mounted guide; (2) 3D preoperative imaging; (3) surgical navigation; and (4) electromagnetic tracking, briefly described in the following. (1) The nail-mounted guide technique uses the proximal portion of the IM nail as the reference to register the distal part (Figure 3-1). The main source of error is the deformation of the IM-nail during surgery: the range of deformation for a 13 mm-diameter slotted nail can be up to 15 mm in translation, 9 in flexion, and 14 in torsion
64
(Krettek et al. 1998). Considering the forces applied (from the surgeon, thermal change and the patient’s cortical bone reaction forces), this method does not provide reliable results for distal locking guidance. Some studies have tried to address the deformation issue by calibrating and compensating for the deformations (Krettek et al. 1997). Some innovative designs of the IM rod provide a means for locating the holes regardless of the deformation that takes place along the length of the rod (Krettek et al. 1998; Abdlslam et al. 2003). The deformation issue is still a challenge with this technique. (2) Three-dimensional imaging modalities such as computed tomography (CT) or magnetic resonance imaging (MRI) have been introduced to provide static information of pathologic or anatomic parts in the preoperative stage. However, their intraoperative performance is limited because of radiation exposure, time consumption and high expense (Zhu et al. 2001). (3) Computer aided navigation modalities for IM-nail insertion surgeries have also been introduced (Viant et al. 1997; Suhm et al. 2000). These methods utilize calibrated fluoroscopic images and register the position of the image intensifier at the time of exposure (relative to a global coordinate system) by using a calibration phantom as well as an optical tracking system. Several X-ray shots are acquired from the IM-nail and a 3D model of the nail subsequently created. A silhouette projection of the IM-nail’s distal part is created in real time; the operator can then continuously change the C-arm view until the virtual distal holes appear as perfect circles. The operator then takes an X-ray image to confirm the recovered pose. Although this method provides a practical solution for reducing time of surgery, it has several limitations such as: the requirement for line of sight for the optical tracking system; cost; and issues regarding projection of the calculated location of the tools onto the X-ray, which is a significant source of error. (4) The Trigen Sureshot Distal Targeting System has been recently introduced (Smith&Nephew, Inc., Memphis, TN, USA). This system utilizes electromagnetic field tracking 65
technology with a sensor on the probe inserted into the nail. Virtual imagery of the distal portion of the nail is then projected onto a screen to provide real-time feedback without the need for fluoroscopy. The main disadvantage of this system is its high expense and additional instrumentation. (5) Image based IM-nail localization techniques have also been introduced (Leloup et al. 2008 and Zheng et al. 2008). These methods have several limitations such as: requirement of physical landmarks attached to the imaging device and bone, the need for projection of calibration fiducial landmarks in the X-rays, high computation time and computational complexity. 𝒁𝑶𝒃𝒋 𝒀𝑶𝒃𝒋 Yaw Pitch Roll
O
𝑿𝑶𝒃𝒋
Distal holes
Figure 3-1: Distal part of the IM-nail and the local object coordinate system.
The limitations and disadvantages of the described techniques result in most surgeons still choosing to use the freehand technique. The freehand technique consists of the following steps: (1) aligning the intensifier to acquire an X-ray shot in which the distal holes appear as a complete circle (one study reported 77 s of radiation technologist (rad-tech) wait time plus 105 s of setup time per fixed nail (Chan et al. 2013)); (2) utilizing a sharp trocar to point to the centre of the circle; (3) using the trocar to penetrate the lateral cortex; and (4) performing the drilling based on the created reference. Multiple drillings are often needed, but subsequent drillings may follow the 66
first drill hole and repeated drilling can damage the cortex and cause weak fixation (Knudsen et al. 1991). Freehand techniques for finding the distal holes are mainly based on fluoroscopic imaging (C-arms) in a trial-and-error manner. Trying to achieve the X-ray shot in which the distal hole appears as a complete circle causes considerable irradiation to the surgical staff and patient: radiation exposure to the surgeon in IM-nail insertion surgery is in the range of 3.1 min to 31.4 min per surgery, in which 31% − 51% of the total irradiation is just for the distal locking step (Leloup et al. 2008). Long-term use of C-arms can lead to cancer or cataracts in the surgical staff (Harstall et al. 2005). For any C-arm pose that does not result in the ideal view of the IM-nail, the distal holes will appear as ellipses; the specific elliptical shape depends on the viewpoint. Once the ideal view is acquired, the remaining steps associated with screw fixation are straightforward. The overall goal of our research is therefore to design and develop an IM-nail distal locking navigation technique that leads to more accurate and faster screw placement with less radiation dose, and with a minimum number of added steps to the operation, to make it more accepted within the orthopaedic community. The specific purpose of this study was to develop and validate an automated technique for identifying the current pose of the IM nail relative to the C-arm.
3.2 Methods The IM-nail’s pose is computed from only two biplane X-rays and reported to the user (surgeon/radiation technologist) relative to a global coordinate system determined from a previously-developed tracked C-arm system (Amiri et al. 2013). The overall work flow, illustrated in Figure 3-2, shows the inputs, outputs and computations within the algorithm. These are
67
described in turn below after an initial description of the coordinate systems and tracked C-arm used.
Biplane X-rays
Feature Extraction
Calibration Files
Tracking System
Pose Estimation
C-arm Movements to Achieve Desired Image
User Interface
User Confirmation
Figure 3-2: Overall work flow. Red: Inputs; Blue: Computations; Green: Outputs.
3.2.1 Coordinate System Definition The image coordinate system (𝑖𝑚) is the original coordinate system in which the pixels are represented. The image local coordinate system (𝐼𝐿) is obtained by moving the 𝑖𝑚 coordinate system into the centre of the image and converting the units from pixels to millimetres. For the image global coordinate system (𝐼𝐺), the origin of the coordinate system is located on the principal point and its Z-axis is parallel to the image normal (Figure 3-3). The global coordinate system (𝐺) is obtained after performing the calibration procedure using a custom-designed phantom (Amiri et al. 2013). The origin of the local object coordinate system (Obj) is at the centroid of the upperlower distal holes set on the nail axis (Figure 3-1). The X-axis of this coordinate system is along the nail’s longitudinal axis; the Y-axis is in the direction of a vector from the origin that passes through the center of the upper distal hole; and the Z-axis is defined to complete the orthogonal right-handed system. These coordinate system definitions are summarized in Table 3-1.
68
Figure 3-3: Configuration of the different coordinate systems.
Table 3-1: Coordinate system definitions. Coordinate System im IL IG Obj G
Origin
X-axis
Y-axis
Z-axis
Units
upper left image centre principal point O (Figure 3-1) from calibration
Xim XIL XIG Xobj XG
Yim YIL YIG Yobj YG
NA NA image normal Zobj ZG
pixels mm mm mm mm
3.2.2 Tracking System The tracked C-arm (TC-arm) system previously developed by our research group (Amiri et al. 2013) utilizes two inertial measurement units (IMUs) attached to the gantry of the C-arm to measure its orbit, tilt and wig wag, as well as two laser beam sensors to measure up-down and inout movements of the gantry (Figure 3-3). The comprehensive calibration protocol and phantoms provide full three-dimensional spatial information of the camera and image intensifier for any
69
arbitrary image acquired by the TC-arm system. After capturing two bi-planar X-rays, the corresponding calibration files describing each image’s intrinsic and extrinsic parameters are interpolated from the previous overall calibration of the imaging space (Figure 3-4).
Figure 3-4: Calibration parameters: principal distance 𝒅 (physical distance from X-ray source to projection plane), principal point (𝒙𝑰𝒑𝒑 , 𝒚𝑰𝒑𝒑 , ) (perpendicular projection of the Xray source onto the projection plane), pixel size 𝒓 (resolution), position of the X-ray source ⃗⃗ and (𝑿𝑮𝑺 , 𝒀𝑮𝑺 ) (relative to the phantom’s global coordinate system), image normal vector 𝒏 ⃗⃗ (the vector that indicates the planar orientation of the X-ray). image up vector 𝒖
3.2.3 Biplane X-ray Acquisition The first step in the procedure is to acquire two calibrated X-ray images of the distal part of the nail. As described above, the TC-arm system provides the full description of the calibration information for any acquired image based on a comprehensive offline calibration process.
70
Accordingly, each of the X-ray images (i.e. intensifier-detector set) can be computationally positioned into the equivalent position as occurred at the time of exposure.
3.2.4 Feature Extraction If enough information were available about conjugate features in the two acquired X-rays, a simple ray-intersection procedure could be performed. However, this is not the case. The acquired IM-nail images do not have enough texture due both to homogeneity of the IM-nail material and to their simple, cylindrical design. However, pseudo-features in both of the X-ray shots can be extracted thanks to the specific geometric design of IM-nails. Pseudo-features are calculated points that are not physically present in the IM-nail, but can be found in common between the two views. After image acquisition, a moving average low-pass filter (with window size of 5 × 5) is applied to reduce the effects of random errors in the image. The Canny edge detection (with a threshold of 0.1 and a value of 1.5) is then performed using the built-in MATLAB (R2013a; MathWorks®, Natick, Massachusetts, US) function to detect the abrupt changes in intensity for the sake of feature extraction. The circular projections of both the upper and lower distal holes are then detected using the circular Hough transform (Hough et al. 1962; Duda et al. 1972). For each projected distal hole, there are three parameters of interest that must be calculated: the location of the centre (𝑥0 , 𝑦0 ) and the radius 𝑟. For a given edge pixel (𝑥, 𝑦) lying on the circular projection of a distal hole, the following transformation is performed so the circle is represented in polar coordinates.
71
(𝑥 − 𝑥0 )2 + (𝑦 − 𝑦0 )2 − 𝑟 2 = 0
(1)
𝑥 = 𝑥0 + 𝑟𝑐𝑜𝑠(𝜃); 𝑦 = 𝑦0 + 𝑟𝑠𝑖𝑛(𝜃)
(2)
The created 3D Hough image (Figure 3-5-C) is then searched for the peak. The prior knowledge about the sizes of the distal holes was used to reduce the Hough space dimensions. The coordinates of the peak location correspond to the parameters of interest for the projected distal holes.
B
A
C
D
Figure 3-5: A) low-pass image, B) edge image, C) Hough image for a specific radius (21 pixels), D) detected circles.
3.2.5 Pose Recognition In order to transform each measured pixel’s coordinates from the local image coordinate system (IL) to the global coordinate system (G), the image global (IG) coordinate system is defined as follows (Figure 3-3) (see Fig. 3-4 for parameter definitions):
72
𝑋𝐼𝐺0 = −𝑑 𝑛𝑥 + 𝑋𝑠𝐺 ; 𝑌𝐼𝐺0 = −𝑑 𝑛𝑦 + 𝑌𝑠𝐺 ; 𝑜𝑟𝑖𝑔𝑖𝑛: ( ) 𝑍𝐼𝐺0 = −𝑑 𝑛𝑧 + 𝑍𝑠𝐺
(3)
𝑋 𝑎𝑥𝑖𝑠: ⃗⃗⃗⃗⃗⃗⃗⃗ 𝑋𝐼𝐺 = 𝑛⃗⃗ × 𝑢 ⃗⃗ ; 𝑌 𝑎𝑥𝑖𝑠: ⃗⃗⃗⃗⃗⃗⃗⃗ 𝑌𝐼𝐺 = 𝑢 ⃗⃗ ; 𝑍 𝑎𝑥𝑖𝑠: ⃗⃗⃗⃗⃗⃗⃗⃗ 𝑍𝐼𝐺 = 𝑛⃗
(4)
For a given measured image pixel lying on the circumference of the projected distal hole (𝑥𝑝𝑖𝑚 , 𝑦𝑝𝑖𝑚 ), the corresponding local image coordinates are calculated as follows: 𝑥𝑝𝐼𝐿 = (𝑥𝑝𝑖𝑚 − 𝑡⁄2) 𝑟, 𝑦𝑝𝐼𝐿 = (𝑦𝑝𝑖𝑚 − 𝑠⁄2) 𝑟
(5)
𝑖𝑚 axes, respectively. Each measured point ⃗⃗⃗⃗⃗⃗⃗⃗ where 𝑡 and 𝑠 are the number of pixels along ⃗⃗⃗⃗⃗⃗⃗⃗ 𝑥 𝑖𝑚 and 𝑦
is then transformed into the global coordinate system (𝑥𝑝𝐺 , 𝑦𝑝𝐺 and 𝑧𝑝𝐺 are the global coordinates of the measured point): 𝑥𝑝𝐺 ⃗⃗⃗⃗⃗⃗⃗⃗ . ⃗⃗⃗⃗⃗⃗ 𝑋𝐼𝐺 𝑋𝐺 𝐺 [𝑦𝑝 ] = [𝑋𝐼𝐺 ⃗⃗⃗⃗⃗⃗⃗⃗ . 𝑌𝐺 ⃗⃗⃗⃗⃗⃗ 𝐺 ⃗⃗⃗⃗⃗⃗⃗⃗ . ⃗⃗⃗⃗⃗⃗ 𝑧𝑝 𝑋𝐼𝐺 𝑍𝐺
⃗⃗⃗⃗⃗⃗⃗⃗ . ⃗⃗⃗⃗⃗⃗ 𝑌𝐼𝐺 𝑋𝐺 ⃗⃗⃗⃗⃗⃗⃗⃗ . 𝑌𝐺 ⃗⃗⃗⃗⃗⃗ 𝑌𝐼𝐺 ⃗⃗⃗⃗⃗⃗⃗⃗ 𝑌𝐼𝐺 . ⃗⃗⃗⃗⃗⃗ 𝑍𝐺
⃗⃗⃗⃗⃗⃗⃗⃗ . ⃗⃗⃗⃗⃗⃗ 𝑥𝑝𝐼𝐿 𝑋𝐼𝐺0 𝑍𝐼𝐺 𝑋𝐺 𝐼𝐿 ⃗⃗⃗⃗⃗⃗⃗⃗ . 𝑌𝐺 ⃗⃗⃗⃗⃗⃗ ] × [𝑦𝑝 ] + [𝑌𝐼𝐺0 ] 𝑍𝐼𝐺 𝑍𝐼𝐺0 ⃗⃗⃗⃗⃗⃗⃗⃗ 𝑍𝐼𝐺 . ⃗⃗⃗⃗⃗⃗ 𝑍𝐺 0
(6)
At this stage, the position of the image intensifier (𝑋𝑆𝐺 , 𝑌𝑆𝐺 , 𝑌𝑆𝐺 ), obtained from the calibration procedure, and the position of each measured distal circle centre are known. Having conjugate points in both images, the corresponding rays can be analytically reconstructed to obtain the object space coordinates. Although each individual upper and lower distal circle might appear as an ellipse rather than a complete circle (even without the occlusion effect), their projections can be approximated as circles within the X-ray because of their small dimension relative to the principal distance. The image-processing module calculates the centres of all the four projected distal holes (upper and lower holes in each X-ray). This method requires two sets of X-ray images taken from two different sides of a plane containing the four holes in the distal part of the IM-nail. This helps us to find the corresponding projected distal-hole centres in the biplane images. The Xrays are cropped by the boundaries in which all the projected distal holes are apparent. 73
In order to find the corresponding projected distal holes in the two acquired X-rays, the following steps are performed: 1) For each detected distal point (centre of the projected distal circle), we find another distal point which has the minimum Euclidean distance to it; these two points are then defined as the projected centres of the upper and lower distal-hole sets. 2) The centroid of each upper-lower distal-hole set is then calculated (projection of the point 𝑂 in Figure 3-1). 3) A line is fit to these 6 points (upper holes, lower holes and centroids of these), by least squares regression, which approximately represents the IM-nail’s axis; 4) The line perpendicular to the fit line at its mid-point is constructed; 5) The side of the fit line on which a distal point is located (the same analysis is done for the perpendicular line) is determined by utilizing Eq. 7. For a given point a line passes through points 𝐴(𝑋𝐴 , 𝑌𝐴 ), 𝐵(𝑋𝐵 , 𝑌𝐵 ) and 𝐶(𝑋𝑐 , 𝑌𝑐 ) and 𝑇 value in Eq. 7 is formed; if it is greater than zero the distal point is located on one side of the line and vice versa; 𝑇 = (𝑋𝐵 − 𝑋𝐴 ) (𝑌𝐶 − 𝑌𝐴 ) − (𝑌𝐵 − 𝑌𝐴 ) (𝑋𝐶 − 𝑋𝐴 )
(7)
6) The 𝑇 values for the fit and the perpendicular lines are compared to determine in which quadrant a distal point is located. Because of the given imaging configuration, and considering each distal hole, the distal point should appear on different sides of the approximated projection of the IMnail axis in two biplane X-rays. This is the cue for our automatic point correspondence extraction. As seen in Figure 3-6, the proposed image-processing algorithm is independent of the actual orientation of the IM-nail in space. Different angles are simulated for any arbitrary IM-nail’s orientation by rotating the original X-ray. The intersection of X-rays for each extracted distal point is performed as shown in Figure 3-7.
74
B
A
C
D
Figure 3-6: Distal-hole centre tracking for A) 60º, B) +30º, C) -30º and D) +90º rotations of a single X-ray.
Figure 3-7: Intersection geometry.
Although, in theory the reconstructed rays for the projected distal points must intersect at the center of the distal hole in object space, because of possible calibration and random errors, they 75
form two skew rays. In order to minimize the effects of these errors on the pose estimation outcome, the centre of a line segment with the shortest length between the skew lines (Figure 3-7) is calculated. The reconstruction of the corresponding ray for the ith measured point (based on the jth X-ray image) can be accomplished by utilizing Eq. 8. 𝐺 𝑋𝑝𝑖 𝑋𝑠𝑗 𝑗 ⃗⃗⃗⃗ 𝑟𝑖 = [ 𝑌𝑠𝑗 ] − [ 𝑌𝑝𝑖𝐺 ] 𝑍𝑠𝑗 𝑍𝐺
(8)
𝑝𝑖
For a pair of conjugate points (in biplane X-rays j=1,2) the centre point of the line with the closest distance to the skew rays can be calculated and considered as the reconstructed object point. The intersection procedure is performed for all 6 conjugate points within biplane X-rays to recover the coordinates of all upper and lower distal hole centres as well as the centroid of each upperlower set. These coordinates are with respect to the global coordinate system, as defined by the Carm tracking system. By performing measurements on the IM-nail CAD models, the object space coordinates of the features of interest can be retrieved. At this stage, the coordinates of all upper, lower and centroid points (6 points) are known in the global coordinate system along and in the object’s local coordinate system (thanks to the acquired 3D model). Since each extracted feature in the acquired image corresponds to a real geometric component of the IM-nail’s distal part, and since the global positions of each of the described features is computed at this stage, transformation parameters between the local (IM-nail coordinate system) and global coordinate systems can be solved by having at least three non-collinear, corresponding points (because of the number of extracted features, we have six conjugate points) using Horn’s Method (Horn, 1987). The rigid body transformation parameters (3 rotations + 3 translations) between the object’s local coordinate
76
system and the global coordinate system are calculated in a closed-form manner. The same pose parameters are therefore produced for each software run for a specific set of biplane images. These transformation parameters represent the IM-nail’s pose relative to the global coordinate system, using the previously-described points.
3.2.6 Acceptable Error Tolerance Definition of an explicit acceptable error tolerance is necessary in order to evaluate the effectiveness of the proposed solution. For this purpose, the method presented in (Zheng et al. 2008) is utilized. This method presents a mathematical framework for computing the acceptable error range for pose recovery of an IM-nail with a known geometry and a drill bit with known radius. Considering an IM-nail with the distal opening height of 𝐻, distal hole radius of 𝑅, and a drill bit with radius of 𝑟, the pose recovery error parameters (translational: 𝑑𝑇; Rotational: 𝛼) have to satisfy the following conditions: {
𝑑𝑇 +
𝑟 𝐻 + . tan(𝛼) ≤ 𝑅; 𝑎𝑛𝑑 𝑐𝑜𝑠𝛼 2 𝑑𝑇 ≤ (𝑅 − 𝑟)
Figure 3-8 demonstrates a schematic cross-section view of an IM-nail and drill bit.
77
(9)
Figure 3-8: Schematic cross-section view of an IM-nail and drill bit.
Figure 3-9 shows the error tolerance for an IM-nail (𝑅 = 2.5 mm, 𝐻 = 10 mm) and different drill bit radii. The area under each of the presented curves corresponds to the acceptable error range. The nail dimensions (𝑅, 𝐻) are taken from the nails utilized in the experimental study of this research. For a drill diameter of 1.5 mm, the maximum acceptable errors are 1 mm if there is no angular error or 11º if there is no translational error, or less when both have errors. For a drill diameter of 2 mm, the corresponding maximum acceptable errors are 0.5 mm or 6º or less for a combined error.
78
Acceptable error tolerance Rotational Error (o)
14 12 10 8 6 4 2 0 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Translational Error (mm) r=1.25 mm
r = 1.5 mm
r=1.75 mm
r=2 mm
Figure 3-9: Acceptable error tolerance for different drill bit diameters for a nail with 𝐡𝐨𝐥𝐞 𝐫𝐚𝐝𝐢𝐮𝐬, 𝑹 = 𝟐. 𝟓 𝐦𝐦, and hole height 𝑯 = 𝟏𝟎 𝐦𝐦.
According to the symmetrical design of the distal holes of an IM-nail, the recovered acceptable error tolerance can be applied for all 3D pose parameters except the 𝑦𝑎𝑤 rotation, which is the rotation along the distal hole axis. It is important to note that the 𝑦𝑎𝑤 rotation does not have any impact on the distal hole localization due to the very nearly circular design of the distal hole openings.
3.2.7 Experimental Methods Experiments were performed with an Arcadic Orbic Iso-C C-arm (Siemens AG, Munich, Germany) retrofitted with the TC-arm system reported in (Amiri et al. 2013). Two
different
metal
IM-nail
models
(A: 10 mm × 34 cm Smith&Nephew,
B: 10 mm Synthes Expert) were utilized for the purpose of pose estimation in this research. The 79
only information needed about the geometry of each nail is the relative configuration of the distal hole centres. The first round of experiments was performed by inserting IM-nail A into a plastic foam cube having an arbitrary orientation in space (Figure 3-10), after which numerous bi-plane X-ray shots were acquired from the IM-nail’s distal part using the tracked C-arm. In the second round of experiments and in order to simulate the actual application scenario, IM-nails A and B were inserted sequentially into a cow femur surrounded by animal soft tissue (to replicate the effect of bone and soft tissue artifacts in the X-rays; Figure 3-11). An additional trial tested the ability to extract the required features in the presence of other surgical instruments.
Figure 3-10: Experimental setup.
80
Figure 3-11: Simulation of soft tissue artifacts
The nail CAD models were created as following: Nail A: the CAD model was generated based on the measurement of the nail dimensions with a vernier caliper (this was sufficient for the relatively simple geometry). Nail B: the nail was 𝜇𝐶𝑇-scanned and the CAD model was produced by performing segmentation and surface fitting procedures.
Relevant coding and user interface development was first done in MATLAB (R2013a; MathWorks®, Natick, Massachusetts, US) software. A later version of the software was developed with OpenCV (Version 2.7) library in Visual C++ 2010 interface (X86, Microsoft Corporation) to reduce the processing time. Image processing and pose computation is accomplished through a user-friendly interface (Figure 3-12). The user first uploads the biplane X-rays to the software with the corresponding calibration files. Then image processing and conjugate feature extraction is performed and the results shown to the user. The image-processing algorithm is entirely automated, saving a 81
substantial amount of time and producing more accurate results for the pseudo-feature extraction. The custom user interface provides a double-check option to the user (surgeon/radiation technologist); if the user chooses not to accept the result, then an enhanced, processed image will be displayed on the monitor so that the pseudo-features can be manually selected by clicking on them. After feature extraction, the 3D, global position of each feature of interest is computed and the results shown in relevant sections of the software. Possible errors (e.g. in introducing the images and calibration files) and warnings as well as computation time are shown in a message box.
Figure 3-12: User interface showing: the biplanar X-ray views, automatically-detected pseudo-features, option to accept the automated result or not, and the resulting calculated translations and rotations of the current view relative to the desired view. In the desired view, the front and back holes line up as circles. 82
Validation was performed by capturing biplanar images of the IM nail, and comparing the calculated 6 degree-of-freedom data to that derived by fitting the 3D model of the IM-nail using 2D-3D matching techniques in the image using JointTrack biplane open-source software (Acker et al. 2011). In order to test the robustness of the solution, we back-transformed the points in the local object coordinate system into the global coordinate system using Horn’s rotation and translation matrices. Considering the nearly linear configuration of the object points, another robustness test was performed by artificially transforming the object points and trying to recover the transformation parameters. The ability of the proposed method in identifying the projected distal holes was examined under a number of conditions: (1) with animal tissue (to generate the effect of bone and soft tissues in the acquired X-rays); (2) with other surgical instruments in the fluoroscopic field of view; (3) with different geometries (IM-nails A and B); (4) with different separation angles; and (5) with repeated measurements, in which two sets of biplanar X-rays (with equal separation angles) from a single nail geometry were acquired in different rounds of the experiment (with and without bone and soft tissue artifacts). An investigation about the computation time was performed using a computer with the following specifications: Intel® core™ i5 processor, 7.87 GB of random access memory and 64bit Windows 7 operating system.
83
3.3 Results The maximum fitting error, as judged by back-transforming the local to global coordinate system, was 0.48 mm (from 6 points and 4 trials). After comparing the recovered parameters to the original (arbitrary) ones, results from the Horn’s method appeared accurate for the described configuration, resulting in the same parameters for the original and transformed sets. Results of the feature extraction in the presence of bone and soft tissue artifacts in the acquired fluoroscopic images show reliable performance of the feature extraction model in this case (Figure 3-13).
Figure 3-13: Bone and soft tissue artifact scenario.
The test of the ability of the proposed method in identifying the distal holes in case of projection of other surgical instruments within the X-ray illustrated the independent performance of the feature extraction method in case of metallic surgical instruments appearing inside the Xrays (Figure 3-14). 84
Figure 3-14: Surgical instrumentation scenario.
The results of the first round of experiments (with IM-nail A), with multiple separation angles, demonstrated translational accuracies better than 0.5 mm in all three directions, rotational accuracies for roll and pitch better than 1º and rotational accuracy for yaw better than 1.5º (Table 3-2).
85
Table 3-2: Pose recovery results. There is only one value for any given set of biplanar images due to the closed-form solution.
Computed Orientations (º)
Computed Translations (mm)
Separation Angle (º)*
45º
35º
30º
25º
Truth**
Tx
-16.97
-16.89
-16.95
-17.02
-17.34
∆𝐓𝐱
0.37
0.45
0.39
0.32
Ty
-36.73
-36.93
-37.15
-36.71
∆𝐓𝐲
0.22
0.02
-0.2
0.24
Tz
12.64
12.64
12.64
12.62
∆𝐓𝐳
0.46
0.46
0.46
0.44
Roll
11.08
11.34
10.8
10.87
∆𝐑𝐨𝐥𝐥
0.10
0.36
-0.18
-0.11
Pitch
-86.94
-86.69
-87.27
-87.29
∆Pitch
-0.39
-0.14
-0.72
-0.74
Yaw
0.78
0.87
0.69
0.78
∆Yaw
1.40
0.71
1.31
1.40
-36.95
12.18
10.98
-86.55
-0.62
* Separation angle = the biplanar angle between the two positions of source-detector sets for two X-ray shots. ** True values based on JointTrack biplane fitting of a 3D model of the IM nail to the 2D X-rays.
The recovered pose parameters from the second round of experiments (with IM-nails A and B) showed that the proposed method is capable of achieving accurate results for both of the investigated geometries (Table 3-3).
86
Table 3-3: Pose recovery results for the second nail (B) geometry. Tx
Ty
Tz
Roll
Pitch
Yaw
-5.73
8.67
-9.02
-8.70
-179.46
-81.04
50o
-5.57
9.01
-8.67
-8.23
179.02
-82.50
150o
-5.51
9.34
-8.59
-8.06
178.59
-83.33
50o
0.16
0.34
0.35
0.47
1.52
-1.46
150o
0.22
0.67
0.43
0.64
1.95
-2.29
Truth
Recovered
Error * True values based on JointTrack biplane fitting of a 3D model of the IM nail to the 2D X-rays.
The results of the repeatability analysis (shown in Table 3-4) illustrate the repeatable performance of this method under different imaging conditions.
Error
Recovered
Truth*
Table 3-4: Repeatability analysis results.
Tx
Ty
Tz
Roll
Pitch
Yaw
Round I*
-17.34
-36.95
12.18
10.98
-86.55
-0.62
Round II**
-3.75
6.96
7.06
-8.96
87.90
-87.85
Round I
-16.98
-37.15
12.64
10.80
-87.27
0.69
Round II
-3.67
7.27
6.78
-8.57
88.05
-88.06
Round I
0.39
0.24
0.44
-0.11
-0.74
1.4
Round II
0.08
0.31
-0.28
0.39
0.15
-0.21
* True values based on JointTrack biplane fitting of a 3D model of the IM nail to the 2D X-rays. * Without bone and soft tissue artifacts, * With bone and soft tissue artifacts.
87
The computation time of the proposed method was in the range of 0.4 s to 0.6 s and less than 0.5 s on average.
3.4 Discussion This novel analysis technique provides an accurate and robust method for determining the pose of an IM nail in C-arm views. In terms of reconstruction accuracy, the described method seems very promising and appears to satisfy the needs of orthopaedic trauma surgeries (according to the analysis described in 3.2.6). Since only two images are required to perform the navigation computations, this method can be easily adapted into current orthopaedic trauma surgeries to have better and more accurate outcomes of IM-nail fixation. The X-ray calibration only needs to be performed once for each C-arm, and does not require a patient-attached reference base, important advantages over previous techniques. The computation time (0.5 s) of this method is considerably lower than the time needed to find the distal holes using current surgical method and can be reduced even further in future implementation of the software using a different integrated development environment (IDE). The evaluation study showed that this method is capable of pose recovery of an inserted IM-nail with translational accuracies of better than 0.5 mm and rotational accuracies of better than 2o for roll and pitch and better than 2.5o for yaw. These ranges of error are lower than the clinically acceptable tolerance (as shown in 3.2.6).The recovered pose data can be incorporated to place the C-arm source to a position in space from which the distal holes appear circular, however, this method is capable of incorporating more X-rays after this stage (for the purpose of fine localization) to improve the pose recognition results. An IM-nail navigation system is reported in (Leloup et al. 2008) with comparable accuracies (translational: 1.5 mm; rotational: 1o), however there is a requirement for presence of physical 88
landmarks attached to the C-arm and the nail (for 3D optical tracking). This method does the calculations within (500 𝑠). The system reported in (Zheng et al. 2008) requires a dynamic reference base rigidly attached to the bone and has a computation time of around 14 𝑠. Both of these methods require projection of calibration fiducial marks in the acquired X-rays. Another great achievement of this work is the closed-form concept of the solution, instead of the iterative or optimization based solutions provided in the previous techniques. These demonstrate higher practical potentiality of the proposed method compared to the related methods in the literature. A magnified image may improve the accuracy even further. Each additional minute added to the time of surgery will notably increase the operation expenses and increases the patient time under anaesthetic. The described system could reduce operating time and radiation dose by reducing the amount of trial and error that currently takes place. The radiation dose time involved in IM-nail navigation using our method is remarkably lower than the time needed to find the distal holes in current methods of orthopaedic trauma surgery. The main limitations of the validation study are that it was only performed for two IM nail geometries and was performed on animal (instead of human) specimens; other geometries will be tested in the future. The next phase of this research is to validate the performance of the proposed method in real clinical scenarios and test it with surgeons and rad-techs. Another limitation associated with the proposed method is the need to have biplane X-rays on either side of the plane containing the four distal points. In conclusion, distal locking of intramedullary nails for fracture fixation is challenging and typically done by a trial-and-error technique using C-arm imaging. Our automated technique of identifying the current pose of the C-arm compared to that required to achieve the ideal view should improve accuracy, reduce operating time and radiation dose, and improve patient outcome. 89
90
Discussion and Conclusions
4.1 Summary and Overview C-arms are sophisticated X-ray machines that are frequently used during orthopaedic procedures, but currently lack the ability to provide quantitative data and are only used as a qualitative assessment tool. They lack sufficient software and hardware that can be supplemented to the C-arm devices to take the available fluoroscopic data into the next phase from which the surgeons can estimate the alignment of the body organs and implants, can perform measurements on the X-rays and can generate long-range panoramic X-ray views and so on. The primary motivation of this research was to enable the currently used C-arms to provide this information by adding several retrofittable modules with minimum hardware and software requirements. The data from a calibrated (tracked) C-arm can be used for 3D reconstruction in addition to standard 2D imaging (Brost et al. 2009). A tracked C-arm is capable of tagging any acquired Xray with its corresponding positional (and orientation) data at the time of exposure. This plays an important role in any 3D reconstruction applications. Localization (tracking) of a C-arm can be divided into two different categories: first, to track the degrees of freedom associated with the mechanical joints in 3D space (joint tracking); and second, to localize the movements of the base of the device in 2D space (relative to the operating room floor; base tracking). An IMU-based joint tracking system has been developed by our research group (Amiri et al. 2013). This system is capable of providing an accurate 3D localization of the C-arm transmitter-detector set in real time, however it is limited to a stationary C-arm base. This means that the C-arm wheels must be locked during the manipulation of the system. This assumption is, however, not true during orthopaedic surgeries in which there is a necessity of long-range movements of the X-ray acquisition device. 91
The first phase of this research was to investigate the performance of a novel odometry system as the C-arm base-tracking. The second phase was to investigate the performance of a novel application method (IM-nail fixation) for a fully tracked C-arm. A successful base-tracking system must be capable of providing accurate 2D navigation results (translational: better than 10 cm error for displacements less than 4 m of movement; rotational: better than 3o). The ideal system aims to be inexpensive and require the lowest possible number of sensors. An online solution is desired that adds a minimum time to the current surgery routines. Accurate performance of the potential tracking system is preferred for a variety of operating room floors. An effective localization technique must incorporate simple mathematical operators to ensure minimum computational resource requirements. This is essential for the final product since the software will be incorporated within a microcontroller framework. These criteria were designed to guarantee a clinically acceptable base-tracking system. Optical flow sensors (standard computer mouse sensors) were studied first for the purpose of base-tracking (Appendix A). These sensors were shown to be able to provide accurate localization (mean translational error of 1.77 mm and mean rotational error of 0.57o) for shortrange displacements (maximum range of movement: 76.5 mm). The dependence of measurement resolution on the distance from the underlying floor and the floor texture caused considerable error accumulation in longer ranges of movement. This emphasizes the fact that the sensory units must be calibrated for any type of operating room floor and can only perform accurately on homogeneous surfaces. Considering the desired criteria, it was concluded that the effectiveness of a system based on optical flow sensors would be limited under clinical situations. Visual odometry systems were then investigated for the specific application of this research. There are a variety of VO systems that are capable of providing real-time navigation 92
solutions for a moving platform. However, a careful selection and combination of the existing VO methods was essential to fulfil the practical requirements. Application-specific advantages of a Carm base tracking are: 1) the camera can be rigidly mounted (nadir looking) to the moving platform, which means that the relative alignment of the camera and the platform is kept constant during movement; 2) only the 2D pose is desired; and 3) a smooth and gentle movement of the Carm base is guaranteed by the C-pilot system. Considering the previously stated criteria and these advantages, the monocular visual odometry systems seemed to best fit the requirements of the application. Different monocular odometry methods such as SFM and SLAM have been introduced in recent years. Many of the SFM systems incorporate forward/side-looking cameras (e. g. Nistér et al. 2006). This will not be applicable for C-arm base tracking since there are a lot of moving objects (surgical staff and instrumentation) around the device that will interfere with feature tracking. There are a variety of omnidirectional camera-based monocular SFM methods available in the literature (e.g. Corke et al. 2004), which are not desired for indoor (and nadir looking) applications. In parallel to the SFM methods, SLAM techniques have been developed that aim to estimate the motion of a moving platform while simultaneously building and updating the environment map (Scaramuzza et al. 2008). These methods are also initially designed for outdoor applications (nonnadir looking cameras; Clemente et al. 2007). Requirement for high computational resources must be added to the limitations of the SFM and SLAM methods. A homography-based visual odometry system was introduced in Chapter Two as the means for C-arm base tracking. This system incorporates a rigidly attached, nadir looking camera with perspective architecture to report the real-time pose of the C-arm base. Feature correspondence was established using a well-known optical flow estimator and these data were guided through a 93
homography-based, 2D egomotion calculator. A loop closure procedure was also introduced and evaluated in this study. The accuracy assessment analysis showed that a maximum translational error of 4% (of the total traveled distance) over 6 m and a maximum rotational error of 2.7% (of the total absolute cumulative changes in the orientation) over 360º can be expected from the proposed method. If a loop closure procedure is used, requiring the appearance of ground landmarks within the camera’s field of view, the translational odometry results can be corrected to less than 0.3%. The proposed C-arm base-tracking system requires minimum computational and sensory resources and can be easily integrated with the TC-arm joint tracking system to provide the full range 6 DOF C-arm localization solutions. The main challenge in intramedullary nail fixation surgery is to find the position and orientation of the distal holes after insertion of the nail (into the bone canal). Considering the common usage of the C-arm devices in these routines, this is equivalent to finding the pose from which an X-ray shot can be acquired containing the completely circular projection of the distal holes. It is critical to accurately find this position to avoid multiple drillings (Knudsen et al. 1991) and to reduce the current considerable radiation exposure (Chan et al. 2013) to find the desired shot. In Chapter Three an accurate and robust method for determining the pose of an IM-nail based on C-arm imagery has been introduced. Since this system only requires a set of biplane X-rays from the TC-arm system, it can be easily adapted to the current orthopaedic routines. Comparing the current trial and error techniques, the proposed method will improve the accuracy, reduce the operating time and reduce the radiation dose associated with finding the distal holes.
94
4.2 Novelty and Contributions Optical flow sensor study (mouse study): A comprehensive performance analysis of optical flow sensors (standard computer mouse sensors) as the main sensory unit for an odometry application was identified to be lacking from the current literature; therefore, in Appendix A the relevant mathematical framework as well as an experimental analysis were provided. Although the evaluation analysis illustrated that these sensors are not adequate candidates for C-arm base tracking, the provided analyses fill the described gap in the literature since the previous studies either missed the analysis on real experimental data (Bonarini et al. 2004; Bell 2011; Palacin et al. 2006; Ross et al. 2012), or could not reach the same level of accuracy (Sekimori et al. 2007). C-arm base tracking (VO) study: The provided C-arm base-tracking solution is capable of accurately estimating the real time 2D position and orientation of the platform with minimal hardware and software resource requirements. Although each of the individual steps of the algorithm is built around a previously established technique, the generated algorithm is unique to our knowledge. The main idea of this method was to take advantage of the application-specific practical prerequisites to avoid a computationally expensive method (e. g. SFM and SLAM). The achieved accuracy level compares to the most relevant works: In (Campbell et al. 2005), a minimum 3.3% translational error (with no report on rotational accuracy) is reported for a forwardlooking camera (containing both the upper and lower parts of the horizon). A study reported in (Forster et al. 2014) introduces a fast monocular odometry system with 12.8% translational and 0.45 deg. s −1 rotational accuracy. Translational odometry error of 6% is reported in (Silva et al. 2012) for another forward-looking monocular odometry system. In comparison to these systems, the proposed method outperformed either in terms of accuracy or practicality since forwardlooking cameras are not a feasible VO configuration for the C-arm base-tracking applications since 95
there exist a lot of moving objects in the operating rooms (surgical staff feet and other surgical instruments) within a relatively short-range distance from the VO camera. If loop closure (absolute tracking) is used, then the accuracy of the proposed system is markedly better than previous systems, with only a small increase in implementation complexity. High localization accuracy of the proposed odometry system along with its low software and hardware resource requirements could be of interest for any other application in which the real-time location of a moving platform is desired. IM-nail pose estimation study: The proposed IM-nail localization method was shown to reach translational accuracy of better than 0.5 mm, roll and pitch rotation accuracy better than 2º and yaw rotational accuracy better than 2.5. This shows that the system can compute the pose of an inserted IM-nail with considerably lower error than the clinically acceptable tolerance. The computation time of this method is less than 0.5 s, which is a breakthrough improvement compared to the time currently spent to find the desired shot. An IM-nail navigation system is reported in (Leloup et al. 2008) with comparable accuracies (translational: 1.5 mm; rotational: 1o), however this method requires a physical landmark to be attached to the imaging device and the nail itself (for the purpose of 3D optical tracking). The computation time of this method (~500 s) is also noticeably higher than for the proposed method. In (Zheng et al. 2008) another IM-nail localization method is introduced, but is limited to a dynamic reference base attached to the bone and with a computation time of around 14 s. Requiring the appearance of fiducial marks in the acquired Xrays (for calibration) is an important limitation to both of these methods. Another important advantage of the proposed method over the existing techniques is the closed form (instead of iterative or optimization) concept of the provided solution. These illustrate that the IM-nail
96
localization method introduced in this thesis has higher practical potentiality and is more proficient in fulfilling the clinical requirements. The integration of the joint tracking system (TC-arm) with the proposed base tracking system will facilitate the quantitative use of C-arm devices for different orthopaedic procedures. In this research, one of the potential fields of application for a fully tracked C-arm has been investigated. This idea can be expanded to other surgical routines to improve the efficiency of C-arm imaging for the stages in which the current procedures result in high radiation time and exposure (e.g. pedicle screw placement in spinal fusion surgery and sacroiliac screw placement in pelvis surgery). Such a package could eventually reduce the radiation exposure and operating time and consequently reduce the risk of cancer or cataracts in the surgical staff.
4.3 Limitations Optical flow sensor study (mouse study): A potential improvement for the investigated system is to increase the number of optical flow sensors. The main limitation of this study was that this hypothesis has not been evaluated. Long-range displacements (e.g. more than 1 m) have also not been included in the accuracy assessment analysis. C-arm base tracking (VO) study: There are several limitations for the base tracking study of this thesis. The proposed system is currently limited to operate in stop-and-go off-line mode, whereas a completely real-time solution is preferred for the specific application of the study. Another limitation for the proposed method is the non-intelligent (trackable) feature selection across the acquired frames. This results in periodical accumulation of the trackable features in a particular part of the image. For example, if the platform is moving in the right direction, the extracted features (which are the main input to the pose estimation) will condense in the left part 97
of the image frame until the total number of features remain above a threshold. Although the ability of the feature detection method was evaluated on real OR room floors, the overall performance of the tracking system has not been tested in a real clinical situation. IM-nail pose estimation study: Several limitations of the IM-nail study include: the validation study was limited to two IM-nail geometries; the X-rays were acquired from animal (instead of human) specimens to simulate the real clinical situation; and biplane X-rays are needed on either side of the nail.
4.4 Future Directions Optical flow sensor study (mouse study): Increasing the number of optical flow sensors must be investigated for any potential improvement in odometry outcomes. Although currently the system has just been evaluated for short-range movements, in future developments, long-range trajectories need to be tested as well. C-arm base tracking study: Developing an on-line version of the base tracking software is scheduled for future phases of the research. Parallel processing methods are of interest in future implementations to reduce the processing time. Selective feature extraction methods should be replaced with the current non-intelligent approach. This can guarantee the presence of a minimum number of trackable features within different parts of a single frame and eventually can improve the odometry results. Evaluation of longer ranges of movement as well as more complex trajectories should be performed in the future. For this purpose, an optical tracking system is employed to provide the real time (reference) 2D pose of the platform. Another set of experimental evaluations of the base-tracking system should be performed in a real clinical situation and for a real C-arm. This is scheduled to be performed after the base-tracking system is integrated with the 98
joint-tracking (TC-arm) system. It has been assumed that the distortion parameters of the lens assembly remain the same during the usage of this system. This assumption has to be validated by performing a stability analysis in the future phases of the research. IM-nail pose estimation study: The performance of the proposed pose estimation method will be tested for other IM-nail geometries in the future. Magnified X-rays will be investigated for potential improvements in accuracy. An in vitro evaluation study with human specimens needs to be performed in the next phase of this research to justify the clinical applicability of the method. Similar concepts of the proposed pose estimation technique are scheduled to be incorporated for other orthopaedic routines (e. g. pedicle screw placement in spinal fusion surgery and sacroiliac screw placement in pelvis surgery).
4.5 Conclusions The main goal of the research presented in this thesis (as individual modules of the overall Smart-C project) was to improve the currently existing C-arms in a way that they can be utilized for quantitative estimations in orthopaedic procedures. After investigation of several odometry sensors, a base-tracking package (software plus hardware) was assembled to provide the real-time 2D pose of the C-arm base. The integration of this system with a joint-tracking system developed in our research group (TC-arm; Amiri et al. 2013) will offer a full range 6 DOF C-arm tracking solution. These data can then be utilized along with the C-arm fluoroscopic imagery to provide quantitative estimation for different orthopaedic routines. One potential field of application of a fully tracked C-arm (intramedullary nail fixation surgery) was investigated in this thesis and consequently a successful IM-nail localization method was introduced. Employing the methods
99
developed in this thesis will ultimately lead these orthopaedic procedures to be undertaken with lower radiation exposure, less time consumption and higher accuracies.
100
Appendix A: Optical Flow Based on Standard Computer Mouse Sensors as a Base Tracking Solution for a Surgical C-arm: Practical Approach and Validation
Introduction It is vital to track the pose (position and orientation) of the C-arm’s transmitter-detector set in order to be able to place it in any desired location and/or compute the location of the object of interest (patient’s organ or implant).C-arm pose estimation can be used in different applications ranging from computer-assisted surgery 2D-3D registration to panoramic X-ray imagery (Amiri et al. 2013). There are two main categories of tracking modalities for any moving device (robot): Absolute tracking (reference based): in this method the current position of the robot is localized with respect to an absolute cue using signal generating beacons (Global navigation satellite systems (GNSS), ultrasonic ant etc.) or vision-based feature extraction and analysis (Borenstein et al. 1997). Internal sensors (dead-reckoning): The location of the robot is inferred based on an accumulation of movements from an initial position using inertial sensors or odometry (Ross et al. 2012). The choice of appropriate sensor to measure each of the individual movement parameters associated with the C-arm is critical since each tracking technology has its own limitations. Global navigation satellite systems are not an option because of the lack of signals, since C-arms are always operated in indoor environments. Since an inexpensive sensor is desired (both in terms of cost and additional setup requirements), active indoor signal generating beacons are not appropriate for this application. Optical tracking systems have been developed and proposed as a tracking solution for the C-arm (Reaungamornrat et al. 2012; Amiri et al. 2011), but are limited by
101
the requirement of line of sight as well as expense. Magnetic tracking methods do not satisfy the sub-millimetre accuracies required in many applications. A novel, robust and low-cost tracking system has been developed (Amiri et al. 2013), which utilizes several inertial measurement units (IMUs) attached to the gantry in order to sense the changes in position and orientation. This method can provide the real-time pose (the centre point of the gantry can be tracked with accuracies better than 1.5 ± 1.2 mm) of the transmitter-detector set after performing an offline calibration procedure using a designed calibration frame. Although this method provides accurate results, it is limited to a locked base; in other words, it assumes that the C-arm base is stationary, which is not the case for many surgical applications. In order to address this issue, this paper investigates the performance of a dead-reckoning system that can act as a supplementary module to provide comprehensive localization outcomes. The desired output of this system is the absolute position and orientation (relative to any arbitrary origin) of the C-arm’s base in the operating room. The first candidate method for a dead-reckoning system is wheel odometry. Typically, odometry relies on the measurement of the covered distance (and velocity of movement) by using shaft-mounted encoders to calculate the 2D movement (as well as orientation) of the robot relative to a starting coordinate system (Bonarini et al. 2004; Bell 2011). It is well known that the wheelbased odometry is subject to: Systematic error: due to factors such as unequal wheel diameters, incorrectly measured wheel diameters and/or wheel distance (Borenstein et al. 1997). Random (or un-modeled systematic) errors: caused by irregularities of the floor, bumps, cracks or wheel slippage (Bonarini et al. 2005). In order to address these issues, optical flow sensors (e. g. a standard optical computer mouse sensor) have been introduced recently. The main idea of developing optical flow sensors is to 102
measure the movement without the need to be in direct contact with the floor underneath. This is a revolutionary advantage over previous mechanically-based encoders (e.g. a conventional trackball computer mouse). Figure 1-4 shows the different parts of an optical mouse in a schematic view. For the purpose of illumination, a light emitting diode (LED) is used to generate an adequate and suitable amount of light. In order to avoid blurry images and to have high contrast frames, the light is reflected by mirrors to the floor in a way that the lowest multipathed light is guided to the sensor and the highest area is illuminated on the floor. Then, all the light beams pass through a lens/prism set-up to be focused on the Charge-Coupled Device (CCD) or Complementary metal–oxide–semiconductor (CMOS) sensor which is commonly mounted on the processing chip itself (e.g. ADNSD 2051, AVAGO Technologies). Flow data are then computed based on comparison between successive frames and finding corresponding features. Changes in the position of the sensor are inferred from the flow data and then transformed into displacement information. Finally, the sensor outputs a two phase pulse through the port (∆𝑥, ∆𝑦; Sekimori et al. 2007).
Error Sources of Optical Flow Sensors Texture The texture (e.g. material, color and brightness) of the surface on which the optical flow sensor is being moved has a great impact on the displacement signals reported by the sensor. As reported in (Minoni et al. 2006) the scale factor (𝑐𝑜𝑢𝑛𝑡𝑠/𝑚𝑚) of the mouse sensor (ADNS 2051) readouts can vary from 11.60 (for a tile surface) to 26.37 (for a parquet surface).
103
Sensor Distance to the Floor Although optical flow sensors are designed to measure the displacement of the device without the need to be in direct contact with the floor, it is critical to keep them within an appropriate range of distances above the surface. It is reported that the sensors’ sensitivity (measurement scale; counts/mm) decreases from 16 to 0 when the distance to the floor is increased from 0.8 to 1.6 mm (Minoni et al. 2006). Another study confirms the previous survey by declaring a systematic error of up to 0.17 counts/mm for a change of 0.1 mm in the distance (4% of the recommended distance by the manufacturer; Palacin et al. 2006). In this part of the research, the performance of optical flow odometry (more specifically, an optical mouse sensor) is investigated as a solution for C-arm base tracking. The desired configuration would be a number (minimum of two sensors for 2D dead-reckoning) of mouse sensors attached to the body of the C-arm and looking downward to the underlying operating room floor. Specific dead-reckoning mathematical models were designed and developed for the mentioned sensory unit as part of this thesis research. Different sources of error were investigated to distinguish their impact on the tracking outcomes. A survey of the optimal configuration of the unit as well as an evaluation study were performed. The purpose of this study was to test the feasibility of optical flow sensors for C-arm basetracking applications. Mathematical framework and evaluation analysis is provided for this purpose.
Methods For any dead-reckoning system, the error accumulation is a big challenge since the localization problem is open loop in concept. All the sources of error associated with the sensor 104
identity, setup configuration and navigation model must be identified, modeled and corrected appropriately. Each error source must be treated differently based on its nature.
Texture An investigation was performed to examine the sensitivity of different optical mouse sensors to different surface textures. For three different optical flow sensors and on four specific textures, ten observations over a 100 cm baseline were collected.
Sensor Distance to the Floor The reason for the dependency of sensor resolution to its height can be described by the resultant effect on the sensor’s field of view. As the sensor increases its distance to the floor, the surface area it covers (sees) will increase, causing the measurement resolution to be decreased. This problem has been addressed in this study by designing a setup unit that guarantees a fixed distance for each of the sensors. A calibration procedure must be performed for each setup to recover each sensor’s resolution (in both the X and Y directions).
Odometry Model Up to this stage, two important sources of systematic error have been introduced, but as described earlier, it is crucial to utilize readings from several optical flow sensors in order to increase the measurement redundancy and consequently eliminate the effect of more random errors. With this being said, it is important to know the relative relationship of these sensors within a unit and develop a navigation model.
105
It can be shown that whenever the device moves along an arc, the rigidly attached mice will also trace out an arc, which is characterized by the same centre and the same arc angle (but with a different radius). During the sampling time, the angle 𝛼 between the 𝑥-axis of the mouse and the tangent to its trajectory does not change (Figure A-1-a). In other words, when a mouse moves along an arc of length 𝑙, it measures the same values, independent of the radius of the arc (Bonarini et al. 2005). The previous statement implies that, regardless of the true mouse movement in the desired global coordinate system, the sensor will record the data in its own coordinate system. For instance, the measured relative coordinates acquired from an optical mouse in either of two trajectories shown in Figure A-1-b and Figure A-1-c will be the same (2πr movement along the local Y-axis).
a
b
c
B
B
Figure A-1: Odometry Principle, a: relative configuration of the sensor’s local coordinate system and the trajectory, b: A circular trajectory with radius of 𝒓, c: a linear trajectory with a length equal to the circumference of the circle in b.
106
Figure A-2: Relative configuration of the main coordinate frames.
Figure A-2 shows the relative configuration of three important coordinate frames for the purpose of designing a proper dead-reckoning navigation model. 𝑋𝐺 and 𝑌𝐺 are the axes of the global coordinate system (which is considered to be identical to the coordinate frame of the moving platform at the starting point); 𝑋𝐶 and 𝑌𝐶 are the coordinate system axes of the moving C-arm; 𝑥𝑖 and 𝑦𝑖 are the local axes of each optical flow sensor in which the measurements are observed; 𝜑𝑖 is the boresight angle for the 𝑖 th sensor; 𝑟𝑖 and 𝜃𝑖 are the lever arm vector components of the 𝑖 th sensor (in polar coordinate system); and 𝑂𝐶 and 𝑂𝑖 are the centres of the C-arm and sensor coordinate frames, respectively. Note that a fixed lever arm (𝑟𝑖 , 𝜃𝑖 ) and boresight (𝜑𝑖 ) are guaranteed for each of the sensors by fabricating a robust housing unit that holds all the sensors at their firm place. In other words the vector 𝑂𝑖𝐶 in Eq. 1 is always constant.
107
𝑂𝑖𝐶 = [
𝑟𝑖 ∗ cos(𝜑𝑖 ) ] 𝑟𝑖 ∗ sin(𝜑𝑖 )
(1)
The kinematics model presented here is an extension of previous work (Cimino et al. 2011). The absolute position of the C-arm can be expressed as the elements of a 2D vector (O) in the global coordinate system. The centre of the sensor’s coordinate system in the C-arm frame 𝑂𝑖𝐶 can be expressed in the global coordinate frame 𝑂𝑖𝐺 by: 𝑂𝑖𝐺 = 𝑅(𝜔)𝑂𝑖𝐶 + 𝑂𝑐𝐺
(2)
in which 𝑅(𝜔) is the rotation matrix that relates the C-arm’s coordinate frame to the global system based on its orientation angle (𝜔). Since the desired parameters are the relative displacement of the C-arm’s coordinate system with respect to the global coordinate frame and because all the measurements are displacement of each sensor, the first order derivative of Eq. 2 is taken to obtain the kinematic relationships: 𝑉𝑂𝐺𝑖 = 𝑀(𝜃𝑖 )𝑅(𝜔)𝑂𝑖𝐶 + 𝑉𝑂𝐺𝐶
(3)
in which 𝑉𝑂𝐺𝑖 is the absolute velocity of the 𝑖 th sensor, 𝑉𝑂𝐺𝐶 is the absolute velocity vector for the Carm and the 𝑅(𝜔) is a skew-symmetric matrix such that 𝑀(𝜃𝑖 )𝑅(𝜔) =
𝑑𝑅(𝜔)⁄ 𝑑𝑡 . Eq. 3 can be
expressed as: 𝑉𝑂𝐺𝑖 = 𝑉𝑂𝐺𝐶 +
𝑑𝜔 −𝑟𝑐𝑜𝑠(𝜃 + 𝜔) [ ] 𝑑𝑡 𝑟𝑠𝑖𝑛(𝜃 + 𝜔)
(4)
Since the sensor measurements are collected at a constant rate (this depends on the type and model of the sensor), Eq. 4 can be revised as:
108
∆𝐺𝑂𝑖 = ∆𝐺𝑂𝐶 + ∆𝜔 ∗ [
−𝑟𝑐𝑜𝑠(𝜃 + 𝜔) ] 𝑟𝑠𝑖𝑛(𝜃 + 𝜔)
The raw sensor displacement measurements (∆𝑖𝑂𝑖 = [∆𝑥 𝑖
(5)
∆𝑦 𝑖 ]𝑇 ) (in the sensor’s local
coordinate system), can be expressed in the global frame (∆𝐺𝑂𝑖 ) as: ∆𝐺𝑂𝑖 = 𝑅(𝜑𝑖 + 𝜔)∆𝑖𝑂𝑖
(6)
Considering Eq. 5 and Eq. 6, it can be shown that: 𝑅(𝜑 + 𝜔)∆𝑖𝑂𝑖 = ∆𝐺𝑂𝐶 + ∆𝜔 ∗ [
−𝑟𝑐𝑜𝑠(𝜃 + 𝜔) ] 𝑟𝑠𝑖𝑛(𝜃 + 𝜔)
(7)
In order to have a clear relationship between the raw observations of each sensor (∆𝑖𝑂𝑖 ) and the desired states (∆𝐺𝑂𝐶 , ∆𝜔) and after some simplification, the following equations are obtained: cos(𝜔 + 𝜑𝑖 ) sin(𝜔 + 𝜑𝑖 ) 1 0 ∆𝑥 𝑖 [ 𝑖] = [ ]∗[ − sin(𝜔 + 𝜑𝑖 ) cos(𝜔 + 𝜑𝑖 ) 0 1 ∆𝑦 𝑖
{
∆𝑋𝐶𝐺
∆𝑋𝐶𝐺 −𝑟𝑖 ∗ 𝑠𝑖𝑛(𝜔 + 𝜃𝑖 ) ] ∗ [∆𝑌𝐶𝐺 ] 𝑟𝑖 ∗ 𝑐𝑜𝑠(𝜔 + 𝜃𝑖 ) ∆𝜔
∆𝑌𝐶𝐺
∆𝑥 = ∗ 𝑐𝑜𝑠(𝜔 + 𝜑𝑖 ) + ∗ 𝑠𝑖𝑛(𝜔 + 𝜑𝑖 ) + ∆𝜔 ∗ 𝑟𝑖 ∗ 𝑠𝑖𝑛(𝜑𝑖 − 𝜃𝑖 ) 𝐺 𝐺 𝑖 ∆𝑦 𝑖 = −∆𝑋𝐶 ∗ 𝑠𝑖𝑛(𝜔 + 𝜑𝑖 ) + ∆𝑌𝐶 ∗ 𝑐𝑜𝑠(𝜔 + 𝜑𝑖 ) + ∆𝜔 ∗ 𝑟𝑖 ∗ 𝑐𝑜𝑠(𝜑𝑖 − 𝜃𝑖 )
(8)
As seen in Eq. 8, at each epoch the 3 unknown parameters cannot be estimated with observations from a single sensor (2 observations). For this reason, a minimum of two sensors is required for the purpose of odometry. By increasing the number of sensors, the redundancy of the system is increased and, consequently, the expected precision will increase. Having 𝑛 optical flow sensors attached to the C-arm base, the observation equations can be formed as follows:
109
𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑚𝑎𝑡𝑟𝑖𝑥: 𝐿 = [∆𝑥1 ∆𝑦1 ⋯ ∆𝑥𝑖 𝑈𝑛𝑘𝑛𝑜𝑤𝑛 𝑚𝑎𝑡𝑟𝑖𝑥: 𝑈 = [∆𝑋𝐶𝐺 ∆𝑌𝐶𝐺 ∆𝜔] cos(𝜔 + 𝜑1 ) sin(𝜔 + 𝜑1 ) sin(𝜔 + 𝜑1 ) cos(𝜔 + 𝜑1 ) ⋮ ⋮ 𝐷𝑒𝑠𝑖𝑔𝑛 𝑚𝑎𝑡𝑟𝑖𝑥: 𝐴 = cos(𝜔 + 𝜑𝑖 ) sin(𝜔 + 𝜑𝑖 ) [ sin(𝜔 + 𝜑𝑖 ) cos(𝜔 + 𝜑𝑖 )
∆𝑦𝑖 ]
𝑟1 ∗ sin(𝜑1 − 𝜃1 ) 𝑟1 ∗ cos(𝜑1 − 𝜃1 ) ⋮ 𝑟𝑖 ∗ sin(𝜑𝑖 − 𝜃𝑖 ) 𝑟1 ∗ cos(𝜑1 − 𝜃1 )]
(9)
By using a least squares optimizer with the Gauss-Markov model, the unknowns can be solved by the following equation: ̂ = (𝐴𝑇 𝑃𝐴)−1 𝐴𝑇 𝑃𝐿 𝑈
,
𝑃 = 𝜎0−2 𝐶𝐿 −1
(10)
in which P is the observation weight matrix (calculated from the variance matrix of observations 𝐶𝐿 ). If a particular measurement comes from a less accurate sensor, it will be given a lower weight and the corresponding observation has a lower influence on the results.
Setup Calibration A calibration procedure must be performed for each setup and each time prior to deadreckoning in order to accurately estimate the setup parameters (𝜑𝑖 , 𝜃𝑖 , 𝑟𝑖 ) and consequently to obtain more accurate tracking results. If by some means the actual movements of the platform (relative to the global coordinate system) are known (e.g. by utilizing an optical tracking system with its targets attached to the platform), the design parameters can be estimated by undertaking a calibration procedure described as follows. Assuming the amount of displacement of the moving platform (the C-arm base) (∆𝑋𝑐𝐺 , ∆𝑌𝑐𝐺 , ∆𝜔) is known, Eq. 8 and Eq. 9 can be revised as:
110
{
𝑓1 (𝑈, 𝐿 + 𝑣) = ∆𝑥𝑖 − 𝑐𝑖 ∗ (∆𝑋𝐶𝐺 ∗ cos(𝜔 + 𝜑𝑖 ) + ∆𝑌𝐶𝐺 ∗ sin(𝜔 + 𝜑𝑖 ) + ∆𝜔 ∗ 𝑟𝑖 ∗ sin(𝜑𝑖 − 𝜃𝑖 )) = 0 𝑓2 (𝑈, 𝐿 + 𝑣) = ∆𝑦𝑖 − 𝑞𝑖 ∗ (−∆𝑋𝐶𝐺 ∗ sin(𝜔 + 𝜑𝑖 ) + ∆𝑌𝐶𝐺 ∗ cos(𝜔 + 𝜑𝑖 ) + ∆𝜔 ∗ 𝑟𝑖 ∗ cos(𝜑𝑖 − 𝜃𝑖 )) = 0 𝐿 = 𝑜𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛 𝑚𝑎𝑡𝑟𝑖𝑥 = [∆𝑥1
∆𝑥1
𝑈 = 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 𝑚𝑎𝑡𝑟𝑖𝑥 = [𝑐1
𝜑𝑖
𝑞1
⋯ 𝜃𝑖
∆𝑥𝑛 𝑟1
⋯
∆𝑦𝑛 𝑐𝑛
∆𝑋𝐶𝐺 𝑞𝑛
∆𝑌𝐶𝐺 𝜑𝑛
∆𝜔]𝑇
𝜃𝑛
(11)
𝑟𝑛 ]𝑇
in which the vector 𝑣 is the residual vector. Up to this stage, the sensor measurements have been assumed to have the same units as the global frame. This is not the case in the real application. In order to address this issue, two scale factors (𝑐𝑖 , 𝑞𝑖 ) have been added (as extra design parameters) to scale the sensor measurements into the global units. Since these equations are not linear and in order to solve for the unknowns, the combined form of the least squares optimizer with the Gauss-Helmert model is used. This method requires initial approximations for the unknown vector, which are measurable in the application of this study (the distance of each sensor and its orientation according to the C-arm’s coordinate system can be approximately measured). 𝑓⃗(𝑈, 𝐿 + 𝑣) = [𝑓1 (𝑈, 𝐿 + 𝑣)
𝑓2 (𝑈, 𝐿 + 𝑣)]𝑇
𝜕𝑓⃗ 𝜕𝑓⃗ 𝐴= |𝑈=𝑈0 ,𝐿=𝐿0 , 𝐵 = | , 𝑊 = 𝑓⃗(𝑈0 , 𝐿0 ) 𝜕𝑈 𝜕𝐿 𝑈=𝑈0 ,𝐿=𝐿0
(12)
̂ ) can then be computed in an iterative procedure: The unknowns (design parameters 𝑈 𝛿̂ = −[𝐴𝑇 (𝐵𝑃−1 𝐵 𝑇 )−1 𝐴]−1 𝐴𝑇 (𝐵𝑃−1 𝐵 𝑇 )−1 𝑊 𝑣̂ = −𝑃 −1 𝐵𝑇 (𝐵𝑃 −1 𝐵 𝑇 )−1 (𝐴𝛿 + 𝑊) ̂ = 𝑈0 + 𝛿̂ 𝑈
&
(13)
𝐿̂ = 𝐿 + 𝑣̂
Optimal Configuration By investigating the effect of changes in the tunable design parameters (3 for each sensor: 𝜑𝑖 , 𝜃𝑖 and 𝑟𝑖 ) on the variance matrix of the states (𝐶𝑈̂ = (𝐴𝑇 𝑃𝐴)−1 ), one can discover the optimum 111
configuration of the dead-reckoning setup from which the highest potential tracking accuracy is expected. A pre analysis based on the variance matrix of the states (𝐶𝑈̂ ) is performed to generate the declared configuration. Different setup configurations were simulated with their resultant localization precision, as follows. Lever arm angular (𝜃𝑖 ) values as well as boresight angles (𝜑𝑖 ) were changed from 0o to 360o in 1o degree increments for each sensor and the lever scalar components (𝑟𝑖 ) increased from 0 mm to 170 mm (due to limitations in the size of the fabricated unit). The planimetric precision is defined as: 𝜎𝑋𝑌 = √𝜎𝑋2 + 𝜎𝑌2 .
Experimental Methods In this study 3 wireless optical flow sensors (model PAW3204DB; PixArt Imaging Inc. Taiwan) were utilized, all attached to a rigid housing unit. Table A-1 provides the nominal specifications of the mentioned sensor. Table A-1: Sensor specifications. Optical lens zoom
Speed
Resolution (default)
Frame rate
Operating current
Power supply
Imaging Sensor
Recommended distance from surface
1:1
28 inches per second
1000 count per inch
3000 frames per second
3mA (while moving)
1.73~1.87 V
CMOS
2.4 mm
For the purpose of calibration (measuring the actual platform movements), a MicroScribe G2X (Solution Technologies, Inc., Maryland, USA) Coordinate Measuring Machine (CMM) with a manufacturer-specified accuracy of 0.34 mm (max) was utilized as the reference data to our model. During the calibration (
112
Figure A-3) and at each control movement, five rigid fiducial points attached to the designed unit were digitized both before and after the movement. The similarity transformation parameters were then estimated for the fiducial points (after reducing the coordinates to the centroid of the unit), which represent the reference translation and orientation of the setup for each movement. The data for 15 control movements (input to the calibration model) and 5 check movements (for accuracy assessment) were collected using custom software in Visual C++ 2010 (X86; Microsoft Corporation) environment. The calibration software was written in the MATLAB R2013a environment.
Fiducial Points
CMM Sensor Unit
Bottom View Top View
Figure A-3: Calibration Setup.
Results As explained in Section A.2.1, the performance of three optical flow sensors was analysed in recovering a known displacement (repeated 10 times) on four different floor textures. It was shown that different sensors models have different measurement resolutions (on the same floor
113
texture) and the measurements resolution for a single sensor is varying from one texture to another. (Table A-2). Table A-2: Repeatability (mean +/- standard deviation) of recovered baseline length from 3 sensors on 4 different floor textures, repeated 10 times (100 cm baseline). Texture Desk Surface
White Paper
Printed Mosaic
Printed Canvas
Mean (pixels)
2068.8
2066.0
2156.0
2108.3
SD (pixels) (%)
14.1 (0.6%)
21.9 (1%)
23.2 (1%)
20.1 (0.9%)
Mean (pixels)
1664.6
1680.2
1820.5
1745.0
SD (pixels) (%)
9.0 (0.5%)
23.6 (1.4%)
19.0 (1.0%)
13.0 (0.7%)
Mean (pixels)
4510.4
2801.0
4253.0
3653.9
SD (pixels) (%)
29.5 (0.6%)
11.9 (0.4%)
24.3 (0.5%)
18.3 (0.5%)
Sensor AT-S5008
ADNS-2051
PAW-103
By assuming a setup that contains 3 sensors and is limited to a size of 30 cm by 30 cm, and after performing the described optimal setup simulation study, the following design parameters were shown to be the ones with which the system can potentially provide the highest possible accuracy. The optimal tunable design parameters (𝑟𝑖 , 𝜃𝑖 , 𝜑𝑖 ; Table A-3) were obtained after performing the simulation analysis. Lever arm scalar parameters (𝑟𝑖 ) were shown to have no effect on the final recovered planimetric precision, however, the more the sensors are separated from each other, one can expect a higher rotational precision (Figure A-4). Although the angular components of the lever arm parameter were shown to have no impact on the angular precision, the sensors must be 120o apart from each other to achieve the highest planimetric precision (Figure A-5). The boresight angles 𝜑𝑖 had no impact on planimetric and angular precision
114
(Figure A-6). According to the results of the configuration analysis, an experimental setup consisting of 3 of the described sensors was fabricated (Figure A-7).
Table A-3: Optimum configuration of 3 sensors. 𝜃1
𝜃2
𝜃3
𝜑1 (optional)
𝜑2 (optional)
𝜑3 (optional)
0
120
240
0
120
240
Dashed: Planimetric Precision, Solid: Angular precision
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
20
40
60
80
100
120
140
160
ri(mm) Planimetric Precision
Angular Precision
Figure A-4: The effect of 𝒓𝒊 on planimetric (red {mm}) and angular (blue {radians}) precision.
115
Dashed: Planimetric Precision, Solid: Angular precision
0.8 0.7 0.6 0.5 0.4
120, 0.47
0, 0.47
0.3
240, 0.47
0.2 0.1 0 0
50
100
150
200
250
300
350
θo θ1
θ2
θ3
θ1& θ2& θ3
Figure A-5: The effect of 𝜽𝒊 on planimetric (red {mm}) and angular (blue {radians}) precision. All the design parameters are set to the optimal values (based on Table A-3)
Dashed: Planimetric Precision, Solid: Angular precision
except the ones being investigated in each curve. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
50
100
150
200
250
300
350
𝜑i Planimetric Precision
Angular Precision
Figure A-6: The effect of 𝝋𝒊 on planimetric (red {mm}) and angular (blue {radian}) precision. All the design parameters are set to the optimal values (based on Table A-3) except 𝝋𝟏 . Similar results are obtained for 𝝋𝟐 and 𝝋𝟑 .
116
Figure A-7: Experimental setup.
After following the described calibration routine (A.2.4), the design parameters and their corresponding precision estimates (from least squares) were calculated (Table A-4).
Table A-4: Recovered calibration parameters. Sensor #1
Sensor #2
Sensor #3
Parameter
𝑠1
𝑞1
𝜃1 o
𝜑1 o
𝑟1 mm
𝑠2
𝑞2
𝜃2 o
𝜑2 o
𝑟2 mm
𝑠3
𝑞3
𝜃3 o
𝜑3 o
𝑟3 mm
Value
54.23
-55.82
-0.61
91.99
90.23
56.08
-55.08
121.17
214.82
103.79
57.60
-57.54
237.42
327.98
108.89
Precision
0.016
0.015
0.011
0.077
0.128
0.016
0.015
0.011
0.067
0.11
0.152
0.017
0.011
0.049
0.098
The overall performance of the proposed odometry system was evaluated by comparing the recovered 2D transformations of the platform with the equivalent, accurately measured (by CMM) reference data. The data sets that were collected for the purpose of accuracy assessment, 117
were independent to the ones utilized for setup calibration. The system was shown to reach translational accuracy of 1.77 (3.2% of the total traveled distance) and rotational accuracy of 0.58o in average. (Table A-5). This study also illustrated the indirect dependency of the estimated pose precision with the total traveled travelled distance, in other words the accuracy of the pose parameters will decrease for longer range trajectories.
118
Table A-5: Accuracy assessment results. ∆𝑋𝐶𝐺 (𝑚𝑚)
∆𝑌𝐶𝐺 (𝑚𝑚)
𝑇𝑜𝑡𝑎𝑙 𝐷𝑖𝑠𝑝𝑙𝑎𝑐𝑒𝑚𝑒𝑛𝑡 (𝑚𝑚)
∆𝜔o
Check
-30.196
29.728
42.373
4.861
Recovered
-29.531
30.773
42.650
4.500
Error
0.66
1.04
1.23 (2.90%)
0.36
Check
35.117
47.487
59.061
-6.074
Recovered
32.310
46.079
56.277
-5.070
Error
2.80
1.40
3.13 (5.29%)
1.00
Check
23.759
36.460
43.518
-3.204
Recovered
23.194
35.702
42.574
-3.391
Error
0.56
0.75
0.93 (2.13%)
0.18
Check
35.666
-17.906
39.90
0.515
Recovered
34.838
-17.976
39.202
0.377
Error
0.82
0.07
0.82 (2.05%)
0.13
Check
76.409
3.855
76.506
-0.317
Recovered
79.147
3.282
79.215
-1.522
Error
2.73
0.57
2.78 (3.63%)
1.20
Max error
2.80 mm
1.40 mm
3.13 mm
1.20º
Min Error
0.56 mm
0.07 mm
0.82 mm
0.13º
Average Error
1.52 mm
0.77 mm
1.77 mm (3.2%)
0.58º
SD Error
1.14 mm
0.50 mm
1.09 mm
0.49º
Random Movement #1
Random Movement #2
Random Movement #3
Random Movement #4
Random Movement #5
119
Discussion A comprehensive investigation on optical flow sensors as a means for C-arm base tracking was performed in this study. Different sources of errors were surveyed and sufficient mathematical framework was provided. An accuracy assessment survey was undertaken to evaluate the efficiency of the sensors for the particular application of this thesis. The texture analysis shows that the sensor measurement scale depends strongly on the reflectivity properties of the floor. This finding illustrates the necessity of undertaking a calibration procedure on the particular texture that will be used. After performing the calibration procedure, the recovered parameters are only valid if the system is used on the same texture. Another important factor to be pointed out is that, since the sensor`s field of view is very small and the sensor calibration depends strongly on the local texture, it is suggested to have a uniform surface texture over the desired area. The results of the configuration analysis revealed that the localization precision does not depend on 𝜑 angle and is a function of 𝜃 angle. Although the change in the scalar component of the lever arms (𝑟𝑖 ) does not affect the planimetric precision, it has a strong influence on the precision of the recovered orientation. The farther the sensors are from the centroid of the unit, the higher the angular precision that can be expected. Minimum translational odometry accuracy of 2.1% (0.82 mm error over 39.90 mm of displacement) and a maximum of 5.3% (3.13 mm error over 59.06 mm) shows that the expected accuracies will be worse in longer range displacements, as expected. This finding is also valid for the estimated orientation parameter of the platform (seen in Table A-5), which reveals a lower angular odometry performance over longer ranges of displacement. The acquired 2D localization
120
accuracies were insufficient, demonstrating the need for an alternative sensor choice (Chapter Two). In this study, we have investigated the performance of optical flow sensors as a navigation alternative for commonly used C-arm devices. The low cost and high accessibility of these sensors are advantageous for the intended real life applications. The designed unit can be easily fabricated and attached to the C-arm body. Some of the possible sources of error have been studied and treated respectively. A rigid housing unit has been designed to hold the mice at a certain distance from the reference surface to avoid the height-sensitivity problem. A comprehensive calibration model has also been developed which eliminates the effects of variable resolution and provides an accurate estimate of the design parameters (lever arms and boresights of all the sensors relative to the C-arm’s coordinate system). The calibration is done in off-line mode and should be performed just once for each unit. This enables disposable manufacturing of the dead-reckoning unit. The optimum configuration of the sensors was also studied. Although a similar survey was performed previously (Cimino et al. 2010) to generate the optimum location of optical flow sensors on a robot, based on singular value analysis, no experimental data were reported. An external precision (accuracy) analysis was also performed to achieve a clear impression about the reliability of the proposed method. Previous studies either missed this analysis on the experimental data (Bonarini et al. 2005; Bell 2011; Palacin et al. 2006; Ross et al. 2012), or could not reach to the same level of accuracy (Sekimori et al. 2007). The localization accuracy can even be improved further by utilizing a higher number of sensors. One of the limitations associated with this study was that the proposed system has not been evaluated for long-range (more than 1 m) movements. This can be done by utilizing external trackers (along with their markers attached to the odometry platform) as the reference to assess the 121
accuracy of the recovered odometry solutions. The effect of increasing the number of optical flow sensors (more than 3) on the tracking results has not been investigated in this study.
122
References
Abdel-Aziz, Y.I., & Karara, H.M. 1971. Direct Linear Transformation from Comparator Coordinates into Object Space Coordinates in Close-Range Photogrammetry. In Proceedings of the Symposium on Close-Range Photogrammetry (pp. 1-18). Falls Church, VA: American Society of Photogrammetry.
Abdlslam, K.M., & Bonnaire, F. 2003. Experimental Model for a New Distal Locking Aiming Device for Solid Intramedullary Tibia Nails. Injury 34(5): p.363–6.
Acker, S., Li, R. Murray, H., St. John, P., Banks, S., Mu, S., Wyss, U., & Deluzio, K. 2011. Accuracy of Single-Plane Fluoroscopy in Determining Relative Position and Orientation of Total Knee Replacement Components. Journal of Biomechanics 44(4): p.784–7.
Alvarez, L., Weickert, J., & Sanches, J. 2000. Reliable Estimation of Dense Optical Flow Fields with Large Displacements. International Journal of Computer Vision 39(1): p.41–56.
Amiri, S., Wilson, D.R., Masri, B.A., Sharma, G., & Anglin, C. 2011. A Novel Multi-Planar Radiography Method for Three Dimensional Pose Reconstruction of the Patellofemoral and Tibiofemoral Joints after Arthroplasty. Journal of biomechanics 44(9): p.1757–64.
Amiri, S., Wilson, D.R., Masri, B.A., & Anglin, C. 2013. A low-cost Tracked C-arm (TC-arm) Upgrade System for Versatile Quantitative Intraoperative Imaging. International Journal of Computer Assisted Radiology and Surgery 9(4): p.695-711.
123
Anandan, P. 1989. A Computational Framework and an Algorithm for the Measurement of Visual Motion. International Journal of Computer Vision 2: p.283–310. Baker, S., & Matthews, I. 2004. Lucas-Kanade 20 Years on : a Unifying Framework. International Journal of Computer Vision 56(3): p.221–255.
Bell, S. 2011. High-Precision Robot Odometry Using an Array of Optical Mice (Technical Report, Oklahoma Christian University).
Bonarini, A., Matteucci, M., & Restelli, M. 2004. Dead Reckoning for Mobile Robots Using Two Optical Mice. In Robotics and Automation, 87–94.
Bonarini, A., Matteucci, M., & Restelli, M. 2005. Automatic Error Detection and Reduction for an Odometric Sensor based on Two Optical Mice. In International conference on Robotics and Automation, 1675–1680. Barcelona
Bonarini, A., Matteucci, M., & Restelli, R. 2004. A Kinematic-independent Dead-reckoning Sensor for Indoor Mobile Robotics. In International Conference on Intelligent Robots and Systems, 3750–3755. Sendai, Japan: Ieee
Bong, M.R., Koval, K.J., & Egol, K.A. 2006. The History of Intramedullary Nailing. Bulletin of the NYU Hospital for Joint Diseases 64(3-4): p.94–7.
Borenstein, J., Everett, H.R., Feng, L., & Wehe, D. 1997. Mobile Robot Positioning: Sensors and Techniques. Journal of Robotic Systems 14(4): p.231–249. 124
Brost, A., Strobel, N., Yatziv, L., Gilson, W., Meyer, B., Hornegger, J., Lewin, J ., & Wacker, F. 2009. Accuracy of X-Ray Image-Based 3D Localization from Two C-Arm Views : A Comparison Between an Ideal System and a Real Device a Chair. Medical Imaging 7261.
Campbell, J., Sukthankar, R., Nourbakhsh, I., & Pahwa, A. 2005. A Robust Visual Odometry and Precipice Detection System Using Consumer-grade Monocular Vision. In IEEE International Conference on Robotics and Automation, 3421 – 3427.
Chan, D.S., Burris, R.B., Erdogan, M., & Sagi, H.C. 2013. The Insertion of Intramedullary Nail Locking Screws Without Fluoroscopy: A Faster and Safer Technique. Journal of Orthopaedic Trauma 27(7): p.363–6.
Chintalapani, G., Jain, A.K., Burkhardt, D.H., Prince, J.L., & Fichtinger, G. 2008. CTREC: C-arm Tracking and Reconstruction using Elliptic Curves. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 1–7. Ieee
Chintalapani, G., & Taylor, R.H. 2007. C-arm Distortion Correction Using Patient CT as a Fiducial. In 4th IEEE International Symposium on Biomedical Imaging, 1180–1183.
Chli, M. 2009. Applying Information Theory to Efficient SLAM. Imperial College London (PhD. Thesis).
Cimino, M., & Pagilla, P.R. 2010. Location of Optical Mouse Sensors on Mobile Robots for Odometry. In International Conference on Robotics and Automation, 5429–5434. Ieee
125
Cimino, M., & Pagilla, P.R. 2011. Optimal Location of Mouse Sensors on Mobile Robots for Position Sensing. Automatica 47(10): p.2267–2272.
Cizik, A.M., Lee, M.J., Martin, B.I., Bransford, R.J., Bellabarba, C., Chapman, J.R., & Mirza, S.K. 2012. Using the Spine Surgical Invasiveness Index to Identify Risk of Surgical Site Infection. The Journal of Bone & Joint Surgery 94-A(4): p.335–342.
Clemente, L.A., Davison, A.J., & Reid, I.D. 2007. Mapping Large Loops with a Single Hand-Held Camera. In Robotics: Science and Systems,
Corke, P., Strelow, D., & Singh, S. 2004. Omnidirectional Visual Odometry for a Planetary Rover. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 4007–4012. Ieee
Davison, A.J., Reid, I.D., Molton, N.D., & Stasse, O. 2007. MonoSLAM: Real-Time Single Camera SLAM. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6): p.1052–67.
Deriche, R., Kornprobst, P., & Aubert, G. 2004. Optical-Flow Estimation while Preserving its Discontinuities : A Variational Approach. In Second Asian Conference on Computer Vision,
Duda, R.O., & Hart, P.E. 1972. Use of the Hough Transformation to Detect Lines and Curves in Pictures. Graphics and Image Processing 15(1): p.11–15.
Fahrig, R., Moreau, M., & Holdsworth, D.W. 1997. Three-dimensional Computed Tomographic Reconstruction using a C-arm Mounted XRII: Correction of Image Intensifier Distortion. Medical Physics 24(7): p.1097.
126
Forster, C., Pizzoli, M., & Scaramuzza, D. 2014. SVO : Fast Semi-Direct Monocular Visual Odometry. In IEEE International Conference on Robotics & Automation, Hong Kong
Grewal, I., & Carter, P. 2012. Focus on Distal Lcoking of Intramedullary Nails. The Journal of Bone & Joint Surgery: p.20–22.
Grzeda, V., & Fichtinger, G. 2011. C-Arm Rotation Encoding Methods and Apparatus. 1(19) (Patent Publication #: US 2011/0311030 A1).
Harstall, R., Heini, P.F., Mini, R.L., & Orler, R. 2005. Radiation Exposure to The Surgeon During Fluoroscopically Assisted Percutaneous Vertebroplasty. Spine 30(16): p.1893–8.
Hartley, R., & Zisserman, A. 2003. Multiple View Geometry in computer vision. Cambridge University Press, ISBN 0-521-54051-8.
Hofstetter, R., Slomczykowski, M., Sati, M., & Nolte, L.P. 1999. Fluoroscopy as an Imaging Means for Computer-Assisted Surgical Navigation. Computer aided surgery : official journal of the International Society for Computer Aided Surgery 4(2): p.65–76.
Horn, B.K.P., & Schunck, B.G. 1981. Determining Optical Flow. Artificial Intelligence 17(1-3): p.185–203.
17. Horn, B.K.P. 1987. Closed-Form Solution of Absolute Orientation Using Unit Quaternions. Joutnal of Optical Society of America 4: p.629-642.
127
Hough, P.V.C., & Arbor, A. 1962. Method and Means for Recognizing Complex Patterns (Patent #: 3069654, US).
Jain, A.K., Mustafa, T., Zhou, Y., Burdette, C., Chirikjian, G.S., & Fichtinger, G. 2005. FTRAC— A Robust Fluoroscope Tracking Fiducial. Medical Physics 32(10): p.3185.
Kahler, D.M. 2004. Image Guidance, Fluoroscopic Navigation. Clinical Orthopaedics and Related Research 421: p.70–76.
Kneip, L., Chli, M., & Siegwart, R. 2011. Robust Real-Time Visual Odometry with a Single Camera and an IMU. In Procedings of the British Machine Vision Conference, 16.1–16.11.
Knudsen, C.J.M., Grobler, G.P., & Close, R.E.W. 1991. Inserting the Distal Screws in a Locked Femoral Nail. The Journal of Bone and Joint Surgery 73-b(4): p.660–661.
Krettek, C., Konemann, B., Miclau, T., Schandelmaier, P., Blauth, M., & Tscherne, H. 1997. A New Technique for the Distal Locking of Solid AO Unreamed Tibial Nails Development of the Distal Aiming System. Journal of Orthopaedic Trauma 11(6): p.446–451.
Krettek, C., MannB, J., Miclau, T., Schandelmaier, P., Linnemann, I,. & Tscherne, H . 1998. Deformation of Femoral Nails with Intramedullary Insertion. The Journal Journal of Bone and Joint Surgery 16(5): p.572–575.
Leloup, T., El Kazzi, W., Schuind, F., & Warzée, N. 2008. A Novel Technique for Distal Locking of Intramedullary Nail Based on Two Non-Constrained Fluoroscopic Images and Navigation. IEEE Transactions on Medical Imaging 27(9): p.1202–12.
128
Livyatan, H., Yaniv, Z., & Joskowicz, L. 2002. Robust Automatic C-Arm Calibration for Fluoroscopy-Based Navigation : A Practical Aproach. In Medical Image Computing and Computer-Assisted Intervention, 5th International Conference, 60–68. Tokyo, Japan
Lucas, B.D., & Kanade, T. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In Imaging Understanding Workshop, 121–130.
Mechlenburg, I., Daugaard, H., & Søballe, K. 2009. Radiation Exposure to the Orthopaedic Surgeon during Periacetabular Osteotomy. International Orthopaedics 33(6): p.1747–51.
Minoni, U., & Signorini, A. 2006. Low-cost Optical Motion Sensors: An Experimental Characterization. Sensors and Actuators 128(2): p.402–408.
Moore, C., & Heeckt, P. 2011. Reducing Radiation Risk in Orthopaedic Trauma Surgery. Bone&Joint Science 2(7).
Muller, L.P., Suffner, J., Wenda, K., Mohr, W., & Rommens, P.M. 1998. Radiation Exposureto the Hans and the Thyroid of the Surgeon During Intramedullary Nailing. Injury 29(6): p.461– 468.
Navad, N., Heining, S.-M., & Traub, J. 2010. Camera Augmented Mobile C-Arm ( CAMC ): Calibration, Accuracy Study, and Clinical Applications. IEEE Transactions on Medical Imaging 29(7): p.1412–1423.
129
Neatpisarnvanit, C., & Suthakorn, J. 2006. Intramedullary Nail Distal Hole Axis Estimation using Blob Analysis and Hough Transform. In IEEE Conference on Robotics, Automation and Mechatronics, Ieee
Nistér, D., Naroditsky, O., & Bergen, J. 2006. Visual Odometry for Ground Vehicle Applications. Journal of Field Robotics 23(1): p.3–20.
Palacin, J., Valga, I., & Pernia, R. 2006. The Optical Mouse for Indoor Mobile Robot Odometry Measurment. Sensors and Actuators 126: p.141–147.
Reaungamornrat, S., Otake, Y., Uneri, A., Schafer, S., Mirota, D.J., Nithiananthan, S., Stayman, J.W., Khanna, A.j., Reh, D.D., Gallia, G.L., Taylor, R.H., & Siewerdsen, J.H. 2012a. Trackeron-C for Cone-Beam CT-Guided Surgery: Evaluation of Geometric Accuracy and Clinical Applications. Medical Imaging: Image-Guided Procedures, Robotic Interventions, and Modeling 8316.
Remondino, F., & Fraser, C. 2006. Digital Camera Calibration Methods: Considerations and Comparisions. In International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences XXXVI(5): p.266–272.
Ross, R., Devlin, J., & Wang, S. 2012. Toward Refocused Optical Mouse Sensors for Outdoor Optical Flow Odometry. IEEE Sensors Journal 12(6): p.1925–1932.
Scaramuzza, B.D., & Fraundorfer, F. 2011. Visual Odometry. IEEE Robotics & Automation Magazine (December).
130
Scaramuzza, D., Fraundorfer, F., & Siegwart, R. 2009. Real-time Monocular Visual Odometry for On-Road Vehicles with 1-Point RANSAC. In 2009 IEEE International Conference on Robotics and Automation, 4293–4299. Kobe: Ieee
Scaramuzza, D., & Siegwart, R. 2008. Appearance-Guided Monocular Omnidirectional Visual Odometry for Outdoor Ground Vehicles. IEEE Transactions on Robotics 24(5): p.1015– 1026.
Sekimori, D., & Miyazaki, F. 2007. Precise Dead-Reckoning for Mobile Robots Using Multiple Optical Mouse Sensors. Informatics in Control, Automation and Robotics II 41: p.145–151.
Shapiro, R. 1978. Direct Linear Transformation Method for Three-Dimensional Cinematography. Research Quarterly 49(2): p.197–205.
Siewerdsen, J.H., Daly, M.J., Bachar, G., Moseley, D.J., Bootsma, G., Brock, K.K., Ansell, S., Wilson, G.A., Chhabra, S., Jaffray, D.A., & Irish, J.C. 2007. Multi-Mode C-arm Fluoroscopy, Tomosynthesis, and Cone-Beam CT for Image-Guided Interventions: From Proof of Principle to Patient Protocols J. Hsieh & M. J. Flynn (eds). Medical Imaging 6510.
Silva, B., Burlamaqui, A., & Goncalves, L. 2012. On Monocular Visual Odometry for Indoor Ground Vehicles. In 2012 Brazilian Robotics Symposium and Latin American Robotics Symposium, 220–225. Fortaleza: Ieee
Singer, G. 2005. Occupational Radiation Exposure to the Surgeon. The Journal of the American Academy of Orthopaedic Surgeons 13(1): p.69–76.
131
Suhm, N., Jacob, a L., Nolte, L.P., Regazzoni, P., & Messmer, P. 2000. Surgical Navigation Based on Fluoroscopy--Clinical
Application for
Computer-Assisted Distal
Locking of
Intramedullary Implants. Computer Aided Surgery 5(6): p.391–400.
Tsalafoutas, I.A., Tsapaki, V., Kaliakmanis, A., Pneumaticos, S., Tsoronis, F., Koulentianos, E.D., & Papachristou, G. 2007. Estimation of Radiation Doses to Patient and Surgeons from Various Fluoroscopically Guided Orthopaedic Surgeries. Raiation Protection Dosimetry 128(1): p.112–119.
Viant, W.J., Phillips, R., Griffith, J.G., Ozanian, T.O., Mohsen, A.M.M.A., Cain, T.J., Karpinske, M.R.K., & Sherman, K.P. 1997. A Computer Assisted Orthopaedic Surgical System for Distal Locking of Intramedullary Nails. Journal of Engineering in Medicine 211(4): p.293– 300.
Yaniv, Z., & Joskowicz, L. 2005. Precise Robot-Assisted Guide Positioning for Distal Locking of Intramedullary Nails. IEEE Transactions on Medical Imaging 24(5): p.624–35.
Zhang, Z. 2000. A Flexible New Technique for Camera Calibration æ. IEEE Transaction on Pattern Analysis and Machine Intelligence 22(11): p.1330–1334.
Zheng, G. Zhang, X., Haschtmann, D., Gedet, P., Dong, X., & Nolte, L.P. 2008. A Robust and Accurate Two-Stage Approach for Automatic Recovery of Distal Locking Holes in Computer-Assisted Intramedullary Nailing of Femoral Shaft Fractures. IEEE Transactions on Medical Imaging 27(2): p.171–87.
132
Zhu, Y., Philips, R., Griffith, J.G., Viant, W., Mohsen, A., & Bielby, M. 2001. Recovery of Distal Hole Axis in IntraMedullary Nail Trajectory Planning. In IEEE International Conference on Robotics & Automation, 1561–1566.
133
