Underwater 3D Mapping: Experiences and Lessons Learned

Underwater 3D Mapping: Experiences and Lessons learned Andrew Hogue, Andrew German, James Zacher, and Michael Jenkin Department of Computer Science and Engineering and the Centre for Vision Research York University, Toronto, Ontario, Canada {hogue, german, zacher, jenkin}@cs.yorku.ca Abstract This paper provides details on the development of a tool to aid in 3D coral reef mapping designed to be operated by a single diver and later integrated into an autonomous robot. We discuss issues that influence the deployment and development of underwater sensor technology for 6DOF hand-held and robotic mapping. We describe our current underwater vision-based mapping system, some of our experiences, lessons learned, and discuss how this knowledge is being incorporated into our underwater sensor. Keywords: Underwater vision, 6DOF, 3D mapping, SLAM, stereo vision.

1

Introduction

Tourism and rising ocean temperatures are among the multitude of threats to aquatic life in general and coral reefs in particular1 . Coral reefs are ecologically important. They provide food and shelter for fish and other aquatic life. Many factors have an impact on the life of coral. For example, a slight rise of temperature (only a few degrees) destroyed 80 percent of the coral reefs in Okinawa in 1998[5]. Monitoring the health of coral reefs is a difficult and time consuming task. Existing measurement processes are based on manually sampling the reef structure over time, a lengthy, tedious, and potentially dangerous process. Tools that could be used to automatically or semiautomatically construct 3D models of such highlyunstructured underwater environments can be used to safely provide accurate data in a relatively inexpensive and timely manner. In addition to monitoring reef health for environmental and global climate change motives, such objective and bias-free tools would also find application in security, environmental evaluation generally, and in underwater construction. In order to address these and other sens1 http://www.starfish.ch/reef/conservation.html

ing/localization applications related to the sub-sea environment, we are developing an underwater vision sensor consisting of digital video cameras and an inertial measurement unit (IMU). The sensor is designed to operate in conjunction with the AQUA robot[2] which has been developed as part of a multiuniversity research collaboration between Dalhousie, McGill and York Universities. Working in parallel with colleagues at McGill and Dalhousie, we have been refining a design for a hand-held underwater stereo system that will be integrated into the AQUA robot. This paper describes the underwater sensor developed within the AQUA project. It examines design decisions made during the various incarnations of the sensor in light of the difficulties involved in conducting sensing tasks in open water. Results obtained with various versions of the sensor are presented, and the paper concludes with some discussion of the sensor currently under construction. We begin by placing the sensor in the context of the AQUA robot and the various tasks that the sensor is designed to perform for the robot itself.

2

The AQUA robot[2, 6, 7, 8] is a six-legged swimming robot based on the terrestrial hexapod RHex[1]. RHEX was developed in part by the Ambulatory Robotics Lab at McGill in collaboration with the University of Michigan, the University of California at Berkeley and Carnegie Mellon University[1]. AQUA’s required capabilities include swimming in open water, diving to a depth of 15m, station keeping, crawling on the bottom of the sea, and entering/exiting the water amphibiously. For propulsion, the vehicle deviates from traditional ROV designs through the use of six paddles, which act as control surfaces during swimming and as legs when walking (see Figure 1). The paddle configuration gives the robot direct control over five of the six degrees of

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006

IEEE

The AQUA Robot

(a)

(b)

Figure 1: The AQUA Robot (a) swimming (b) walking with amphibious legs. freedom of motion; surge, heave, pitch, roll and yaw. An inclinometer and a compass are used to assist in the control of the robot’s motion underwater.

3

The AQUASENSOR

Although the AQUA vehicle has on-board cameras facing both fore and aft, the cameras and on-board computational power of the robot are designed for teleoperational and servoing tasks, rather than for more higher level functions such as 3D map building. In order to provide a higher level of autonomy the AQUASENSOR is being developed in parallel with the AQUA vehicle. Initial versions of the AQUASENSOR operated off-board the vehicle, but the intent is to integrate the AQUASENSOR and AQUA vehicle in the future. The AQUASENSOR’s primary task is to collect rich and robust depth information through a multi-camera vision system coupled with an IMU. A reality of the underwater world is that although visual features may be very rich in some locations, they can be very sparse in others. Inertial data can be used to address ego-motion estimation issues when the visual information field is not sufficiently rich and to aid in the processing of visual data when both are present. It is also important to recognize the realities of the 6DOF environment encountered by a submersible. The environment is truly 6DOF, a situation which is rarely encountered except perhaps in robotic flying vehicles[4, 3], and outer space applications[14]. Developing a sensor for underwater usage is a non-trivial task. There are many challenges for such a system; although ensuring the electronics are protected by a watertight housing is perhaps the most critical. During the course of this project three different camera housing designs have been built and tested. Each of these housings provide different constraints on mobility and ease of deployment and are described below.

Figure 2: AQUASENSOR V1.0. Trinocular stereo rig and example reference images with corresponding dense disparity.

3.1

AQUASENSOR V1.0 (see Figure 2) is an underwater camera sensor manageable by divers that can be used to collect underwater stereo video and synchronized inertial data. To simplify the design of the sensor, the data storage and computing unit is land-based and is connected via a communications tether. The housing is based on a commercial underwater camera housing from Aquatix2 and this housing provides a watertight enclosure for the sensor electronics. Within the sensor housing are a triad of FirewireTM cameras that, after processing the 640x480 grayscale video streams, provide dense underwater depth images synchronized with inertial data from a Crossbow IMU300CC3 . The IMU is used for accurate relative 6DOF motion estimation in the model building phase. The land-based computing unit was a standard desktop PC powered by a set of lead-acid batteries and a power inverter. The underwater housing is connected to the computing unit via a custom-built 62m fiber-optic data cable eliminating concerns of signal degradation, EMI, and the use of a powered tether. Off-the-shelf Firewire and USB to optic converters to transform the Firewire signal for the camera video streams and the USB serial data from the IMU into optical signals. All electronics inside of the camera housing are powered by an on-board 16V NiMH battery source. Once imagery is collected at the base station, it is processed offline using a novel 3D SLAM algorithm[9] (described below). Images are combined binocularly, and then temporally linked to construct 3D models of underwater structures (see Section 4). Image Processing First, a stereo disparity map is computed from the stereo images using a standard stereo algorithm [12] and a set of 3D points 2 http://www.aquatix.com 3 http://www.xbow.com


IEEE

AQUASENSOR V1.0

(a)

(b)

(c)

(d)

(e)

Figure 3: AQUASENSOR V2.0 (a-c) and V2.1(d-e). (a) Hand-held unit (b) Hand-held unit with Bumblebee (c) Complete unit (d) Final system (e) Closeup of revised Hand-held unit are extracted from the resulting map. Next, salient features, based upon the Kanade-LucasTomasi features[11, 13], are found and tracked from frame to frame. This establishes temporal correspondence and optimally coupled with IMU data allows us to compute the pose of the camera relative to the previous frame. We prune the temporal correspondences by specifying that each feature tracked requires that a corresponding 3D point be available in both frames. These 3D points (extracted from the stereo images) are then used to compute the 6DOF pose using Horn’s method[10]; namely a translation vector and quaternion are computed. The pose is accumulated to bring the measurements (3D point cloud) into the global frame. The entire algorithm operates at 7fps using the highest accuracy stereo/rectification settings and low resolution (160x120) images and is mainly limited by the feature tracking stage. Using 320x240 images, the algorithm operates at approximately 1fps. Sampling reconstructions can be found in Section 4. Although V1.0 of the sensor was capable of local 3D map construction, operation of the device identified a number of critical operational issues. Namely, • Cable Management • Camera Synchronization Managing a 62 metre cable in high current and surge was extremely difficult. The cable is slightly positively buoyant and is thus subject to wind and wave action on the water surface. This negatively impacts the divers’ ability to control the orientation of the sensor housing. The fiber-optic cable is fragile and bending it close to a ninety degree angle can damage the cable. To reduce the fragility of the cable, the optical fiber was enclosed in a more rigid protective sheath that is less prone to bending. Spooling and unspooling the cable is also a problem

and long lengths of cable are unmanageable especially by divers operating at depth requiring a significant number of people to provide surface support. Cable management is time-consuming and requires practice in order to properly collect the cable at the end of the trial. The number of divers and surface support personnel required to deploy and wrangle the cable became an issue in sensor deployment. A secondary issue coupled with cable management was the use of a land-based PC. When deploying from a beach site this is not an issue. When deploying from a boat however, care must be taken to ensure that the components are not exposed to weather conditions. Monitoring the data capture on the display also increases the chance of motion sickness. Another problem identified by our field trials is camera synchronization. Although the cameras supposedly support automatic synchronization, they did not operate in a fully synchronized manner. While the stereo cameras were unsynchronized (by possibly a frame or two), it was still possible to extract valid disparity maps, however the disparities are scaled improperly. The temporal framerate drift between the cameras becomes equivalent to moving the baseline and slightly rotating the cameras. It is possible to ‘synchronize’ (or frame-sync) the image streams in a post-processing stage, however the cameras can still be off by several milliseconds due to the images not being captured at the same time instant. When registering point clouds, the effect of the desynchronization may not be apparent for many frames but results in a very low frequency curvature in the resulting model. This can be seen in the recovered models shown in Figure 7(b). In the following sections, we discuss how we addressed and overcame these issues for the next generation sensor design.

3.2

To overcome the problems with Version 1.0 of the sensor, we needed to address the cable management


IEEE

AQUASENSOR V2.0

(a)

(b)

Figure 4: Collecting Ground Truth Data (a) a single marker. (b) measuring from the reference point. and camera synchronization issues. The goal of this version of the sensor was to enable a single diver to deploy and operate the sensor autonomously. This called for a complete re-design of both the sensor and the data storage unit. To enable the diver to work autonomously, the data storage and processing components of AQUASENSOR V1.0 needed to be placed with the diver underwater. To accomplish this, we placed all electronics and computing components in a second underwater housing fastened to the diver. This allowed us to shorten the tether between the cameras and the computing unit to a manageable length of 1.5m thereby eliminating the effect of ocean currents on the tether. Maneuverability of the sensor was further improved upon by reducing the size of the camera housing to a hand-held form-factor manageable by the same diver (see Figures 3(a)-3(c)). To solve the camera synchronization issues, we adopted a Point Grey BumblebeeTM colour stereo camera. This unit provides left and right synchronized stereo video streams suitable for stereo image processing. Using the Bumblebee had the added benefit of a smaller form-factor than the cameras used in AQUASENSOR V1.0. To further reduce the size of the hand-held camera housing, we replaced the Crossbow IMU with a smaller 3DOF Inertiacube3TM from Intersense4 . The hand-held unit was fabricated from a standard aluminum tube with an embedded optical-grade acrylic view-port (see Figures 3(a) and 3(b)). Neoprene gasket material provided the watertight seal around both the end caps and the view-port. The two housings were connected via a 16 conductor cable to provide Firewire and USB communication. We mounted the cable to each housing through watertight cord glands that also provided strain-relief. The complete system was capable of being deployed and operated by a single diver and was tested during 4 http://www.isense.com

Figure 5: The Folkstone Marine Reserve Sunken Barge. These are the ground truth markers of the barge that was surveyed using two reference points, namely markers 11 and 17. The concentric circles indicate the radial distance measurements we used to determine an accurate measurement for each point. our pool trials. Although, this version of the sensor addressed the mobility issues, We identified three major problems with this version of the sensor, namely • Watertight seal issues • Heating concerns • Cable electrical specifications Due to the design of the end-caps, it was difficult to create a reliable watertight seal. There were many screws that needed to be inserted at an appropriate tension. The neoprene gasket continually compresses which loosens the other screws making the sealing process time consuming and error prone. One lesson learned from this experience is that a design using tension on an O-ring is a must for underwater applications. Furthermore, gasket re-use is a mistake. Re-using the material from previous trials results in a housing that is susceptible to leakage due to the compression and tearing of the gasket. Another major concern was heat dissipation for the electronic components in the housing that contained the computer and electronics. Due to its fibreglass construction, heat does not dissipate very effectively through the material itself. We fabricated an aluminum port for one end of the tube to act as a heat transfer point. We used aluminum due to its availability, low cost, and high thermal conductance value. The aluminum port provides a large surface area that is in direct contact with the water providing an effective way to transfer heat from the inside of the tube to the surrounding water. The mainboard and components were all mounted to an aluminum plate which in turn was attached to the aluminum port to transfer the heat created by the CPU to the water. Initial bench testing indicated


IEEE

(a)

(b)

(c)

(d)

Figure 6: (a-b) Sample 3D Reconstruction of short sequence using AQUASENSOR V1.0. (a) Shows the recovered camera trajectory as well. (c-d) Sample images from sequence that the heat transfer was sufficient to maintain an acceptable working temperature range. In the field however, the humidity and local temperatures overloaded some of the subcomponents on the motherboard into an unstable state. The next generation sensor housing is currently under development. It is being designed to maximize heat transfer utilizing proper airflow and connecting various electrical components directly to the aluminum port on the end of the housing. In addition, a combination of fans will be used to circulate the ambient air in the housing to dissipate pockets of hot air are believed to be the major cause of component failure.

3.3

Version 2.1

To overcome the concerns with the watertight seal on AQUASENSOR V2.0, we opted to modify the sensor hand-held unit. For AQUASENSOR V2.1 (see Figures 3(d) and 3(e)), the custom aluminum tube that housed the cameras was replaced with a small fibre-glass commercial housing from Aquatix and we built an aluminum port for one end to act as a connection for the cable cord gland and as a heat transfer point. This design was basically a shorter, but wider version of the computer housing. This modification provided a better watertight seal and was much simpler to close than the custom housing. The only drawback was that it was slightly larger than the previous version and thus a marginally less maneuverable. We tested AQUASENSOR V2.1 in the ocean during the recent AQUA robot field trials.

4

Evaluation of Sensor Performance

A critical issue in the development of a new sensor is performance. Although portions of this evaluation

can be done on land, a true test of the system requires an underwater test using a structure for which ground truth is available. Acquiring ground truth data underwater is a challenging task. During the AQUA robot field trials, two test sites have been identified for our mapping purposes • A manually mapped reef approximately 100m2 just off-shore Holetown, Barbados and • A sunken barge at the Folkstone marine reserve in Holetown, Barbados. Tests to date have concentrated on the barge test site and the manually mapped reef will be used in future trials. The barge dimensions were manually found to be approximately 23m x 9m x 2m. In order to obtain ground truth measurements, we surveyed the wreck using ten markers manually placed around the edge of the wreck, and measured the distance and bearing to all markers from a common reference point (see Figure 4). The distance was measured using a standard tape measure and the bearing was measured using a standard diver’s compass. Depth of the markers was measured using a standard diving depth gauge. To reduce measurement error, this measurement process was performed using multiple reference points. The resulting model can be seen in Figure 5. Reconstructions using AQUASENSOR V2.0 of some underwater structures on the Barge are shown in Figure 6 using AQUASENSOR V1.0. Longer sequences from V1.0 can be seen in Figure 7 which show the low frequency curvature of the registration due to the camera desynchronization. To show results from later versions of the sensor, landbased reconstructions are shown in Figure 8.

5


IEEE

Summary and Future Work

In this paper, we have outlined our experiences designing an underwater vision-based tool to aid in

(a)

(b)

Figure 7: Underwater Reconstructions using AQUASENSOR V1.0. Note the low frequency surface errors due to camera desynchronization. (b) Also shows the recovered camera trajectory. coral reef monitoring. The designs for the sensor, will eventually be miniaturized and integrated with the AQUA robot. We have identified several issues with each design and outlined our solutions for them. The design has evolved from an initial solution requiring at least two divers to control the sensor and a multitude of surface support personnel to a design that is manageable by a single diver. We have also shown results from some underwater mapping experiments and more recent land-based 3D reconstructions, however a full evaluation of the accuracy of the sensor remains for future work. The next generation sensor design will incorporate several design changes to address robustness and usability. First, a more effective heat-transfer solution is being designed and internal fans are being added to circulate air, eliminating uneven heat pockets. Two new designs of the sensor are currently being evaluated on the bench. One is a single-tube solution incorporating lessons learned from AQUASENSOR V1.0 and V2.1 while the other is a two tube solution similar to AQUASENSOR V2.1 which utilizes a a custom sealed cable for communications between the two housings. This sealed cable is necessary to eliminate the possibility of water being transferred from one housing to the other in case of flooding. The new cable will incorporate standard Firewire and USB cables to avoid EMI and be specially terminated with watertight military spec circular connectors5 . A final addition to the sensor will enable the diver to more easily control the hardware operation. We are incorporating external watertight toggle switches to control both power and recording state. The new designs include a WiFi interface 5 http://www.underwatersystems.com

which will allow us to monitor temperatures inside the sensor and ensure that everything is initialized prior to deployment into the water. The sensor can be fully controlled through a web interface that is accessed and controlled using a standard PDA in the field trials. Future evaluation of the sensor will include a full mapping of the aforementioned test sites. Designs to integrate the AQUASENSOR with the AQUA robot are currently being developed. Acknowledgments: We graciously acknowledge the funding provided by NSERC and IRIS NCE for the AQUA project. We also thank the McGill AQUA team for engineering support and the McGill Bellairs Research Institute for providing a positive atmosphere during the field research trials. We also thank Mark Obsniuk for agreeing to stand still while we modeled him.

References [1] R. Altendorfer, N. Moore, H. Komsuoglu, M. Buehler, H. Jr., D. McMordie, U. Saranli, R. Full, and D. Koditschek. RHex: A Biologically Inspired Hexapod Runner. Autonomous Robots, 11:207–213, 2001. [2] G. Dudek, M. Jenkin, C. Prahacs, A. Hogue, J. Sattar, P. Giguere, A. German, H. Liu, S. Saunderson, A. Ripsman, S. Simhon, L. Torres, E. Milios, P. Zhang, and I. Rekleitis. A visually guided swimming robot. In Intelligent Robots and Systems, 2005. (IROS 2005). Proceedings. 2005 IEEE/RSJ International Conference on, Aug. 2-6 2005.


IEEE

(a)

(b)

(c)

(d)

(e)

Figure 8: Surface reconstruction sequence. (a-d) show the development of the mesh from start to finish. The camera trajectory is also shown in green. (e) shows a close-up of the resulting mesh from the side.


IEEE

[3] A. Elfes, S. Bueno, M. Bergerman, and J. Ramos. A semi-autonomous robotic airship for environmental monitoring mission. In Robotics and Automation, 1998. Proceedings. 1998 IEEE International Conference on, pages 3449 – 3455, May 16-20 1998. [4] A. Elfes, S. Bueno, M. Bergerman, J. Ramos, and S. Gomes. AURORA: development of an autonomous unmanned remote monitoring robotic airship. Journal of the Brazilian Computer Society, 4(3):70–78, April 1998. [5] Y. Furushima, H. Yamamoto, T. Maruyama, T. Ohyagi, Y. Yamamura, S. Imanaga, S. Fujishima, Y. Nakazawa, and A. Shimamura. Necessity of bottom topography measurements in coral reef regions. In OCEANS ’04. MTS/IEEE TECHNO-OCEAN ’04, pages 930 – 935, Nov. 9-12 2004.

[12] M. Okutomi and T. Kanade. A MultipleBaseline Stereo. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 15(4):353– 363, 1993. [13] J. Shi and C. Tomasi. Good Features to Track. In Computer Vision and Pattern Recognition, 1994. Proceedings CVPR ’94., 1994 IEEE Computer Society Conference on, pages 593 – 600, Jun. 21-23 1994. [14] Y. Tsumaki, M. Yokohama, and D. Nenchev. Intra-Vehicular Free-Flyer System. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 2547–2552, Oct. 27 - 31 2003.

[6] C. Georgiades, A. German, A. Hogue, H. Liu, C. Prahacs, A. Ripsman, R. Sim, L. Torres, P. Zhang, M. Buehler, G. Dudek, M. Jenkin, and E. Milios. AQUA: an aquatic walking robot. In The 7th Unmanned Underwater Vehicle Showcase, Proceedings UUVS’04, 2004. [7] C. Georgiades, A. German, A. Hogue, H. Liu, C. Prahacs, A. Ripsman, R. Sim, L. Torres, P. Zhang, M. Buehler, G. Dudek, M. Jenkin, and E. Milios. AQUA: an aquatic walking robot. In Intelligent Robots and Systems, Proceedings IROS’04, IEEE/RSJ International Conference on, pages 3525–3531, Sept 28 - Oct 2 2004. [8] C. Georgiades, A. Hogue, H. Liu, A. Ripsman, R. Sim, L. Torres, P. Zhang, C. Prahacs, M. Buehler, G. Dudek, M. Jenkin, and E. Milios. AQUA: an aquatic walking robot. Technical Report CS-2003-08, Dalhousie University, Nova Scotia, Canada, Sept 28 - Oct 2 2003. [9] A. Hogue. SensorSLAM. PhD thesis, York University, Toronto, Ontario, Canada, 2006. In Preparation. [10] B. Horn. Closed-form solution of absolute orientaiton using unit quaternions. AI Magazine, A(4):629, April 1987. [11] B. Lucas and T. Kanade. An Iterative Image Registration Technique with an Application to Stereo Vision. In Artificial Intelligence, 1981. Proceedings IJCAI’81., 1981 International Joint Conference on, pages 674–679, 1981.


IEEE