Tsai hand-eye calibration in a modified and automated configuration, while the high-speed calibration is based on a com- bination of laser triangulation and ...
Calibration Techniques for Industrial Mobile Manipulators: Theoretical configurations and Best practices Mads Hvilshøj, Simon Bøgh, Ole Madsen and Morten Kristiansen Department of Production, Aalborg University, Denmark {mh, sb, i9om, morten}@production.aau.dk www.machinevision.dk
Abstract At the moment autonomous industrial mobile manipulation (AIMM) is a subject of major focus in development and research environments, as it is a technology with significant potential. However, we are experiencing a lack in calibration techniques, in order to obtain industrially acceptable localization and manipulation tolerances. In this paper two calibration methods (high-precision and high-speed) are proposed, with the purpose of defining sufficient localization of a mobile manipulator in relation to a workstation. Initially the techniques are presented in their theoretical forms, and subsequently the focus is on verification (experiments) and practical use (applications). The high-precision calibration is based on the Tsai hand-eye calibration in a modified and automated configuration, while the high-speed calibration is based on a combination of laser triangulation and image processing. Experiments and applications, based on the proposed techniques, have been carried out with a full-scale mobile manipulator ("Little Helper" from the Department of Production, Aalborg University). The obtained tolerances and processing times are promising - high-precision; [± 0.1 mm, ± 0.1◦ , 60 seconds] and high-speed; [± 1.0 mm, ± 1.0◦ , 10 seconds]- in relation to industrial integration of mobile manipulators.
1 Introduction Increased customer needs and intensified global competition require intelligent and flexible automation. The interaction technology mobile manipulation addresses this, so it holds great potential within the industry. Already a lot of research has been carried out in this field, e.g. [1]-[6], but we are experiencing a lack in calibration techniques, in order to obtain acceptable localization and manipulation tolerances for execution of typical industrial tasks. This paper focuses on workstation calibration1 for autonomous industrial mobile manipulators (AIMM), based on the mobile manipulator "Little Helper" (Figure 1).
The purpose of AIMM is to develop autonomous manufacturing assistants, which can be easily integrated into various applications in typical production environments (Figure 2). These mobile manipulators must be able to; work with or alongside people, serve usual production equipment, and carry out operations at different workstations[6].
Remote desktop Operator High-speed calibration target
Charging station
Highprecision calibration target
Workstation 1
z
Mobile manipulator Data Lens Light controlbox controlbox acquisition module Air compressor
Air tank
Workstation 2 Toolchange
Tools
Tooling
Lens
Camera
Light
Vision
6 DoF manipulator
Platform
Communication Power/Signal module module
Manipulator
Figure 1: The mobile manipulator "Little Helper". 1 Unique
Workstation 3
Laser
Inverter
Mobile platform
No calibration target
localization of a mobile manipulator in relation to a workstation.
Figure 2: A typical production environment with workstations, calibration targets and a mobile manipulator. The required characteristics of autonomy and mobility are challenging, because they are conflicting in achieving industrially acceptable localization tolerances. The mobile platform, alone, is rarely able to achieve the desired localization for task execution. Therefore subsequent calibration is often necessary, which ensures sufficient localization of the manipulator, and thereby the entire mobile
manipulator, in relation to a workstation. Already a lot of calibration techniques have been proposed in the literature, e.g. [7]-[9], but they are mainly concentrated on internal calibration of stationary manipulators - that is; manipulator, camera, and hand-eye calibration. However, techniques for external calibration of mobile manipulators (workstation calibration) have been proposed, e.g. [10] and [11], but the focus has not been on industrial use. Therefore new calibration approaches are necessary, in order of introducing mobile manipulators in the industry, although inspiration will be sought in existing techniques. As we focus on industrial integration of mobile manipulators, it is vital that the calibration methods are; automatic, robust, adaptable, flexible, precise, and/or rapid[6].
2 Workstation calibration The proposed calibration methods, in this paper, are based on dedicated calibration targets, placed on the workstation. The general principle is shown in Figure 3.
Manipulator locations are configured in the workstation frame {W} for later task execution.
1. Setup phase {C} {E}
{O}
{W}
{M} M O
T
The mobile manipulator returns to the workstation for task execution, based on the earlier configured locations. This requires calibration, in order to deM termine O Tnew
2. Operation phase
M O
Tnew
dx, dy, dӨ Figure 3: The general principle of a workstation calibration, divided in a setup and operation phase, respectively. 2 E.g.
by use of teaching, learning by demonstration, etcetera.
The frames of the workstation calibration (Figure 3) are: • {M }: The manipulator frame is located in the base of the manipulator. Also {M } represents the reference frame of the mobile manipulator, since it is the control of the manipulator that is of general interest. • {E}: The end-effector frame is located in the endeffector (or wrist) of the manipulator, and is defined relative to the manipulator base; {E} = M E T. • {C}: The camera frame is located in the refraction point (the location of the front lens), and represents the reference frame for the overall vision system. • {O}: The calibration frame is located at the origin of the calibration target, which corresponds to the zero of a configured workstation. • {W }: The workstation frame is located at the workspace of interest at a workstation, and is defined relative to the calibration target; {W } = O W T. The general purpose of a workstation calibration is to determine the transformation between the manipulator and a calibration target placed on the workstation: M O T
=
M E T
·
E CT
·
C OT
(1)
This calibration approach makes the mobile manipulator able to return to a configured workstation and carry out the intended tasks, as shown in Figure 3. In the upper part of Figure 3, the setup phase is shown. Here a task is configured2 at a workstation, relative to a calibration target. This can be done for various reasons; authentication of locations within a specific range, realization of narrow tolerances, and/or execution of tasks with limited work space. An example could be taking out parts from an injection molding machine (Figure 3). When the setup phase is completed, the configurations of the workstation (all the O W T’s) are saved, resulting in a library of tasks that the mobile manipulator can carry out[6]. In the lower part of Figure 3, the operation phase is shown. Here the mobile manipulator is in automatic mode, and returns to an already configured workstation. The mobile manipulator is now located different to the workstation, due to the localization tolerances of the mobile platform (as indicated by dx, dy and dΘ). To compensate for this, a workstation calibration is carried out, where the transformation M O Tnew is found. Based on this new determined location of the mobile manipulator, relative to the configured workstation, the transformations M W Tnew can be found: M W Tnew
O =M O Tnew · W T
(2)
In this way, the intended tasks at a workstation can be carried out, within the specified tolerances.
2.1
Assumptions
To perform a workstation calibration, based on the principles of Figure 3, the following assumptions must be valid: • A camera calibration must be performed in advance, which determines the intrinsic parameters; focal length, lens distortion, pixel skew, and principal point[8]. • A hand-eye calibration must be performed in advance, which expresses the constant transformation between the end-effector and the camera; E C T [9]. • There must be sufficient space at the workstation to actually perform the calibration procedure, e.g. necessary workspace for the manipulator. • During a workstation calibration, the origin of the calibration target must be visible3 , within the field of view of the camera.
2.2
General approach
As already mentioned, the goal of a workstation calibration M is to establish M O T or O Tnew , based on the transformation E M matrices; C T , E T and C O T , as shown in Formula 1 and 2. In the following, two workstation calibration methods (high-precision and high-speed) are presented, from a theoretical point of view. The primary focus is on M E T and C O T , as determination of these transformations represent the differences between the proposed methods.
2.3
Calibration procedure During high-precision calibration, the camera is located differently to the calibration target4 , by pre-defined movements of the manipulator, as shown in Figure 4. In this way, M E1..i..n T is determined by conversion of the Cartesian locations of the manipulator, and O C1..i..n T is establish by image acquisition and subsequent image processing[9]. The use of the Tsai calibration method requires focus on the image acquisition locations, since these are essential for the results obtained. In Figure 4, the optimal image acquisition locations (of {E} and {C}) are illustrated, corresponding to a star formation. A minimum of 3 images is required, in order of obtaining a unique solution[9]. Calibration target The calibration target used to perform a high-precision workstation calibration is shown in Figure 5. The black pattern on white background is easy separable from the surroundings, which minimize the demands on lighting and image processing. Based on this, automated, robust, and inexpensive calibration procedures can be realized.
dy
Lx
dx
Ly
Origin
Figure 5: High precision calibration target.
High-precision calibration
Calibration principle The high-precision calibration approach is based on the well-known and robust Tsai calibration method[9], in a modified and automated configuration[12]. However, unlike the original Tsai calibration, here the method is used to establish a workstation calibration. The general principle of the high-precision calibration is shown in Figure 4.
2.4
High-speed calibration
Calibration principle The high-speed calibration approach is based on a combination of laser triangulation5 and image processing. The general principle is shown in Figure 6. Parameters found with laser: z, yaw, pitch
Parameters found with camera: x, y, roll
Location 1 Location i {E}i
{E} M E iT
{M} M OT
E CT
{C}i
{C} C OT i
Location n
{O}
Figure 4: High-precision calibration principle. 3 Is
End-effector alignment to the plane
360 n
z 3 laser measurements
θ Image acquisition y and processing
Figure 6: High-speed calibration principle.
highly dependent on the composition of; camera, lens and lighting. locations of image acquisition, corresponding to n locations of the end-effector, are used. 5 Therefore this method requires use of a laser rangefinder, as shown in Figure 1. 4n
x
Calibration procedure As indicated in Figure 6, the high-speed calibration method is divided in the following two steps: • Laser rangefinder ⇒ (z, yaw, pitch)
3.1
Experimental setup
The experiments have been conducted in an emulated industrial environment at Aalborg University with (a prototype of) the mobile manipulator "Little Helper"[6][13]. The used hardware and software are shown in Figure 8.
• Camera ⇒ (x, y, roll) As a starting point, the end-effector is located in front of the calibration target. Then the camera is aligned perpendicular to the calibration target plane, by use of laser triangulation (as indicated by the three laser measurements, in the left side of Figure 6). Since the end-effector is now oriented in the same way as the calibration target, it is possible to perform a series of measurements. Based on these measurements, the orientation of (yaw and pitch) and the distance to (z) the calibration target can be found - corresponding to the definition of M E T. The camera is then moved to a pre-defined distance from the calibration target (as indicated in the right side of Figure 6), which gives a fixed image field. The perpendicular alignment of the camera minimizes the perspective errors. In this location a single image is acquired, followed by image processing, which results in the x and y coordinates and their related orientation Θ (the roll value of the calibration target). Based on this, C O T can be found. Calibration target The calibration target used to perform a high-speed workstation calibration is shown in Figure 7. The black bars on white backgrounds are easy separable from the surroundings, which minimize the demands on lighting and image processing. Compared to high-precision, the target is simpler, due to initial use of laser triangulation, before image acquisition and processing. In fact, the high-precision calibration target can also be used for the high-speed calibration, but for the sake of understanding, a dedicated target for the high-speed calibration is used, in this paper.
”Little Helper”
R2007b Main control and HMI System
Hardware Software
Vision
Manipulator + tooling
Mobile platform
Auxiliaries
Prosilica EC 1380 firewire camera
Adept Viper s650
Neobotix MP-L655
Laser Light Lens
NI Vision Builder 3.5
Adept Desktop 4.0
Neobotix PltfCtrl 4.610
NI LabVIEW 7.5
PlatformCtrl
Figure 8: Experimental equipment: HW and SW. In addition, the following MATLAB tools have been used in a modified and automated configuration: • Robotics Toolbox[14][15] • Camera Calibration Toolbox[16] • Calibration Add-on[12][17] In Figure 9 the experimental environment is shown, consisting of the mobile manipulator "Little Helper" and a workstation with a dedicated calibration setup. The calibration setup consists of a combined high-precision/highspeed calibration target and an x-shaped reference object6 .
Lx High-speed target
Ly
High-precision target
Origin
Figure 7: High-speed calibration target.
ViaPoint
Reference object
3 Experiments In order of verifying the proposed workstation calibration methods (high-precision and high-speed), a series of experiments have been conducted, with the purpose of determining the repeatability and process speed, and in addition, the advantages and disadvantages. 6 Both
printed in a quality of 2400 dpi.
StationFinal StationStop Figure 9: The experimental environment.
The used settings of the experiments are shown in Table 1. High-precision 10 images of the calibration target Calibration target: 8 mm dot pattern and central bars High-speed Three laser measurements and one image Calibration target: only central bars Table 1: Used settings of the conducted experiments.
3.2
4 Results During verification of the proposed workstation calibration methods, a lot of different experiments have been conducted. The presented results, in this paper, are related to a worst-case scenario7 , based on 30 repetitions. In Figure 11, 30 determined coordinate sets for highprecision and high-speed calibration are shown, based on acquired images of the reference object. The data have been normalized (referenced to (0,0)), in order to compare the results of the two calibration methods directly.
Experimental procedure
High-speed calibration
High-precision calibration 1,5
The experimental procedure is divided in two main steps: 1,0
• Initial calibration • Subsequent localization of the reference object Delta Y
The initial step is dependent on the calibration method used (high-precision or high-speed), while the subsequent step is identical. A flowchart of the experimental procedure is illustrated in Figure 10, and videos of the conducted experiments can be found at [13].
0,5
-1,5
0,0 -1,0
-0,5
0,0
0,5
1,0
1,5
-0,5
-1,0
-1,5 Delta X
Starting point: ViaPoint
Mobile platform moves to starting point
Figure 11: Normalized experimental data. Mobile platform moves to StationStop
Based on Figure 11, the repeatability (σpyth 8 and α9 ) of the proposed calibration methods can be found. The results are listed in Table 2 with average processing times10 .
Mobile platform moves to StationFinal Performing workstation calibration
Step 1
Acquire image of reference object
Step 2
YES
Run experiment again?
NO
Analysis - Processing time - Repeatability - (Dis)advantages
Figure 10: Flowchart of the experimental procedure.
7 Localization
◦ p accuracy of the platform system; ± 20 mm and ± 10 . 2 2 (σx ) + (σy ) ; the linear displacement. pyth = 9 α; the angular displacement, is determined in the same way as x and y. 10 The time used for actual calibration; image acquisition and processing. 8σ
σpyth : α: Processing time:
High-precision ± 0.1 mm ± 0,1◦ 60 seconds
High-speed ± 1.0 mm ± 1,0◦ 10 seconds
Table 2: General results of the calibration methods. From Table 2 it is seen that the high-precision calibration has the highest repeatability, while the high-speed calibration has advantages in the processing time. Both the repeatability and the processing time depend on the applied settings (e.g. the number of images acquired, the amount of available data in the calibration target, etcetera), and the used hardware and software (e.g. manipulator, processing unit, image processing algorithms, etcetera). In practice, a typical high-precision calibration takes 60 seconds, while a corresponding high-speed calibration takes 10 seconds.
5 Applications Based on the results of the conducted experiments, the proposed workstation calibration methods have been tested in suitable (industrial) applications, as shown in Figure 12. 1) High precision calibration at CNC milling/turning center.
Picking up part inside the machine.
• Application 1; "Little Helper" inserts and removes parts from a CNC milling/turning center, based on the high-precision calibration target and method. In this way, fully automatic machining is realized. • Application 2; "Little Helper" picks up highly reflective metal parts from a storage area, based on the high-speed calibration target and method. In this way, inventory management can be automated. • Application 3; "Little Helper" feeds a metrology machine with parts mounted on pallets with a standardized manipulation interface. The interface is based on the high-speed calibration target and method, in a modified setup, as only the imaging part is used. • Application 4; "Little Helper" picks up easy recognizable parts from an AnyFeeder. In this case it is sufficient just to use the integrated vision system, which means that no calibration is necessary.
2) High-speed calibration at storage area.
Picking up part from area.
The four applications show that the proposed workstation calibrations are suited for different types of tasks, and that they can be used in modified forms. It is also evident that a workstation calibration is not always necessary. Based on this, it is possible to establish a set of guidelines for workstations calibration of industrial mobile manipulators: • What is the required accuracy/precision of the task? • What is the required speed/velocity of the task? • Are the locations of the task fully defined, or not? • What is the required tact time/cycle time of the task?
3) High-speed calibration target on product pallets.
Placing pallet with part, in metrology machine.
• What is the length of stay at the workstation? From these parameters, typical industrial tasks and the proposed calibration methods can be mapped, as shown in Figure 13. The mapping is based on the fact that the actual task is carried out by a mobile manipulator.
Product
Pattern
Sample testing Pallet
Task locations fully defined
Assembly
Teach
1.
Picking up part using vision system.
No calibration
4.
High-speed calibration
High-precision calibration
High precision
4) No calibration used.
High speed
2.
3.
Vision
Pick-and-place Task locations not fully defined
Quality control
Figure 13: Mapping; tasks (typical industrial tasks and the performed applications) versus calibration methods.
Figure 12: Performed (industrial) applications[13].
From Figure 13, it is possible to choose an appropriate calibration method for a current task at hand, if any.
6 Conclusion and Future Work In this paper we proposed two calibration methods - highprecision and high-speed - to establish sufficient spatial description between autonomous industrial mobile manipulators (AIMM) and workstations. Also a set of guidelines for mapping of typical industrial tasks to the proposed calibration methods have been suggested. The high-precision calibration method is partly based on the Tsai hand-eye calibration, while the high-speed calibration method rely on a combination of laser triangulation and image processing. Both methods use dedicated calibration targets placed on the workstation. Experiments and applications have been performed with the full-scale mobile manipulator "Little Helper". The obtained results are promising - high-precision; [± 0.1 mm, ± 0.1◦ , 60 seconds] and high-speed; [± 1.0 mm, ± 1.0◦ , 10 seconds], which enable plug-and-produce functionality of mobile manipulators in industrial environments. In the future, we will focus on developing a compromise between precision (repeatability) and speed (processing time), to achieve a general workstation calibration technique for AIMM. We believe that great advances in technology, particularly in processing power and sensoring, provide significant opportunities for realizing this.
References [1] Stopp, A. et al: The Manufacturing Assistant: Safe, Interactive Teaching of Operation Sequences, IEEE Proceedings of International Workshop of Robot and Human Interactive Communication, Vol. 50. pp. 705707, 2003 [2] Coma, C. et al: An Autonomous Robot for Indoor Light Logistics, Proceedings of International Conference on Intelligent Robots and Systems, Vol. 3, pp. 3003-3008, 2004 [3] Katz, D. et al: The UMass Mobile Manipulator UMan: An Experimental Platform for Autonomous Mobile Manipulation, Workshop on Manipulation in Human Environments at Robotics: Science and Systems, 2006 [4] Datta, S. et al: Development of autonomous mobile robot with manipulator for manufacturing environ-
ment, International Journal of Advanced Manufacturing Technology, Vol. 38, pp. 536-542, 2008 [5] Ellekilde, L. and Christensen, H.: Control of Mobile Manipulator using the Dynamical Systems Approach, IEEE Proceedings of International Conference on Robotics and Automation, pp. 337-343, 2009 [6] Hvilshøj, M. et al: The Mobile Robot "Little Helper": Concepts, ideas and working principles, IEEE Proceedings of Emerging Technologies and Factory Automation, 2009 [7] Alici, G., and Shirinzadeh, B.: A systematic technique to estimate positioning errors for robot accuracy improvement using laser interferometry based sensing, Mechanism and Machine Theory, IEEE Transactions on Robotics and Automation, Vol. 40, pp. 879-906, 2005 [8] Zhang, Z.: A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, pp. 1330-1334, 2000 [9] Tsai, R. Y., and Lenz, R. K.: New technique for fully autonomous and efficient 3D robotics hand/eye calibration, IEEE Transactions on Robotics and Automation, Vol. 5, pp. 345-358, 1989 [10] Maeda, Y. et al: "Plug & Produce" functions for an easily reconfigurable robotic assembly cell, Assembly Automation, Vol. 27, pp. 253-260, 2007 [11] Seelinger, M., and Yoder, J. D.: Automatic visual guidance of a forklift engaging a pallet, Robotics and Autonomous Systems, Vol. 54, pp. 1026-1038, 2006 [12] Wengert, C. et al: Fully Automatic Endoscope Calibration for Intraoperative Use, Bildverarbeitung für die Medizin, pp. 419-423, 2006 [13] Download section on; http://www.machinevision.dk [14] Corke, P. I.: A robotics toolbox for MATLAB, IEEE Robotics & Automation Magazine, Vol. 3, pp. 24-32, 1996 [15] http://petercorke.com/Robotics_Toolbox.html [16] http://www.vision.caltech.edu/bouguetj/calib_doc [17] http://www.vision.ee.ethz.ch/~cwengert/calibration_toolbox.php
