Calibration of a Structured Light System: A Projective Approach Du Q. Huynh Centre for Sensor Signal and Information Processing Signal Processing Research Institute Technology Park, The Levels, SA 5095, Australia
[email protected]
Abstract We present in this paper a novel calibration method that uses cross ratio t o compute world points falling onto any given light stripe plane of a structured light system. We show that, by using 4 known non-coplanar sets of 3 collinear world points, the direct 4 x 3 imageto-world transformation matrix for each light stripe plane can also be recovered from plane-to-plane homography. Preliminary experiments conducted with a calibration target and a mannequin suggest that this novel calibration method is robust and is applicable t o many shape measurement tasks.
Figure 1. A structured light system. where M and m are the homogeneous coordinates of a world point (relative to a given coordinate frame) and its image point, and s is an unknown scalar. It is known that P can be estimated if at least 6 known noncoplanar world points and their corresponding image projections are available [2, 41. The projector calibration stage involves estimating, for each light stripe plane, the unknown coefficients a, b, c, and d such that
1. Introduction The two basic components of a structured light system are a camera and a light (or laser) stripe projector (Figure l), displaced either laterally or vertically relative to each other in space. Because energy is emitted into the scene, the term active triangulation is often used in conjunction with such a system. Unlike passive triangulation with two cameras, self-calibration cannot be applied to active triangulation, as “image points” are not available from the projector and “correspondences” cannot be established. The absence of this difficult matching problem makes active triangulation an attractive method (see [5]for a general review) for many practical shape measurement tasks - with the trade-off that the system must be fully calibrated prior to use. The traditional approach to calibrating a structured light system takes two separate stages: Camera calibration and projector calibration. The first stage is to compute the 3 x 4 world-to-image perspective projection matrix P as given below:
PM =sm,
1063-6919/97$10.00 0 1997 IEEE
aX1+ bXa + c X ,
+d =
0.
(2)
Since the coefficients in (2) are defined up to a scale, at least 3 non-collinear world points that fall onto the stripe plane must be known. The problem with this is that it is not practical to set up a large number of known world points for all light stripe planes, and moreover, as discussed in [3, 71, known world points on the calibration target often fall between, rather than exactly on, the light stripe planes. In the traditional calibration approach, this problem can be overcome because matrix P is known from the first calibration stage. For each image point that lies on a given light stripe, 2 equations are available from (1). Furthermore, because calibration targets often consist of two or three mutually orthogonal surfaces whose plane equations in the world coordinate frame are known, if the image point is known to correspond to a world point that lies on one of these surfaces, then the plane equation of the surface provides 1 additional equation. By in-
(1)
225
corporating these 3 equations together (i.e. triangulation), the world coordinates of the given image point can be estimated. Repeating this process for a number of non-collinear world points yields an over-constrained system of equations in the form given in (2); solving for the plane coefficients is then a trivial procedure. Approaching this calibration problem from a different angle, we propose in this paper a novel projective calibration method that uses 4 known non-coplanar sets of 3 collinear world points and computes, for each light stripe plane illuminated from the projector, a unique 4 x 3 image-to-world transformation matrix. These 12 calibration points can either be marked on a calibration target such as that shown in Figure 4 or be marked on 4 non-coplanar vertical strips permanently fixed in the workspace. The rest of the paper is organized as follow: A review on projectivity is outlined in section 2, the proposed projective calibration method is detailed in section 3, experiments are then given in section 4, and conclusion is drawn in section 5.
to p on the line. The cross ratio {p, q;r, m} of these points is then defined as follows:
It is known that collinearity and cross ratio are invariant under perspective projection [87. That is, if 4 world points P , Q , R , M are collinear and have cross ratio equal to T , then their image projections p, q, r, m are also collinear and have the same cross ratio. The above indicates that given 3 known collinear world points P , Q, R on a line and an unknown world points M also on 5, the coordinates of M can be computed from the parameters %p,OQ, %R and the cross ratio T of the image projections of these 4 points.
c
2.2. Homography Given 4 or more points mi in general positions, where i = l...n, n 2 4, on a projective plane II and their projections ma on another projective plane 11' under a perspective centre C, there exists a homography H I i.e. a unique 3 x 3 non-singular matrix, defined up to a scale, such that
2. Projectivity Throughout this paper we denote the known world coordinate frame by F,,,; the image plane by n; the k-th light stripe plane by IIk; world points by bold upper case 3-vectors, e.g. M = ( M i ,M2, M3)'; world lines, each of which is internally represented as a 3vector for the line direction and a world point that lies on it, are denoted by, bold upper case letters with an overhead arrow, e.g. L;image points by bold lower case 2-vectors, e.g. m = (m1,mz)'; matrices by upper case. In addition to the above notations, homogeneous or projective coordinates are denoted by adding a - above the entities, e.g. M = (MT,1)' = (Ml,M2,M3,1)' and m = ( m l , m z , l ) ' for the homogeneous coordinates of a world point and an image point. The readers are reminded that M and A M , for any non-zero scalar A, denote the same point.
Hmi = p i m a , where pi's are unknown scalars. The estimation of H involves solving a linear system of equations or finding the vector that corresponds to the smallest singular value of the associated system.
3. A projective approach 3.1. Related work Chen and Kak [3] proved that in active triangulation systems, image coordinates can be directly transformed to world coordinates via a 4 x 3 image-to-world transformation matrix that is defined up to a scale. They showed that to determine the 11 unknown elements of the matrix at least 6 world lines are required, and that each of these lines can be defined as the intersection of 2 known world planes. Since their laser range finder was incorporated with a robot arm, world lines could be generated by moving the robot arm that carried the scanner to different known positions. Reid [7] later extended the concept of known world line to image point correspondences proposed by Chen and Kak to known world plane to image point correspondences. The laser range finder used in [7] was mounted on an AGV whose position relative to the world coordinate frame was known. In addition, the
2.1. Cross ratio Given 4 collinear points p, q, r, m lying on a line 1, there is an infinite number of ways to parameterize 1. A simple parameterization is to express points on 1 in the following form: (q - PI %
+ P,
where 9 is the parameter that defines points on the line. = 0, 6, = 1, 0,. and ,6 This parameterization gives any real numbers depending on their positions relative
226
relate the coordinate frame FTk2, which has yet to be determined, with the world coordinate frame 3w, we define a 3D coordinate frame FTk3 such that the X3axis of 3Tk3 coincides with the normal vector of II,. It is apparent from this definition that both the XIand Xz-axes of 3*,, would lie on &, although their directions are still not determined at this stage. The arbitrary 2D coordinate frame FTkz is now defined as having its two axes coinciding with the XI- and X2axes of 3Tk3. Given that the 4 world points Mi, i = 1...4 lying on IIk have been determined, the following simple procedure shows how points in Fw can be transformed to points in 3Tk3 and how the XI- and X2-axes of 3Tk3 and 3TkZ are defined:
transformations from sensor coordinate frame to vehicle coordinate frame, from the fixed light stripe coordinate frame to sensor coordinate frame, etc, were all known. This reduces the problem to solving a 4 x 3 image-to-stripe transformation matrix whose first row is identically zero. Thus, in his system, 8 elements of the matrix must be determined, and a number of known world planes are required.
3.2. Computing the image-to-world transformation Inspired by [3, 71, we propose that the image-toworld transformation matrix for each light stripe plane illuminated from the projector can be computed from known world points. Our structured light system consists of a camera and a projector. The set up is similar to that depicted in Figure 1. As mentioned in section 1, our projective calibration method requires 4 non-coplanar sets of 3 collinear points known relative to a world coordinate frame (Figure 2).
0
compute, and normalize to unit length, the normal vector n = (n1,n2,n3)T of n k using the 4 computed world points Ma, i = 1 . . .4. In practice, Mi, V i may not be exactly coplanar in space, so a plane must be first fitted to the computed world points, vector n is then defined as the normal vector of the fitted plane.
+
C
compute the angle between n and the X3-axis (=(0,0,1)T) o f F w : b?, = COS-'(Il. x 3 ) = COS-l(n3). 0
Figure 2. Projectivecalibrationusing 4 known non-coplanar sets of 3 collinear world points (dark circles {Pi, Q i , Ri I i = 1...4}). Open circles Mi's are unknown world points lying on I I r ; c and C' are the perspective centres of the camera and projector.
0
compute, and normalize to unit length, the vector a that is orthogonal to n and X3:
compute the rotation matrix R that brings X3 to align with n . It can be perceived from Figure 3 that a rotation of angle II, about an axis passing n
We have shown in section 2.1 how the coordinates of an unknown world point that is collinear with 3 known world points can be computed. Here, we extend this to 4 known non-coplanar sets of 3 collinear world points.
As shown in Figure 2, these known collinear world points {Pi, Q;, R.},i = 1...4 do not fall onto the light stripe plane IIk, but the 4 world points Mi, i = 1...4
Figure 3. The relationship between 3Tk3 and Fw. The X3-axisof FTk3 Is parallel to the normal vector n of &.
that do fall onto IIk can be computed. Let 3Tk2 be a 2D coordinate frame on I l k . Then, as given in section 2.2, any 4 or more points in general positions defined in FTk2 and their corresponding image projections determine a unique homography Ha. Since the homography exists regardless of the choice of 2D coordinate frames in both projective planes, without loss of generality, 3Tk2 can be arbitrarily defined. To
through the origin and parallel to a would do the job. The matrix for this rotation can be derived from Rodrigues formula and can be found in most graphics and kinematics books, e.g. [6]. So, define
R = aaT + c o s + ( I - aaT) + s i n 4 ( I x a),
227
0
where m = (mT,l)T, m' E FTkZ, and p is a nonzero scalar. Suppose that pm' = (mi, mi,m $ ) T ,then relative to Frk3,this is simplily (mi, mg, 0, m$)T. Let M be this point in FTk3then it follows that
where I is the 3 x 3 identity matrix and I x a is the skew-symmetric matrix formed by a. If n and Xs are parallel then II, = 0 and R = I . define the 4 x 4 matrix
x
M' = S T ( pm'), "
(3)
where S is given in ( 5 ) . Since Tk is in the group of Euclidean transformations, TI,is invertible. Substituting (4) and (7) gives
m
where is the centroid of { M i l i = 1...4}. Ideally, matrix TI, would transform any world point M = ( M I ,M2, M3, l)T lying on III, to the form M' = ( M i ,M i , 0,l)'. TI, is thus a transformation from F, to FTk3:
M = T , - ~H~s m. ~
(8)
E,,
(9)
Define :F
Tk
w
+
FTk3
TI,M= M',
(4)
This is the 4 x 3 image-to-world transformation matrix that maps any given image point lying on the k-th light stripe directly to projective coordinates in Fw.
and it has also defined the XI-and Xa-axes of both FTk3and FTk2. Note that the origin of FTk3can be chosen arbitrarily. We have chosen the centroid as the origin for practical reason. Because the computed Mi,Vi may not be exactly coplanar in space, the third Component of M i , i = 1...4 after the TI, transformation may not be identically zero. This third component of each Mi, although non-zero, has been found to be insignificant in comparison with the other two components, and choosing the centroid as the origin of FTk3helps average out these small non-zero values. relative to the local 2D coordinate frame FTkZ on H I , , M' can be expressed in homogeneous coordinates as m' = ( M I , M l , I ) ~ .Points in F,,, can now be directly computed from world points lying on IIr, via the following 3 x 4 transformation:
3 3 . Summary and discussion In summary, for each light stripe plane IIk illuminated from the projector, the procedure for projective calibration is as follows:
1. compute the 4 image points ma that the k-th light stripe intersects with la, i = 1...4.
m
0
Tw,,
T,,,
F w
=
[
-
Frka
1 0 0 0 0 1 0 0 0 0 0 1
]
2. compute, using cross ratio, the 4 points Mi E Fw that III,intersects with Li.
3. follow the procedure described above and compute the 4 x 4 transformation Tj defined in (3). 4. compute T,,, given in (5), and then mi = T W r k M ii, = 1...4. 5. compute the homography Hk that maps mi to mi, i = 1...4. 6. compute Tjwkdefined in (9).
TI,= S T I , . ( 5 ) Our method is different from those given in [3, 71 in the following respects:
Given that Mi E F, ,V i can be transformed to m: E FTk2 via T,,, , the homography HI, that satisfies
H~,mj= pi&:,
i = 1...4
0
(6)
can be computed. Here, m i ' s are the intersection points of the k-th light stripe with Id's (Figure 2). Our aim now is to estimate the image-to-world transformation Ti,, for every image point m that is known to be the projection of a world point M lying on nk. Since HI, and Turk have been estimated from the above procedure, given m E 11 and using (6) yields H k m
= pm',
=T , - ~ s ~ H ~ .
0
(7)
228
While world line to image point correspondences are used in [3] and world plane to image point correspondences are used in [7], world point t o image point correspondences are used here. Because our method uses world point to image point correspondences, we are able to compute the world to stripe plane transformation TI, and homography H k , and estimate the image-to-world transformation from explicit matrix multiplications. In [3] (or [7]), this matrix is estimated from world line (or world plane) to image point correspondences.
The origin and coordinate axes of our 2D coordinate frame FTk2 on each stripe plane are chosen such that any errors that may arise from the world points, Mj V i , not being perfectly coplanar can be kept within acceptable bound. In [3, 71, the issue on such errors is not considered.
4. Experiments Figure 4. The calibration target used for testing the projective calibration method.
Two of the experiments for testing the projective calibration method are detailed here. To carry out the experiments, the output data produced by the SHAPE measurement system [l] were used. Because the SHAPE system is protected under copyright, to completely avoid any involvement with the SHAPE software, we have written our own Matlab programs and used only the output data of SHAPE in text format. For the completeness of this paper, the SHAPE system is briefly described below, interested readers may refer to [l] for more detail. The basic components of the SHAPE measurement system are a camera and a projector. The 64 light stripes, labelled as 0 to 63, in the modified 35” slide projector are horizontal in direction. The calibration target used in SHAPE consists of two perpendicular surfaces, corresponding to the planes X I = 0 and X2 = 0. The 6 calibration points on each surface are arranged into 2 sets of 3 collinear points parallel to the X3-axis. These points are known with high accuracy (to the order of and are ideal for testing the calibration method proposed in this paper. SHAPE can generate up to 8192 data points (i.e. 128 points per stripe). Each data point contains information such as the stripe number to which the data point belongs, the image coordinates, and the computed world coordinates. Only the stripe number and the image coordinates of each data point are required for our experiments. The SHAPE world points are only used for comparison with the world points obtained from our method. An image of the calibration target, placed at approximately l m away from the camera and illuminated by all the light stripe planes, is shown in Figure 4. The coordinates of the 12 known world points {Pi,Qj, R, I i = 1...4} for calibration were firstly loaded into Matlab, the parameter 0 (see section 2.1) of each point was then estimated. The image coordinates of these world points were manually detected, the fitting of the image lines lj, i = 1...4 then followed. To determine the intersection point of a light stripe with each lj (the open circles mi, i = 1...4 shown in Figure 2), a neighbourhood of 20 pixels on either side of each lj was specified. Image points on the stripe falling
into this neighbourhood were automatically selected, an image line 1 was then fitted to these points, the intersection point was finally computed by solving la and 1 simultaneously. On average, 6 image points were selected for this neighbourhood size. After this, the world points Mi, Vi were computed from cross ratio. Following the summary of the calibration method given in the previous section, the imageto-world matrix for the k-th stripe was computed. The final procedure was to transform all image points on the stripe to projective coordinates using Vlc. The data file from SHAPE in this first experiment contains a total of 5681 points, ranging from stripe 9 to stripe 63. Our method gave a total of 5348 points, ranging from stripe 14 to stripe 60. The reason that our method missed the first 5 and the last 3 stripes is due to the intersection point, mi for some i, for these stripes being outside the image boundary and so the Tiwkmatrix for these stripes could not be determined. Since the computed world points are too dense for visualization, only alternate world points from alternate stripes are selected and displayed in Figure 5. The plan view shows a very good coplanarity of the computed points on each surface of the calibration target; points on both surfaces also form a good right angle.
zwk
xwk,
(a) plan view
(b) perspective view
Figure 5. World points computed by the projective calibration method. The unit of each axis is metre. We should note that lens distortion has not been considered in our calibration method. However, for the type of lens used in our experiments, lens distor-
229
tion has been minimized to a negligible level (observe the straightness of the line stripes in Figure 4). The location of image points on each stripe is also a major factor that determines how well world points can be computed. In the SHAPE system, the centre line of each line stripe is detected to sub-pixel accuracy, so the image coordinates that we used in the experiments have been accurately determined. SHAPE uses the Direct Linear Transform (DLT) described in section l to calibrate the camera and projector and computes world points by triangulation. The availability of the SHAPE world points makes it convenient for us to assess our calibration method. We compared the coordinates of world points obtained from the two methods by computing the Euclidean distance between each pair of corresponding world points produced by both methods. In the first experiment, the computed mean and standard deviation of all these distances were 0.4269" and 0.3292". (a)
(b)
5 . Conclusion We have presented a novel projective calibration method for a structured light system. The method uses 4 known non-coplanar sets of 3 collinear world points and, based on the invariance of cross ratio under perspective projection, computes world points that fall onto each light stripe plane illuminated from the projector. The computed world points and the corresponding image points determine a unique homography between the image plane and each light stripe plane. Finally, for each light stripe plane, the 4 x 3 imageto-world transformation is recovered from matrix multiplications. Experiments conducted for the proposed projective calibration method suggest that it is robust and can be applied to many shape measurement tasks.
Acknowledgements This research was accomplished while the author was visiting the Department of Computer Science, The University of Western Australia. The author would like to thank the following people for making it possible to carry out this research: Assoc. Prof. Owens, Prof. Hartmann, Leon Robert Mitoulas, and Jackie Kent.
References [l] B. F. Alexander and K. C. Ng. 3D Shape Measurement by Active Triangulation using an Array of Coded Light Stripes. In Proc. SPIE: Optics, Illumination and Image Sensing for Machine Vision 11,volume 850, pages 199209, Nov 1987. [2] R. C. Bolles, J. H. Kremers, and R. A. Cain. A Simple Sensor to Gather Three-Dimensional Data. Technical Report 249, SRI International, Stanford University, 17 Jul 1981. [3] C. H. Chen and A. C. Kak. Modeling and Calibration of a Structured Light Scanner for 3-D Robot Vision. In Proc. IEEE Conf. Robotics and Automation, volume 2, pages 807-815, 1987. [4] 0. Faugeras. Three-Dimensional Computer Vision A Geometric Viewpoint. The MIT Press, Cambridge, 1993. [5] R. A. Jarvis. A Perspective on Range Finding Tech-
Figure 6. Image of a mannequin and its reconstruction using projective calibration. Figure 6(a) shows an image of a mannequin, in our second experiment, with all light stripes superimposed. Using the computed Tiwkmatrices from the previous experiment for stripes 14 to 60, the reconstructed world points are displayed in Figure 6(b). In this experiment, the numbers of data points from SHAPE (stripes 9 to 63) and our method (stripes 14 to 60) were 4560 and 4027, respectively. The mean and standard deviation of the Euclidean distances of corresponding world points, excluding those on the wall in the far background, were computed to be 0.7240" and 0.4447". The SHAPE reconstructions show no visible differences from those shown above. In fact, an examination on experiment 1 reveals that the world points from the projective method have a better fit onto both surfaces of the calibration target, i.e. the absolute values of the X1- or X2-components are smaller. The small values of the computed means and standard deviations in both experiments show that the projective method performs as good as the traditional DLT.
niques for Computer Vision. IEEE Trans. on Pattern
Analysis and Machine Intelligence, 5(2):122-139, Mar 1983. [6] T. Pavlidis. Algorithms for Graphics and Image Processing. Springer-Verlag Berlin-Heidelberg, 1982. [7] I. D. Reid. Projective Calibration of a LaserStripe Range Finder. Image and Viston Computing, 14(9):659-666, Oct 1996. [8] J. G. Semple and G. T. Kneebone. Algebraic Projective Geometry. Oxford University Press, 1952.
230
