Projector Calibration for 3D Scanning Using Virtual ... - Springer Link

5 downloads 0 Views 827KB Size Report
through this way the projector can be calibrated like a camera. ... KEYWORDS: Projector calibration, Intrinsic parameters, Extrinsic parameters, Homography.
INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 13, No. 1, pp. 125-131

JANUARY 2012 / 125

DOI: 10.1007/s12541-012-0017-3

Projector Calibration for 3D Scanning Using Virtual Target Images Hafeez Anwar1, Irfanud Din1 and Kang Park2,# 1 Graduate School of Mechanical Engineering, Myongji University, San 38-2, Namdong, Cheoin-gu, Yongin, South Korea, 449-728 2 Department of Mechanical Engineering, Myongji University, San 38-2, Namdong, Cheoin-gu, Yongin, South Korea, 449-728 # Corresponding Author / E-mail: [email protected], TEL: +82-31-330-6344, FAX: +82-31-321-4959 KEYWORDS: Projector calibration, Intrinsic parameters, Extrinsic parameters, Homography

In this work we propose a novel method to calibrate the projector. Calibration of the projector deals with the calculation of geometric parameters of the projector which are intrinsic and extrinsic parameters. An un-calibrated camera assists the process of finding the geometric parameters for projector. This method exploits the fact that a projector can be treated as inverse camera. The camera captures the scene and saves it as an image while the projector projects an image on the scene. Camera to projector transformation is used to make the projector able to see the calibration pattern. Then the projector is calibrated in a similar fashion like camera. Real data is used to evaluate the proposed method of projector calibration and good results are obtained. Manuscript received: December 30, 2010 / Accepted: August 2, 2011

1. Introduction 3D shape measurement and reconstruction or simply 3D scanning, has become one of the hottest fields in Computer Vision and Robotics during past few years. Researchers from various fields like Computer Vision, Robotics, Mechatronics, Intelligent Manufacturing Systems, and Applied Optics have worked enormously to find more robust, less complex, and fast 3D scanning techniques. These techniques are being adapted by medical, rapid prototyping, defense and other numerous industries. In Manufacturing where automatic inspection is used these days, 3D scanning can perform well.1 On the basis of their characteristics, these techniques are divided into two subgroups; passive scanning and active scanning. Passive scanning does not require direct control of illumination source, instead relies entirely on ambient light.2 This system usually makes use of two or more cameras to measure and recover the 3D geometry. The image of the object is captured by both the cameras from different positions and orientations simultaneously. Triangulation is then used for measurement of the 3D geometry. The bottle neck in the stereovision system is the correspondence. This is to find the corresponding points in the projection of the scene in one camera to the points in the other camera. To cope with the correspondence problem, various image processing techniques are used. The correspondence problem is not involved in active scanning techniques. In this method the projector projects a structured light © KSPE and Springer 2012

on a 3D geometry which is captured by a single camera. During the past few years a lot of work has been done on this technique and many people have come up with some very diverse ideas. This technique of reconstructing and measuring the 3D geometry is fast, robust, and inexpensive. Especially these days the decreasing prices of projectors and CCD cameras have made it easy to have a 3D measurement system. But before doing any re-construction and measurement process the projector and the camera system must be calibrated. The camera calibration has been heavily studied by the researchers therefore a vast variety of algorithms exist for it. This research focuses on the geometric calibration of the projector which deals with the calculation of the intrinsic and extrinsic parameters of the projector. Many researchers have worked on the geometric calibration of the projector. Zhang and Huang3 have given the idea to capture images with a projector. The projector is used to capture images like a camera, through this way the projector can be calibrated like a camera. They have made the projector see the chessboard pattern with help of a camera by using the sinusoidal pattern projection technique. The main difficulty lies in making the special setup of white and red light illumination. Li and Shi4 also proposed the calculation of the DMD image i.e. the image captured by the projector using vertical and horizontal fringe patterns, which makes it a time consuming method. Gao and Wang5 have done the projector calibration using Homographies. The problem involved in this idea is the use of red and blue pattern like Zhang and Huang.3 They also use too many

126 / JANUARY 2012

INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 13, No. 1

images of the printed chessboard because when they move the screen to a different position they must take the image of the printed chessboard that is attached to the screen in order to calculate the extrinsic matrix. Liao and Cai6 have proposed the projector calibration by uncoupling the projector and the camera. The results they had are good but they use a calibrated camera as well. The other limitation is that they use the image of the printed chessboard for each position of the plane. In this paper the projector calibration for the 3D measurement system is done based on the principles of the camera calibration. The proposed technique uses an un-calibrated camera for projector calibration. The camera and the screen are both fixed while the projector is moved to different positions and orientations in order to generate the virtual images. Virtual images are those images which are seen by the projector. This seems a bit impossible but in fact these images can be extracted with the help of the camera. The proposed technique uses an un-calibrated camera for projector calibration thus avoiding the errors of camera calibration to accumulate in the calculated parameters of the projector. Secondly, in this method the screen and the camera are both stationary due to which the transformation between the camera and the screen is constant. The extrinsic parameters of the camera are not calculated again and again as is done in5,6 when the screen is moved to different positions and orientations.

2. Background Knowledge 2.1 Camera Model Implication to Projector Fig. 1 shows the perspective projection models of the camera and the projector along with their coordinate systems. Let Pw = [Xw Yw Zw 1]T be a 3D point in world coordinate system and Pc = [uc vc 1]T be the 2D point in the camera image plane. Pw is projected on Pc up to a scale factor ‘s’ according to the pinhole camera model, as follows; P c = sM c [ R c t c ] P w

(1)

c

where ‘M ’ is the set of intrinsic parameters of the camera and is given as  f x 0 cx    M =  0 f y cy  0 0 1    c

In intrinsic matrix (cx , cy) are the co-ordinates of the principal focus and (fx , fy) are the focal lengths along the x and y axes of the image plane respectively. [Rc tc] represents the transformation called extrinsic parameters. It gives the rotation and translation between the world and the camera coordinate systems. The projector can be considered as a camera that acts in reverse manner. It projects the image instead of capturing it. Therefore the pinhole camera model can be applied to the projector as well. Let Pp = [up vp 1]T be a 2D point in projector’s image plane which is projected to a 3D point Pw = [Xw Yw Zw 1]T in world plane. For this 2D-3D point pair Eq. (1) can be modified as P p = sM p [ R p t p ]P w

(2)

Here Mp is the set of intrinsic parameters and [Rp tp] is the set of extrinsic parameters of the projector.

2.2 Zhang’s method A lot of work has been done on camera calibration during the past few decades. The latest technique that is used by most of the researchers is Zhang’s method.7 This technique uses pinhole camera model which has focal length, pixel size, and skews factor as intrinsic parameters and the translation and rotation of the camera reference frame with respect to the world reference frame as extrinsic parameters. The calibration is simply a process that finds the intrinsic and extrinsic parameters of the camera. This method assumes the camera with no distortions while solving for calibration parameters. Based on Zhang’s method the calibration process is programmed using the following steps. 1. A regular shaped object like a chessboard is attached to a flat and smooth plank. 2. Images of the object are captured at different positions and orientations. 3. The feature points in the image are detected using a function of OpenCV and stored as ‘image points’. 4. The coordinates of the feature points in world coordinate system are stored as ‘object points’. 5. Both of the matrices are provided to the main calibration function in OpenCV to find the intrinsic and distortion parameters of the camera. 6. The set of extrinsic parameters of the camera are then determined with the help of the intrinsic parameters.

2.3 Planar Homography Pp=Point in the projector image plane Pw= Point in the world coordinate system

Screen

Y

w

Z

w

X

w

c H Hwwp

P

world

Z

c

Z

PPcam c

PPproj p

X

c

Y

c

p HH w

p

Homography is a nonsingular 3×3 matrix, which defines a homogeneous linear transformation from a plane to another in a projective space.10 Let the xy plane of the world coordinate system coincide with the screen’s plane as shown in Fig. 1, then a 3D point on the screen becomes Pw = [Xw Yw Zw 1]T and according to pinhole camera model

X

p

Hpcp

Y

p

H Fig. 1 Mathematical Model of the Camera and the Projector

 r11c u c   c  c c v  = sM  r21  c 1     r31

X w r12c r13c t xc     Yw r22c r23c t yc       0 r32c r33c t zc    1 

INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 13, No. 1

or  r11 u   c  c c v  = sM  r21  c 1     r31 c

c

r12 t   r22c t yc   r32c t zc  c

c x

X  w Y 1 

w

    

Let H wc be the Homography from world plane to the camera image plane which is given as c  r11c  h11 h12 h13     H wc =  h21 h22 h23  = sM c  r21c  c  h31 h32 h33   r31 w

r12c t xc   r22c t yc   r32c t zc 

A point Pp is projected to Pw, a point on the world plane (i.e. screen). And this point is re-projected to another point Pc in the camera image plane as shown in Fig. 1. Then Pw and Pc are related to each other according to the following relation.

P c = H wc P w

(3)

As it is stated before that the projector can be considered as a camera acting in reverse order so the pinhole camera model can also be applied to it. Therefore a point Pp on projector’s image plane can be related to a point Pw on world plane and this relationship can be presented in the following form

P p = H wp P w

(4)

where H wp is the planar Homography from the world plane to projector image plane. Now combining Eq. (3) and Eq. (4), we get

P p = H wp ( H w ) P c −1 c

or P p = H wp H cw P c

(5)

Let H cp be the Homography from the camera to the projector which is given as

H cp = H wp H cw

(6)

Now putting Eq. (6) in Eq. (5) we get

P p = H cp P c

JANUARY 2012 / 127

virtual calibration patterns. These are the images of the printed chessboard seen by the projector. In this section the whole procedure for generating the virtual calibration patterns will be discussed. The method proposed in this research uses the rotation and translation of the projector instead of the screen. The screen and the camera both are fixed. The camera is tilted and fixed at 20º with respect to the stationary screen. First of all the printed chessboard is attached to the screen and its image is captured with the camera. The printed chessboard is then removed from the screen. The projector is roughly brought to a fronto-parallel position w.r.t the screen. A known chessboard pattern is then projected by the projector on the screen and captured with the camera. p be the known pattern that will be projected by the Let Pprojected c be the image of the projected projector on the screen and Pprojected pattern captured by the camera as shown in Fig. 2. Then Eq. (7) can be written as p c Pprojected = H cp Pprojected

The corners of both the chessboards are then detected using the functions in OpenCV. These corners are then used to determine the Homography H cp between the fronto-parallel position of the projector and the camera. Let the image of the printed chessboard pattern captured by the c p and the virtual calibration pattern be Pprnted , then camera be Pprnted Eq. (7) can be written as p c Pprnted = H cp Pprnted

Since the Homograph H cp is known, it can be applied to the image of the printed chessboard captured by the camera to generate the virtual calibration pattern. The images of the printed chessboard are captured by the camera and the corresponding virtual calibration pattern is shown in Fig. 3. Screen

Screen

As the Homography H cp between the camera image plane and the projector image plane is a 3×3 matrix with eight unknowns therefore at least four pairs of corresponding points in both the planes are sufficient to find it. Homography is calculated using the function in OpenCV which chooses normalization where ninth parameter of the Homography matrix is 1.8

w Pprntd

P

w projected

(7)

c Pprojected

Camera

Projector

For Projected Pattern

3. Virtual calibration pattern generation The images of the chessboard with different positions and orientations can be used to calibrate the camera. As it is mentioned earlier that the projector can be considered as a reverse camera, so if the projector is made able to capture the images of the printed chessboard pattern then it can also be calibrated like a camera using Zhang’s method. It sounds a bit impossible but in fact it can be done with the help of a camera which would work along with the projector to generate these virtual images which are named as

c Pprntd

p Pprojected

Camera

p Pprnted

Projector

For Printed Pattern

Fig. 2 Projected and printed chessboard points

p H c * c Pprnted

p Pprnted

Fig. 3 Image of the printed chessboard seen by the camera and virtual calibration pattern for fronto-parallel pose of the projector

128 / JANUARY 2012

INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 13, No. 1

In Fig. 4, let the fronto-parallel pose of the projector be named as {p1}, then for this pose Eq. (6) can be written as H wp = H cp H wc H wp1 = H cp1H wc or H = H pc1H wp1

Now the projector is moved to another pose. Let this pose of projector be named as {p2}, then for this position Eq. (6) can be written as

or H wc = H pc 2 H wp 2

H pc1H wp1 = H pc 2 H wp 2 H wp 2 H pw1 = H cp 2 H pc1 H

Screen

p2 c

=H H

(10)

c p1

printed chessboard

{w}

The Homography between two poses of the projector can be calculated using Eq. (10) and the virtual calibration pattern for the fronto-parallel pose is already calculated which will allow calculating the virtual calibration pattern for the second pose easily. To write Eq. (10) more generally,

H ppi1 = H cpi H cp1

(9)

Since L.H.S of Eq. (8) and Eq. (9) are the same which means that camera to screen Homography is constant, so combining both of them gives,

p2 p1

p2 p1 Pprntd = H pp12 Pprntd

(8)

c w

H wp 2 = H cp 2 H wc

Now let the virtual calibration pattern at fronto-parallel position {p1} p1 be Pprnted , and the virtual calibration pattern at new pose {p2} be p2 Pprnted then Eq. (7) can be written as

(11)

From Eq. (11) it can be summarized that the Homography between the first pose and any arbitrary pose of the projector can be calculated using the projector to camera Homographies of both the poses. Then using this Homography the virtual calibration pattern for any arbitrary pose can be calculated from the virtual calibration pattern for the fronto-parallel pose. In Fig. 5, some virtual calibration pattern for some arbitrary projector poses are shown which are generated on the Homography calculated by Eq. (11) and the virtual calibration pattern for the first pose. start : Bring projector to fronto-parallel position w.r.t the screen {p1}

STEP_1

: Take image of the printed chessboard attached to c the screen. {Pprnted }

H wc

STEP_2

HHcpcp1

pro p1 _ 1

ZZpro_1 Z ZZ

Z2pro _ 2

Projector

Pose 2

pose 1

{P2}

P

2

pro_ 2

X X2

{c }

i

{P1} X1

X P YY1 pro _ 1

Y Y2pro _ 2 pro_ 2 p _1 p pro pro _2

H pp

p_2 p2 2

p1 p_1

H

STEP_3

i

Pose i

pro_1

H pro _ 1

p {Pi} X X

Projector

1

H

Camera Camera

: Project a known chessboard pattern from projector and capture with camera. {P p1 }

_

i pro i

1

Projector

_i _1

p_i pi i pro

p1 p_1 pro

P

projected

i

YY

i

pro

pro

pro

_i

: Calculate camera to projector Homography using image set from step 3. {H p1 }

_i

STEP_4

c

_1

: Transform the image taken in step 2 to projector image plane using Homography from step 4.

STEP_5

Fig. 4 Projector poses and Homographies from camera to projector pose 1, from projector pose 1 to pose 2 and pose i

{P } p1 prnted

: Move projector to another position i {pi}

STEP_6

p_2 p2

pro = Hp_1 p1 pro

p_3 p3 p_1 pro Hp1 pro p_4 p_1 p4 pro p1 pro

= H p4 prnted

P

STEP_7

p1 Pprnted

= p3 Pprnted

: Project a known pattern and capture with camera and calculate camera to projector Homography. {H cpi }

*

p2 Pprnted

.

: Calculate Homography from first projector position to projector position using camera to projector Homographies of both positions. {H 1 }

STEP_8

ith

*

pi

p

: Calculate the new image of the printed chessboard for

STEP_9

p1 Pprnted

YES

i=i+1

*

new position using Homography from previous step and image from step 5. i