A method is proposed for estimating the spectral reflectance function of an object surface by using a six-color scanner. The scanner is regarded as a six-band ...
Spectral Reflectance Estimation Using a Six Color Scanner Shoji Tominagaa, Satoshi Kohnob, Hirokazu Kakinumaa, Fuminori Noharaa, Takahiko Horiuchia a Division of Information Sciences, Graduate School of Advanced Integration Science b Department of Information and Image Science, Faculty of Engineering Chiba University, 1-33, Yayoi-cho, Inage-ku, Chiba 263-8522, Japan ABSTRACT A method is proposed for estimating the spectral reflectance function of an object surface by using a six-color scanner. The scanner is regarded as a six-band spectral imaging system, since it captures six color channels in total from two separate scans using two difference lamps. First, we describe the basic characteristics of the imaging systems for a HP color scanner and a multiband camera used for comparison. Second, we describe a computational method for recovering surface-spectral reflectances from the noisy sensor outputs. A LMMSE estimator is presented as an optimal estimator. We discuss the reflectance estimation for non-flat surfaces with shading effect. A solution method is presented for the reliable reflectance estimation. Finally, the performance of the proposed method is examined in detail on experiments using the Macbeth Color Checker and non-flat objects. Keywords: Spectral reflectance estimation, color scanner, six sensors, multispectral imaging
1. INTRODUCTION A scanner is considered as a precise imaging device for document and objects with flat surface. The device can acquire images with high resolution and without camera lens distortion. Traditional three color scanners could suffer from color reproduction errors due to the significant mismatch between their spectral sensitivities and those of the human visual systems. Recently a scanner was developed for capturing additional color channels to reduce the color reproduction errors [1]. The novel scanner captures six color channels in total from two separate scans using two different fluorescent lamps. The present paper proposes a multiband spectral imaging system with high spatial resolution by using this type of scanner. We regard the scanner as a six-band spectral imaging device, because it has three color sensors and can capture six spectral images by two scans of two light sources. The sensors have three broad-spectral sensitivities in the range of visible wavelength. The light sources are fluorescent lamps with spiky spectra of white and blue. We use raw data of the six channel outputs for estimating the surface-spectral reflectance of an object at every pixel point. The performance of the spectral reflectance estimation is compared with the estimation results by using another multiband imaging system based on a digital camera. First, we describe the basic characteristics of imaging systems. The scanner we used in this study is a HP Scanjet G4050 color scanner. We examine the spectral signals of the scanner outputs, which are normalized with the spectral reflectance of a white reference. We show the spectral response functions and the output linearity of the scanner. A multiband camera system with six color filters is used for comparison in this study. Second, we describe a computational method for recovering surface-spectral reflectances from noisy sensor observations. Our algorithm provides an optimal estimator in the sense of a linear minimum mean-squared error (LMMSE). Moreover, we discuss the reflectance estimation for non-flat objects. A method is devised for scanning a non-flat object with shading effect and obtaining a reliable estimate of the surface-spectral reflectance. Finally, the performance of the proposed method is evaluated on experiments in detail. We first check the basic statistical characteristics of the scanner outputs. We use the Macbeth Color Checker for recovering surface-spectral reflectances of the color patches from the scanner output data and in comparison the camera image data. The reflectance estimation of non-flat objects is also examined.
Color Imaging XIV: Displaying, Processing, Hardcopy, and Applications, edited by Reiner Eschbach Gabriel G. Marcu, Shoji Tominaga, Alessandro Rizzi, Proc. of SPIE-IS&T Electronic Imaging SPIE Vol. 7241, 72410W · © 2009 SPIE-IS&T · CCC code: 0277-786X/09/$18 · doi: 10.1117/12.805725 SPIE-IS&T/ Vol. 7241 72410W-1
2. BASIC CHARACTERISTICS OF IMAGING SYSTEMS 2.1 Scanner Figure 1 draws a schematic diagram of the six-color scanner. An object on the scanning plane is illuminated by light bulbs L1 and L2 with two different spectral properties from two directions. The scanner system uses a single three color CCD and two different cold cathode fluorescent lamps for two scans. The phosphor selection of the lamps is described in Ref.[1] in detail. We measured the spectral power distributions of two scanner lamps. In Figure 2, the solid curve (Bulb1) shows the spectral distribution of a standard white lamp. The dashed curve (Bulb2) shows the different spectral distribution of the second blue lamp. The spectral intensity of this lamp is higher in shorter wavelength of the visible range than one in longer wavelengths. Figure 3 shows the overall scanner spectral response functions by combing the spectral power distributions of the lamps and the spectral sensitivities of RGB sensors. The solid curves (Bulb1) represent the red, green, and blue responses for the standard white lamp, and the dashed curves (Bulb2) represent the red, green, and blue responses for the second blue lamp. z
7
"7
Scanning plane
-
Light bulb
Light bulb
Mirror
CCD sensor
Fig. 1. Schematic diagram of the scanner used in this study.
0.50
0.40
Bulbi
Bulbi Bu1b2
Bulb2 0.40 0.30
500 600 Wavelength (nm)
IR
600 500 Wavelength (nm)
700
Fig. 2 Spectral power distributions of scanner lamps.
G
------- - -
Normalized v
0.30
700
Fig. 3 Overall scanner spectral response functions.
The scanner outputs are quantized in 16 bits and normalized with a white reference at every scan. We investigated the linearity of the responses for reflecting objects. The Munsell Neutral Value Scale with 37-step scale was used as a set of gray scale samples. Figure 4 shows the relationships between the reflectance Y values of the gray samples and the RGB values of the scanner outputs. A good linear relationship is obtained in both scans.
SPIE-IS&T/ Vol. 7241 72410W-2
2.2 Camera The above scanner can be regarded as a six-band spectral imaging system. For the purpose of performance comparison in spectral reflectance estimation, we use a multiband camera with six channels. Figure 5 shows the camera system, which is composed of a monochromatic camera, a standard lens, and color filters (Wratten gelatin filters). The camera is a Toshiba Teli CCD camera with the image size of 1636x1236 and the quantization level of 10 bits. Figure 6 represents the overall spectral-sensitivity functions, which are determined by combining the spectral sensitivity function of the monochrome camera and the spectral transmittances of six color filters. The six spectral sensitivities are well separated in the visible wavelength range, compared with the six spectral responsivities of the scanner in Figure 3. This is because we have freedom to select filters for deterring the spectral sensitivity functions.
65535
60000
Bulb 1
Bulb2
-
Scanner outputs
50000
10000
0'0
20
40
60
80
100
Reflectance Y(%) Fig. 4 Relationship between the reflectance Y values and the output RGB values.
1.0
0.8
Shutter control Standard lens I
*
CCD camera
>06 N
500 600 Wavelength (nm)
Color filter Fig. 5 Six-band spectral camera system.
700
Fig.6 Overall spectral sensitivity functions.
3. REFLECTANCE ESTIMATION METHOD 3.1 Algorithm for reflectance recovery The problem is to recover the surface-spectral reflectance of a scan object from the scanner outputs by knowing the spectral response functions. The scanner output signals are normalized with a standard white reference in the range of 16 bits. The normalized RGB sensor outputs for the two scans are modeled as a linear system
SPIE-IS&T/ Vol. 7241 72410W-3
⎡ ri ⎤ ⎢g ⎥ = ⎢ i⎥ ⎣⎢ bi ⎦⎥
∫
700
400
⎡ R (λ ) ⎤ S (λ ) Ei (λ ) ⎢⎢G (λ ) ⎥⎥ d λ , ⎢⎣ B(λ ) ⎥⎦
(i=1, 2)
(1)
where S (λ ) is the surface-spectral reflectance, Ei (λ ) (i=1, 2) are the spectral power distributions of the white and blue light sources shown in Figure 2, and ( R(λ ) , G (λ ) , B(λ ) ) are the spectral sensitivity functions of RGB sensors. When we take any noisy observations into account, two sets of senor outputs are summarized as a six-dimensional linear system
ρi =
∫
700
S (λ ) H i (λ )d λ + ni ,
(i=1, 2, ..., 6)
(2)
400
where the spectral response functions H i (λ ) , shown in Figure 3, are defined as
H1 (λ ) = E1 (λ ) R(λ ) , H 2 (λ ) = E2 (λ ) R(λ ) , H 3 (λ ) = E1 (λ )G (λ ) , H 4 (λ ) = E2 (λ )G (λ ) , H 5 (λ ) = E1 (λ ) B(λ ) , H 6 (λ ) = E2 (λ ) B(λ ) , and ni are the noise component with zero mean, including image sensor noise and an approximation error in the model.
Assume that each spectral function is sampled at n points with an equal interval ∆λ in the region [400, 700nm]. Let s
be an n-dimensional column vector representing the spectral reflectance S (λ ) , s be the mean reflectance vector, H be a 6xn matrix with the element hij = hi (λ j ) ∆λ , and define ρ be a six-dimensional column vector representing the scanner outputs. Then the imaging relationships are summarized in a matrix equation ρ=Hs +n .
(3)
When the signal component s and the noise component n are uncorrelated, a linear minimum mean-squared error (LMMSE) estimate of s is given as sˆ = s + K ( ρ − H s ) ,
(4)
K = Css H T (HCss H T + Σ) −1
(5)
where CSS is an n × n covariance matrix that is statistically defined as Css = E[(s − s )(s − s )T ] (see [2]). The covariance matrix of the estimation error is then given as Cerror = Css − K HCss .
(6)
To determine s and CSS , we used a database of many surface-spectral reflectances for different paints and natural objects (e.g., see [3]). We can assume that the noises are statistically independent and each noise element has the same variance σ 2 . In this case, the covariance matrix is diagonal as Σ = σ 2 I . In this paper, we determined the noise variance empirically. 3.2 Reflectance estimation for shading effect The reflectance estimation method described above works well if the scan object is perfectly flat. However, if the object surface is not flat and has difference in level, the senor output signals reflected from the surface in the two scans vary according to the surface shape. Figure 7 illustrates the effect of shading for an object with difference in surface level. The slanting rays of either light source casts the shadow of the higher surface onto the lower surface. The sensor outputs for the shadow area decrease much, compared with ones in any other surface area, because the surface is shielded by another surface from the light source.
SPIE-IS&T/ Vol. 7241 72410W-4
In such an area, the six sensor outputs by the two scans are too unreliable to estimate the surface-spectral reflectance function. We examine that each point of the surface is properly illuminated by a light source. Notice that the shadows occurs only once in the two scans. If the pixel point belongs to the shadow area, we discard the RGB outputs of the corresponding scan and use the remaining RGB outputs of the other scan for spectral reflectance estimation. Examination of shading can be performed easily by intensity comparison between two RGB vectors by the two scans as follows: 2
2
2
r1 + g1 + b1
r2 + g 2 + b2 ≥ T 2
2
2
(7)
where T is a certain threshold. If a surface point satisfies the above condition, we take only the senor outputs ( r1 , g1 , b1 ) under the light source 1 for the estimation.
Light source 2
Light source I
Shadow area
Object
Shadow area
Object
Fig. 7 Shading effect for an object with difference in surface level.
4. EXPERIMENTS 4.1 Basic Characteristics We first check the basic statistics of output signals of the six color scanner and confirm the stability of output signals between two RGB sensors. A color printed paper was scanned with 600 DPI to capture the image with 2640x2122 pixels. Then the intensity ratio between two RGB vectors by the two scans, defined in Eq.(7), was calculated for every pixel. Figure 8 shows the ratio vs. accumulation value for the whole image. The average ratio was 1.040, and almost 95% pixels existed within the ratio of 1.100. Next, we compare the appearance of the scanner image with the camera image. A plastic plate including two kinds of minute texture was used as a test object. The size of the object was about 75x49mm. As the monochrome camera, we used the CCD camera in Sec.2.2 without color filters. For both systems, the sampling pitch was adjusted around 65 µm. Figure 9 shows the two captured images. We can see that the image captured with the scanner is more uniform in the entire region and expressible of the detailed texture patterns, compared with the camera image under the same resolution. The camera image may include the spatial non-uniformity of illumination and the effect of lens distortion. 100
80
60 C
C 40 C Q
0
I
1.0
1.1
1.2
I
I
1.3
1.4
1.5
1.6
Intensity ratio of RGB Vectors
Fig.8 Accumulation value as a function of the intensity ratio of two RGB vectors for an actual color image.
SPIE-IS&T/ Vol. 7241 72410W-5
(a) Scanner
(b) Camera
Fig. 9 Comparison of appearance between the scanner image and the camera image for a plastic plate with textures.
Measured Estimated 2 Os
0.8
0.8
0.8
U.b
3.0
(lb
0.8
0.4
04
0.4
0.4
0.4
0.4
Z 0.2
02
0.2
0.2
0.2
0.2
C
0400
500
o 2o
600
500
600
7000400
500
700 0400
600
500
o 04o
600
500
700 0400
600
jo
600
700
12
Nonnalized val
H
500
11
40U
UU
/UU 4U0
OUU
13 -
UU
IOU 400
OUU
0.8
0.6
0.6
0.6
0.4
0.4
0.4
600
700 400
500
600
19 0.8
0.61
0.6
0.4
0.4
0.4
Z 0.2
0.2
0.2
600
700
400
500
600
IUU 4UU
OUU
17
J 500
600
0.6
02''/
0.4
000
IOU
18
700 400
500
600
700 400
500
ft2
22
400
700
600
0.8
0.6
0.6
0.6
0.6
0.4 0.2
0.4
0----
0.4
0.2
0.2
500
600
700
Wavelength (nm)
0. 400
500
600
700 400
Wavelength (nm)
500
600
700
Wavelength (nm)
700
600
24
0.8
r
500
23 0.8
0
DUO
1
0.8
700 400
Wavelength (nm)
DUU
21
-
500
:2
20
0.8
Wavelength (nm)
DUO
0.8
700 400
-
0'400
400
15
0.8
500
/00
000
14
1
0.8
400
00
0
400
500
600
700
Wavelength (nm)
Fig. 10 Estimation results of spectral reflectances of 24 color patches in the Macbeth Color Checker.
4.2 Flat Object We examine the feasibility of the proposed method for estimating surface-spectral reflectance by using the six-color scanner. First, the Macbeth Color Checker was used as a test object. The object surface was scanned with 200 DPI and the sensor outputs were averaged over 50x50 pixels for each color patch. We assume the noise variance at each channel so that the signal-to-noise ratio of the imaging system is about 30 decibels (dB). Figure 10 shows the estimation results
SPIE-IS&T/ Vol. 7241 72410W-6
of spectral reflectances of 24 color patches. The estimated spectra are depicted in bold curves. In comparison, the broken curves represent the direct measurements by a spectrophotometer. We see that, although a good coincidence is obtained between the two curves for most patches, there is a discrepancy between two curves for Color Patch 2 of beige. Next, we performed the reflectance estimation of the Macbeth Color Checker by using the six-band camera system in Figure 5. The object surface was captured in a multiband image with about 1200x1600 pixels, and the sensor outputs were averaged over 100x100 pixels for each color patch. The same algorithm as the above was used for reflectance estimation. The noise variance was then set to be the signal-to-noise ratio of the range 25-30 dB. Table 1 shows the performance comparison between the two methods using the scanner and the camera. The performance is evaluated by the two indices of reflectance error and color difference. The first index is the root mean squared error s − sˆ and the second is the CIE-L*a*b* color difference under Illuminant D65. The two indices lead to the suggestion that the color error by the scanner is close to the one by the camera, although the spectral reflectance error by the scanner is larger than the camera. Table. 1. Performance comparison of reflectance estimation between two imaging systems.
Scanner
Camera
Average RMSE
Average ∆Lab
Average RMSE
Average ∆Lab
0.04043
4.90
0.02780
4.74
4.3 Object with shading effect Figure 11 shows the scanned images of a thick cardboard. A chip of the yellow cardboard with a thickness of about 0.5 mm is placed on a white paper. The first scan captures an image of the non-flat object illuminated by the first white lamp as shown in Figure 11 (a). The second scan captures an image of the same object illuminated by the second blue lamp as shown in Figure 11 (b). Shadows appear on the white back paper in opposite directions at the upper and lower edge portions.
Dl
2D
(b) Fig. 11 Scanned images of a yellow thick cardboard. (a) First scan image under white light source. (b) Second scan image under blue light source.
Use of the six-sensor outputs resulted in a quite poor estimate of the surface-spectral reflectance for the back paper in the shadow area. So we discarded the dark RGB image data of one scan and used only the bright RGB image data of the other scan. Figure 12 shows the estimation results of spectral reflectances of four areas with 3x3 pixels. All the sixsenor outputs by both scans were used for the estimates 1 and 3, and only the three-sensor outputs of a single scan were used for the estimates 2 and 4. The broken curves represent the direct measurements. Although we used the single scan data, the spectral reflectances are well recovered for the shadow areas. We applied the proposed algorithm to the whole image including shadows, scanned in 600 DPI. According to the analysis in Sec.4.1, the threshold was set to T=1.10. The surface-spectral reflectance at each pixel was sampled at ∆λ = 5 nm . So the 61-dimensional spectral-reflectance image was created using the corrected six-sensor outputs. Figure 13 demonstrates the reproduced image in the sRGB coordinates for the object scene rendered under D65.
SPIE-IS&T/ Vol. 7241 72410W-7
P ©
Normali zed value
P
Normali zed value
Estimated Measured
Estimated Measured
(1
400
600 Wavelength (nm)
500
700
400
(a)
500 600 Wavelength (nm) (b)
Fig. 12 Estimation results of spectral reflectances for four areas 1, 2, 3, and 4 shown in Figure 11.
Fig. 13 Reproduced image of the scan object rendered under D65.
5. CONCLUSIONS We have proposed a method for estimating the spectral reflectance function of an object surface by using a six-color scanner. The scanner was regarded as a six-band spectral imaging system, since it captured six color channels in total from two separate scans using two difference lamps. First, the basic characteristics of the imaging systems were described for a HP color scanner and a multiband camera used for comparison. Second, the LMMSE estimator was described as an optimal estimator for recovering surface-spectral reflectances from the noisy sensor outputs. We discussed the reflectance estimation for non-flat surfaces with shading effect, and proposed a solution method to the reliable reflectance estimation. Finally, the feasibility of the proposed method was confirmed on experiments using the Macbeth Color Checker and non-flat objects.
ACKNOWLEDGEMENT We would like to thank Dr. Jeffrey DiCarlo, HP Lab for providing a HP color scanner and helping this work.
REFERENCES [1] [2] [3]
Hunter, A., DiCarlo, J. and Pollard, S., "Six Color Scanning," Proc. CGIV 2008 and MCS'08, 570-574 (2008). Kay, S., [Fundamentals of Statistical Signal Processing: Estimation Theory], Prentice-Hall, New Jersey, 379 (1993). Tominaga, S., Nakagawa, M. and Tanaka, N., "Image Rendering of Art Paintings -Total Archives Considering Surface Properties and Chromatic Adaptation," Proc. Twelfth Color Imaging Conference, 70-75 (2004).
SPIE-IS&T/ Vol. 7241 72410W-8
