Absolute colorimetric characterization of a DSLR camera

Absolute Colorimetric Characterization of a DSLR Camera Giuseppe Claudio Guarneraa , Simone Biancoa and Raimondo Schettinia a DISCo

(Dipartimento di Informatica, Sistemistica e Comunicazione) Universit´a degli Studi di Milano-Bicocca, Viale Sarca 336, 20126 Milano, Italy ABSTRACT A simple but effective technique for absolute colorimetric camera characterization is proposed. It offers a large dynamic range requiring just a single, off-the-shelf target and a commonly available controllable light source for the characterization. The characterization task is broken down in two modules, respectively devoted to absolute luminance estimation and to colorimetric characterization matrix estimation. The characterized camera can be effectively used as a tele-colorimeter, giving an absolute estimation of the XYZ data in cd/m2 . The user is only required to vary the f − number of the camera lens or the exposure time t, to better exploit the sensor dynamic range. The estimated absolute tristimulus values closely match the values measured by a professional spectro-radiometer. Keywords: Camera characterization, High Dynamic Range imaging, DSLR Cameras, Colorimetric Characterization

1. INTRODUCTION In digital photography the incident photons on the sensor pixel area cause charge to accumulate at each pixel location, thus forming a picture. The relation between the sensor irradiance and the RGB triplet is called Opto-Electronic Conversion Function (OECF). In analogy with the human visual system, a sensor should have a set of color filters in order to mimic the trichromatic color matching functions, actually mosaicing the captured image. However, consumer Digital Single Lens Reflex (DSLR) cameras are typically designed to produce pleasing images, in which the importance of obtaining good contrast and vivid colors is more important than an accurate colorimetric reproduction of the scene. A specific design of the spectral sensitivities of the color filter arrays is usually employed with this purpose. It affects the way the sensor collects the charges which will form the RAW picture, on which a set of algorithms for demosaicing, white balancing, gamut mapping, color enhancement and so on is subsequently applied. Color characterization of imaging devices establishes the relationship between the camera responses to a set of colors and the corresponding colorimetric values. Various techniques have been proposed to find such a relationship, which typically either requires the acquisition of a reference color target (e.g. a GretagMacbeth R with known spectral reflectance1 or the use of specific equipment such as monochromators, as ColorChecker ) recommended by the standard ISO:17321-1:20062 . Empirical DSLR characterization directly relates measured colorimetric data from a target and the corresponding camera RGB data, obtained from a picture of the target itself. Many of these techniques are limited to linear regression on training samples, whereas more advanced methods use a white point preserving maximum ignorance assumption.3 In Ref. 4 several empirical techniques to compute the optimal 3 × 3 transformation matrix are compared. It is an useful approach when the camera spectral sensitivities are unknown, assuming that the characterized camera will be used in illumination conditions very similar to those encountered during the characterization.5 However, the limited set of color patches on the target can lead to inaccurate results and they are generally limited to low dynamic range imaging, since the measurement devices and the model used discard the intensity scale of the illuminant, preserving only the relative spectral power distribution. In Ref. 6 a transparent target is proposed, which allows the use of the characterization method also for High Dynamic Range (HDR) imaging. Giuseppe Claudio Guarnera: [email protected], Simone Bianco: [email protected], Raimondo Schettini: [email protected]

Digital Photography X, edited by Nitin Sampat, Radka Tezaur, et. al. Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 9023, 90230U · © 2014 SPIE-IS&T CCC code: 0277-786X/14/$18 · doi: 10.1117/12.2042172 Proc. of SPIE-IS&T/ Vol. 9023 90230U-1 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 07/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

Characterization approaches based on a monochromator generally prove to be more accurate and of general application. Since a white integrating sphere is illuminated with a monochromatic light, with a single wavelength at a time, the main drawback of monochromator-based techniques is the time required to collect the pictures of the sphere for each single wavelength. Moreover, the absence of an absolute scale for the luminance represents a limitation. To overcome this limitation in Ref. 7 an adaptation algorithm is described, to obtain absolute measurements of the tristimulus values by exploiting the variation of the lens aperture. The inverse OECFs are obtained from a simulated spectrally neutral grayscale pattern, assuming the equal energy illuminant CIE E. The estimated data are further fitted with true measurements to derive a linear model of color correction. In this paper we propose a simple but effective technique, which offers a large dynamic range requiring just a single, off-the-shelf target and a commonly available controllable light source for the characterization. We separate the characterization task into two modules, an absolute luminance estimation module and a colorimetric characterization matrix estimation. The characterized camera can be effectively used as a tele-colorimeter, giving an absolute estimation of the XYZ data in cd/m2 , just varying the f − number of the camera lens or the shutter time t. The estimated absolute tristimulus values closely match the values measured by a professional spectroradiometer, as confirmed by experimental results.

2. ABSOLUTE CHARACTERIZATION OF THE DSLR CAMERA Each pixel of a RAW image depicting a diffuse reflective surface, with spectral reflectance S(λ), acquired by means of a camera sensor with spectral sensitivities CR|G|B (λ) under an illuminant with spectral power distribution I(λ), can be modeled as an integration over the visible part ω of the spectrum: Z [R, G, B] ∝ I(λ)S(λ)CR|G|B (λ)dλ (1) ω

To obtain the absolute tristimulus values CIE − XY Z we can use a similar expression, taking into account the CIE-1931 2-degree Color Matching Function (CM F ) fx¯ (λ), fy¯(λ) and fz¯(λ): Z [X, Y, Z] ∝

I(λ)S(λ)fx¯|¯y|¯z(λ)dλ

(2)

ω

The aim of a absolute camera characterization is to find a proper mapping between the captured RGB triplets and the absolute CIE − XY Z values, generally acquiring a training set with pictures of scenes with known tristimulus values. Comparing Eq.1 and Eq.2 it is evident that the only difference is the use of the CM F s instead of the sensor spectral sensitivities. Hence, a proper setup must use the same illuminant both to measure the scene absolute colorimetric values and to capture the training set, in order to obtain a mapping which is robust to illumination changes. An important aspect to take into account is that, in theory, a camera needs to satisfy the Luther’s condition in order to faithfully capture the colors of a scene:     fx¯ CR fy¯  = T · CB  (3) fz¯ CG where T is a 3 × 3 matrix. This assumption usually does not hold in digital photography, thus reducing the colorimetric accuracy of a camera. The usefulness of camera characterization is often limited by the low dynamic range of the sensor, further reduced by the noise floor. A better solution would allow the possibility to change the integration time or to vary the lens aperture, extending the measurement range of the system. In the following we assume that the reciprocity law holds for the system (camera and lens) to be characterized. Taking into account the OECFs

Proc. of SPIE-IS&T/ Vol. 9023 90230U-2 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 07/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

FR|G|B , the lens properties, the dark noise n and the shutter time t, the discrete representation of Eq.1 in a small area close to the optical axis can be written as: ! X t T (λ)I(λ)S(λ)CR|G|B (λ)4λ + n [R, G, B] = kR|G|B 2 FR|G|B (4) N λ∈ω

where T (λ) is the lens transmittance, N is the f − number and kR|G|B enclose some constant scaling factors. We separate the characterization task into two modules, an absolute luminance estimation module and a colorimetric characterization matrix estimation: T

[XY Z] cd/m2 = M · [L(R) L(G) L(B)]T cd/m2

(5)

where M is the colorimetric profile of the camera, L(R), L(G), L(B) are the luminance estimates obtained −1 individually from each channel of the camera, as the inverse functions FR|G|B : −1 L(R|G|B) = FR|G|B .

(6)

It should be noted that the areas beneath the sensor spectral sensitivities curves are different from each other, whereas the CIE-CMFs have equal areas. We should take into account this information in order to obtain a meaningful relation between the values measured in image channel. Since the curves are fixed for each sensor and do not vary with the exposure or the aperture, we can embed these balancing factors in the kR|G|B terms.

2.1 Luminance estimation To obtain an absolute estimation of the luminance we must link the measured RGB values with a set of known radiance values, including the intensity scale and not only the relative spectral power distribution. Most reflectancebased techniques for camera characterization make use of a gray-scale target, either real or simulated, illuminated by a known light source. These techniques are inherently low dynamic range. A better strategy would make use of a light source with equal CIE XYZ tristimulus values, with increasing intensity. Such an illuminant would allow an accurate modeling of the OECFs, without introducing a bias due to the combined effect of the light chromaticity and the channel-dependent sensor spectral sensitivities. We propose a simple but effective technique, which offers a large dynamic range employing just a single, off-the-shelf target and a commonly available controllable light source. A 2 × 2 inches white diffuse patch is placed in front of a DLP projector, roughly at the center of the light beam. A DSLR camera is aligned with the light beam, with the white patch placed in the middle of the image formation plane, to reduce the influence of the vignetting effect. The lens should allow the selection of the f − number to use, with standard full-stop scale. The projector should have been previously calibrated with a spectro-radiometer, to correct for the gamma and to obtain a equal tristimulus scale, with a measured luminance Y on the white patch approximately in the range 0 − 1000 cd/m2 . The projected light beams hence have the same relative spectral distribution but a different absolute intensity. A sequence of pictures of the white patch is acquired using the widest entrance pupil available, with increasing luminance. All images are saved as RAWs files, with no debayering or white balancing. The ISO setting should be fixed beforehand to the native ISO value of the sensor. Once saturation is detected on a pixel, the entrance pupil is shrunk by a full-stop, and the gray scale is projected back from the lowest value. The process is repeated until a picture of the patch illuminated by the brightest gray value is captured. The noise floor n is estimated with a sequence of pictures acquired with the cap on the lens, at different times and averaging them. We need to account for the different areas of the spectral sensitivities curves, which we need to estimate. We used the technique described in Ref. 8, which just requires a picture of a reflective color R target with known relative spectral reflectances (e.g. a Color Checker ), acquired under daylight illumination. Each RAW pixel value rv in the acquired images is then converted into a normalized value f p using one of the following formulas, depending on the color filter mounted on the pixel itself:  (rv−n)A G   M RV ·AR , if the pixel belongs to the Red channel (rv−n) fp = (7) if the pixel belongs to the Green channel M RV ,   (rv−n)AG M RV ·AB , if the pixel belongs to the Blue channel


where M RV is the maximum RAW intensity value representable by the camera (i.e. saturation level), n is the estimated noise floor and AR , AG and AB are the areas beneath the spectral sensitivities curves of the Red, Green and Blue sensor channels respectively. The normalized values are then fitted with the measured −1 luminance, to derive the inverse OECFs FR|G|B . In fig. 1(a) the normalized values from the three channels are plotted versus the measured luminance Y of the white patch, for a fixed f − number and shutter time t. It is evident that the response is almost linear. In fig.1(b) the same settings for the lens and camera are used, but the tristimulus values X and Z of the projected light has been slightly altered with respect to the Y value (random perturbations, in the range X, Z = Y ± 5%). It is evident that the sensor response is no longer linear and such an illuminant should not be used to characterize the camera with our technique. 0.35

042

0.3

04

025

0.38

02

m 0.36

l6

(.9

re

_

0.15

- 0.34

0.1

Z 0.32

005

0.3

E

E

ó

0 0

50

100

150

200

250

300

280

300

Luminance Y (cd/m2)

320

340

360

380

400

Luminance Y (cd/m2)

(a) (b) Figure 1. (a) Sensor response measured with the corrected, normalized RGB RAW values; the illuminant in the scene has equal tristimulus values. (b) Sensor response measured using an illuminant with tristimulus values slightly changed from a pure gray value. (X, Z = Y ± 5%)

2.2 Colorimetric Characterization Previous work use a reflective color target, illuminated by a limited set of illuminants (typically an incandescent lamp, an halogen lamp and a daylight lamp) to derive the matrix M in Eq. 5. We propose the use of the same setup discussed in Sec. 2.1, in which we can modulate the relative and absolute spectral distribution of the light beam thus enabling a virtually infinite number of illuminants useful to colorimetrically characterize the camera. A set of illuminants are then projected on the white reflective patch, and the corresponding absolute tristimulus values are annotated. For each illuminant a picture is captured, and used to estimate the L(R|G|B) values. The matrix M is found by solving the following optimization problem:9 M = arg

min median(E) + mean(E) + max(E)

M ∈R3×3

(8)

where E = [XY Z]T − M · [L(R) L(G) L(B)]T 1

(9)

and ||·||1 denotes the L1 norm. Since the objective function is non-linear and non-differentiable the optimization problem is solved using the Pattern Search Method (PSM). PSMs are a class of direct search methods for nonlinear optimization.10 PSMs are simple to implement and do not require any explicit estimate of derivatives. Furthermore, global convergence can be established under certain regularity assumptions of the function to minimize.11 The estimated absolute tristimulus values can now be obtained using Eq.5.


3. EXPERIMENTS AND RESULTS 3.1 Experimental Setup A Canon 40D camera, with a standard 28 − 135mm f /3.5 − 5.6 zoom lens, was used in this work; the focal length was set to 30mm and kept fixed during the experiments. The lens f-number at the chosen focal length ranges from f /4 to f /22, with a standard full-stop scale; a slip-on lens hood is used on the lens, in order to reduce stray light, thus increasing the measurements accuracy. A previously calibrated Dell S300 DLP projector, with a maximum brightness of 2200 ANSI lumens was aligned with the camera and employed as controllable light source. The reflective targets are fixed to a blackboard placed in front of the projector, about 0.6m away from it. A spectro-radiometer with a luminance measurement range of 0.2−1200cd/m2 and spectral range 380−730nm is employed for the system characterization and to obtain the ground truth in our experiments. In order to reduce the number of acquisitions and at the same time to allow a fair comparison with previous work,1, 7 we decided to mimic the color target approach by projecting the 24 illuminants which would produce on the white reflective R patch a set of tristimulus values similar to a physical ColorChecker target. A schematic representation of our setup is reported in Fig.2. While a single reflective white target can suffice for the characterization, we evaluated our method using a set of 8 different color targets, illuminated by a set of 30 illuminants which differ in terms of spectral power distribution.

Blackboard Target

ectrophotoi eter

PC

Projector Figure 2. Setup used for the camera characterization and testing.

The camera sensor is equipped with a standard color filter array, hence we need to account for the reduced spatial resolution camera of each color channel, which have a sampling factor of 1 : 2 for the Green and 1 : 4 for both the Red and Blue values. Since the targets we used can be considered roughly lambertian surfaces and they are aligned with the center of the projector beam and orthogonal, the surface reflectance varies smoothly and a simple interpolation, performed individually for each channel, suffice to recover the missing data.

3.2 Results and Discussion To evaluate the goodness of the proposed characterization we computed the ∆E94 and the CIEDE2000 (∆E00 ) color differences between our estimated absolute tristimulus values and the ground-truth, for all the 8×30 samples in our test set. In order to compute the color differences formula all values are converted into the CIELab color space. In table 1 we report the computed differences for both the training set and test set, whereas in Fig. 3 the estimated X, Y and Z values are plotted versus their respective ground truths. In particular, on the luminance Y we obtained a median relative error of 2.37% and a median absolute error of 2.7 cd/m2 .

4. CONCLUSIONS In this work a simple but effective technique for absolute colorimetric camera characterization has been proposed. The proposed technique permits to obtain a large dynamic range characterization requiring just a single, offthe-shelf target and a commonly available controllable light source. The characterization task is broken down in two modules: the former devoted to absolute luminance estimation, the latter to colorimetric characterization.


Table 1. Color accuracy of the proposed method

∆E94 median mean

min

max

min

∆E00 median mean

max

Training Set

0.2514

0.6703

0.7213

1.7659

0.2853

0.7228

0.7434

2.2197

Test Set

0.0618

1.7428

1.7622

3.9997

0.0617

1.6317

1.6352

4.5783

zoo

400

600

Measured % values -cNm2

800

1000

zoo

400

600

000

1000

Measured Y crues -cNm2

200

400

600

000

1000

Measured Z Values - Wm2

Figure 3. Estimated absolute tristimulus values versus the ground truth. All measures are in cd/m2 .

The characterized camera can be effectively used as a tele-colorimeter, giving an absolute estimation of the XYZ data in cd/m2 . In particular, on the luminance Y we obtained a median relative error of 2.37% and a median absolute error of 2.7 cd/m2 . As future work we plan to investigate if different colorimetric characterizations12, 13 can further improve the results, as well as the applicability of the proposed technique in more complex processing pipelines.14, 15 We also plan to investigate the usability of the characterized camera to recover reflectance spectra from the estimated tristimulus values.16

REFERENCES [1] Bianco, S., Schettini, R., and Vanneschi, L., “Empirical modeling for colorimetric characterization of digital cameras,” in [Image Processing (ICIP), 2009 16th IEEE International Conference on ], 3469–3472 (2009). [2] ISO, “Iso/17321-1:2006: Graphic technology and photography – colour characterisation of digital still cameras (dscs) – part 1: Stimuli, metrology and test procedures,” ISO. [3] Finlayson, G. D. and Drew, M. S., “Constrained least-squares regression in color spaces,” Journal of Electronic Imaging 6(4), 484–493 (1997). [4] Hubel, P., Finlayson, G., Holm, J., and Drew, M. S., “Matrix calculations for digital photography,” in [Fifth Color Imaging Conference: Color Science, Systems and Applications], 105–111 (november 1997). [5] Sharma, G., [Digital Color Imaging Handbook], CRC Press, Inc., Boca Raton, FL, USA (2003). [6] Kim, M. H. and Kautz, J., “Characterization for high dynamic range imaging,” Computer Graphics Forum 27(2). [7] Martinez-Verdu, F., Pujol, J., Vilaseca, M., and Capilla, P., “Characterization of a digital camera as an absolute tristimulus colorimeter,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series], Eschbach, R. and Marcu, G. G., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 5008, 197–208 (Jan. 2003). [8] Jiang, J., Liu, D., Gu, J., and Susstrunk, S., “What is the space of spectral sensitivity functions for digital color cameras?,” in [Applications of Computer Vision (WACV), 2013 IEEE Workshop on], 168–179 (2013). [9] Bianco, S., Gasparini, F., Russo, A., and Schettini, R., “A new method for rgb to xyz transformation based on pattern search optimization,” IEEE Transactions on Consumer Electronics 53(3), 1020–1028 (2007). [10] Lewis, R. M. and Torczon, V., “Pattern search methods for linearly constrained minimization,” SIAM J. Optim 10(3), 917–941 (2000).


[11] Lewis, R. M. and Torczon, V., “On the convergence of pattern search algorithms,” SIAM J. Optim 7(1), 1–25 (1997). [12] Bianco, S. and Schettini, R., “Two new von kries based chromatic adaptation transforms found by numerical optimization,” Color Research and Application 35(3), 184–192 (2010). [13] Bianco, S., Gasparini, F., Schettini, R., and Vanneschi, L., “Polynomial modeling and optimization for colorimetric characterization of scanners,” Journal of Electronic Imaging 17(4), 043002–043002 (2008). [14] Bianco, S., Bruna, A., Naccari, F., and Schettini, R., “Color space transformations for digital photography exploiting information about the illuminant estimation process,” Journal of the Optical Society of America A 29(3), 374–384 (2012). [15] Bianco, S., Bruna, A., Naccari, F., and Schettini, R., “Color correction pipeline optimization for digital cameras,” Journal of Electronic Imaging 22(2), 023014 (2013). [16] Bianco, S., “Reflectance spectra recovery from tristimulus values by adaptive estimation with metameric shape correction,” Journal of the Optical Society of America A 27(8), 1868–1877 (2010).
