Recovery from Simulated RGB Camera Responses. Eva M. Valero, Natalia .... free and noisy data: We have calculated the RGB camera responses without and ...
Clustering Algorithms Applied to Spectral Reflectance Recovery from Simulated RGB Camera Responses Eva M. Valero, Natalia Lyubova, Juan L. Nieves and Javier Romero Clustering algorithms applied to spectral reflectance recovery from simulated RGB camera responses Departamento de Óptica, Facultad de Ciencias, Universidad de Granada
Eva M. Valero, Natalia Lyubova, Juan L. Nieves, Javier Romero Departamento de Óptica, Facultad de Ciencias, Universidad de Granada ABSTRACT: Clustering techniques have been applied to spectral images mostly to classify materials or objects included in the scenes. We present in this study new data concerning the use of clustering algorithms as a preprocessing step for obtaining spectral data from camera responses. Four different clustering algorithms have been used, and the Kohonen maps seem to provide very promising results for spectral recovery based on simulated digital counts wihout or with a high level of additive noise included. We have considered also the how the addition of a wide-band color filter in front of the camera lens to increase the number of channels of the camera from three to six affects the results. The influence of the number of classes for all four algorithms and map configurations for the Kohonen map have been studied as well. 1. INTRODUCTION: Spectral imaging is used in many applications because it offers the possibility of access to the spectral information on a pixel-by-pixel basis of a scene1-2. The hyperspectral imaging approach involves usually employing more than 30 narrow-band filters in different spectral bands, and with this method, radiance or reflectance can be recovered exactly3. If one is interested only in the visible range, then 30-33 bands are enough to obtain a good-quality recovery of the spectral reflectance or radiance for each pixel, using for instance a monochrome camera coupled with a LCTF (liquid crystal tunable filter) device2. There are many alternative approaches involving the use of only a few color filters (or even just a conventional digital RGB camera) to capture the image of the scene of interest, and afterwards using some method for recovering spectral information from camera responses. Spectral function recovery is an ill-posed problem in this case, so the methods for spectral recovery from camera
responses (like direct reconstruction, reconstruction by interpolation and learning-based reconstruction or pseudoinverse method)2,4,5,6 commonly involve a previous training step to develop the mapping or relation matrix between the camera responses and the spectral information. In the socalled “direct pseudo-inverse method”6, the recovery process includes the calculation of an estimation matrix D using a set of camera responses for which the spectral radiance or reflectance is known:
D St t
(1)
where t is the set of camera responses for the training data and St is the set of training spectral radiances or reflectances. The + sign indicates pseudoinverse matrix. Afterwards, D can be used to recover other sets of spectral signals not including in the training from camera responses:
S D
(2)
Previous computational results regarding reflectance and radiance6 or illuminant5 recovery from natural scenes indicate that the spectral and colorimetric quality of the recovered signals improves when one or two coloured filters are added to a conventional RGB digital camera. The addition of color filters allows an increase in the number of camera responses from which the spectral information is afterwards recovered. The improvement is not so clear when the RGB signals are noisy7 or for illuminant estimation5. The direct pseudo-inverse method has also been applied to develop a system for measuring of spectral power distributions (SPDs) of skylight with 3-5 sensors, showing good accuracy and a reduced computation complexity in comparison to other spectral recovery methods. Recently, clustering techniques have been applied to hyperspectral imaging systems to improve classification 1
79
results in satellite images, and as a preprocessing step for spectral information recovery from camera responses. Clustering algorithms perform an unsupervised classification of a data set in a number of classes (indicated by the user), so they are very useful when a priori knowledge of the data structure is not available. This is the case for images of natural scenes which usually contain many different surfaces with different reflectance functions. By forming clusters before the spectral information is recovered from the camera responses, accuracy is improved because the training set is subdivided into several different groups with similar camera response values. So the training is afterwards performed cluster by cluster and the estimation matrices obtained from eq. (1) are able to work with higher accuracy when applied for spectral recovery. Two among the most widely used clustering algorithms are the k-means and the more recently developed SelfOrganizing Map or SOM, also known as Kohonen’s map. The k-means algorithm is based on grouping the nearest points (those with lower sum of squared distances) to a given cluster center, beginning usually from random initialization values. The SOM is based on neural networks. The neurons are connected to adjacent neurons by a neighbourhood relation, which dictates the topology, or structure, of a map. The SOM training algorithm resembles vector quantization (VQ) algorithms, such as k-means. The important distinction is that the neurons on the grid become ordered, and neighbouring neurons have similar weight vectors. The main aim of this study is to provide additional and more complete data using simulated digital counts calculated from a set of natural reflectances, to test the hypothesis of a possible improvement in recovery quality of spectral reflectances from camera responses by using clustering techniques as a pre-processing step. So we will first simulate the capture of a natural scene with or without a colour filter in front of an RGB digital camera, then classify the camera responses using clustering with variable number of classes, and obtain a set of estimation matrices using only data belonging to one class. Afterwards, we check the recovery quality using a test data set not included in the training data. Each member of the test set is assigned
80
a class according to its distance from the different class centres obtained for the training data. In addition, we have analyzed how the introduction of a high level of additive noise and the use of a color filter in front of the lens affect the spectral recovery. So we have four capture conditions: noise-free without filter, noise-free with filter, noisy without filter and noisy with filter. For each condition, we study the influence of the number of classes and kind of algorithm used in the recovery quality results. 2. METHOD: 2.1 Simulated camera responses for noisefree and noisy data: We have calculated the RGB camera responses without and with a color filter in front of the camera, using a set of hyperspectral data from a highspatial-resolution database3 which included rural scenes in the region of Minho (Portugal). The simulated camera was a Retiga 1300 (QImaging Corp., Canada) with 12-bit intensity resolution per channel. When the effect of added noise was studied, we introduced a 5% variance level of additive noise, simulating thermal and shot noise. The noise N was introduced as shown in eq. (3).
n N
(3)
where n are the noisy digital counts and N is a column vector of three (without filter) or six (with added filter) values7. Using the camera spectral sensitivities, we have calculated the camera responses for a set of 228010 reflectances in the training group and a set of 37210 reflectances in the test group (none of them included in the training set). Then, we have successively used eq. (1) and (2) to obtain the recovered reflectances for the test set. This recovery will serve as a reference for testing the effect of applying clustering algorithms to the camera responses. 2.2 Clustering algorithms and quality indexes: Once the camera responses for each capture condition (noise-free or noisy, without or with filter) were computed, we have used several clustering algorithms (standard k-means, Fuzzy-C means, Gaussian Mixture Model and SOM) with random initialization values to divide the training set into 2, 4, 8, 16 and 32 classes. 2
Then we have computed the recovery matrix D for each class and obtained the recovered reflectances for the test set (having previously classified the test data according to the algorithm’s output). Given the known dependence of the kmeans results on initialization values, we have run the kmeans algorithm five times and selected the output giving a better quality (as derived from the Xie and Beni and Separation indexes). Two spectral (Goodness-of-Fit-Coefficient or GFC, defined as the cosine of the angle between original and recovered signals in the reflectance vector space; and Root Mean Square Error or RMSE) and one colorimetric (CIELAB color difference) quality measures were used to assess similarity between original and recovered reflectances. The global quality result for a given algorithm and number of classes were the weighted average values for these three quality indexes. 3. RESULTS: 3.1 Effect of the number of classes and capture condition on recovery quality: In Figure 1 we can see weighted mean GFC for reflectance recovery in the four experimental conditions as a function of the number of classes used as input to the four different clustering algorithms. The use of noisy camera responses leads to lower recovery quality, as expected, given the high level of added noise. It also influences the performance as the number of classes varies, as for noisy data the improvement in performance from 0 to 8 classes is more dramatic than for the noise-free data. We can see that differences tend to become stationary from 8 classes on, although the GFC values vary less in general for the noise-free data, as we pointed out in a previous study. In the noise-free data, recovery with filter is better than without filter, while the opposite trend is found for the noisy data. This is in agreement with other recent experimental results regarding reflectance and illuminant estimation with noisy data and the previous results obtained only for the k-means algorithm3,7. For the FCC and GMM algorithms, the increase in computation time from 8 to 32 classes is quite dramatic. So 8 classes would be the best value for these algorithms, because the improvement in quality obtained would not compensate for the increase in computation time.
K-means, GFC
SOM, GFC
0,98
0,98
0,96
0,96
0,94
0,94
0,92
0,92
0,90
0,90
0,88
0,88 0,86
0,86 0
2
4
8
16
0
32
FCC, GFC
0,98
2
4
8
16
32
Number of classes Without filter, non-noisy Magenta filter, non-noisy Without filter, noisy Magenta filter, noisy
Number of classes Without filter, non-noisy Magenta filter, non-noisy Without filter, noisy Magenta filter, noisy
GMM, GFC
0,98
0,96
0,96
0,94
0,94
0,92
0,92
0,90
0,90 0,88
0,88
0,86
0,86 0
2
4
8
16
32
Number of classes Without filter, non-noisy Magenta filter, non-noisy Without filter, noisy Magenta filter, noisy
0
2
4
8
16
32
Number of classes Without filter, non-noisy Magenta filter, non-noisy Without filter, noisy Magenta filter, noisy
Figure 1. GFC values as a function of the number of classes.
We have performed a one-way ANOVA for each condition and algorithm and the factor number of classes was always significant (p
