Spatialchromatic Foveation for Gaze Contingent Displays Sheng Liu Hong Hua 3DVIS Lab, College of Optical Sciences, University of Arizona 1630 E University Blvd., Tucson, AZ 85721 E-mail: [sliu, hhua]@optics.arizona.edu Abstract
decrease of saturation of many unique hue components with the increasing eccentricity. Watson et al. [1997] conducted a visual search study on gaze contingent HMD with degraded chrominance details by using grayscale periphery and a 24-bit color inset at the foveated FOV. The experimental results suggested that the task performance is not greatly affected by removing chrominance on the peripheral visual field. Reingold raised the question about whether a multi-resolution GCD can be constructed using a hue resolution drop-off function, similar to the spatial resolution drop-off matching with that human visual acuity, which is just imperceptible from a full color image [Reingold et al. 2003]. Duchowski and Çöltekin [2007] presented a hardware-accelerated fragment programming technique allowing for real time processing of images/videos for GCDs with degraded spatial and chromatic details on the periphery.
Spatially variant resolution method has been widely explored for Gaze Contingent Displays (GCDs). Recently several studies suggested that spatial chromatic foveation can further improve the sampling efficiency and save computational resources and communication bandwidths in GCDs. In this paper, we explore the spatial variance of the contrast sensitivity function (CSF) of the human visual system (HVS) to examine the potential of spatialchromatic foveation in GCDs. The proposed algorithm reveals that, not only the spatial resolution, but also the chrominance complexity can be monotonically degraded from the center of the field of view (FOV) to the periphery of a GCD. A perceptually-based spatialchromatic foveation metric is derived. Applying the proposed hue-resolution foveation metric, we demonstrate that over 65% of bandwidth can be saved.
In this paper, we will particularly focus on the spatial chromatic foveation rather than the well explored domain of spatially variant resolution. In order to derive a metric for spatialchromatic foveation, in Section 2, we firstly define the sampling efficiency and bandwidth savings of a GCD in general by integrating the spatial and chromatic resolution maps in 2D. The ultimate goal of the chromatic foveation metric is to derive the hue resolution drop-off curve that not only satisfies the human perception but also maximizes the bandwidth savings of the GCD compared to single modality displays (SMDs). In Section 3, an optimal hueresolution drop-off curve is derived by comparing the contrast modulation changes when applying chromatic foveation on a general purpose display device to the contrast modulation thresholds at different peripheral eccentricities in the HVS. Finally in Section 4, we apply the proposed hue resolution dropoff mask to demonstrate a perceptually based spatialchromatic foveation example which yields over 65% of bandwidth savings.
CR Categories: I.3.7 [Computer Graphics]:Three-Dimensional Graphics and Realism - Virtual reality; H.5.2 [Information Interfaces and Presentation] User Interfaces - User-centered design, Screen design; I.4.8 [Image Processing and Computer Vision] Scene Analysis - Color Keywords: Gaze contingent display, contrast sensitivity, color bytes, hue
1 Introduction A Gaze Contingent Display (GCD) allocates limited computational resources by spatially variant sampling in image or video encoding and processing. It attempts to present the highest level of detail (LOD) to the viewer’s gaze direction by coupling the system with an eye tracking device. Different types of foveation imaging system [Kortum and Geisler 1996] and GCDs [Geisler and Perry 1998; Murphy and Duchowski 2001; Luebke et al. 2001; Reddy 2001] have been presented by managing the spatial LOD on the periphery of an image or display. Integrative reviews can be found in [Parkhurst and Niebur 2002; Reingold et al. 2003; Duchowski et al. 2004].
2 Sampling efficiency of GCD In order to quantify the total amount of information presented by a display, we extend the measures of bandwidth by Hua and Liu [2007] and computational costs by Parkhurst and Niebur [2002], by incorporating the chromatic resolvability per pixel. The total amount of information by a display is quantified as the integration of the relative spatial resolution across the horizontal and vertical FOVs, θ X and θY , respectively, multiplied by the hue resolution per pixel, and is given as:
While the current study of GCD metrics is particularly concerned about the spatial resolution degradation and the resulted perceptual artifacts, spatialchromatic foveation emerges for perceptually driven GCD in the color domain. Studies of the color zone map [Sakurai et al. 2003] indicated the shift of hue and the
B=
————————————————
θ X / 2 θY / 2
∫ ∫ H (e , e ) F x
y
2
(ex , ey )dex dey .
(1)
−θ X / 2 −θY / 2
e-mail: [sliu, hhua]@optics.arizona.edu
H defines the hue resolution, analogous to the spatial resolution F, N by the total number of resolvable color given by H (ex , ey ) = 2 , where N is the bit depth of a display pixel. F (ex , e y ) is the relative spatial resolution distribution, which is well defined by the reciprocal of the angular resolution written as e2 , (2) F ( ex , e y ) = e2 + ex2 + e y2
Copyright © 2008 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
[email protected]. ETRA 2008, Savannah, Georgia, March 26–28, 2008. © 2008 ACM 978-1-59593-982-1/08/0003 $5.00
where e2 = 2.3o is a well-accepted half resolution constant [Geisler and Perry 1998; Loschky et al. 2005]. For a SMD display
139
with constant spatial and hue resolutions across its entire FOV, Eq. (1) is simplified to: 2 BSMD = H SMD FSMD θ X θY (3) In contrast, the HVS is an imaging system with spatially varying resolution F (ex , e y ) as well as hue resolution H (ex , ey ) . Without loss of generality, the sampling efficiency E of a display or imaging system can be derived by the ratio between the bandwidth cost of the HVS to that of the display: B (4) E = HVS B Rather than providing constant spatial and chromatic resolvability comparing to SMDs, a GCD aims to improve the sampling efficiency by approximating the spatially varying resolution and chromaticity acuity of the HVS. Compared to the SMD, the bandwidth saving S of a GCD is quantified by comparing the bandwidth requirements of GCD to a SMD and is given as: B − BGCD S = SMD (5) BSMD
n−1−i
3 Spatialchromatic foveation metric A GCD may be optimized to maximize its sampling efficiency E or the bandwidth saving S by approximating various properties of the HVS as a function of eccentricity, such as visual acuity, color sensitivity and contrast sensitivity. In this section, we will focus on deriving spatialchromatic foveation metrics that approximate the color sensitivity of the HVS in such a way that a spatially variant color sampling method can be readily applied to a GCD which is implemented with a general purpose display device.
n−1−k
1
N=8,I =1/256 0
Ci,j,k(N,M)|max
0.8
3.1 Contrast modulation by spatialchromatic foveation
0.6
0.4
0.2
Generally chromatic foveation from a higher bit depth level to a lower one will cause a contrast modulation to the original image and yield artifacts, such as toning and banding effects. Naturally, spatialchromatic foveation in GCDs need to satisfy the condition in which the contrast modulation introduced by bit-depth reduction is not perceivable by a human observer while the bit depth around the gaze direction should remain unchanged.
(a)
n−1− j
I (i, j , k ) = 0.299 I 0 n−1 + 0.587 I 0 n−1 + 0.114 I 0 n−1 , (6) N where n = 2 is the total number of gray levels per color channel, I0 is the lowest attainable luminance level which also determines the dynamic range of the display pixel, and i, j , k = 1, 2,...n . The above equation assumes that all of the RGB channels have the same dynamic range and the National Television System Committee (NTSC) RGB color system. After applying bit-depth reduction on the peripheral pixel P from N to M bits, the original RGB CLs (i,j,k) are remapped to a new set CLs of (i ', j ', k ' = 1, 2,...m) where m = 2M (M
