Pixels Grouping and Shadow Cache for Faster Integral 3D Ray-Tracing Osama H. Youssefa, Amar Aggouna, Wayne Wolfb , and Malcolm McCormicka a Faculty of Computing Sciences and Engineering, De Montfort University, UK b Department of Electrical Engineering, Princeton University, USA Email:
[email protected]
ABSTRACT This paper presents for the first time, a theory for obtaining the optimum pixel grouping for improving the coherence and the shadow cache in integral 3D ray-tracing in order to reduce execution time. A theoretical study of the number of shadow cache hits with respect to the properties of the lenses and the shadow size and its location is discussed with analysis for three different styles of pixel grouping in order to obtain the optimum grouping. The first style traces rows of pixels in the horizontal direction, the second traces similar pixels in adjacent lenses in the horizontal direction, and the third traces columns of pixels in the vertical direction. The optimum grouping is a combination of all three dependant up on the number of cache hits in each. Experimental results show validation of the theory and tests on benchmark scenes show that up to a 37% improvement in execution time can be achieved by proper pixel grouping. Keywords: 3D Imaging, Computer graphics, Ray-tracing.
1. INTRODUCTION Integral photography is one of the auto-stereoscopic techniques for capturing and displaying true 3D images. It is well known for its viewer independent spatial solid image and continuous perspective. Computer generation of Integral photography (CGIP) has been reported in literatures1,2,3. The Imaging Technologies research group in De Montfort University had developed CGIP using Ray-Tracing as the base renderer. However, ray-tracing is computationally expensive due to the huge number of intersection tests of the spawned rays and the objects of the scene. One way of accelerating ray-tracing is to reduce the number of intersection tests for shadow rays using a shadow cache algorithm. Shadow cache requires coherent grouping of pixels i.e. tracing (successively) pixels whose primary rays share spatial coherence in order to increase the number of hits to the shadow cache and save large number of intersection tests. This concept is applied to integral 3D ray-tracing and a study of the optimum grouping of pixels, in order to improve the efficiency of the shadow cache, is presented in this paper
2. INTEGRAL 3D RAY-TRACING Ray-tracing algorithms have been developed for many years. Early versions were presented by Apple4 in 1968 and Goldstein and Nagel5, in 1971. By 1980, a refined ray-tracing procedure had been implemented6. Ray-tracing traces light rays in a backward fashion. The ray-tracer spawns reflection, transmission, and shadow rays to generate reflection, transmission, and shadows. When a ray intersects with an object, the colour of the intersection point on the object surface is recorded on the pixel associated with that ray. Ray-tracing produces photo-realistic images of excellent quality but unfortunately ray-tracing suffers from lengthy computation times. In ray-tracing nearly all of the execution time is spent performing intersection tests to test visibility and shadowing6. More intersection tests are performed for shadow determination than for primary rays, reflection rays, and transmission rays. The number of shadow rays is proportional to the number of light sources. For each primary ray that hits an object, surface shading calculations of the intersection point take into account the contribution of each light source to the intensity of that point (i.e. whether the light source is occluded by an object or not). The shadow cache was introduced by Hains and Greenberg7 to decrease the number of the shadowing intersection tests by using object coherence. They considered the observation that if an object prevents a light source from contribution to a pixel’s intensity, then the same object is likely to block the same light for an adjacent pixel. Shadow cache also operates on the assumption that
consecutive rays share some amount of spatial coherence and statistically they tend to intersect objects at points which are near previous intersections. Shadow cache works by keeping a cache of the most recent shadow casting object for each light source. The next time a light is tested for occlusion, the associated shadow casting object is first tested for intersection rather than traversing the objects within the scene in the usual manner. If the shadow test of the cached object fails, then testing must continue on the objects in the usual manner. If the shadow test on the cached object succeeds, significant computation can be saved. In general CGIP requires higher resolution display i.e. larger number of pixels than normal computer images to produce a 3D image with acceptable quality1. The efficiency of the shadow cache increases as the resolution of the image increases7. The reason is that when an environment is rendered at higher resolution, its objects cast shadows on a larger number of pixels. 2.1 Cache Calculations The number of intersection tests of a shadow ray missing the shadow cache can be in thousands or in millions depending on the scene complexity while a shadow ray hitting the shadow cache is tested only once for intersection with the object in the cache. Let N be the number of shadow rays, Cm be the average computation cost of the intersection tests required when a shadow ray misses the shadow cache, and Ch be the average computation cost of the intersection test required when a shadow ray hits the shadow cache. The total computation cost Ct can be expressed as follow. Ct = NhCh + NmCm,
(1)
where Nh is the number of shadow rays that hit the shadow cache and Nm is the number of shadow rays that miss it. The total number of the shadow rays is N = Nh + Nm, and since, Cm >> Ch, the total computation cost Ct becomes Ct = N Cm – Nh Cm.
(2)
In order to minimize the total computation cost of shadow rays, the number of shadow rays hitting the shadow cache must increase. Consider a number of pixels N whose primary rays hit shadowed points in a scene. For only one of the light sources, the number of shadow rays that hit a shadow casting object is Mh and their associated pixels are called H pixels. The number of shadow rays that do not hit that shadow casting object is Mm and their associated pixels are called M pixels. Hence, N = Mh + Mm.
(3)
The order of tracing the pixels has a great effect on Ct. Obviously the best case is when H pixels are traced successively before switching to M pixels. This results in the shadow casting object being kept in the cache once it is hit and the only pixels which miss the cache are the M pixels. Therefore Nm = Mm, and Nh = N – Mh. By substituting in equation 2 Ct = Mm Cm.
(4)
The worst case is when pixels of each group are traced in alternating order (i.e. H pixel then M pixel then H pixel and so on), which results into two cache misses for each pixel M. Therefore Nm = 2Mm (double the number of shadow rays that do not hit the object in the cache), and Nh = N - 2Mh. From equation 2 it can be seen that Ct = 2Mm Cm.
(5)
Therefore losing the cache while tracing pixels is very computationally expensive and could duplicate the computation cost of shadow rays. 2.2 Camera model implications Due to the nature of the recording process of integral photography, many changes on the camera model in the ray-tracer are required. In this paper, lenticular lenses, which produce integral images with parallax in the horizontal direction only are used. Each lens acts like a cylindrical camera. A strip of pixels is associated with each lens forming a sub-image. Each lens records a sub-image of the scene from a different angle as shown in the Figure 1. In the lateral cross section of the lenticular lens, a pinhole model with refraction is used. The pinhole forms a straight line parallel to the axis of the
cylindrical lens in the vertical direction. For each pixel, a primary ray is spawned and refracted through the associated pinhole as shown in Figure 2. The recording path of the primary ray draws a straight line going forward towards the image plane and backward away from the image plane. Similar primary rays of neighbour lenses are spawned to similar directions parallel to each other as shown in Figure 3. Therefore highly correlated sub-images are produced which is a property of an integral image. As an object is positioned away from the image plane, it is recorded by more lenses and a point on the surface of that object can be hit by more than one ray.
Figure 1: Camera model in Integral 3D Ray-tracer
1 2 3 4 5 6 7 8
field of view pixel new direction point
Figure 2. Tracing the ray after being refracted
Figure 3: Similar ray directions creating correlated sub-images
3. PIXEL GROUPING 3.1 Overview The structure of the lenses and the camera model in the integral 3D ray-tracing affects the way primary rays are spawned and traced as well as the spatial coherence among consecutive rays. From the shadow cache calculations it is shown that grouping all of H pixels before switching to M pixels could maximise the number of hits to the shadow cache and accordingly minimises the number of misses to the shadow cache. In the remainder of this paper, tracing of successive H pixels is termed “continuous cache hits” and is considered as a measure of the efficiency of the different style of grouping pixels. 3.2 Analysis of the number of continuos cache hits and shadows In order to find out the optimal grouping of image pixels for the maximum number of continuous cache hits, the relationship between the pixels and their primary rays and shadows is analysed. The analysis consists of the way primary rays relate to the shadow size and location (assuming the shadow is projected on a flat surface facing the lens sheet) and its affect on the number of the continuous cache hits. Due to the lens structure in the encoded integral image, the resulting number of continuous cache hits for the serial grouping of consecutive pixels and of similar pixels in the adjacent lenses in the horizontal direction are different. Both are different to grouping of consecutive pixels in the vertical direction. In this section, each case is analysed.
3.2.1 Grouping in the horizontal direction 3.2.1.1 Style 1: Grouping of consecutive pixels Pixels in the horizontal direction within the cross section of the lenses are traced from the lens pinhole by spawning primary rays in both directions: forward to the pixels, and backward towards the aperture. Figure 4 shows that the angle of view of the lens, α, forms a horizontal cross section of a space volume that bounds the rays traced from the lens through the object space. α is given by α = 2*tan-1 (P / 2f),
(6)
Where P and f are the pitch and the focal length of the lens, respectively. The number of pixels behind the lens Nlens = P/ps, where ps is the pixel size, assuming that the pixels size in the horizontal direction is the same for the vertical one. The spacing between the centres of the adjacent pixels equals the pixel size ps.
Figure 4: lateral cross section of a lens in Integral 3D Ray-tracing
Assuming that β is small enough to consider the top view of the image plane at the lens backside a line not an arc, the angle between two adjacent rays β is given approximately by β = α / Nlens.
(7)
In addition assume that a shadow with width A is projected on some surface at distance D from a lens. Then a primary ray R1 hits that shadow at some point, causing a shadow casting object to be saved in the shadow cache. From Figure 4, the angle between two rays, which cover the width A of the object is given by θ = tan-1 A/D.
(8)
In order that R2 hits the cache object’s shadow, R2 must be bounded by the angle θ, i.e. β < θ. If θ < α then the number of rays hitting that shadow is θ/β. Therefore, the number of continuous cache hits within a lens (CCWLhit) is given by CCWLhit = θ/β .
(9)
Substituting equation 6 and equation 8 in equation 9 gives CCWLhit = Nlens * (tan-1 (A/D)) / (2*tan-1(Nlens*ps/2f)).
(10)
Therefore CCWLhit depends on A and D. While tracing pixels in the horizontal direction across the lenses, switching between lenses (i.e. between the last ray in a lens and the first ray in the adjacent lens) has a big effect on the cache status with respect to the size and location of the shadow. The cache status depends on whether the first ray of the adjacent lens hits the shadow or not. In Figure 5, R1 is the last ray of a lens, which hits a shadow and R2 is the first ray of the adjacent lens. The intersection point of R2 and the surface where the shadow is projected, X2, is given by X2 = LAstart + P * n – D * tan (α/2). Where LAstart is the lens array start point, and n is the lens index in the lens array. R2 hits the cache if X2 X Xstart = Xend – A. by substituting equation 11 the condition becomes
(11) start, where
Figure 5: Crossing to an adjacent lens
[P * n + LAstart – D * tan (α/2)] [X
end
– A].
(12)
Therefore the cache status of crossing to an adjacent lens (CSTAL) depends on A and D 1 CSTAL = 0
X 2 ≥ X start X 2 < X start
(13)
Considering multiple lenses as shown in Figure 6. Then the number of continuous cache hits per lens (CCLhit) is given by CCLhit = CCWLhit + CSTAL,
(14)
assuming that the last ray in this lens hits the shadow. CCLhit depends on A and D because CCWLhit and CSTAL depend on A and D. By grouping consecutive pixels in the horizontal direction, CChit is expressed as follows. n
CChit = CCLhit 0 +
∑
CSTALn-1 * CCLhit n.
lensID =1
All of CChit’s parameters depend on A and D, therefore CChit depends on A and D.
Figure 6: Grouping in the horizontal direction.
Figure 7: Grouping similar pixels in adjacent lenses
(15)
Style 2: Grouping similar pixels in adjacent lenses: Tracing similar pixels in adjacent lenses yields parallel rays as shown in Figure 7. The distance between two similar rays in adjacent lenses is P, where each ray starts at the lens pinhole and goes through the pixel centre. The continuous number of cache hits of similar pixels in adjacent lenses CChit can be expressed as CChit = A/P,
(16)
where A = Xend – Xstart. Therefore CChit in the parallel direction depends on A only. Notice that P/ ps could be 10, 20 or bigger depending on the used lens array. 3.2.1.3 Combining Style 1 and 2 in the horizontal direction: This style of tracing pixels is a combined version of tracing within a lens and tracing similar pixels in adjacent lenses. Thus every lens is treated separately by tracing rays that hit the shadow only and then repeat that for all the lenses as shown in Figure 8. Therefore the continuous number of cache hits CChit is expressed by CC hit =
n
∑ CCWL hit n + n .
(17)
lensID =1
Using CCWLhit from equation 10 for each lens. The number of continuous cache hits depends on A and D. 3.2.2 Grouping in the vertical direction Tracing pixels in the vertical direction and parallel to the lens structure makes the successive rays parallel to each other. Each ray starts at the centre of the pixel and goes parallel to the horizontal planes as shown in Figure 9. The continuous number of cache hits in the vertical direction CChit is given by CChit = A/ps, where A = Yend – Ystart. Therefore CChit depends on A only.
Figure 8: The combined style in the horizontal direction
Figure 9: Grouping in the vertical direction.
(18)
3.3 The number of continuous cache hits: In this section, the effect of the shadow size and its location on the continuous number of shadow cache hits in each style of grouping is studied. 3.3.1. Grouping in the horizontal direction: 3.3.1.1 Style 1: For each value of D there is a function CChit(A) represented by a curve on the surface of the CChit(A, D) function in Figure 10. The values of CChit(A) increase and D decreases. The CChit values reach a maximum when D = 0. The curve CChit(A) D=0 is a linear function and is equal to A/ps. This is similar to CChit obtained for style 3 of tracing. When D is not 0 as shown in Figure 11, the CChit(A) curve consists of two regions. In the first region, the curve is linear with A because not every ray in the first lens hits the shadow (θ < α). In this region CChit(A) is less than (Nlens – 1). When θ α, the curve is similar to a staircase increasing with A the step size depending up on D. The worst case is when the step size equals Nlens. 180 160
Continuous Cache Hits
140 120 100 80 60 40 20 0 0 D = 13 mm
Figure 10: CChit(A, D) of style 1.
2
4
6
8
10 12 Size (mm)
14
16
18
20
Figure 11: CChit(A)D=13 of style 1.
The reason for the existence of the steps is that when the last ray in a lens hits or misses the shadow, the number of rays of the adjacent lens hitting the shadow is added to or subtracted from CChit expressed by equation 16. When A is varied by P, it covers the last ray of a lens, which makes CChit(A) jump by one step. Again for each value of A there is a function CChit(D) represented by a curve as shown in Figure 12. The values of CChit(D) decrease with A. This curve also consists of two region but operates oppositely to CChit(A). The first section is a staircase decreasing as D increases. When D is varied by 2f, A will cover the last ray in a lens, which makes CChit(D) jump by one step. The step size increases as D increases. The maximum step size is Nlens as described earlier. In the second region where θ < α (within the first lens only), the curve behaves as an inverse proportional relationship of CChit and D. 3.3.1.2 Style 2: In this style, CChit(A, D) is independent of D, and is linearly proportional to A because only similar rays in adjacent lenses, which are parallel, are examined against the shadow. Therefore CChit(A) increases or decreases, if and only if, A increases or decreases by an amount P as shown in Figure 13. 3.3.1.3 Combining style 1 and style 2: In Figure 14, the surface representing CChit(A, D) depends on A and D. CChit(A, D) is proportional to A and inversely proportional to D. The gradient of CChit(A, D) increases with A or with 1/D because rays of additional lenses that hit the shadow add a new term to the series( CC hit =
n
∑ CCWL hit n + n )
lensID =1
with each new lens. The surface has no sudden
change or staircase appearance because there are no conditions where hits from more lenses are added while some rays from current lenses (n lenses) still miss the shadow. For fixed values of D or A, there is no special effect of θ being smaller than α on CChit(A) and CChit(D) curves, therefore they have no separate regions. This is shown in Figures 15 and 16.
300
Continuous Cache Hits
250
200
150
100
50
0 0
10
20
30
40 50 60 Distance (mm)
70
80
90
100
Figure 12: CChit(D)A=20 of style 1
Figure 13: CChit(A, D) of style 2 250
Continuous Cache Hits
200
150
100
50
0 0
2
4
6
8
10
12
14
16
18
20
Size (mm)
Figure 14: CChit(A, D) of the combined style.
Figure 15: CChit(A)D=13 of the combined style.
3.3.2 Grouping in the vertical direction: The function CChit(A, D) shown in Figure 18 is independent of D therefore it could be written as CChit(A) which is linearly proportional to A. Rays in vertical directions are structured in a parallel form spaced by the distance between centres of pixels which is equals to the pixel size ps. As A is varied, CChit(A) is altered by adding or subtracting the number of new rays that hit or miss the shadow. 4. OPTIMUM PIXEL GROUPING FOR MAXIMUM NUMBER OF CONTINUOUS CACHE HITS: This section studies the issues that affect the number of continuous cache hits and how to find the optimum grouping using a combination of the different styles of tracing pixels. 4.1 Loosing (missing) the cache while tracing pixels Missing the cache even once during tracing pixels can be very expensive as described earlier. If there are a number of pixels whose rays hit the shadow, the optimum style pixel tracing is the one, which traces all those pixels successively before switching to other pixels that miss the cache.
300
250
Continuous Cache Hits
200
150
100
50
0 0 A = 20 mm
10
20
30
40
50
60
70
80
90
100
Distance (mm)
Figure 16: CChit(D)A=20 of the combined style.
Figure 17: CChit(A, D) of style 3.
4.1.1 The Horizontal direction: Assuming the arrangement shown in Figure 18. Nlens = 5 and there are six rays that hit the shadow, three from the first lens, two from the second and one from the last. 4.1.1.1 Style 1: Using style 1 yields CChit being equal to three out of six possible hits as shown in Figure 19. Therefore three pixels are traced and hit the shadow before pixels from the grouping start to miss it. Thus grouping traces a group of three pixels which hit the shadow before it starts on another group for a different shadow if there is any.
Figure 18: Grouping pixels
Figure 19: Style 1 of grouping pixels
4.1.1.2 Style 2: In style 2, CChit is equal to three. In Figure 20, grouping makes a group of three similar pixels in the adjacent lenses. Also it misses the cache before tracing all pixels that hit the shadow. 4.1.1.3 Combining style 1 and style 2: Figure 21 shows the combined style where CChit is the maximum possible hits because it groups only those pixels that hit the particular shadow and gives the optimum solution. The combined style gives the highest and the maximum value of CChit when compared to the other two styles for the horizontal directions. 4.1.2 Grouping in the vertical direction: Tracing in the vertical direction does not include the properties of the lenses. Therefore, the continuous cache hit is found to be always equal to the number of rays that hit the shadow, which is the maximum achievable.
Figure 20: Style 2 of grouping pixels
Figure 21: Combined Style of grouping pixels
4.2 Grouping in two directions In section 3, the horizontal and the vertical grouping of image pixels and their relationship to shadows has been discussed. Shadows are projected areas on the surfaces of the objects and tracing pixels must combine both directions to deal correctly with shadows. For simplicity, an area of shadow projected on a flat surface facing the lens sheet is considered. This area contains a big rectangle or could be formed by a number of small rectangles. Assume there is a shadow projected as a rectangle, which is hit by 6 pixels in the horizontal direction and 6 pixels in the vertical direction. Therefore the actual number of pixels, which hit that shadow is 36. Using the combined style of tracing in the horizontal direction results in the CChit being equal to 6 out 36. Using the vertical direction style will result in same value of CChit (i.e. 6). If all the pixels that hit the shadow are traced successively, the CChit would reach the maximum of 36. 4.2.1 Finding the optimum grouping Having a two dimensional style that uses the combined style in the horizontal direction and the vertical style produces groupings of pixel strips, a strip for every lens as shown in Figure 22. The width of the strip is evaluated using the combined version and the length of the strip is evaluated using the vertical style. This gives the maximum number of continuous cache hits. The number of continuous cache hits is expressed as follow CChit2D = CChitCom * CChitV Figure 23 shows CChit2D(A,D).
Traced Strips
Figure 22: Grouping in two directions
Figure 23: CChit(A, D) of the two dimension grouping.
(19)
5. TESTS AND RESULTS The lens parameters of the modelled ray-tracing camera are the same as that of the lenticular lens sheet used in displaying the output images. The relevant parameters are: lens’s pitch = 1.124 mm, lens’s refractive index = 1.6, and focal lens = 3.03 mm. The same single shadow projected on a flat square facing the lens sheet, which is used in the theory is also used in testing. The results of testing the different styles of pixels grouping are identical to those in the theory as shown in Figure 24. The different grouping of pixels have been tested on benchmark scenes, which are used in testing ray-tracers[10] such as the gears and the balls scenes. The image resolution is 2000 x 2000 and Nlens = 15. It was found that using grouping in the vertical direction gives up to 37% improvement when compared to style 1and style 2. However this was not the case for the other combination. The reason is the width and the length of the strips of pixels used in testing are fixed when the scenes contain thousands of objects with different sizes and at various depths which creates randomly projected shadows.
6. CONCLUSION In this paper, the effect of the characteristics of the lenses on the coherence among the rays has been discussed. In the horizontal direction the spatial coherence of the rays is less than the one of the vertical direction. In style 1 and the combined style in the horizontal direction CChit depends on D. This has a big effect on tracing successive rays as in style 1 (staircase effect), especially when θ < α in the first lens (CChit always less than (Nlens – 1)). Style 2 of pixel grouping misses other pixels in lenses that hit the shadow. On the other hand tracing successive rays in the vertical direction always traces those pixels that hit the shadow. The combined style corrects the missing bits of style 1 and style 3 by not tracing all successive pixels in a lens. The grouping in two-dimensions combines the combined style in the horizontal direction and style 2 in the vertical direction in order to extend the shadow cache to use the area covered by shadow as a whole and makes CChit multiple of CChit in each separate direction.
7. REFERENCES 1.
2. 3. 4. 5. 6. 7. 8.
Sung-Wook Min et al, “Three-dimensional display system based on computer-generated integral photography”, Stereoscopic Display and Virtual Reality Systems VIII, A. J. Woods; M.T. Bolas; J. O. Merritt and S. A. Benton, 4297, pp 187-195, SPIE, 2001. Michael W. Halle et al, “Fast computer graphics rendering for full parallax spatial displays”, Practical Holography XI and Holographic Materials III, S. A. Benton; J. T. Trout, 3011, SPIE, 1997. T Naemura et al, “3-D computer graphics based on the integral photography”," Optics Express 8, pp. 255-262, 2001. Apple, “Some Techniques for Shading Machine Renderings of Solids,” Proc. SJCC, pp.37-45, Thompson Books, Washington DC, 1968. R. Goldstein et al, “3-D Visual Simulation,” Simulation, pp.25-31, Jan. 1971. T. Whitted, “An Improved Illumination Model for Shaded Display,” Comm. ACM 23, pp.343-349, 1980. Eric A. Hains et al, “The Light Buffer: A Shadow-Testing Accelerator,” IEEE Graphics and Applications 6, pp.616, 1986. Eric A. Hains et al, “A Proposal for Standard Graphics Environments,” IEEE Computer Graphics and Applications 7, pp. 3-5, 1987.
Figure 24(a): Continuous cache hits of style 1
Figure 24(c): Continuous cache hits of style 3.
Figure 24(b): Continuous cache hits in the vertical direction.
Figure 24(d): Continuous cache hits of the combined style
Figure 24(e): Continuous cache hits of the two-dimension grouping.
