a liquid crystal scanning aperture to transmit image data from different ... content and continuous parallax within the field of view (2)-. (3). In an integral imagingĀ ...
Compression of integral 3D TV pictures
M. Forman, A. Aggoun and M. McCormick
De Montfort University, UK
compression
spatial information and produces images with full colour
algorithm for reduction of the transmission bit rate of 3D
content and continuous parallax within the field of view (2)-
integral TV pictures is discussed. The proposed full parallax
(3).
3D image compression algorithm takes advantage of the
is formed around the capture microlens array, each
cross correlation between the multiple images recorded on
individual microlens sees a unique (directional) part of the
a CCD placed behind a directionally selective microlens
total image field. Adjacent microlenses record an angularly
array as well as the correlation inherent within each image.
displaced view of the same image field. In this way each
Tests carried out show that by using a hybrid DPCM/DCT
image point is represented by a series of related image
coding scheme to code a still 3D integral TV image, high
intensity distributions. A full 3D image is achieved by the
compression performance can be obtained while retaining
integration of all the directionally intersecting, intensity
all the necessary information to reconstruct a faithful
modulated beams of light generated by the lens array.
ABSTRACT:
The
development of a
In an integral imaging system, in which the image
duplicate of the original 3D image. A coding scheme for Since the image is recorded as an intensity distribution on a
moving integral 3D images is also described.
flat plane and in real time, it is possible to use electronic capture.
INTRODUCTION
Electronic capture, transmission and display of
integral 3D images call for a large increase in resolution outperforms
over that available in existing TV equipment. This results
Its growth has
in an enormous increase in bandwidth requirements. In this
been slower than many predicted, but current technology
paper the development of a compression algorithm for
advances could make 3D TV much more common in the
reduction of the transmission bit rate is considered. The aim
near future. Recently, television researchers have shown
is to record and display sufficient information to allow the
strong interest in 3D TV technology, and are currently
images to have sufficient solidity and parallax content
attempting
to approximate the original real optical model.
Three
dimensional
television
(3D
TV)
conventional TV for many applications.
to
realise
a
3D
TV
system
capable
of
reproducing images which generate a good sensation of reality and existence. The main issues that are being
In this paper, a hybrid DPCM/DCT coding system is used to
investigated by most research groups for the development of
compress still 3D TV images. This coding scheme has been
3D TV systems are bandwidth compression techniques and
extensively used to compress 2D moving images by
display and capture technologies. Work carried out on high
decorrelating the image information in spatial and temporal
definition TV
great deal in
domains (4). The hybrid DPCM/DCT coding scheme
accelerating the progress of some of these research areas.
compresses still 3D images by taking advantage of the
(HDTV)
has helped a
redundancies present both within a microlens image and Autostereoscopic 3D TV displays using multiple image
neighbouring microlens views.
viewpoints have been demonstrated by several groups (1).
objective tests carried out, it is shown that the coding
Most designs have used either lenticular sheet decoders or
and
a liquid crystal scanning aperture to transmit image data
DPCM/DCT coding system result in minimal degradation
from different viewpoints into the left and right eyes by
of visual quality compared to the original image. A
either spatial or temporal multiplexing. The need for
compression algorithm for real-time moving integral 3D TV
expensive multi-camera capture and the complexity of the
pictures is suggested.
decoding
of
still
3D
From the subjective and
images
using
the
hybrid
image processing electronics to generate the interlaced banded images have to date prevented such systems from
ELECTRONIC
becoming commercially available.
INTEGRAL 3D IMAGES
An alternative method of capturing a continuous parallax 3D
Several methods have been examined for the capture of
image based on an advanced form of integral imaging is
integral and lenticular-integral images for evaluation of
considered to have a number of advantages over multi
compression algorithms. Broadly, these fall into two
image systems. The system captures and displays volume
categories:
CAPTURE
AND
DISPLAY
OF
1. Direct capture. A video camera is used together with the
lenses.
integral imaging equipment (3); Compression schemes are under development which take 2. Indirect capture.
An integral or lenticular-integral
photograph is scanned either in transparency or print form.
advantage of the extra degree of redundancy involved in the
inter-subimage correlation. As each subimage is
effectively a low resolution image of the object scene, For the purposes of this investigation, a print of a lenticular-
DCT coding can be used in the same way as with
integral photograph was captured using a desktop scanner
conventional 2D images. A DPCM coder can be used to
operating at a resolution of 300dpi. The resulting image file
take advantage of the integral image specific inter-subimage
was processed in order to produce an image with a suitable
redundancy.
resolution per lens band for compression (8 pixels/band). The combination of has
the DPCM and the DCT coding
Re-display of integral or lenticular-integral images has
algorithms
been
been carried out by outputting the image file to various
compression
previously
used
hardcopy devices after appropriate scaling to match the
removing redundances inherent within an image and
device's output resolution. Dye sublimation, thermal wax,
between image frames.
colour ink jet and laser printers have been used, with the
DPCM/DCT
former two giving the highest quality results since they do
compensation is shown in figure 1.
not apply comparatively low resolution dithering techniques
hybrid DPCM/DCT encoding scheme (4) is used to
to generate colour tones. A thermal wax printer, which most
compress still integral 3D TV pictures.
closely simulates the pixel characteristics of an LCD device,
subimages generated by the microlens array are fed one by
was used to demonstrate the potential of the method (6).
one into the encoder. The previously coded subimage is
of conventional 2D
to
achieve
moving images by
A block diagram of the hybrid
encoding
scheme
without
motion
In this paper, the The small
then subtracted from the present subimage and the resultant COMPRESSION OF STILL INTEGRAL 3D IMAGES
difference subimage is passed to the DCT unit. Taking the difference picture substantially reduces the amount of view
Recent advances in HDTV create an optimistic view of the
data, since most of the multiple images recorded on a CCD
future implementation of full colour, continuous parallax
placed behind the directionally selective microlens array, are
broadcast 3D TV systems. The minimum bandwidth initially
highly correlated.
believed to be required for the transmission of integral 3D
redundancies within each subimage. The DCT coding takes
images was 42 GHz (5).
The DCT is used to remove the
From recent developments in
advantage of the fact that few transform coefficients need to
integral 3D imaging, it is believed that a full colour 3-D
be transmitted - usually only the DC coefficient and a few
display should be possible using a receiver which has a
low frequency coefficients have significant magnitude. The
horizontal resolution of 2,048 pixels and a vertical resolution of 1,536 lines. In this case the transmission data rate required for integral 3D TV is approximately 1.5 times that for HDTV.
As a result, a compression ratio of
+
Input subimage
DCT
Entropy Coder
Quantise
approximately 16 : 1 is required for the transmission of
Reconstructed Previously coded subimage
integral 3D TV pictures. However, this should not be a
De-Quantise
major obstacle since the correlation between neighbouring
IDCT
image sections within the integral 3D TV images is much
+ +
higher than that of HDTV pictures. Subimage Store
Traditional compression algorithms for HDTV, such as transform
and
subband
coding
techniques,
Coded subimage
-
achieve
a. Encoder
compression by decorrelating in the spatial and/or temporal correlation domains (4). In the integral image recording process, each microlens produces a small image (subimage) of the subject from its particular viewpoint.
Coded subimage
Entropy Decoder
+
De-Quantise
Reconstructed subimage
IDCT +
Therefore,
compression of integral 3D TV images involves a new
Subimage
Store
dimension, namely the cross correlation between the
b. Decoder
multiple small images generated by the microlens array. The 1:
Hybrid
DPCM/DCT
cross image correlation is expected to be very high due to
Figure
the small angular disparity between microlenses, except
compression of integral 3D images
coding
scheme
for
where an object in the scene is close to the recording plane such that it is in the field of view of only a small number of
subsequent quantisation process sets all small values to zero
and quantises all non-zero transformed values to a set of
varying
nearest preferred magnitudes ready for transmission. Further
reconstructed image at the receiver.
q
leads
to
variable
visual
quality
of
the
gain in compression is obtained by the use of entropy coding which takes advantage of the statistics of the occurrence
Figure 2 shows the results of the objective and subjective
of non-zero coefficient values.
tests, and the average concentration of zeros per quantised coefficient block for different values of q. In the simulation,
The decoder merely uses the entropy coded quantised
the entropy coding of the
coefficient magnitudes to reconstruct each coded subimage
magnitudes has not been carried out.
in turn. A block diagram of the decoder is shown in figure
comparison purposes the bit rate is measured by the
1b.
The entropy decoded coefficient values are passed
concentration of zeros per quantised coefficient block. The
through the inverse DCT unit and then added to the
figure shows that for q > 20 there is little gain in the bit rate
previously decoded subimage.
quantised DCT coefficient
reduction. It also shows that for q
As a result, for
# 20, the impairments
caused by the coding scheme are imperceptible. SIMULATION RESULTS
As
q
increases from this point, the visual quality decreases and
In this section, simulation results obtained by applying the hybrid DPCM/DCT
encoding scheme for bandwidth
reduction of still integral 3D TV pictures are discussed. Both subjective and objective tests were carried out to assess
the quality of reconstruction of images. The
subjective tests were carried out using the five point image quality scale shown in Table 1. Quality and impairment are measured looking at parallax information as well as the overall quality. The objective tests were carried out using the peak signal-to-noise ratio (PSNR) as a measure. TABLE 1: Five-point image quality scale Quality 5 Excellent (parallax and general quality) 4 Good
(a)
3 Fair; Parallax present, but not strong 2 Poor 1 Bad; Parallax seriously degraded
Although the coding scheme was designed to deal with integral 3D images, the simulations were carried out using lenticular-integral images with only horizontal parallax. To form a subimage, the vertical lenticular bands are split into blocks of 8 pixels each. The key parameter in the proposed coding scheme is the quantisation process in which loss of information occurs.
As a result, the efficiency of the
algorithm depends mainly on the choice of the step size of the quantiser. For the purpose of this paper, the DCT coefficients are uniformly quantised using a 8x8 quantiser matrix. component
of
the
quantiser
matrix
Each
determines
the
(b)
quantisation step size; larger values correspond to larger
Figure 2: Performance of the coding scheme using a fixed
quantisation steps.
quantiser as a function of q
The quantisation step size can be
chosen to be constant for all DCT coefficients.
In this
case, it is possible to scale all quantisation levels using a single parameter, the quantiser scale, q.
(a) Objective quality and zero concentration; (b) Subjective quality and zero concentration.
The bit rate of the
encoded image can be varied by varying q.
However,
the PSNR value decreases below 30dB.
Another way to design the quantiser is to define the quantisation step size for each DCT coefficient in the transformed subimage separately.
This was achieved by
using the JPEG luminance quantisation matrix (7), which ensures that the low frequency DCT coefficients are quantised more accurately (with small step size) while the high frequencies are quantised more coarsely. There are different ways to vary the bit rate of the encoded image. One is to change the quantisation step size for the AC coefficients while keeping the quantisation of the DC coefficient fixed. However, in this paper, the quantisation step size for all coefficients is scaled by a constant factor, p. Figure 3 shows the results of the objective and subjective quality tests carried out, and the average concentration of zeros per quantised coefficient block for different values of p.
(a)
The results shown in figure 3 are similar to those
obtained with the fixed quantiser (figure 2). Figure 3 shows that there is a little gain in bit rate of the encoded image when p > 2 . It also shows that for p
# 2, the impairments
are imperceptible. Figures 2a and 3a show that for the same bit rate of the encoded image a fixed quantiser can achieve better objective quality than the JPEG variable quantiser investigated. COMPRESSION OF MOVING INTEGRAL 3D TV PICTURES
In order to extend compression to real-time moving 3D images, a modification of the hybrid DPCM/DCT coding scheme
previously described is proposed. To
effect
decorrelation in the temporal in addition to the intra- and
(b)
inter-subimage spatial domains, a second DPCM stage is required. The encoding and decoding processes are shown
Figure 3: Performance of the coding scheme using JPEG
in figure 4. First, the previous subimage is subtracted from
quantisation as a function of p.
the present one in the current frame, having been stored in
(a) Objective quality and zero concentration;
a buffer for one pass of the encoder. This intra-frame
(b) Subjective quality and zero concentration.
subimage difference is passed to the second stage, where the previous frame subimage difference is subtracted, having been reconstructed from the previous pass, stored and motion-compensated.
This
data
is
then
intra-subimage redundancies.
DCT-coded,
quantised and entropy coded. In the decoder, the inverse
CONCLUSIONS
transformed data is added to the previous inter-frame subimage difference, and the resulting intra-frame subimage
A compression algorithm for reduction of the transmission
difference added to the previously-decoded subimage.
bit rate of 3D integral TV pictures is presented. It uses a hybrid DPCM/DCT coding scheme to take advantage of the
This scheme uses DPCM to decorrelate in the temporal and
redundancies present within each subimage and between
inter-subimage domains, and the DCT to decorrelate
neighbouring subimages. Subjective tests have shown that the impairments caused by the coding and decoding of still integral 3D images using the hybrid DPCM/DCT coding scheme are imperceptible for a suitable choice of quantiser. An extension of this algorithm for application to moving integral 3D images is also presented.
Input subimage
+
+
DCT
Coded subimage
Entropy Coder
Quantise
-
-
Previous subimage Reconstructed Previously coded frame subimage difference
+ +
De-Quantise
IDCT Subimage Store
+ +
Motion Compensation
Frame Store
a. Encoder
Coded subimage
+
Entropy Decoder
De-Quantise
+
Reconstructed subimage
IDCT +
+
Frame
Subimage
Store
Store
b. Decoder Figure 4: Suggested coding scheme for moving integral 3D images.
6.
REFERENCES
M, 1. M,
Moor J R, Travis A R L,. Lang S R, and Castle O 1992,
"The
Autostereoscopic
Implementation Display"
IEE
of
a
Colloq.
Multi-View
McCormick M, Davies N, Aggoun A and Brewin 1994,
7.
McCormick M, Davies N and Chowanietz E G,
1992, "Towards Continuous Parallax 3D Images" European Workshop on 3D TV Signal Processing, Rome (Italy), 9-10.
"Three
Davies N, McCormick M and Yang L, 1988, Dimensional
Imaging
Systems:
A
New
Development", Appl. Opt., 27, 4520-4528, 1988. 4.
Netravali A N, and Haskell B G, 1988, "Digital
Pictures: Representation and Compression" Plenum Press, New York 5.
Okoshi T, 1976, "'Three Dimensional Imaging
Techniques" Academic Press, London, UK
the
Requirements
for
397, 477-482.
Image
3.
of
'Stereoscopic
Television', No 1992/173, 4/5-4/16. 2.
"Examination
Autostereoscopic Full Parallax 3D-TV", Conf. Proc. IEE
Pennebaker W B, Mitchell J L, 1993, "JPEG: Still Data
Compression
Reinhold, New York
Standard"
Van
Nostrand