Compression of integral 3D TV pictures - CiteSeerX

1 downloads 0 Views 147KB Size Report
a liquid crystal scanning aperture to transmit image data from different ... content and continuous parallax within the field of view (2)-. (3). In an integral imagingĀ ...
Compression of integral 3D TV pictures

M. Forman, A. Aggoun and M. McCormick

De Montfort University, UK

compression

spatial information and produces images with full colour

algorithm for reduction of the transmission bit rate of 3D

content and continuous parallax within the field of view (2)-

integral TV pictures is discussed. The proposed full parallax

(3).

3D image compression algorithm takes advantage of the

is formed around the capture microlens array, each

cross correlation between the multiple images recorded on

individual microlens sees a unique (directional) part of the

a CCD placed behind a directionally selective microlens

total image field. Adjacent microlenses record an angularly

array as well as the correlation inherent within each image.

displaced view of the same image field. In this way each

Tests carried out show that by using a hybrid DPCM/DCT

image point is represented by a series of related image

coding scheme to code a still 3D integral TV image, high

intensity distributions. A full 3D image is achieved by the

compression performance can be obtained while retaining

integration of all the directionally intersecting, intensity

all the necessary information to reconstruct a faithful

modulated beams of light generated by the lens array.

ABSTRACT:

The

development of a

In an integral imaging system, in which the image

duplicate of the original 3D image. A coding scheme for Since the image is recorded as an intensity distribution on a

moving integral 3D images is also described.

flat plane and in real time, it is possible to use electronic capture.

INTRODUCTION

Electronic capture, transmission and display of

integral 3D images call for a large increase in resolution outperforms

over that available in existing TV equipment. This results

Its growth has

in an enormous increase in bandwidth requirements. In this

been slower than many predicted, but current technology

paper the development of a compression algorithm for

advances could make 3D TV much more common in the

reduction of the transmission bit rate is considered. The aim

near future. Recently, television researchers have shown

is to record and display sufficient information to allow the

strong interest in 3D TV technology, and are currently

images to have sufficient solidity and parallax content

attempting

to approximate the original real optical model.

Three

dimensional

television

(3D

TV)

conventional TV for many applications.

to

realise

a

3D

TV

system

capable

of

reproducing images which generate a good sensation of reality and existence. The main issues that are being

In this paper, a hybrid DPCM/DCT coding system is used to

investigated by most research groups for the development of

compress still 3D TV images. This coding scheme has been

3D TV systems are bandwidth compression techniques and

extensively used to compress 2D moving images by

display and capture technologies. Work carried out on high

decorrelating the image information in spatial and temporal

definition TV

great deal in

domains (4). The hybrid DPCM/DCT coding scheme

accelerating the progress of some of these research areas.

compresses still 3D images by taking advantage of the

(HDTV)

has helped a

redundancies present both within a microlens image and Autostereoscopic 3D TV displays using multiple image

neighbouring microlens views.

viewpoints have been demonstrated by several groups (1).

objective tests carried out, it is shown that the coding

Most designs have used either lenticular sheet decoders or

and

a liquid crystal scanning aperture to transmit image data

DPCM/DCT coding system result in minimal degradation

from different viewpoints into the left and right eyes by

of visual quality compared to the original image. A

either spatial or temporal multiplexing. The need for

compression algorithm for real-time moving integral 3D TV

expensive multi-camera capture and the complexity of the

pictures is suggested.

decoding

of

still

3D

From the subjective and

images

using

the

hybrid

image processing electronics to generate the interlaced banded images have to date prevented such systems from

ELECTRONIC

becoming commercially available.

INTEGRAL 3D IMAGES

An alternative method of capturing a continuous parallax 3D

Several methods have been examined for the capture of

image based on an advanced form of integral imaging is

integral and lenticular-integral images for evaluation of

considered to have a number of advantages over multi

compression algorithms. Broadly, these fall into two

image systems. The system captures and displays volume

categories:

CAPTURE

AND

DISPLAY

OF

1. Direct capture. A video camera is used together with the

lenses.

integral imaging equipment (3); Compression schemes are under development which take 2. Indirect capture.

An integral or lenticular-integral

photograph is scanned either in transparency or print form.

advantage of the extra degree of redundancy involved in the

inter-subimage correlation. As each subimage is

effectively a low resolution image of the object scene, For the purposes of this investigation, a print of a lenticular-

DCT coding can be used in the same way as with

integral photograph was captured using a desktop scanner

conventional 2D images. A DPCM coder can be used to

operating at a resolution of 300dpi. The resulting image file

take advantage of the integral image specific inter-subimage

was processed in order to produce an image with a suitable

redundancy.

resolution per lens band for compression (8 pixels/band). The combination of has

the DPCM and the DCT coding

Re-display of integral or lenticular-integral images has

algorithms

been

been carried out by outputting the image file to various

compression

previously

used

hardcopy devices after appropriate scaling to match the

removing redundances inherent within an image and

device's output resolution. Dye sublimation, thermal wax,

between image frames.

colour ink jet and laser printers have been used, with the

DPCM/DCT

former two giving the highest quality results since they do

compensation is shown in figure 1.

not apply comparatively low resolution dithering techniques

hybrid DPCM/DCT encoding scheme (4) is used to

to generate colour tones. A thermal wax printer, which most

compress still integral 3D TV pictures.

closely simulates the pixel characteristics of an LCD device,

subimages generated by the microlens array are fed one by

was used to demonstrate the potential of the method (6).

one into the encoder. The previously coded subimage is

of conventional 2D

to

achieve

moving images by

A block diagram of the hybrid

encoding

scheme

without

motion

In this paper, the The small

then subtracted from the present subimage and the resultant COMPRESSION OF STILL INTEGRAL 3D IMAGES

difference subimage is passed to the DCT unit. Taking the difference picture substantially reduces the amount of view

Recent advances in HDTV create an optimistic view of the

data, since most of the multiple images recorded on a CCD

future implementation of full colour, continuous parallax

placed behind the directionally selective microlens array, are

broadcast 3D TV systems. The minimum bandwidth initially

highly correlated.

believed to be required for the transmission of integral 3D

redundancies within each subimage. The DCT coding takes

images was 42 GHz (5).

The DCT is used to remove the

From recent developments in

advantage of the fact that few transform coefficients need to

integral 3D imaging, it is believed that a full colour 3-D

be transmitted - usually only the DC coefficient and a few

display should be possible using a receiver which has a

low frequency coefficients have significant magnitude. The

horizontal resolution of 2,048 pixels and a vertical resolution of 1,536 lines. In this case the transmission data rate required for integral 3D TV is approximately 1.5 times that for HDTV.

As a result, a compression ratio of

+

Input subimage

DCT

Entropy Coder

Quantise

approximately 16 : 1 is required for the transmission of

Reconstructed Previously coded subimage

integral 3D TV pictures. However, this should not be a

De-Quantise

major obstacle since the correlation between neighbouring

IDCT

image sections within the integral 3D TV images is much

+ +

higher than that of HDTV pictures. Subimage Store

Traditional compression algorithms for HDTV, such as transform

and

subband

coding

techniques,

Coded subimage

-

achieve

a. Encoder

compression by decorrelating in the spatial and/or temporal correlation domains (4). In the integral image recording process, each microlens produces a small image (subimage) of the subject from its particular viewpoint.

Coded subimage

Entropy Decoder

+

De-Quantise

Reconstructed subimage

IDCT +

Therefore,

compression of integral 3D TV images involves a new

Subimage

Store

dimension, namely the cross correlation between the

b. Decoder

multiple small images generated by the microlens array. The 1:

Hybrid

DPCM/DCT

cross image correlation is expected to be very high due to

Figure

the small angular disparity between microlenses, except

compression of integral 3D images

coding

scheme

for

where an object in the scene is close to the recording plane such that it is in the field of view of only a small number of

subsequent quantisation process sets all small values to zero

and quantises all non-zero transformed values to a set of

varying

nearest preferred magnitudes ready for transmission. Further

reconstructed image at the receiver.

q

leads

to

variable

visual

quality

of

the

gain in compression is obtained by the use of entropy coding which takes advantage of the statistics of the occurrence

Figure 2 shows the results of the objective and subjective

of non-zero coefficient values.

tests, and the average concentration of zeros per quantised coefficient block for different values of q. In the simulation,

The decoder merely uses the entropy coded quantised

the entropy coding of the

coefficient magnitudes to reconstruct each coded subimage

magnitudes has not been carried out.

in turn. A block diagram of the decoder is shown in figure

comparison purposes the bit rate is measured by the

1b.

The entropy decoded coefficient values are passed

concentration of zeros per quantised coefficient block. The

through the inverse DCT unit and then added to the

figure shows that for q > 20 there is little gain in the bit rate

previously decoded subimage.

quantised DCT coefficient

reduction. It also shows that for q

As a result, for

# 20, the impairments

caused by the coding scheme are imperceptible. SIMULATION RESULTS

As

q

increases from this point, the visual quality decreases and

In this section, simulation results obtained by applying the hybrid DPCM/DCT

encoding scheme for bandwidth

reduction of still integral 3D TV pictures are discussed. Both subjective and objective tests were carried out to assess

the quality of reconstruction of images. The

subjective tests were carried out using the five point image quality scale shown in Table 1. Quality and impairment are measured looking at parallax information as well as the overall quality. The objective tests were carried out using the peak signal-to-noise ratio (PSNR) as a measure. TABLE 1: Five-point image quality scale Quality 5 Excellent (parallax and general quality) 4 Good

(a)

3 Fair; Parallax present, but not strong 2 Poor 1 Bad; Parallax seriously degraded

Although the coding scheme was designed to deal with integral 3D images, the simulations were carried out using lenticular-integral images with only horizontal parallax. To form a subimage, the vertical lenticular bands are split into blocks of 8 pixels each. The key parameter in the proposed coding scheme is the quantisation process in which loss of information occurs.

As a result, the efficiency of the

algorithm depends mainly on the choice of the step size of the quantiser. For the purpose of this paper, the DCT coefficients are uniformly quantised using a 8x8 quantiser matrix. component

of

the

quantiser

matrix

Each

determines

the

(b)

quantisation step size; larger values correspond to larger

Figure 2: Performance of the coding scheme using a fixed

quantisation steps.

quantiser as a function of q

The quantisation step size can be

chosen to be constant for all DCT coefficients.

In this

case, it is possible to scale all quantisation levels using a single parameter, the quantiser scale, q.

(a) Objective quality and zero concentration; (b) Subjective quality and zero concentration.

The bit rate of the

encoded image can be varied by varying q.

However,

the PSNR value decreases below 30dB.

Another way to design the quantiser is to define the quantisation step size for each DCT coefficient in the transformed subimage separately.

This was achieved by

using the JPEG luminance quantisation matrix (7), which ensures that the low frequency DCT coefficients are quantised more accurately (with small step size) while the high frequencies are quantised more coarsely. There are different ways to vary the bit rate of the encoded image. One is to change the quantisation step size for the AC coefficients while keeping the quantisation of the DC coefficient fixed. However, in this paper, the quantisation step size for all coefficients is scaled by a constant factor, p. Figure 3 shows the results of the objective and subjective quality tests carried out, and the average concentration of zeros per quantised coefficient block for different values of p.

(a)

The results shown in figure 3 are similar to those

obtained with the fixed quantiser (figure 2). Figure 3 shows that there is a little gain in bit rate of the encoded image when p > 2 . It also shows that for p

# 2, the impairments

are imperceptible. Figures 2a and 3a show that for the same bit rate of the encoded image a fixed quantiser can achieve better objective quality than the JPEG variable quantiser investigated. COMPRESSION OF MOVING INTEGRAL 3D TV PICTURES

In order to extend compression to real-time moving 3D images, a modification of the hybrid DPCM/DCT coding scheme

previously described is proposed. To

effect

decorrelation in the temporal in addition to the intra- and

(b)

inter-subimage spatial domains, a second DPCM stage is required. The encoding and decoding processes are shown

Figure 3: Performance of the coding scheme using JPEG

in figure 4. First, the previous subimage is subtracted from

quantisation as a function of p.

the present one in the current frame, having been stored in

(a) Objective quality and zero concentration;

a buffer for one pass of the encoder. This intra-frame

(b) Subjective quality and zero concentration.

subimage difference is passed to the second stage, where the previous frame subimage difference is subtracted, having been reconstructed from the previous pass, stored and motion-compensated.

This

data

is

then

intra-subimage redundancies.

DCT-coded,

quantised and entropy coded. In the decoder, the inverse

CONCLUSIONS

transformed data is added to the previous inter-frame subimage difference, and the resulting intra-frame subimage

A compression algorithm for reduction of the transmission

difference added to the previously-decoded subimage.

bit rate of 3D integral TV pictures is presented. It uses a hybrid DPCM/DCT coding scheme to take advantage of the

This scheme uses DPCM to decorrelate in the temporal and

redundancies present within each subimage and between

inter-subimage domains, and the DCT to decorrelate

neighbouring subimages. Subjective tests have shown that the impairments caused by the coding and decoding of still integral 3D images using the hybrid DPCM/DCT coding scheme are imperceptible for a suitable choice of quantiser. An extension of this algorithm for application to moving integral 3D images is also presented.

Input subimage

+

+

DCT

Coded subimage

Entropy Coder

Quantise

-

-

Previous subimage Reconstructed Previously coded frame subimage difference

+ +

De-Quantise

IDCT Subimage Store

+ +

Motion Compensation

Frame Store

a. Encoder

Coded subimage

+

Entropy Decoder

De-Quantise

+

Reconstructed subimage

IDCT +

+

Frame

Subimage

Store

Store

b. Decoder Figure 4: Suggested coding scheme for moving integral 3D images.

6.

REFERENCES

M, 1. M,

Moor J R, Travis A R L,. Lang S R, and Castle O 1992,

"The

Autostereoscopic

Implementation Display"

IEE

of

a

Colloq.

Multi-View

McCormick M, Davies N, Aggoun A and Brewin 1994,

7.

McCormick M, Davies N and Chowanietz E G,

1992, "Towards Continuous Parallax 3D Images" European Workshop on 3D TV Signal Processing, Rome (Italy), 9-10.

"Three

Davies N, McCormick M and Yang L, 1988, Dimensional

Imaging

Systems:

A

New

Development", Appl. Opt., 27, 4520-4528, 1988. 4.

Netravali A N, and Haskell B G, 1988, "Digital

Pictures: Representation and Compression" Plenum Press, New York 5.

Okoshi T, 1976, "'Three Dimensional Imaging

Techniques" Academic Press, London, UK

the

Requirements

for

397, 477-482.

Image

3.

of

'Stereoscopic

Television', No 1992/173, 4/5-4/16. 2.

"Examination

Autostereoscopic Full Parallax 3D-TV", Conf. Proc. IEE

Pennebaker W B, Mitchell J L, 1993, "JPEG: Still Data

Compression

Reinhold, New York

Standard"

Van

Nostrand